diff --git a/.agents/references/provider-profile-inventory.md b/.agents/references/provider-profile-inventory.md index 44f112124bc..5bd53f841e6 100644 --- a/.agents/references/provider-profile-inventory.md +++ b/.agents/references/provider-profile-inventory.md @@ -39,7 +39,8 @@ IPv6 endpoints remain excluded. Literal private IPv4 endpoints retain their sing The dual-stack boundary has a distinct profile and provider identity. Existing IPv4-only receipts remain valid for verification and cleanup; resume and rebuild retain their recorded permissions. -To use IPv6 after upgrading, reselect the local provider with `nemoclaw inference set --provider --model `. +For OpenClaw and Hermes, reselect the local provider with `nemoclaw inference set --sandbox --provider --model ` to use IPv6 after upgrading. +Deep Agents Code and Pi require sandbox recreation to change inference. If onboarding has not finished, resume with `nemoclaw onboard --resume`; if it retains the old attachment, reselect local inference before retrying the proof. Do not overwrite or delete an attached profile to change its permissions. Reselection uses the existing ownership and cleanup @@ -91,7 +92,7 @@ evidence that its create branch works with an empty catalog. | [vllm-local.ts](../../src/lib/onboard/inference-providers/vllm-local.ts) | Standalone native vLLM is intercepted. Local NIM is deliberately excluded and retains the shared route, owned by #13116 and #12975. It still depends on `openai`. | | [bedrock-runtime.ts](../../src/lib/onboard/bedrock-runtime.ts) | Legacy pending retirement for existing shared-route custom sandboxes. Fresh hosted Bedrock uses native custom setup, including a profile check before adapter startup. The leaf `openai` upsert is not evidence of a fresh native gap. | | [openrouter-runtime.ts](../../src/lib/onboard/openrouter-runtime.ts) | Legacy runtime-adapter branch. The fixed hosted OpenRouter selection returns through native setup first. The retained branch uses catalog type `openai`. | -| [routed-inference.ts](../../src/lib/onboard/routed-inference.ts) | OpenClaw's `nvidia-router` selection uses the native local lifecycle through `selectedUpsertProvider`. Fresh setup and resume reconcile the managed router, then import and verify the restricted profile before creation. They do not mutate a shared route. Other agents and custom blueprint provider names retain their existing catalog path. | +| [routed-inference.ts](../../src/lib/onboard/routed-inference.ts) | The OpenShell `0.1.5` migration uses the native local lifecycle for Model Router selections on OpenClaw, Hermes, Deep Agents Code, and Pi. The configured routed blueprint supplies the provider name and endpoint. Fresh setup and resume verify router ownership and credentials, then import and verify the restricted profile before creation. They do not mutate a shared route. Live qualification remains pending. | | [compatible-endpoint-gateway-route.ts](../../src/lib/onboard/inference-providers/compatible-endpoint-gateway-route.ts) | Legacy recovered-provider URL repair. It first checks that the provider exists; a later disappearance can still expose the helper's create branch. No owned profile import is established by this wrapper. | | [hermes-provider-auth.ts](../../src/lib/hermes-provider-auth.ts) | Native Hermes setup injects a native credential registrar. The fallback `openai` create remains for explicit custom Hermes endpoints and legacy rebuilds without a native receipt. The [rebuild credential preflight](../../src/lib/actions/sandbox/rebuild-credential-preflight.ts) reaches it when the Hermes Provider is missing before rebuild. These are retained legacy catalog paths. | | [hermes-tool-gateway-broker.ts](../../src/lib/hermes-tool-gateway-broker.ts) | Supported owned-profile create, imported before upsert. Exact legacy `generic` reuse passes `knownExists: true`; it does not create a new unowned profile. | @@ -138,25 +139,35 @@ used commit `19d994a0086c59e945a2456b1264ca1444a37fc7`. Their proposed behavior is absent from this baseline. #13504 owns native-provider authority cleanup. - [#13560](https://github.com/NVIDIA/NemoClaw/issues/13560) supplies the native OpenClaw Model Router profile described below. -## OpenClaw Model Router +## Native Model Router -The accepted `nvidia-router` path reuses the native local profile and ownership lifecycle. -The profile binds the configured router port, managed host alias, API prefix, and OpenClaw executables. -It permits model listing and chat completions. Generic endpoint selections cannot use router port authority. +The native Model Router path reuses the local profile and ownership lifecycle. +The OpenShell `0.1.5` migration applies it to OpenClaw, Hermes, Deep Agents Code, and Pi. +`nativeModelRouterEndpoint` requires `nvidia-router` or the provider name configured by the routed blueprint. +It verifies the configured host port and API prefix before normalizing the endpoint to the managed host alias. +The profile binds that endpoint and the selected agent executables. +It permits model listing and chat completions. +Generic endpoint selections cannot use router port authority. Other control-plane ports remain denied, including when the router is configured on one. Fresh setup and resume hold the gateway and router-port locks while reconciling the router and preparing its provider. -The sandbox receives an opaque credential reference and the native endpoint. The key stays in the managed provider boundary. -Matching profiles and owned provider instances are reused. Import failure, profile collision, or missing ownership stops mutation. -Existing shared-route sandboxes require the explicit recreation procedure owned by #12558. +Router process ownership and its host credential must match before and after provider preparation. +The sandbox receives an opaque credential reference and the native endpoint. +The host credential store and OpenShell provider retain the key outside the sandbox. +Router startup reads the host credential into its process environment. +Matching profiles and owned provider instances are reused. +Import failure, profile collision, or missing ownership stops mutation. +Existing shared-route sandboxes require explicit recreation on OpenShell `0.1.5`; legacy data remains available for recovery. OpenClaw `inference set` reuses a verified native router attachment for model changes. -Without that attachment, it stops before mutation and directs the user to explicit onboarding and recreation. +When switching from a different local or hosted provider, it can prepare a router attachment after verifying router process and credential ownership. +If a legacy sandbox already selects that router but lacks its native attachment, the command stops and requires explicit recreation. [Native local tests](../../src/lib/inference/native-local.test.ts) cover profile ordering, attachment, reuse, credential isolation, temporary-file cleanup, configured ports, and denied endpoints. [Routed setup tests](../../src/lib/onboard/routed-inference.test.ts) cover endpoint normalization and shared-route containment. [Resume tests](../../src/lib/onboard/machine/handlers/provider-inference-native-local-resume.test.ts) require native setup on resume. The shared lifecycle retains ambiguous-creation reconciliation and owned-only cleanup. +These source contracts do not establish live Model Router qualification for this upgrade. [#14066](https://github.com/NVIDIA/NemoClaw/issues/14066) owns replacement-model API checks and complete native router inference E2E. ## Warning evidence and focused follow-up diff --git a/.github/workflows/base-image.yaml b/.github/workflows/base-image.yaml index 5bdb0d51814..5f33125e961 100644 --- a/.github/workflows/base-image.yaml +++ b/.github/workflows/base-image.yaml @@ -59,6 +59,7 @@ on: - "src/lib/extra-agents-validation.ts" - "src/lib/core/json-types.ts" - "src/lib/core/ports.ts" + - "src/lib/inference/native-provider/catalog.ts" - "src/lib/messaging/**" - "src/lib/onboard/**" - "src/lib/security/credential-hash.ts" diff --git a/.github/workflows/managed-images.yaml b/.github/workflows/managed-images.yaml index 511d83c9987..05649c6c898 100644 --- a/.github/workflows/managed-images.yaml +++ b/.github/workflows/managed-images.yaml @@ -54,6 +54,7 @@ on: - "src/lib/extra-agents-validation.ts" - "src/lib/core/json-types.ts" - "src/lib/core/ports.ts" + - "src/lib/inference/native-provider/catalog.ts" - "src/lib/messaging/**" - "src/lib/onboard/**" - "src/lib/security/credential-hash.ts" @@ -1927,7 +1928,7 @@ jobs: # correctly. Start the published digest through its declared entrypoint # with no command, matching a real launch, # and prove PID 1 drops to the sandbox user, hardens its resource limits, - # and persists the trusted proxy environment before it holds the sandbox + # and persists the TLS environment before it holds the sandbox # open. - name: Exercise the Pi candidate through its declared entrypoint shell: bash @@ -2024,20 +2025,15 @@ jobs: )) process.exit(1); ' "$merged_ca" docker exec --user 999:999 "$entrypoint_container" /bin/bash -ceu ' - unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy unset SSL_CERT_FILE CURL_CA_BUNDLE REQUESTS_CA_BUNDLE GIT_SSL_CAINFO NODE_EXTRA_CA_CERTS source /tmp/nemoclaw-proxy-env.sh - proxy_url=http://10.200.0.1:3128 - for proxy_variable in HTTP_PROXY HTTPS_PROXY http_proxy https_proxy; do - if [ "${!proxy_variable}" != "$proxy_url" ]; then - echo "ERROR: an independent Pi shell did not load $proxy_variable from the runtime environment." >&2 - exit 1 - fi - done - no_proxy_value=localhost,127.0.0.1,::1,10.200.0.1 - for no_proxy_variable in NO_PROXY no_proxy; do - if [ "${!no_proxy_variable}" != "$no_proxy_value" ]; then - echo "ERROR: an independent Pi shell did not load $no_proxy_variable from the runtime environment." >&2 + for proxy_variable in \ + HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy \ + ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY NODE_USE_ENV_PROXY \ + NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT + do + if [ "${!proxy_variable+x}" = x ]; then + echo "ERROR: an independent Pi shell retained $proxy_variable after loading the runtime environment." >&2 exit 1 fi done diff --git a/.github/workflows/podman-cpu-proof.yaml b/.github/workflows/podman-cpu-proof.yaml index 24783d1d02a..d4acbdbc067 100644 --- a/.github/workflows/podman-cpu-proof.yaml +++ b/.github/workflows/podman-cpu-proof.yaml @@ -138,7 +138,7 @@ jobs: E2E_SOURCE_REVISION: ${{ github.event.pull_request.head.sha }} E2E_TARGET_ID: podman-cpu-lifecycle NEMOCLAW_RUN_LIVE_E2E: "1" - NEMOCLAW_OPENSHELL_PIN_VERSION: "0.0.116" + NEMOCLAW_OPENSHELL_PIN_VERSION: "0.1.7" PODMAN_APT_VERSION: "5.7.0+ds2-3build1" steps: - name: Checkout diff --git a/.github/workflows/pr-self-hosted.yaml b/.github/workflows/pr-self-hosted.yaml index b8aa5d896e3..375231ef7f5 100644 --- a/.github/workflows/pr-self-hosted.yaml +++ b/.github/workflows/pr-self-hosted.yaml @@ -75,9 +75,11 @@ jobs: or .filename == ".github/workflows/sandbox-images.yaml" or .filename == "package.json" or .filename == "package-lock.json" - or .filename == "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.116.json" + or .filename == "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.5.json" + or .filename == "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json" or .filename == "src/lib/actions/sandbox/runtime/hermes-lifecycle.ts" or .filename == "src/lib/hermes-managed-route.ts" + or .filename == "src/lib/inference/native-provider/catalog.ts" or .filename == "src/lib/providerless-inference.ts" or .filename == "src/lib/tool-disclosure.ts" or .filename == "test/e2e/live/hermes-root-entrypoint-smoke.test.ts" diff --git a/Dockerfile b/Dockerfile index d48aa599374..2ee442c9031 100644 --- a/Dockerfile +++ b/Dockerfile @@ -671,6 +671,7 @@ COPY nemoclaw-blueprint/ /opt/nemoclaw-blueprint/ FROM scratch AS openclaw-patch-payload COPY scripts/patch-openclaw-tool-catalog.mts /usr/local/lib/nemoclaw/patch-openclaw-tool-catalog.mts +COPY scripts/lib/patch-openclaw-native-network.mts /usr/local/lib/nemoclaw/patch-openclaw-native-network.mts COPY scripts/lib/patch-openclaw-container-restart.mts scripts/lib/patch-openclaw-agent-message-input.mts /usr/local/lib/nemoclaw/ COPY scripts/lib/patch-openclaw-npm12-pack-json.mts /usr/local/lib/nemoclaw/npm12.mts COPY scripts/patch-openclaw-mcp-npx.mts /usr/local/lib/nemoclaw/patch-openclaw-mcp-npx.mts @@ -698,7 +699,7 @@ COPY --from=managed-startup-runtime-builder /out/managed-startup-image-runtime.c COPY src/lib/extra-agents-validation.ts src/lib/tool-disclosure.ts src/lib/providerless-inference.ts src/lib/inference-credential.ts /src/lib/ COPY src/lib/core/endpoint-url-safety.ts /src/lib/core/ COPY src/lib/inference/native-local/agent-config.ts /src/lib/inference/native-local/ -COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/hosted.ts src/lib/inference/native-provider/contract.ts /src/lib/inference/native-provider/ +COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/catalog.ts /src/lib/inference/native-provider/ COPY nemoclaw-blueprint/openclaw-plugins/ /usr/local/share/nemoclaw/openclaw-plugins/ COPY --from=mcp-tool-discovery-runtime /opt/mcp-tool-discovery-runtime/dist/ /usr/local/lib/nemoclaw/mcp-tool-discovery-runtime/ @@ -881,6 +882,7 @@ COPY --from=wechat-npm-cache /out/wechat-npm-cache/ /usr/local/share/nemoclaw/we COPY --from=openclaw-patch-payload / / RUN chmod 755 /usr/local/lib/nemoclaw/patch-openclaw-tool-catalog.mts \ + /usr/local/lib/nemoclaw/patch-openclaw-native-network.mts \ /usr/local/lib/nemoclaw/npm12.mts \ /usr/local/lib/nemoclaw/patch-openclaw-container-restart.mts \ /usr/local/lib/nemoclaw/patch-openclaw-mcp-npx.mts \ @@ -1049,79 +1051,12 @@ RUN --mount=type=secret,id=nemoclaw-mcporter-audit-receipt,required=false \ fi; \ rm -f /tmp/mcporter-npm-audit.json /tmp/mcporter-npm-audit-policy.json -# Patch OpenClaw media fetch for proxy-only sandbox (NVIDIA/NemoClaw#1755). -# Route strict-mode requests through the trusted OpenShell egress proxy. -# The proxy enforces policy before connecting to the destination. - -# === Patch 2: env-gated bypass for assertExplicitProxyAllowed === -# OpenClaw 2026.4.2 added assertExplicitProxyAllowed() in fetch-guard, -# which validates the explicit proxy URL by passing the proxy hostname -# through resolvePinnedHostnameWithPolicy() with the *target's* SsrfPolicy. -# When the target uses hostnameAllowlist (Telegram media policy: -# `["api.telegram.org"]`), the proxy hostname (e.g. 10.200.0.1) gets -# rejected with "Blocked hostname (not in allowlist)". This is an upstream -# OpenClaw design flaw: a proxy is infrastructure, not a fetch target, and -# should not be filtered through the target's allowlist. -# -# Inject an early-return guarded by `process.env.OPENSHELL_SANDBOX === "1"` -# so the bypass only activates inside an OpenShell sandbox runtime, which -# is what NemoClaw deploys into. OpenShell injects this env var when it -# starts a sandbox pod; any consumer running the same openclaw bundle -# outside an OpenShell sandbox (bare-metal, another wrapper) does not have -# OPENSHELL_SANDBOX set and keeps the full upstream SSRF check. The L7 -# proxy itself enforces per-endpoint network policy inside the sandbox, -# so the trust boundary for SSRF protection is unchanged. -# -# Image-level `ENV` does NOT work here: OpenShell controls the pod env at -# runtime and image ENV vars set by Dockerfile are stripped. OPENSHELL_SANDBOX -# is the only marker reliably present in the runtime. -# -# === Patch 2b: allow OpenShell host gateway through web_fetch guard === -# OpenClaw's web_fetch SSRF guard blocks *.internal hostnames before the -# OpenShell L7 proxy sees the request. NemoClaw users legitimately reach -# host-local approved services through host.openshell.internal after the -# OpenShell policy explicitly allows that host:port. Add this hostname -# only to the web_fetch trusted-env-proxy policy, only inside an OpenShell -# sandbox. The generic SSRF helper and strict/direct DNS-pinned paths remain -# unmodified, so metadata/link-local/private IP literals are unchanged. -# -# === Patch 4: route unconfigured strict SSRF fetches through the egress proxy === -# (NVIDIA/NemoClaw#4687). fetchWithSsrFGuard builds a per-request DNS-pinned -# *direct* undici dispatcher for STRICT-mode fetches that pass no explicit -# dispatcherPolicy — e.g. the @openclaw/googlechat inbound JWT signing-cert -# fetch from www.googleapis.com/service_accounts/v1/metadata/x509/.... A direct -# dispatcher ignores the global EnvHttpProxyAgent installed by -# NODE_USE_ENV_PROXY=1, so the request never reaches the OpenShell L7 proxy and -# fails in the proxy-only sandbox netns — rejecting every inbound Google Chat -# webhook. OpenClaw already has a "managed proxy" branch that routes such -# fetches through the env proxy (createHttp1EnvHttpProxyAgent) while still -# resolving + SSRF-validating the target hostname, but it is gated on -# isManagedProxyActive() (OPENCLAW_PROXY_ACTIVE=1), which NemoClaw does not set. -# Inside an OpenShell sandbox the configured egress proxy IS the managed proxy, -# so extend that activation to OPENSHELL_SANDBOX=1 for fetches that supply no -# explicit dispatcherPolicy. Explicit-proxy and direct(mTLS) dispatcher policies -# (Google auth proxy / client-cert paths) keep their existing behavior, and -# resolvePinnedHostnameWithPolicy still blocks private/link-local targets. -# -# === Removal criteria === -# Patch 1: drop when OpenClaw deprecates withStrictGuardedFetchMode or -# when all media-fetch callsites unconditionally pass useEnvProxy. -# Patch 2: drop when OpenClaw fixes assertExplicitProxyAllowed to skip the -# target hostname allowlist for the proxy hostname check (or exposes config -# to disable the check). -# Patch 2b: drop when OpenClaw ships a reviewed web_fetch trusted-proxy SSRF -# policy surface that can allow host.openshell.internal without allowing -# broader private/special-use hostnames. -# Patch 4: drop when OpenClaw routes unconfigured strict fetches through the -# env proxy in proxy-only environments without OPENCLAW_PROXY_ACTIVE, or when -# NemoClaw sets OPENCLAW_PROXY_ACTIVE=1 in the sandbox runtime instead. -# -# SYNC WITH OPENCLAW: these patches classify the compiled OpenClaw dist at -# build time. They apply the legacy patch when the old target exists, skip -# only when the dist shape proves OpenClaw no longer needs that patch, and -# fail with the OpenClaw version plus dist path for mixed or unknown shapes. -# When bumping OPENCLAW_VERSION, verify the new dist -# takes the expected branch and update the regex / sed replacement if needed. +# Patch OpenClaw network boundaries for OpenShell 0.1.3. +# Current images use native socket mediation and synthetic DNS answers. Preserve +# upstream SSRF checks except for the reviewed mapped-address boundary. +# Legacy proxy patches below apply only to the existing E2E fixture selector; +# the reviewed-package install above restricts its accepted versions. +# Remove the native patch when OpenClaw supports these OpenShell DNS mappings. # hadolint ignore=SC2016,DL3059,DL4006 RUN set -eu; \ OC_DIST=/usr/local/lib/node_modules/openclaw/dist; \ @@ -1134,6 +1069,10 @@ RUN set -eu; \ echo " Inspect ${OC_DIST} and update the Dockerfile patch rules for this OpenClaw layout." >&2; \ exit 1; \ }; \ + if [ "$OC_VERSION" = "2026.9.2" ]; then \ + node /usr/local/lib/nemoclaw/patch-openclaw-native-network.mts "$OC_DIST"; \ + echo "INFO: OpenClaw ${OC_VERSION} native network patch applied"; \ + elif [ "${NEMOCLAW_E2E_FIXTURE_LEGACY_OPENCLAW:-0}" = "1" ]; then \ # --- Patch 1: rewrite fetch-guard export --- \ fg_export="$(grep -RIlE --include='*.js' 'export \{[^}]*withStrictGuardedFetchMode as [a-z]' "$OC_DIST" || true)"; \ if [ -n "$fg_export" ]; then \ @@ -1309,6 +1248,9 @@ RUN set -eu; \ patch_fail "Patch 6 cannot safely skip"; \ fi; \ fi; \ + else \ + patch_fail "Native network patch requires the reviewed current version"; \ + fi; \ # --- Patch 3: follow symlinks in plugin-install path checks (#2203) --- \ # Replace legacy lstat checks with stat while retaining realpath containment. \ # Current safe-path delegates to fs-safe; directory guards remain native. \ @@ -1563,12 +1505,6 @@ ARG NEMOCLAW_BUILD_ID=default # starts NemoClaw as the non-root sandbox user, so uid-owned 770/660 paths # become unreadable unless this Darwin-only compatibility mode is enabled. ARG NEMOCLAW_DARWIN_VM_COMPAT=0 -# Sandbox egress proxy host/port. Defaults match the OpenShell-injected -# gateway (10.200.0.1:3128). Operators on non-default networks can override -# at sandbox creation time by exporting NEMOCLAW_PROXY_HOST / NEMOCLAW_PROXY_PORT -# before running `nemoclaw onboard`. See #1409. -ARG NEMOCLAW_PROXY_HOST=10.200.0.1 -ARG NEMOCLAW_PROXY_PORT=3128 # Non-secret web-search selection from onboard. The actual API key is injected # at runtime via openshell:resolve:env, never baked into the image. ARG NEMOCLAW_WEB_SEARCH_ENABLED=0 @@ -1577,8 +1513,8 @@ ARG NEMOCLAW_OPENCLAW_OTEL=0 # The default local OTEL endpoint is intentionally the single host-gateway # collector path covered by the openclaw-diagnostics-otel-local policy preset. # @openclaw/diagnostics-otel@2026.9.1 exports through OpenTelemetry's OTLP -# trace exporter path, not OpenClaw web_fetch, so Patch 2b's host gateway -# exception remains scoped to user-requested web_fetch proxy calls. +# trace exporter path. The native DNS patch separately handles web_fetch +# requests to the host gateway. ARG NEMOCLAW_OPENCLAW_OTEL_ENDPOINT=http://host.openshell.internal:4318 ARG NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME=openclaw-gateway ARG NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE=1.0 @@ -1611,8 +1547,6 @@ ENV NEMOCLAW_MODEL=${NEMOCLAW_MODEL} \ NEMOCLAW_WSL_DASHBOARD_EXPOSURE=${NEMOCLAW_WSL_DASHBOARD_EXPOSURE} \ NEMOCLAW_DISABLE_DEVICE_AUTH=${NEMOCLAW_DISABLE_DEVICE_AUTH} \ NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE=${NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE} \ - NEMOCLAW_PROXY_HOST=${NEMOCLAW_PROXY_HOST} \ - NEMOCLAW_PROXY_PORT=${NEMOCLAW_PROXY_PORT} \ NEMOCLAW_WEB_SEARCH_ENABLED=${NEMOCLAW_WEB_SEARCH_ENABLED} \ NEMOCLAW_WEB_SEARCH_PROVIDER=${NEMOCLAW_WEB_SEARCH_PROVIDER} \ NEMOCLAW_OPENCLAW_OTEL=${NEMOCLAW_OPENCLAW_OTEL} \ @@ -1671,12 +1605,7 @@ USER sandbox # Generate base openclaw.json from environment variables. Messaging build # steps run through src/lib/messaging/applier/build/messaging-build-applier.mts. # -# OpenClaw's managed proxy config activates process-wide HTTP_PROXY/HTTPS_PROXY -# for child npm processes. During image build the OpenShell gateway is not -# available at the runtime sandbox proxy address yet, so defer the final proxy -# block until after build-time OpenClaw doctor/plugin commands complete. RUN NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION=0 \ - NEMOCLAW_OPENCLAW_MANAGED_PROXY=0 \ node /scripts/generate-openclaw-config.mts # Validate the patched OpenClaw tool-search contract against real generated @@ -1694,7 +1623,6 @@ RUN set -eu; \ NEMOCLAW_PRIMARY_MODEL_REF=inference/test-model \ NEMOCLAW_TOOL_DISCLOSURE="$mode" \ NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION=0 \ - NEMOCLAW_OPENCLAW_MANAGED_PROXY=0 \ node /scripts/generate-openclaw-config.mts; \ node /scripts/validate-openclaw-tool-search.mts \ /usr/local/lib/node_modules/openclaw/dist \ @@ -1946,21 +1874,13 @@ ENV NPM_CONFIG_OFFLINE=false # SECURITY: Clear any gateway auth token that openclaw doctor/plugins may have # auto-generated. The real token is created at container startup by the # entrypoint (generate_gateway_token) and never stored in openclaw.json. -# Also add the final OpenClaw managed proxy config after build-time OpenClaw -# commands are done, so runtime Discord/WebSocket traffic uses the OpenShell -# gateway proxy without forcing image-build npm traffic through that proxy. +# Remove obsolete application proxy state from the generated config. RUN python3 -c "\ import json, os; \ path = os.path.expanduser('~/.openclaw/openclaw.json'); \ cfg = json.load(open(path)); \ cfg.setdefault('gateway', {}).setdefault('auth', {})['token'] = ''; \ -proxy_host = os.environ.get('NEMOCLAW_PROXY_HOST') or '10.200.0.1'; \ -proxy_port = os.environ.get('NEMOCLAW_PROXY_PORT') or '3128'; \ -cfg['proxy'] = { \ - 'enabled': True, \ - 'proxyUrl': f'http://{proxy_host}:{proxy_port}', \ - 'loopbackMode': 'gateway-only', \ -}; \ +cfg.pop('proxy', None); \ json.dump(cfg, open(path, 'w'), indent=2); \ os.chmod(path, 0o600)" @@ -2236,38 +2156,6 @@ RUN if [ "$NEMOCLAW_DARWIN_VM_COMPAT" = "1" ]; then \ chmod a+rw /sandbox/.nemoclaw/config.json; \ fi -# Temporary workaround for OpenTelemetry JS OTLP/HTTP proxy handling. -# When diagnostics OTEL is enabled, patch the bundled exporter so Node's -# NODE_USE_ENV_PROXY=1 handling can apply instead of forcing the default agent. -# Remove once https://github.com/open-telemetry/opentelemetry-js/issues/6638 -# is fixed in @opentelemetry/otlp-exporter-base. -# hadolint ignore=DL4006 -RUN set -eu; \ - if [ "$NEMOCLAW_OPENCLAW_OTEL" = "1" ]; then \ - target="$(find /sandbox/.openclaw \ - -path '*/@opentelemetry/otlp-exporter-base/build/src/transport/http-transport-utils.js' \ - -print -quit 2>/dev/null || true)"; \ - if [ -z "$target" ]; then \ - echo "ERROR: NEMOCLAW_OPENCLAW_OTEL=1 but otlp-exporter-base transport was not found" >&2; \ - exit 1; \ - fi; \ - if grep -q 'NODE_USE_ENV_PROXY' "$target"; then \ - echo "INFO: OpenTelemetry OTLP proxy patch already present in $target"; \ - else \ - owner="$(stat -c '%u:%g' "$target")"; \ - mode="$(stat -c '%a' "$target")"; \ - cp -p "$target" "$target.bak"; \ - sed -i "0,/^[[:space:]]*agent,$/s// agent: process.env.NODE_USE_ENV_PROXY === '1' ? undefined : agent,/" "$target"; \ - grep -q 'NODE_USE_ENV_PROXY' "$target" || { \ - echo "ERROR: failed to patch OpenTelemetry OTLP transport at $target" >&2; \ - exit 1; \ - }; \ - chown "$owner" "$target"; \ - chmod "$mode" "$target"; \ - echo "INFO: patched OpenTelemetry OTLP proxy handling in $target"; \ - fi; \ - fi - RUN check_metadata() { \ metadata_path="$1"; \ expected_metadata="$2"; \ diff --git a/agents/hermes/Dockerfile b/agents/hermes/Dockerfile index a78032634e3..d826740ede1 100644 --- a/agents/hermes/Dockerfile +++ b/agents/hermes/Dockerfile @@ -93,7 +93,7 @@ COPY agents/hermes/managed_policy.py /opt/nemoclaw-hermes-config/managed_policy. COPY agents/hermes/host/managed-tool-gateway-matrix.json /opt/nemoclaw-hermes-config/managed-tool-gateway-matrix.json COPY src/lib/hermes-managed-route.ts src/lib/hermes-switchyard-routing.ts /src/lib/ COPY src/lib/tool-disclosure.ts src/lib/providerless-inference.ts src/lib/inference-credential.ts /src/lib/ -COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/hosted.ts src/lib/inference/native-provider/contract.ts /src/lib/inference/native-provider/ +COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/catalog.ts /src/lib/inference/native-provider/ COPY src/lib/inference/native-nvidia/contract.ts /src/lib/inference/native-nvidia/contract.ts COPY src/lib/inference/native-local/agent-config.ts /src/lib/inference/native-local/ COPY src/lib/core/endpoint-url-safety.ts /src/lib/core/ @@ -124,9 +124,10 @@ COPY agents/hermes/patch-langfuse-credentials.mts /usr/local/lib/nemoclaw/patch- COPY agents/hermes/runtime-config-guard.py /usr/local/lib/nemoclaw/hermes-runtime-config-guard.py COPY agents/hermes/migrate-dashboard-state.py /usr/local/lib/nemoclaw/migrate-hermes-dashboard-state.py COPY agents/hermes/finalize-tirith-marker.py /usr/local/lib/nemoclaw/finalize-tirith-marker.py +COPY agents/hermes/startup-completion.py /usr/local/lib/nemoclaw/hermes-startup-completion.py COPY agents/hermes/mcp-config-transaction.py /usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py COPY agents/hermes/cron-restore-control.py /usr/local/lib/nemoclaw/hermes-cron-restore-control.py -COPY src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.116.json /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.116.json +COPY src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.1.7.json COPY nemoclaw-blueprint/scripts/*.js /usr/local/lib/nemoclaw/preloads/ FROM scratch AS hermes-wrapper-payload @@ -381,13 +382,10 @@ FROM hermes-system # Hermes' WeChat adapter (gateway/platforms/weixin.py:_make_ssl_connector) # builds aiohttp with ssl.create_default_context(cafile=certifi.where()), -# bypassing SSL_CERT_FILE. Symlink certifi's bundle at SSL_CERT_FILE so -# weixin TLS trusts the OpenShell L7 proxy CA. Done at image build (as -# root, which has write perms on the venv) because at runtime root doesn't -# see SSL_CERT_FILE and the gateway user can't write the venv. -# The target doesn't need to exist at build time — OpenShell mounts it at -# sandbox start; the forward-referencing symlink resolves then. -ARG SSL_CERT_FILE=/etc/openshell-tls/ca-bundle.pem +# bypassing SSL_CERT_FILE. Link certifi to the workload CA bundle installed by +# OpenShell 0.1.3 before launch. The image build owns the venv; the sandbox user +# cannot change the link. The target is absent until OpenShell starts the sandbox. +ARG SSL_CERT_FILE=/run/openshell-supervisor-ca/material/ca-bundle.crt # hadolint ignore=DL3059 RUN _hermes_certifi=$(/opt/hermes/.venv/bin/python -c 'import certifi; print(certifi.where())') \ && ln -sf "$SSL_CERT_FILE" "$_hermes_certifi" @@ -504,11 +502,11 @@ RUN chmod -R a+rX /opt/nemoclaw-blueprint/ # minimum supported Hermes sandbox base tag guarantees those artifacts and # test/runtime/sandbox/sandbox-rlimit-hooks.test.ts covers that base. RUN chmod 755 /usr/local/bin/nemoclaw-start /usr/local/bin/nemoclaw-managed-startup-hold /usr/local/lib/nemoclaw/sandbox-init.sh /usr/local/lib/nemoclaw/validate-hermes-env-secret-boundary.py /usr/local/lib/nemoclaw/patch-hermes-sqlite-temp-store.py /usr/local/lib/nemoclaw/patch-hermes-discord-recovery-permissions.py /usr/local/lib/nemoclaw/hermes-runtime-config-guard.py /usr/local/lib/nemoclaw/migrate-hermes-dashboard-state.py /usr/local/lib/nemoclaw/finalize-tirith-marker.py /usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py \ - && chown root:root /usr/local/lib/nemoclaw/hermes-cron-restore-control.py /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.116.json \ + && chown root:root /usr/local/lib/nemoclaw/hermes-cron-restore-control.py /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.1.7.json \ && chmod 700 /usr/local/lib/nemoclaw/hermes-cron-restore-control.py \ && chmod 444 /usr/local/lib/nemoclaw/corporate-ca-runtime.sh /usr/local/lib/nemoclaw/entrypoint-env-wrapper.sh /usr/local/lib/nemoclaw/managed_policy.py \ && chmod 444 /usr/local/lib/nemoclaw/patch-hermes-langfuse-credentials.mts \ - && chmod 444 /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.116.json \ + && chmod 444 /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.1.7.json \ && /opt/hermes/.venv/bin/python3 -I -c 'import runpy, yaml; assert yaml.safe_load("ready: true")["ready"] is True; runpy.run_path("/usr/local/lib/nemoclaw/hermes-runtime-config-guard.py", run_name="nemoclaw_runtime_config_guard_probe")' \ && if [ -d /usr/local/lib/nemoclaw/preloads ]; then \ chown -R 0:0 /usr/local/lib/nemoclaw/preloads \ @@ -821,9 +819,9 @@ RUN node \ ARG NEMOCLAW_HERMES_CLI_ADAPTER_SHA256=d892e57585485880e41c9bd0afe5b851e131997ab6ac86956a58db0704147781 ARG NEMOCLAW_HERMES_WRAPPER_SHA256=ca37193cfa1fd972f64db1e3feb77a0b605e2dec507ee8414310ce002aba6985 ARG NEMOCLAW_HERMES_CLI_ADAPTER_VALIDATOR_SHA256=5205a5446d0a509c5bc60f77fd97ec579449c2c7291dcc2878a788f25f078d6f -ARG NEMOCLAW_HERMES_VALIDATOR_SHA256=8a034b4ecd32b69487d7524200c59446cf8df40a19e92b064d81f052dd8a51c4 +ARG NEMOCLAW_HERMES_VALIDATOR_SHA256=4ffe5790879d54a93705e218fe886ad8916b9e92766cbe38a48fa452366a93af ARG NEMOCLAW_HERMES_TIRITH_FINALIZER_SHA256=a1e6b1c53ab297569abb87c29d15c294d729e46005bfd022136b4c447a791819 -ARG NEMOCLAW_HERMES_CRON_RESTORE_CONTROLLER_SHA256=408c80aacb38ef52cad87278c73f4b0f8863eea75bc7f0ee4f80864137ebf0b0 +ARG NEMOCLAW_HERMES_CRON_RESTORE_CONTROLLER_SHA256=d5b11ec89367cdbdf60390e6fb102fef55cf0cb4b9f4359414c3c3352b962cd4 # hadolint ignore=DL4006 RUN printf '%s %s\n' \ "$NEMOCLAW_HERMES_VALIDATOR_SHA256" /usr/local/lib/nemoclaw/validate-hermes-env-secret-boundary.py \ diff --git a/agents/hermes/config/managed-policy.ts b/agents/hermes/config/managed-policy.ts index 118715b2fe8..23e9fa7d348 100644 --- a/agents/hermes/config/managed-policy.ts +++ b/agents/hermes/config/managed-policy.ts @@ -284,13 +284,17 @@ export function buildHermesManagedPolicy( }; if (settings.model !== null) - applyHermesManagedRoute(config, { - model: settings.model, - baseUrl: settings.baseUrl, - upstreamProvider: settings.upstreamProvider, - inferenceApi: settings.inferenceApi, - contextWindow: settings.contextWindow, - }); + applyHermesManagedRoute( + config, + { + model: settings.model, + baseUrl: settings.baseUrl, + upstreamProvider: settings.upstreamProvider, + inferenceApi: settings.inferenceApi, + contextWindow: settings.contextWindow, + }, + env.NEMOCLAW_LOCAL_INFERENCE_TOKEN, + ); const managedToolGatewayPresets = effectiveManagedToolGatewayPresets(settings); if (managedToolGatewayPresets.length > 0) { diff --git a/agents/hermes/cron-restore-control.py b/agents/hermes/cron-restore-control.py index 48d8b186b68..8b157620fde 100644 --- a/agents/hermes/cron-restore-control.py +++ b/agents/hermes/cron-restore-control.py @@ -50,6 +50,7 @@ CONTROL_ERROR_PREFIX = "NEMOCLAW_HERMES_CRON_RESTORE_ERROR_V1:" CONTROL_ERROR_CODE = "control-failure" DRAIN_MARKER_ROLLBACK_FAILED_CODE = "drain-marker-rollback-failed" +GENERATION_OWNER_READ_REQUIRED_CODE = "gateway-generation-owner-read-required" BEGIN_TIMEOUT_SECONDS = 60.0 RELEASE_TIMEOUT_SECONDS = 15.0 POLL_SECONDS = 0.1 @@ -404,7 +405,14 @@ def _write_owned_record( staged_path.unlink(missing_ok=True) -def _read_gateway_recovery_generation() -> str | None: +def _generation_identity(metadata: os.stat_result) -> list[int]: + return [ + metadata.st_dev, metadata.st_ino, metadata.st_uid, metadata.st_mode, + metadata.st_nlink, metadata.st_size, metadata.st_mtime_ns, metadata.st_ctime_ns, + ] + + +def _read_gateway_recovery_generation() -> dict[str, Any] | None: """Read the unprivileged supervisor's opaque, one-use generation.""" try: expected_uid = HERMES_HOME.lstat().st_uid @@ -417,6 +425,11 @@ def _read_gateway_recovery_generation() -> str | None: descriptor = os.open(GATEWAY_RECOVERY_WAITING_PATH, flags) except FileNotFoundError: return None + except PermissionError as error: + raise ControlError( + "Hermes gateway recovery generation requires its owner to read it", + code=GENERATION_OWNER_READ_REQUIRED_CODE, + ) from error except OSError as error: raise ControlError("Hermes gateway recovery generation is unreadable") from error try: @@ -445,10 +458,14 @@ def _read_gateway_recovery_generation() -> str | None: or any(character not in "0123456789abcdef" for character in payload[3:-1]) ): raise ControlError("Hermes gateway recovery generation is invalid") - return payload[3:-1] + return { + "version": 1, + "generation": payload[3:-1], + "identity": _generation_identity(metadata), + } -def _wait_for_gateway_recovery_generation() -> str | None: +def _wait_for_gateway_recovery_generation() -> dict[str, Any] | None: deadline = time.monotonic() + GATEWAY_RECOVERY_WAIT_SECONDS while True: generation = _read_gateway_recovery_generation() @@ -459,6 +476,36 @@ def _wait_for_gateway_recovery_generation() -> str | None: time.sleep(POLL_SECONDS) +def _validate_recovery_observation(raw: str) -> dict[str, Any] | None: + """Revalidate the owner's read before the root controller publishes a request.""" + try: + observed = json.loads(raw) + metadata = GATEWAY_RECOVERY_WAITING_PATH.lstat() + except FileNotFoundError: + if observed is None: + return None + raise ControlError("Hermes gateway recovery generation changed after observation") + except (OSError, ValueError) as error: + raise ControlError("Hermes gateway recovery observation is invalid") from error + if ( + not isinstance(observed, dict) + or set(observed) != {"version", "generation", "identity"} + or type(observed["version"]) is not int + or observed["version"] != 1 + or not isinstance(observed["generation"], str) + or len(observed["generation"]) != 64 + or any(c not in "0123456789abcdef" for c in observed["generation"]) + or observed["identity"] != _generation_identity(metadata) + or not stat.S_ISREG(metadata.st_mode) + or metadata.st_uid != HERMES_HOME.lstat().st_uid + or stat.S_IMODE(metadata.st_mode) != 0o600 + or metadata.st_nlink != 1 + or metadata.st_size != 68 + ): + raise ControlError("Hermes gateway recovery generation changed after observation") + return observed + + def _self_process_identity() -> tuple[int, int]: """Return the controller PID and Linux process start time.""" pid = os.getpid() @@ -1067,15 +1114,19 @@ def _prepare_owned_drain() -> str | None: return drain_token -def prepare_recovery() -> None: +def prepare_recovery(observation: str | None = None) -> None: """Re-establish any persisted NemoClaw gate before host gateway repair.""" with _control_lock(): drain_acquired = _prepare_owned_drain() is not None - generation = _wait_for_gateway_recovery_generation() + generation = ( + _wait_for_gateway_recovery_generation() + if observation is None + else _validate_recovery_observation(observation) + ) if generation is not None: requester_pid, requester_start_time = _self_process_identity() _publish_gateway_recovery_request( - generation, requester_pid, requester_start_time + generation["generation"], requester_pid, requester_start_time ) _prepare_recovery_receipt(drain_acquired, generation is not None) @@ -1231,7 +1282,8 @@ def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser() subparsers = parser.add_subparsers(dest="action", required=True) subparsers.add_parser("begin") - subparsers.add_parser("prepare-recover") + prepare = subparsers.add_parser("prepare-recover") + prepare.add_argument("--generation-observation-stdin", action="store_true") subparsers.add_parser("recover") for action in ("validate", "observe", "complete"): subparser = subparsers.add_parser(action) @@ -1255,7 +1307,10 @@ def main() -> int: if args.action == "begin": begin_drain() elif args.action == "prepare-recover": - prepare_recovery() + observation = sys.stdin.read(2049) if args.generation_observation_stdin else None + if observation is not None and len(observation) > 2048: + raise ControlError("Hermes gateway recovery observation is too large") + prepare_recovery(observation) elif args.action == "recover": recover_drain() elif args.action == "validate": diff --git a/agents/hermes/mcp-config-transaction.py b/agents/hermes/mcp-config-transaction.py index d0257f21f16..cd0aca1cbe0 100755 --- a/agents/hermes/mcp-config-transaction.py +++ b/agents/hermes/mcp-config-transaction.py @@ -88,7 +88,7 @@ OPENSHELL_STABLE_CREDENTIAL_NAME_RE = re.compile( r"^s[a-f0-9]{64}_[A-Za-z0-9_]+$" ) -BOUNDARY_MANIFEST_NAME = "openshell-child-visible-credentials.v0.0.116.json" +BOUNDARY_MANIFEST_NAME = "openshell-child-visible-credentials.v0.1.7.json" ANSI_ESCAPE_RE = re.compile( r"\x1b(?:\[[0-?]*[ -/]*[@-~]|\][^\x07]*(?:\x07|\x1b\\)|[@-_])" ) @@ -241,7 +241,7 @@ def _load_credential_boundary_manifest() -> dict[str, object]: manifest = json.loads(manifest_path.read_text(encoding="utf-8")) if ( not isinstance(manifest, dict) - or manifest.get("openshellVersion") != "0.0.116" + or manifest.get("openshellVersion") != "0.1.7" ): raise RuntimeError("Hermes MCP credential boundary manifest is invalid") return manifest @@ -426,7 +426,7 @@ def _validate_payload(action: str, payload: dict[str, object]) -> None: } if action == "add" and hostname in host_aliases: raise ValueError( - "Authenticated MCP OpenShell host aliases are unavailable with OpenShell v0.0.116" + "Authenticated MCP OpenShell host aliases require an attested driver gateway address" ) # Host preflight owns destination trust and binds every accepted endpoint to # exact OpenShell address pins. This in-sandbox check revalidates canonical diff --git a/agents/hermes/plugin/__init__.py b/agents/hermes/plugin/__init__.py index 66b5d2a4043..736582cbc20 100644 --- a/agents/hermes/plugin/__init__.py +++ b/agents/hermes/plugin/__init__.py @@ -545,7 +545,6 @@ def submit(self, *args, **kwargs): _BROWSER_USE_CDP_TUNNEL_SCRIPT = r""" -import base64 import os import select import socket @@ -557,8 +556,7 @@ def submit(self, *args, **kwargs): remote_url = sys.argv[1] remote = urlparse(remote_url) -proxy = urlparse(os.environ.get("HTTPS_PROXY") or os.environ.get("HTTP_PROXY") or "") -if remote.scheme != "wss" or not remote.hostname or not proxy.hostname: +if remote.scheme != "wss" or not remote.hostname: raise SystemExit(2) target_host = remote.hostname @@ -566,7 +564,6 @@ def submit(self, *args, **kwargs): target_path = remote.path or "/" if remote.query: target_path += "?" + remote.query -proxy_port = proxy.port or 8080 idle_timeout = int(os.environ.get("NEMOCLAW_CDP_TUNNEL_IDLE_SECONDS", "600")) last_activity = time.monotonic() @@ -584,24 +581,13 @@ def _read_headers(sock): def _connect_remote(): - raw = socket.create_connection((proxy.hostname, proxy_port), timeout=15) - host_header = f"{target_host}:{target_port}" - lines = [ - f"CONNECT {host_header} HTTP/1.1", - f"Host: {host_header}", - ] - if proxy.username: - user = proxy.username or "" - password = proxy.password or "" - token = base64.b64encode(f"{user}:{password}".encode()).decode() - lines.append(f"Proxy-Authorization: Basic {token}") - raw.sendall(("\r\n".join(lines) + "\r\n\r\n").encode("ascii")) - response = _read_headers(raw) - first = response.split(b"\r\n", 1)[0] - if b" 200 " not in first: - raise OSError("proxy CONNECT failed") - context = ssl.create_default_context() - return context.wrap_socket(raw, server_hostname=target_host) + raw = socket.create_connection((target_host, target_port), timeout=15) + try: + context = ssl.create_default_context() + return context.wrap_socket(raw, server_hostname=target_host) + except Exception: + raw.close() + raise def _rewrite_request(data): @@ -741,13 +727,10 @@ def _start_browser_use_cdp_tunnel(cdp_url): def _install_browser_cdp_tunnel_patch(): - """Route Browser Use CDP sockets through OpenShell's HTTP proxy. + """Bridge Browser Use CDP through verified TLS under OpenShell network policy. - ``agent-browser`` can use the OpenShell proxy for ordinary HTTP requests, - but its native CDP websocket connector currently dials WSS endpoints - directly. Direct egress is blocked in the sandbox. Browser Use session - creation still goes through the Nous gateway; this tunnel only bridges the - returned short-lived CDP capability URL through the already-enforced proxy. + Session creation still uses the Nous gateway. Preserve the loopback bridge + for the returned capability URL and use the sandbox's configured CA trust. """ if not _broker_mode_enabled(): return False diff --git a/agents/hermes/runtime-config-guard.py b/agents/hermes/runtime-config-guard.py index a18f79269a5..ab4489d73f9 100755 --- a/agents/hermes/runtime-config-guard.py +++ b/agents/hermes/runtime-config-guard.py @@ -524,6 +524,139 @@ def _startup_process_identity_is_live( os.close(proc_root_fd) +def _runtime_drm_groups(expected_uid: int) -> set[str]: + # NVIDIA CDI adds nonzero groups for device nodes that are not world read/write. + # Landlock denies /dev directory reads. Discover kernel device names through + # sysfs, then inspect metadata under pinned /dev directories without opening devices. + try: + flags = os.O_PATH | _directory_flag() | _no_follow_flag() | _cloexec_flag() + dev_fd = os.open("/dev", flags) + try: + directory = os.fstat(dev_fd) + if directory.st_uid != 0 or directory.st_gid != 0 or directory.st_mode & 0o022: + return set() + device_directory = os.open("dri", flags, dir_fd=dev_fd) + try: + directory = os.fstat(device_directory) + if directory.st_uid != 0 or directory.st_gid != 0 or directory.st_mode & 0o022: + return set() + sysfs_directory = _open_directory("/sys/class/drm") + try: + directory = os.fstat(sysfs_directory) + if directory.st_uid != 0 or directory.st_gid != 0 or directory.st_mode & 0o022: + return set() + groups: set[str] = set() + device_count = 0 + with os.scandir(sysfs_directory) as entries: + for index, entry in enumerate(entries): + # Sysfs includes connectors and devices not assigned to this container. + if index >= 32768: + return set() + match = re.fullmatch(r"(?:card|renderD)([0-9]+)", entry.name) + if not match: + continue + try: + device = os.stat( + entry.name, dir_fd=device_directory, follow_symlinks=False + ) + except FileNotFoundError: + continue + device_count += 1 + if device_count > 256: + return set() + if ( + stat.S_ISCHR(device.st_mode) + and os.major(device.st_rdev) == 226 + and os.minor(device.st_rdev) == int(match[1]) + and device.st_uid in (0, expected_uid) + and device.st_gid > 0 + and device.st_mode & 0o006 != 0o006 + ): + groups.add(str(device.st_gid)) + return groups + finally: + os.close(sysfs_directory) + finally: + os.close(device_directory) + finally: + os.close(dev_fd) + except (OSError, UnsafePathError): + return set() + + +def _capability_free_supervisor_matches( + cmdline: bytes, + status_bytes: bytes, + status_metadata: os.stat_result, + expected_uid: int, +) -> bool: + # OpenShell 0.1.2's workload boundary runs without saved root IDs or + # capabilities, using bootstrap directly or Podman's rootless launcher. + # It becomes nondumpable before reading bootstrap material, which + # makes its proc status file root-owned and prevents same-user ptrace/argv + # replacement. Accept only that pinned launch contract, not arbitrary + # same-user supervisors. PID/parent, lifetime and unique child checks remain + # in the descriptor-pinned callers below; argv alone is never authority. + try: + sandbox = pwd.getpwnam("sandbox") + except KeyError: + return False + if ( + expected_uid <= 0 + or sandbox.pw_uid != expected_uid + or sandbox.pw_gid <= 0 + or not stat.S_ISREG(status_metadata.st_mode) + or status_metadata.st_uid != 0 + or status_metadata.st_gid != 0 + ): + return False + rootless_argv = ( + OPENSHELL_SUPERVISOR_ARGV0, + b"launch-capability-free", + str(expected_uid).encode("ascii"), + str(sandbox.pw_gid).encode("ascii"), + b"/.openshell/channel/sandbox/bootstrap.json", + b"/sandbox", + ) + bootstrap = b"/.openshell/channel/sandbox/bootstrap.json" + expected_argvs = ( + rootless_argv, + (OPENSHELL_SUPERVISOR_ARGV0, b"--bootstrap", bootstrap), + (b"/.openshell/runtime/openshell-sandbox", b"--bootstrap", bootstrap), + ) + if cmdline not in tuple(b"\0".join(argv) + b"\0" for argv in expected_argvs): + return False + expected_fields = { + "Uid": [str(expected_uid)] * 4, + "Gid": [str(sandbox.pw_gid)] * 4, + "Groups": [], + "NoNewPrivs": ["1"], + **{name: ["0000000000000000"] for name in ( + "CapInh", "CapPrm", "CapEff", "CapBnd", "CapAmb" + )}, + } + observed: dict[str, list[str]] = {} + try: + for line in status_bytes.decode("ascii").splitlines(): + name, separator, value = line.partition(":") + if separator and name in expected_fields: + if name in observed: + return False + observed[name] = value.split() + except UnicodeDecodeError: + return False + # Docker preserves the primary group and CDI device groups. The rootless + # launcher still must clear supplementary groups. Reject unexplained groups. + if cmdline != b"\0".join(rootless_argv) + b"\0": + groups = observed.get("Groups", []) + allowed = {str(sandbox.pw_gid)} + if set(groups) - allowed: + allowed |= _runtime_drm_groups(expected_uid) + if len(set(groups)) == len(groups) and set(groups) <= allowed: + expected_fields["Groups"] = groups + return observed == expected_fields + + def _openshell_supervisor_identity( expected_effective_uid: int, ) -> tuple[str, int | None] | None: @@ -533,12 +666,18 @@ def _openshell_supervisor_identity( proc_root_fd = _open_proc_root() proc_pid_fd = _open_proc_pid(proc_root_fd, 1) pinned_before = os.fstat(proc_pid_fd) + # Linux deliberately leaves the world-readable /proc/ directory + # owned by the effective UID even for a nondumpable process. Its status + # entry follows task_dump_owner's nondumpable ownership rule instead. + # Resolve it without symlinks beneath the already-pinned PID directory. + first_status_metadata = os.stat("status", dir_fd=proc_pid_fd, follow_symlinks=False) first_cmdline = _read_proc_pid_file( proc_pid_fd, "cmdline", f"{PROC_ROOT}/1/cmdline" ) - first_status = _process_status_identity( - _read_proc_pid_file(proc_pid_fd, "status", f"{PROC_ROOT}/1/status") + first_status_bytes = _read_proc_pid_file( + proc_pid_fd, "status", f"{PROC_ROOT}/1/status" ) + first_status = _process_status_identity(first_status_bytes) first_stat = _read_proc_pid_file( proc_pid_fd, "stat", f"{PROC_ROOT}/1/stat" ) @@ -547,18 +686,36 @@ def _openshell_supervisor_identity( second_cmdline = _read_proc_pid_file( proc_pid_fd, "cmdline", f"{PROC_ROOT}/1/cmdline" ) - second_status = _process_status_identity( - _read_proc_pid_file(proc_pid_fd, "status", f"{PROC_ROOT}/1/status") + second_status_bytes = _read_proc_pid_file( + proc_pid_fd, "status", f"{PROC_ROOT}/1/status" ) + second_status = _process_status_identity(second_status_bytes) second_stat = _read_proc_pid_file( proc_pid_fd, "stat", f"{PROC_ROOT}/1/stat" ) second_start_time = _parse_process_start_time(second_stat) second_namespace_inode = _proc_pid_namespace_inode(proc_pid_fd) + second_status_metadata = os.stat("status", dir_fd=proc_pid_fd, follow_symlinks=False) pinned_after = os.fstat(proc_pid_fd) + if expected_effective_uid == 0: + supervisor_matches = ( + _cmdline_is_openshell_supervisor(first_cmdline) + and _cmdline_is_openshell_supervisor(second_cmdline) + ) + else: + supervisor_matches = ( + first_cmdline == second_cmdline + and first_status_metadata.st_dev == second_status_metadata.st_dev + and first_status_metadata.st_ino == second_status_metadata.st_ino + and _capability_free_supervisor_matches( + first_cmdline, first_status_bytes, first_status_metadata, expected_effective_uid + ) + and _capability_free_supervisor_matches( + second_cmdline, second_status_bytes, second_status_metadata, expected_effective_uid + ) + ) if not ( - _cmdline_is_openshell_supervisor(first_cmdline) - and _cmdline_is_openshell_supervisor(second_cmdline) + supervisor_matches and first_status is not None and second_status is not None and first_status[0] == expected_effective_uid @@ -676,7 +833,11 @@ def _openshell_supervised_nonroot_start_is_live( expected_sandbox_uid: int, required_pid: int | None = None, ) -> bool: - supervisor_identity = _openshell_supervisor_identity(expected_root_uid) + supervisor_uid = expected_root_uid + supervisor_identity = _openshell_supervisor_identity(supervisor_uid) + if supervisor_identity is None and expected_root_uid == 0 and expected_sandbox_uid > 0: + supervisor_uid = expected_sandbox_uid + supervisor_identity = _openshell_supervisor_identity(supervisor_uid) if supervisor_identity is None: return False proc_root_fd = -1 @@ -705,7 +866,7 @@ def _openshell_supervised_nonroot_start_is_live( return bool( matches == 1 and (required_pid is None or matched_pid == required_pid) - and _openshell_supervisor_identity(expected_root_uid) + and _openshell_supervisor_identity(supervisor_uid) == supervisor_identity ) except OSError: @@ -2909,18 +3070,29 @@ def _validate_restored_native_credential( upstream = source["_nemoclaw_upstream"] provider = upstream["provider"] model = source["model"] - managed = source["providers"][provider] + provider_key = upstream["provider_key"] + managed = source["providers"][provider_key] custom = source["custom_providers"] matches = [entry for entry in custom if isinstance(entry, dict) and entry.get("name") == provider] previous = model["api_key"] reference = candidate["model"]["api_key"] nvidia = provider == "nvidia-prod" - env_key = "NVIDIA_INFERENCE_API_KEY" if nvidia else "NEMOCLAW_LOCAL_INFERENCE_TOKEN" - pattern = rf"openshell:resolve:env:(?:v[0-9]{{1,20}}|s[a-f0-9]{{64}})_{env_key}" + local = provider in {"ollama-local", "vllm-local", "llama-cpp-local", "compatible-endpoint"} + previous_reference = re.fullmatch( + r"openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_([A-Z][A-Z0-9_]{0,255})", + previous if isinstance(previous, str) else "", + ) + env_key = "NVIDIA_INFERENCE_API_KEY" if nvidia else "NEMOCLAW_LOCAL_INFERENCE_TOKEN" if local else ( + previous_reference.group(1) if previous_reference else "" + ) + pattern = rf"openshell:resolve:env:(?:v[0-9]{{1,20}}|s[a-f0-9]{{64}})_{re.escape(env_key)}" if ( - provider not in {"nvidia-prod", "ollama-local", "vllm-local", "llama-cpp-local", "compatible-endpoint"} + not isinstance(provider, str) + or re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}", provider) is None + or provider_key != re.sub(r"-+", "-", provider.lower()).strip("-") + or not env_key + or (not local and not nvidia and previous_reference is None) or (nvidia and model.get("base_url") != "https://integrate.api.nvidia.com/v1") - or upstream.get("provider_key") != provider or model.get("provider") != "custom" or upstream.get("model") != model.get("default") or managed.get("default_model") != model.get("default") @@ -2929,7 +3101,7 @@ def _validate_restored_native_credential( or matches[0].get("base_url") != model.get("base_url") or not isinstance(previous, str) or ( - previous != "${" + env_key + "}" + not ((local or nvidia) and previous == "${" + env_key + "}") and re.fullmatch(pattern, previous) is None and not (recover_stripped_nvidia and nvidia and previous == "[STRIPPED_BY_MIGRATION]") ) @@ -2938,11 +3110,11 @@ def _validate_restored_native_credential( or not isinstance(reference, str) or re.fullmatch(pattern, reference) is None ): - raise ValueError("invalid managed local slots") + raise ValueError("invalid managed native slots") expected = { **source, "model": {**model, "api_key": reference}, - "providers": {**source["providers"], provider: {**managed, "api_key": reference}}, + "providers": {**source["providers"], provider_key: {**managed, "api_key": reference}}, "custom_providers": [ {**entry, "api_key": reference} if entry is matches[0] else entry for entry in custom @@ -2951,7 +3123,7 @@ def _validate_restored_native_credential( if json.dumps(expected, sort_keys=True, allow_nan=False) != json.dumps(candidate, sort_keys=True, allow_nan=False): raise ValueError("candidate changed non-credential configuration") except (AttributeError, KeyError, TypeError, ValueError, yaml.YAMLError) as exc: - raise UnsafePathError("restored Hermes write may change only its three native local credential slots") from exc + raise UnsafePathError("restored Hermes write may change only its three native credential slots") from exc def _with_issued_nvidia_slots(text: str, template: str, reference: str) -> bytes: @@ -2991,17 +3163,10 @@ def verify_nvidia_credential(hermes_dir: str, expected_model: str) -> None: raise NvidiaCredentialRejectedError("Hermes NVIDIA credential is invalid or stale; rebuild the sandbox") from exc -def _adopt_restored_native_local_source( - hermes_dir: str, hash_file: str, state_file: str, - expected_config_sha256: str | None, candidate_bytes: bytes, - hermes_meta: dict[str, int], file_states: dict[str, dict[str, object]], -) -> None: - """Accept only a restored mutable source for an exact local-reference update. - - The namespace is already frozen by seal_restart. General config writes keep - their strict-anchor requirement; this operation cannot change route or user - settings and must validate the restored environment before adopting it. - """ +def _validate_restored_native_local_source( + hermes_dir: str, expected_config_sha256: str | None, candidate_bytes: bytes, +) -> tuple[str, FileSnapshot, str, FileSnapshot]: + """Validate archived inputs and an exact three-slot credential replacement.""" config_path = os.path.join(hermes_dir, "config.yaml") env_path = os.path.join(hermes_dir, ".env") source_text, config_snapshot = _read_text(config_path, MAX_CONFIG_INPUT_BYTES) @@ -3016,6 +3181,25 @@ def _adopt_restored_native_local_source( if os.path.abspath(__file__) != INSTALLED_RUNTIME_CONFIG_GUARD: validator = os.path.join(os.path.dirname(os.path.abspath(__file__)), "validate-env-secret-boundary.py") _validate_env_text_with_boundary(env_text, validator, isolated=True) + return source_text, config_snapshot, env_text, env_snapshot + + +def _adopt_restored_native_local_source( + hermes_dir: str, hash_file: str, state_file: str, + expected_config_sha256: str | None, candidate_bytes: bytes, + hermes_meta: dict[str, int], file_states: dict[str, dict[str, object]], +) -> None: + """Accept only a restored mutable source for an exact native-reference update. + + The namespace is already frozen by seal_restart. General config writes keep + their strict-anchor requirement; this operation cannot change route or user + settings and must validate the restored environment before adopting it. + """ + config_path = os.path.join(hermes_dir, "config.yaml") + env_path = os.path.join(hermes_dir, ".env") + source_text, config_snapshot, env_text, env_snapshot = _validate_restored_native_local_source( + hermes_dir, expected_config_sha256, candidate_bytes, + ) for path, expected_snapshot in ((config_path, config_snapshot), (env_path, env_snapshot)): opened = _open_regular(path) try: diff --git a/agents/hermes/start.sh b/agents/hermes/start.sh index ce70e2c9156..791395bab8c 100755 --- a/agents/hermes/start.sh +++ b/agents/hermes/start.sh @@ -296,6 +296,10 @@ _HERMES_TIRITH_MARKER_FINALIZER="/usr/local/lib/nemoclaw/finalize-tirith-marker. if [ ! -f "$_HERMES_TIRITH_MARKER_FINALIZER" ]; then _HERMES_TIRITH_MARKER_FINALIZER="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/finalize-tirith-marker.py" fi +_HERMES_STARTUP_COMPLETION="/usr/local/lib/nemoclaw/hermes-startup-completion.py" +if [ ! -f "$_HERMES_STARTUP_COMPLETION" ]; then + _HERMES_STARTUP_COMPLETION="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/startup-completion.py" +fi _HERMES_GUARD_TIMEOUT=(timeout --signal=TERM --kill-after=5s 12m) _HERMES_BOUNDARY_TIMEOUT=(timeout --signal=TERM --kill-after=2s 15s) _HERMES_DASHBOARD_STATE_MIGRATION_TIMEOUT=(timeout --signal=TERM --kill-after=5s 30m) @@ -2353,17 +2357,10 @@ restore_hermes_config_permissions_after_dashboard_start() { # primary process whose exit status is returned). # Each code path below sets these before registering the trap. -# ── Proxy environment ──────────────────────────────────────────── -PROXY_HOST="${NEMOCLAW_PROXY_HOST:-10.200.0.1}" -PROXY_PORT="${NEMOCLAW_PROXY_PORT:-3128}" -_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}" -_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}" -export HTTP_PROXY="$_PROXY_URL" -export HTTPS_PROXY="$_PROXY_URL" -export NO_PROXY="$_NO_PROXY_VAL" -export http_proxy="$_PROXY_URL" -export https_proxy="$_PROXY_URL" -export no_proxy="$_NO_PROXY_VAL" +# OpenShell 0.1.3 mediates native egress. Clear obsolete application proxy settings. +unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy +unset ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY +unset NODE_USE_ENV_PROXY NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT # Corporate proxy CA merge (NemoClaw#6210). # OpenShell injects SSL_CERT_FILE for its own L7 proxy CA at runtime. When a @@ -2422,13 +2419,10 @@ _PROXY_ENV_FILE="/tmp/nemoclaw-proxy-env.sh" write_runtime_shell_env() { { cat < bool: + try: + fd = os.open(root_lifecycle, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) + except FileNotFoundError: + return True + try: + info = os.fstat(fd) + if (not stat.S_ISREG(info.st_mode) or info.st_nlink != 1 + or info.st_uid != 0 or info.st_gid != 0 + or stat.S_IMODE(info.st_mode) != 0o444 + or os.read(fd, 64) != b"root-separated\n"): + raise RuntimeError("Invalid Hermes root lifecycle marker") + finally: + os.close(fd) + return False + + +def process_identity(pid: int) -> dict[str, int | str] | None: + if type(pid) is not int or pid <= 0: + return None + try: + proc = Path(f"/proc/{pid}") + before = (proc / "stat").read_text() + fields = before[before.rindex(")") + 2:].split() + uid = next(line.split()[2] for line in + (proc / "status").read_text().splitlines() if line.startswith("Uid:")) + if fields[0] == "Z" or int(uid) != os.geteuid(): + return None + identity = {"pid": pid, "parent": int(fields[1]), "start": fields[19]} + after = (proc / "stat").read_text() + current = after[after.rindex(")") + 2:].split() + if current[0] == "Z" or current[19] != fields[19] or current[1] != fields[1]: + return None + return identity + except (OSError, ValueError, IndexError, StopIteration): + return None + + +def read_completion(path: str = RECEIPT) -> dict | None: + try: + fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) + except FileNotFoundError: + return None + try: + before = os.fstat(fd) + if (not stat.S_ISREG(before.st_mode) or before.st_nlink != 1 + or before.st_uid != os.geteuid() or before.st_gid != os.getegid() + or stat.S_IMODE(before.st_mode) != 0o600 or before.st_size > 1024): + raise RuntimeError("Unsafe Hermes startup completion receipt") + data = os.read(fd, 1025) + after = os.fstat(fd) + named = os.stat(path, follow_symlinks=False) + if any(getattr(before, field) != getattr(info, field) + for info in (after, named) + for field in ("st_dev", "st_ino", "st_mode", "st_uid", "st_gid", + "st_nlink", "st_size", "st_mtime_ns", "st_ctime_ns")): + raise RuntimeError("Hermes startup completion receipt changed while reading") + finally: + os.close(fd) + try: + value = json.loads(data) + if (set(value) != {"version", "supervisor", "gateway"} + or type(value["version"]) is not int or value["version"] != 1): + raise ValueError("invalid receipt fields") + for role in ("supervisor", "gateway"): + identity = value[role] + if (set(identity) != {"pid", "parent", "start"} + or type(identity["pid"]) is not int or identity["pid"] <= 0 + or type(identity["parent"]) is not int or identity["parent"] < 0 + or not isinstance(identity["start"], str) or not identity["start"].isdigit()): + raise ValueError("invalid process identity") + if value["gateway"]["parent"] != value["supervisor"]["pid"]: + raise ValueError("gateway has a different supervisor") + except (ValueError, TypeError, KeyError) as error: + raise RuntimeError("Invalid Hermes startup completion receipt") from error + if any(process_identity(value[role]["pid"]) != value[role] + for role in ("supervisor", "gateway")): + return None + return value + + +def wait_for_completion(timeout: float = 10, path: str = RECEIPT) -> dict: + deadline = time.monotonic() + timeout + while True: + receipt = read_completion(path) + if receipt is not None: + return receipt + if time.monotonic() >= deadline: + raise RuntimeError("Hermes supervisor startup completion was not observed") + time.sleep(min(0.1, max(0, deadline - time.monotonic()))) + + +def publish_completion(gateway_pid: int, path: str = RECEIPT) -> None: + if os.geteuid() == 0: + raise RuntimeError("Hermes startup completion requires the non-root supervisor") + supervisor = process_identity(os.getppid()) + gateway = process_identity(gateway_pid) + if supervisor is None or gateway is None or gateway["parent"] != supervisor["pid"]: + raise RuntimeError("Hermes startup completion requires a live supervised gateway") + # Refuse unsafe existing paths before replacing a prior generation. + read_completion(path) + receipt = {"version": 1, "supervisor": supervisor, "gateway": gateway} + fd, temporary = tempfile.mkstemp(prefix=".hermes-startup-", dir=os.path.dirname(path)) + try: + with os.fdopen(fd, "w", encoding="utf-8") as output: + json.dump(receipt, output, separators=(",", ":")) + if (process_identity(gateway_pid) != gateway + or process_identity(supervisor["pid"]) != supervisor): + raise RuntimeError("Hermes processes changed before startup completion") + os.replace(temporary, path) + finally: + if os.path.lexists(temporary): + os.unlink(temporary) + + +if __name__ == "__main__": + if len(sys.argv) != 2: + raise SystemExit("expected supervised Hermes gateway PID") + publish_completion(int(sys.argv[1])) diff --git a/agents/hermes/validate-env-secret-boundary.py b/agents/hermes/validate-env-secret-boundary.py index 3bb9b2ee8a1..d7be57fe77c 100755 --- a/agents/hermes/validate-env-secret-boundary.py +++ b/agents/hermes/validate-env-secret-boundary.py @@ -191,6 +191,7 @@ def _validate_directory_descriptor(path: str, fd: int) -> tuple[int, int, int, i sandbox_uid, sandbox_gid = sandbox_identity allowed.update( { + (sandbox_uid, sandbox_gid, 0o700), (sandbox_uid, sandbox_gid, 0o755), (sandbox_uid, sandbox_gid, 0o770), (0, sandbox_gid, 0o1775), @@ -198,7 +199,8 @@ def _validate_directory_descriptor(path: str, fd: int) -> tuple[int, int, int, i ) if (st.st_uid, st.st_gid, mode) not in allowed: raise UnsafeEnvInputError( - "/sandbox does not match a trusted owner/group/mode posture" + "/sandbox does not match a trusted owner/group/mode posture " + f"(uid={st.st_uid}, gid={st.st_gid}, mode={mode:04o})" ) elif path == os.path.dirname(INSTALLED_ENV_PATH): allowed = {(0, 0, 0o500), (0, 0, 0o700), (0, 0, 0o755)} diff --git a/agents/langchain-deepagents-code/Dockerfile b/agents/langchain-deepagents-code/Dockerfile index 837a6b962b1..cfe4727b5c5 100644 --- a/agents/langchain-deepagents-code/Dockerfile +++ b/agents/langchain-deepagents-code/Dockerfile @@ -169,7 +169,7 @@ COPY src/lib/inference/managed-dcode/identity.ts /opt/nemoclaw-deepagents-code/s COPY src/lib/inference/native-nvidia/contract.ts /opt/nemoclaw-deepagents-code/src/lib/inference/native-nvidia/contract.ts COPY src/lib/core/endpoint-url-safety.ts /opt/nemoclaw-deepagents-code/src/lib/core/endpoint-url-safety.ts COPY src/lib/inference/native-local/agent-config.ts /opt/nemoclaw-deepagents-code/src/lib/inference/native-local/agent-config.ts -COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/hosted.ts src/lib/inference/native-provider/contract.ts /opt/nemoclaw-deepagents-code/src/lib/inference/native-provider/ +COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/catalog.ts /opt/nemoclaw-deepagents-code/src/lib/inference/native-provider/ COPY agents/langchain-deepagents-code/managed-dcode-runtime.py /opt/nemoclaw-deepagents-code/managed-dcode-runtime.py COPY agents/langchain-deepagents-code/patch-managed-deepagents-code.py /opt/nemoclaw-deepagents-code/patch-managed-deepagents-code.py COPY agents/langchain-deepagents-code/patch-managed-quickjs.py /opt/nemoclaw-deepagents-code/patch-managed-quickjs.py @@ -250,8 +250,6 @@ ARG NEMOCLAW_DCODE_AUTO_APPROVAL=disabled ARG NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION=0 ARG NEMOCLAW_BUILD_ID=default ARG NEMOCLAW_DARWIN_VM_COMPAT=0 -ARG NEMOCLAW_PROXY_HOST=10.200.0.1 -ARG NEMOCLAW_PROXY_PORT=3128 RUN case "$NEMOCLAW_TOOL_DISCLOSURE" in \ progressive|direct) ;; \ @@ -275,15 +273,13 @@ RUN case "$NEMOCLAW_TOOL_DISCLOSURE" in \ # hermetic so base-image or builder observability variables cannot mask the # diagnostic under test; the post-build workflow probes the real managed chain. RUN install -d -m 0755 /usr/local/share/nemoclaw \ - && printf '%s\n' "$NEMOCLAW_PROXY_HOST" > /usr/local/share/nemoclaw/dcode-proxy-host \ - && printf '%s\n' "$NEMOCLAW_PROXY_PORT" > /usr/local/share/nemoclaw/dcode-proxy-port \ && printf '%s\n' "$NEMOCLAW_INFERENCE_BASE_URL" > /usr/local/share/nemoclaw/dcode-inference-base-url \ && printf '%s\n' "$NEMOCLAW_UPSTREAM_PROVIDER" > /usr/local/share/nemoclaw/dcode-upstream-provider \ && printf '%s\n' "$NEMOCLAW_INFERENCE_PROVIDER_ID" > /usr/local/share/nemoclaw/dcode-provider-route \ && printf '%s\n' "$NEMOCLAW_DCODE_AUTO_APPROVAL" > /usr/local/share/nemoclaw/dcode-auto-approval \ && printf '%s\n' "$NEMOCLAW_REASONING_EFFORT" > /usr/local/share/nemoclaw/dcode-reasoning-effort \ - && chown root:root /usr/local/share/nemoclaw/dcode-proxy-host /usr/local/share/nemoclaw/dcode-proxy-port /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval /usr/local/share/nemoclaw/dcode-reasoning-effort /usr/local/share/nemoclaw/dcode-provider-route \ - && chmod 0444 /usr/local/share/nemoclaw/dcode-proxy-host /usr/local/share/nemoclaw/dcode-proxy-port /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval /usr/local/share/nemoclaw/dcode-reasoning-effort /usr/local/share/nemoclaw/dcode-provider-route \ + && chown root:root /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval /usr/local/share/nemoclaw/dcode-reasoning-effort /usr/local/share/nemoclaw/dcode-provider-route \ + && chmod 0444 /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval /usr/local/share/nemoclaw/dcode-reasoning-effort /usr/local/share/nemoclaw/dcode-provider-route \ && /opt/venv/bin/python3 -I /opt/nemoclaw-deepagents-code/validate-read-only-mcp-call.py \ && rm -f /opt/nemoclaw-deepagents-code/validate-read-only-mcp-call.py \ && unset OTEL_EXPORTER_OTLP_TRACES_ENDPOINT \ diff --git a/agents/langchain-deepagents-code/dcode-launcher.sh b/agents/langchain-deepagents-code/dcode-launcher.sh index ed16b26b3b4..b90d19fa2dd 100755 --- a/agents/langchain-deepagents-code/dcode-launcher.sh +++ b/agents/langchain-deepagents-code/dcode-launcher.sh @@ -2,7 +2,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # -# Proxy-normalizing launcher for every managed Deep Agents Code entry point. +# Environment guards for every managed Deep Agents Code entry point. set -euo pipefail unset BASH_ENV ENV @@ -14,7 +14,7 @@ unset _nemoclaw_auto_approval_env readonly MANAGED_DCODE_WRAPPER="/usr/local/lib/nemoclaw/dcode-wrapper.sh" readonly MANAGED_EXEC_LAUNCHER="/usr/local/lib/nemoclaw/dcode-managed-exec" readonly MANAGED_OBSERVABILITY_MARKER="/sandbox/.deepagents/.nemoclaw-observability-enabled" -readonly MANAGED_FETCH_CA_BUNDLE_FILE="/etc/openshell-tls/ca-bundle.pem" +readonly MANAGED_FETCH_CA_BUNDLE_FILE="/run/openshell-supervisor-ca/material/ca-bundle.crt" readonly MANAGED_SESSION_SUPERVISOR="/usr/local/lib/nemoclaw/dcode-session-supervisor.py" export HOME=/sandbox export PATH="/usr/local/bin:/opt/venv/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin" @@ -60,47 +60,8 @@ if [ -f "$MANAGED_OBSERVABILITY_MARKER" ] \ export NEMOCLAW_OBSERVABILITY=1 fi -# Raw OpenShell exec processes do not inherit the entrypoint's environment or -# source shell startup files. Rebuild the proxy-only dcode contract here so a -# direct exec cannot retain the host seed and bypass the managed proxy for a -# direct inference.local DNS lookup. This stays at the agent runtime boundary -# because the shared seed is still required for OpenShell host-side chaining. -# Remove it only when OpenShell normalizes every sandbox exec/login process or -# dcode no longer uses inference.local. -readonly MANAGED_PROXY_HOST_FILE="/usr/local/share/nemoclaw/dcode-proxy-host" -readonly MANAGED_PROXY_PORT_FILE="/usr/local/share/nemoclaw/dcode-proxy-port" -readonly MANAGED_PROXY_OWNER_UID=0 - -managed_proxy_file_metadata() { - local file="$1" - local metadata - if metadata="$(stat -c '%u:%a' "$file" 2>/dev/null)"; then - printf '%s' "$metadata" - else - stat -f '%u:%Lp' "$file" 2>/dev/null - fi -} - -read_managed_proxy_value() { - local file="$1" - local name="$2" - local metadata - local value - if [ ! -f "$file" ] || [ -L "$file" ] || [ ! -r "$file" ]; then - printf 'Missing or unsafe trusted managed proxy %s file.\n' "$name" >&2 - return 1 - fi - metadata="$(managed_proxy_file_metadata "$file")" || { - printf 'Cannot inspect trusted managed proxy %s file.\n' "$name" >&2 - return 1 - } - if [ "$metadata" != "${MANAGED_PROXY_OWNER_UID}:444" ]; then - printf 'Unsafe ownership or mode on trusted managed proxy %s file.\n' "$name" >&2 - return 1 - fi - value="$(<"$file")" - printf '%s' "$value" -} +# OpenShell projects this read-only bundle as the workload identity. +readonly MANAGED_CA_OWNER_UID="$EUID" managed_fetch_ca_bundle_metadata() { local file="$1" @@ -133,66 +94,21 @@ validate_managed_fetch_ca_bundle() { || [[ ! "$owner" =~ ^[0-9]+$ ]] \ || [[ ! "$mode" =~ ^[0-7]{3,4}$ ]] \ || [[ ! "$size" =~ ^[0-9]+$ ]] \ - || [ "$owner" != "$MANAGED_PROXY_OWNER_UID" ] \ + || [ "$owner" != "$MANAGED_CA_OWNER_UID" ] \ || [ "$size" -le 0 ] \ - || (((8#$mode & 0022) != 0)); then + || ((8#$mode != 0444)); then printf '%s\n' 'Unsafe ownership or mode on managed fetch CA bundle file.' >&2 return 1 fi } -# Onboard validates the build args and the Dockerfile stores them in root-owned -# files. Runtime env is untrusted and cannot override those image-baked values. -PROXY_HOST="$(read_managed_proxy_value "$MANAGED_PROXY_HOST_FILE" "host")" -PROXY_PORT="$(read_managed_proxy_value "$MANAGED_PROXY_PORT_FILE" "port")" validate_managed_fetch_ca_bundle -unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT -# Generic proxy fallbacks are outside the managed dcode contract and may carry -# host credentials even after the scheme-specific proxy values are normalized. -unset ALL_PROXY all_proxy OPENAI_PROXY - -# This validator is applied only to image-baked values that onboard writes -# into root-owned files at build time; runtime env is explicitly unset above -# and never reaches this check. That scope is why underscores remain accepted -# for controlled internal/container aliases such as proxy_name — public DNS -# hostnames should still remain RFC 1123 names without underscores. Cross- -# boundary parity tests prevent this standalone boundary from drifting from -# start.sh or from the host-side TypeScript validator. -is_valid_proxy_host() { - local value="$1" - [[ "$value" =~ ^[A-Za-z0-9._-]+$ ]] -} - -is_valid_proxy_port() { - local value="$1" - [[ "$value" =~ ^[0-9]{1,5}$ ]] || return 1 - ((10#$value >= 1 && 10#$value <= 65535)) -} - -if ! is_valid_proxy_host "$PROXY_HOST"; then - printf '%s\n' 'Invalid NEMOCLAW_PROXY_HOST for the managed runtime proxy.' >&2 - exit 1 -fi -if ! is_valid_proxy_port "$PROXY_PORT"; then - printf '%s\n' 'Invalid NEMOCLAW_PROXY_PORT for the managed runtime proxy.' >&2 - exit 1 -fi - -_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}" -_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}" -# fetch_url cannot use its direct DNS-pinning transport inside OpenShell's -# proxy-only network namespace. Opt only this managed launch into the explicit -# trusted-proxy transport, using the same root-owned values as inference and -# shell egress. The managed package patch still ignores ambient proxy values. -export DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL="$_PROXY_URL" -export HTTP_PROXY="$_PROXY_URL" -export HTTPS_PROXY="$_PROXY_URL" -export NO_PROXY="$_NO_PROXY_VAL" -export http_proxy="$_PROXY_URL" -export https_proxy="$_PROXY_URL" -export no_proxy="$_NO_PROXY_VAL" +# OpenShell 0.1.3 mediates sockets and DNS. Ambient proxies must not replace that route. +unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy +unset ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY NODE_USE_ENV_PROXY +unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL -# Diagnostics need this launcher's image-baked proxy normalization and optional +# Diagnostics need this launcher's environment guards and optional # observability bit, but must not invoke the stateful sandbox entrypoint. Keep # the mode bound to a separate root-owned regular-file install so older images # fail before launching anything, then exact-exec without shell evaluation. diff --git a/agents/langchain-deepagents-code/generate-config.ts b/agents/langchain-deepagents-code/generate-config.ts index 8de8a58c7ed..5c6b7a9feee 100644 --- a/agents/langchain-deepagents-code/generate-config.ts +++ b/agents/langchain-deepagents-code/generate-config.ts @@ -32,6 +32,7 @@ type Settings = { providerKey: string; upstreamProvider: string; upstreamEndpointUrl: string | null; + nativeLocalCredentialHandle?: string; inferenceApi: string; reasoningEffort: ReasoningEffort | null; }; @@ -57,6 +58,7 @@ function readSettings(env: NodeJS.ProcessEnv): Settings { env.NEMOCLAW_INFERENCE_BASE_URL || "https://inference.local/v1", ), providerKey, + nativeLocalCredentialHandle: env.NEMOCLAW_LOCAL_INFERENCE_TOKEN, upstreamProvider: normalizeCommentMetadata( env.NEMOCLAW_UPSTREAM_PROVIDER || env.NEMOCLAW_INFERENCE_PROVIDER_ID || @@ -165,14 +167,19 @@ function providerConfigLines( baseUrl: string, reasoningEffort: ReasoningEffort | null, upstreamProvider: string, + nativeLocalCredentialHandle?: string, ): string[] { const nativeHosted = nativeHostedAgentConfig(upstreamProvider, baseUrl); - const apiKeyEnv = nativeLocalCredentialReference(upstreamProvider, baseUrl) + const apiKeyEnv = nativeLocalCredentialReference( + upstreamProvider, + baseUrl, + undefined, + nativeLocalCredentialHandle, + ) ? NATIVE_LOCAL_CREDENTIAL_ENV - : (nativeHosted?.credentialEnv ?? - (baseUrl === NVIDIA_HOSTED_NATIVE_ENDPOINT - ? ATTACHED_PROVIDER_API_KEY_ENV - : MANAGED_INFERENCE_API_KEY_ENV)); + : upstreamProvider === "nvidia-prod" && baseUrl === NVIDIA_HOSTED_NATIVE_ENDPOINT + ? ATTACHED_PROVIDER_API_KEY_ENV + : (nativeHosted?.credentialEnv ?? MANAGED_INFERENCE_API_KEY_ENV); return [ `[models.providers.${provider}]`, `models = ${tomlArray([model])}`, @@ -221,6 +228,7 @@ function buildConfig(settings: Settings): ManagedDeepAgentsConfig { settings.baseUrl, settings.reasoningEffort, settings.upstreamProvider, + settings.nativeLocalCredentialHandle, ), "", "[update]", diff --git a/agents/langchain-deepagents-code/managed-dcode-runtime.py b/agents/langchain-deepagents-code/managed-dcode-runtime.py index 761c902ada3..9157afebf7f 100644 --- a/agents/langchain-deepagents-code/managed-dcode-runtime.py +++ b/agents/langchain-deepagents-code/managed-dcode-runtime.py @@ -12,6 +12,7 @@ import json import os import re +import socket import stat import sys from collections.abc import Callable @@ -29,12 +30,6 @@ _UPSTREAM_PROVIDER_FILE = Path( "/usr/local/share/nemoclaw/dcode-upstream-provider" ) -_MANAGED_PROXY_HOST_FILE = Path( - "/usr/local/share/nemoclaw/dcode-proxy-host" -) -_MANAGED_PROXY_PORT_FILE = Path( - "/usr/local/share/nemoclaw/dcode-proxy-port" -) _AUTO_APPROVAL_FILE = Path( "/usr/local/share/nemoclaw/dcode-auto-approval" ) @@ -100,11 +95,9 @@ {"OPENSHELL_TLS_CA", "OPENSHELL_TLS_CERT", "OPENSHELL_TLS_KEY"} ) _UPSTREAM_PROVIDER_ENV = "NEMOCLAW_UPSTREAM_PROVIDER" -_FETCH_URL_TRUSTED_PROXY_ENV = ( - "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL" -) +# OpenShell installs its mediation CA and upstream roots before starting the workload. _MANAGED_FETCH_CA_BUNDLE_FILE = Path( - "/etc/openshell-tls/ca-bundle.pem" + "/run/openshell-supervisor-ca/material/ca-bundle.crt" ) # Keep this managed adapter allow-list in sync with generate-config.ts and the # patch-managed-deepagents-code.py provider guards injected into Deep Agents Code. @@ -114,11 +107,6 @@ ) _OPENROUTER_DISPLAY_PROVIDER_ALIASES = frozenset({"openrouter", "openrouter-api"}) _DISPLAY_PROVIDER_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}") -# Match the launchers' root-owned, image-baked proxy validator. Its deliberate -# RFC 1123 deviation permits underscores only for controlled internal/container -# aliases such as `proxy_name`; the cross-boundary cases in -# test/agents/deepagents/langchain-deepagents-code-proxy-launcher.test.ts prevent validator drift. -_MANAGED_PROXY_HOST = re.compile(r"[A-Za-z0-9._-]+") _MCP_SERVER_NAME = re.compile(r"[A-Za-z][A-Za-z0-9_-]{0,63}") _MCP_ENV_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]{0,127}") _MCP_DNS_NAME = re.compile( @@ -1082,7 +1070,8 @@ def managed_inference_api_key(base_url: str) -> str: """Select the non-secret credential for the validated image-owned route.""" parsed = urlparse(base_url) host = parsed.hostname or "" - local_host = host in {"host.openshell.internal", "host.docker.internal"} + local_alias = host in {"host.openshell.internal", "host.docker.internal"} + local_host = local_alias try: address = ipaddress.IPv4Address(host) local_host = any(address in ipaddress.IPv4Network(network) for network in ( @@ -1092,9 +1081,10 @@ def managed_inference_api_key(base_url: str) -> str: # Keep the host-alias result above; every other non-IPv4 name remains untrusted. pass upstream_provider = _managed_upstream_provider() - if parsed.scheme == "http" and local_host and upstream_provider in { - "ollama-local", "vllm-local", "llama-cpp-local", "compatible-endpoint" - }: + if parsed.scheme == "http" and local_host and ( + upstream_provider in {"ollama-local", "vllm-local", "llama-cpp-local"} + or (upstream_provider == "compatible-endpoint" and local_alias) + ): value = os.environ.get("NEMOCLAW_LOCAL_INFERENCE_TOKEN", "") if re.fullmatch( r"openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_NEMOCLAW_LOCAL_INFERENCE_TOKEN", @@ -1102,7 +1092,7 @@ def managed_inference_api_key(base_url: str) -> str: ) is None: raise RuntimeError("native local inference requires an issued OpenShell credential reference") return value - if base_url == "https://integrate.api.nvidia.com/v1" and upstream_provider != "compatible-endpoint": + if base_url == "https://integrate.api.nvidia.com/v1" and upstream_provider in _NVIDIA_DISPLAY_PROVIDER_ALIASES: name = "NVIDIA_INFERENCE_API_KEY" provider = "NVIDIA" elif base_url == "https://openrouter.ai/api/v1" and upstream_provider in ( @@ -1114,11 +1104,29 @@ def managed_inference_api_key(base_url: str) -> str: else "OPENROUTER_API_KEY" ) provider = "OpenRouter" - elif upstream_provider == "compatible-endpoint" and host not in { - "inference.local", "localhost", "127.0.0.1", "::1" - }: - name = "COMPATIBLE_API_KEY" - provider = "compatible endpoint" + elif upstream_provider in {"compatible-endpoint", "compatible-anthropic-endpoint"}: + normalized_host = host.lower().rstrip(".") + if normalized_host in { + "inference.local", "host.openshell.internal", "host.docker.internal", + "host.containers.internal", "localhost", + } or normalized_host.endswith(".localhost"): + return "nemoclaw-managed-inference" + try: + if ipaddress.ip_address(normalized_host).is_loopback: + return "nemoclaw-managed-inference" + except ValueError: + # DNS hostnames continue through hosted credential-reference validation. + pass + name = ( + "COMPATIBLE_API_KEY" if upstream_provider == "compatible-endpoint" + else "COMPATIBLE_ANTHROPIC_API_KEY" + ) + value = os.environ.get(name, "") + if re.fullmatch( + rf"openshell:resolve:env:(?:v[0-9]{{1,20}}|s[a-f0-9]{{64}})_{name}", value, + ) is None: + raise RuntimeError("custom hosted inference requires an issued OpenShell credential reference") + return value else: return "nemoclaw-managed-inference" value = os.environ.get(name, "") @@ -1127,53 +1135,12 @@ def managed_inference_api_key(base_url: str) -> str: return value -def managed_fetch_proxy_url() -> str | None: - """Return the explicit OpenShell proxy delegated to managed ``fetch_url``. - - The variable is absent when this helper is imported outside the managed - launcher, in which case the upstream direct DNS-pinning transport remains - authoritative. When present, every conventional HTTP(S) proxy variable - must carry the same launcher-derived value. This prevents a mutable ambient - proxy or ``NO_PROXY`` rule from silently replacing the root-owned route. - """ - value = os.environ.get(_FETCH_URL_TRUSTED_PROXY_ENV) - if value is None: - return None - expected_proxy_url = _managed_fetch_proxy_url_from_files() - if ( - not value - or len(value) > 2048 - or value != value.strip() - or any(ord(character) < 32 for character in value) - ): - raise RuntimeError("managed fetch URL proxy is invalid") - try: - parsed = urlparse(value) - port = parsed.port - except ValueError as exc: - raise RuntimeError("managed fetch URL proxy is invalid") from exc - if ( - parsed.scheme != "http" - or not parsed.hostname - or port is None - or port < 1 - or parsed.username is not None - or parsed.password is not None - or parsed.path not in {"", "/"} - or parsed.params - or parsed.query - or parsed.fragment - or _MANAGED_PROXY_HOST.fullmatch(parsed.hostname) is None - ): - raise RuntimeError("managed fetch URL proxy is invalid") - if value != expected_proxy_url: - raise RuntimeError( - "managed fetch URL proxy does not match root-owned proxy" - ) - for name in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"): - if os.environ.get(name) != value: - raise RuntimeError("managed fetch URL proxy does not match runtime proxy") - return value +def _managed_fetch_uses_openshell() -> bool: + """Identify the managed image through its existing root-owned provider file.""" + if not _UPSTREAM_PROVIDER_FILE.exists() and not _UPSTREAM_PROVIDER_FILE.is_symlink(): + return False + _managed_upstream_provider() + return True def _read_managed_file_value(path: Path, label: str) -> str: @@ -1204,20 +1171,6 @@ def _read_managed_file_value(path: Path, label: str) -> str: return value -def _managed_fetch_proxy_url_from_files() -> str: - """Derive the trusted proxy URL independently from root-owned files.""" - host = _read_managed_file_value(_MANAGED_PROXY_HOST_FILE, "proxy host") - port = _read_managed_file_value(_MANAGED_PROXY_PORT_FILE, "proxy port") - if _MANAGED_PROXY_HOST.fullmatch(host) is None: - raise RuntimeError("managed proxy host file has invalid contents") - if ( - re.fullmatch(r"[0-9]{1,5}", port) is None - or not 1 <= int(port, 10) <= 65535 - ): - raise RuntimeError("managed proxy port file has invalid contents") - return f"http://{host}:{port}" - - def _managed_upstream_provider() -> str: """Return the root-owned upstream provider.""" value = _read_managed_file_value( @@ -1246,8 +1199,8 @@ def _managed_fetch_ca_bundle() -> tuple[int, str]: metadata = os.fstat(descriptor) if ( not stat.S_ISREG(metadata.st_mode) - or metadata.st_uid != _MANAGED_FILE_OWNER_UID - or stat.S_IMODE(metadata.st_mode) & 0o022 + or metadata.st_uid != os.geteuid() + or stat.S_IMODE(metadata.st_mode) != 0o444 or metadata.st_size <= 0 ): raise RuntimeError("managed fetch CA bundle is invalid") @@ -1293,21 +1246,21 @@ def managed_fetch_with_redirects( max_redirects: int, original_fetch: Callable[..., Any], validation_error: type[ValueError], + pin_dns: Callable[..., Any], + is_blocked_ip: Callable[..., bool], ) -> Any: - """Fetch through only the launcher-delegated OpenShell proxy. + """Fetch through OpenShell's mediated network with fixed TLS trust. - Outside the managed launcher, preserve the pinned upstream transport. In - the managed image, avoid forbidden direct DNS while keeping requests' - ambient proxy discovery and ``NO_PROXY`` disabled. OpenShell's proxy then - remains the authoritative network-policy and SSRF boundary for every hop. + Outside the managed image, preserve the pinned upstream transport. + Reject local destinations before connection and pin each DNS result. + OpenShell applies policy to mediated destinations; requests ignores ambient + proxies, netrc credentials, and CA overrides. """ try: - proxy_url = managed_fetch_proxy_url() + managed = _managed_fetch_uses_openshell() except RuntimeError as exc: - # Keep runtime-integrity failures inside fetch_url's structured - # validation result instead of surfacing an opaque tool exception. raise validation_error(str(exc)) from exc - if proxy_url is None: + if not managed: return original_fetch(url, timeout=timeout) try: @@ -1323,7 +1276,7 @@ def managed_fetch_with_redirects( except RuntimeError as exc: raise validation_error(str(exc)) from None - def validate_url(candidate: str) -> None: + def validate_url(candidate: str) -> tuple[str, str, list[str]]: try: parsed = urlparse(candidate) hostname = parsed.hostname @@ -1342,39 +1295,78 @@ def validate_url(candidate: str) -> None: # password. An `@` or `:` after the authority is ordinary path data # (including valid repository refs/files), never authentication; # rejecting that shape would create false positives. These validation - # errors never echo the candidate URL, and the explicit OpenShell proxy - # remains the destination-policy and SSRF authority for every hop. + # errors never echo the candidate URL. if parsed.username is not None or parsed.password is not None: raise validation_error("URL credentials are not allowed") try: - hostname.encode("idna").decode("ascii") + hostname = hostname.encode("idna").decode("ascii") except UnicodeError: raise validation_error("URL hostname is not valid IDNA") from None + if "%" in hostname: + raise validation_error("URL hostname is invalid") + try: + literal = ipaddress.ip_address(hostname.rstrip(".")) + except ValueError: + try: + literal = ipaddress.IPv4Address(socket.inet_aton(hostname.rstrip("."))) + except OSError: + literal = None + if literal is not None and is_blocked_ip(literal): + raise validation_error("URL destination is blocked") + try: + answers = socket.getaddrinfo( + hostname, None, type=socket.SOCK_STREAM, proto=socket.IPPROTO_TCP + ) + except socket.gaierror: + raise validation_error("URL hostname could not be resolved") from None + addresses = [] + for answer in answers: + raw_ip = str(answer[4][0]).split("%", 1)[0] + try: + address = ipaddress.ip_address(raw_ip) + except ValueError: + raise validation_error("URL hostname returned an invalid address") from None + if isinstance(address, ipaddress.IPv6Address): + address = address.ipv4_mapped or address.sixtofour or address + # OpenShell admits synthetic DNS addresses through its policy broker. + # Ordinary loopback sockets bypass that broker and must stop here. + if address.is_loopback or address.is_unspecified: + raise validation_error("URL destination is blocked") + addresses.append(raw_ip) + if not addresses: + raise validation_error("URL hostname resolved to no addresses") + # Send the same ASCII hostname that was resolved and pinned. This avoids + # a different IDNA interpretation in the HTTP client's URL preparation. + authority = f"[{hostname}]" if ":" in hostname else hostname + if parsed.port is not None: + authority += f":{parsed.port}" + return parsed._replace(netloc=authority).geturl(), hostname, addresses current_url = url - proxies = {"http": proxy_url, "https": proxy_url} + proxies: dict[str, str] = {} try: with requests.Session() as session: # Disable every requests environment-derived session setting, including # proxy/NO_PROXY, netrc, and CA-bundle discovery. Each request receives - # the sole root-verified proxy mapping explicitly below. The separately + # an empty proxy mapping explicitly below. The separately # selected CA bundle establishes TLS transport trust only; it cannot # choose a proxy or authorize a destination under OpenShell policy. session.trust_env = False for _hop in range(max_redirects + 1): - validate_url(current_url) + current_url, hostname, addresses = validate_url(current_url) try: _rewind_managed_fetch_ca_bundle(ca_descriptor) except RuntimeError as exc: raise validation_error(str(exc)) from None - response = session.get( - current_url, - timeout=timeout, - headers={"User-Agent": "Mozilla/5.0 (compatible; DeepAgents/1.0)"}, - allow_redirects=False, - proxies=proxies, - verify=ca_bundle, - ) + with pin_dns(hostname, addresses): + response = session.get( + current_url, + timeout=timeout, + headers={"User-Agent": "Mozilla/5.0 (compatible; DeepAgents/1.0)"}, + allow_redirects=False, + proxies=proxies, + verify=ca_bundle, + ) if 300 <= response.status_code < 400: location = response.headers.get("Location") if not location: @@ -1551,7 +1543,6 @@ def assert_safe_runtime() -> None: else: os.environ.pop("NEMOCLAW_ATTACHED_PROVIDER_API_KEY", None) os.environ[_UPSTREAM_PROVIDER_ENV] = _managed_upstream_provider() - managed_fetch_proxy_url() base_url = managed_inference_base_url() os.environ["OPENAI_BASE_URL"] = base_url os.environ["NEMOCLAW_INFERENCE_BASE_URL"] = base_url @@ -1561,6 +1552,20 @@ def assert_safe_runtime() -> None: os.environ["LANGGRAPH_CLI_NO_ANALYTICS"] = "1" os.environ["OTEL_ENABLED"] = "false" for name in ( + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "ALL_PROXY", + "all_proxy", + "grpc_proxy", + "GRPC_PROXY", + "NODE_USE_ENV_PROXY", + "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL", + "NEMOCLAW_PROXY_HOST", + "NEMOCLAW_PROXY_PORT", "OPENAI_PROXY", "OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", diff --git a/agents/langchain-deepagents-code/patch-managed-deepagents-code.py b/agents/langchain-deepagents-code/patch-managed-deepagents-code.py index a6a19365148..ab135f467d1 100644 --- a/agents/langchain-deepagents-code/patch-managed-deepagents-code.py +++ b/agents/langchain-deepagents-code/patch-managed-deepagents-code.py @@ -438,7 +438,7 @@ def _preview_dotenv_environ(*, start_path=None) -> dict[str, str]: def _load_dotenv(*, start_path=None, refresh_loaded=False) -> bool: - """Disable all dotenv loading so it cannot supply the trusted fetch proxy.""" + """Disable project dotenv loading in managed processes.""" del start_path, refresh_loaded _dotenv_loaded_values.clear() return False @@ -512,15 +512,10 @@ def _get_provider_kwargs(provider: str, *, model_name: str | None = None) -> dic return kwargs ''' -# Source-of-truth boundary: upstream Deep Agents Code 0.1.87 resolves and pins -# destination DNS locally, then disables environment proxies. That is a sound -# standalone SSRF defense but cannot operate in OpenShell's proxy-only network -# namespace, where direct DNS and direct target connections are rejected. The -# managed launcher supplies an explicit, root-owned proxy URL. `trust_env=False` -# disables every Requests environment-derived session setting (proxy/NO_PROXY, -# netrc, and CA discovery); each hop receives only the explicit proxy mapping -# and separately validated fixed CA bundle. The proxy's network policy and SSRF -# checks remain authoritative. +# Reuse Deep Agents Code 0.1.87 connection pinning on every redirect. Managed +# validation permits OpenShell synthetic DNS but rejects local socket bypasses. +# Requests uses fixed TLS trust and ignores ambient proxies and netrc. +# Outside managed images, the upstream transport remains authoritative. TOOLS_PATCH = r''' # NemoClaw-managed Deep Agents Code hardening v2. @@ -528,7 +523,7 @@ def _get_provider_kwargs(provider: str, *, model_name: str | None = None) -> dic def _fetch_with_redirects(url: str, *, timeout: int): - """Use only the launcher-delegated OpenShell proxy when configured.""" + """Use OpenShell network mediation in a managed image.""" from deepagents_code._nemoclaw_managed import managed_fetch_with_redirects return managed_fetch_with_redirects( @@ -537,6 +532,8 @@ def _fetch_with_redirects(url: str, *, timeout: int): max_redirects=_MAX_FETCH_REDIRECTS, original_fetch=_nemoclaw_original_fetch_with_redirects, validation_error=_UrlValidationError, + pin_dns=_pinned_dns, + is_blocked_ip=_is_blocked_ip, ) ''' @@ -1354,6 +1351,20 @@ def _build_server_env() -> dict[str, str]: env["LANGGRAPH_CLI_NO_ANALYTICS"] = "1" env["OTEL_ENABLED"] = "false" for name in ( + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "ALL_PROXY", + "all_proxy", + "grpc_proxy", + "GRPC_PROXY", + "NODE_USE_ENV_PROXY", + "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL", + "NEMOCLAW_PROXY_HOST", + "NEMOCLAW_PROXY_PORT", "OPENAI_PROXY", "OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", @@ -1480,6 +1491,20 @@ def discover_mcp_config_sources(*, project_context=None) -> list[Path]: env["LANGGRAPH_CLI_NO_ANALYTICS"] = "1" env["OTEL_ENABLED"] = "false" for name in ( + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "ALL_PROXY", + "all_proxy", + "grpc_proxy", + "GRPC_PROXY", + "NODE_USE_ENV_PROXY", + "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL", + "NEMOCLAW_PROXY_HOST", + "NEMOCLAW_PROXY_PORT", "OPENAI_PROXY", "OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", @@ -1994,7 +2019,7 @@ def main() -> None: }, ) tools_tree = _require_functions( - paths["tools"], texts["tools"], {"_fetch_with_redirects"} + paths["tools"], texts["tools"], {"_fetch_with_redirects", "_pinned_dns", "_is_blocked_ip"} ) _require_symbols( paths["tools"], tools_tree, {"_MAX_FETCH_REDIRECTS", "_UrlValidationError"} diff --git a/agents/langchain-deepagents-code/policy-additions.yaml b/agents/langchain-deepagents-code/policy-additions.yaml index f7718712b85..fa6de95e1e9 100644 --- a/agents/langchain-deepagents-code/policy-additions.yaml +++ b/agents/langchain-deepagents-code/policy-additions.yaml @@ -20,7 +20,7 @@ filesystem_policy: - /app - /etc - /var/log - - /var/lib/dpkg # Allow package-version inspection without package mutation. + - /var/lib/dpkg # Allow package-version inspection without package mutation. read_write: - /sandbox - /sandbox/.deepagents @@ -30,10 +30,10 @@ filesystem_policy: landlock: # Deep Agents Code is a terminal coding harness, so filesystem policy must - # fail closed when Landlock cannot be applied. `strict` makes the OpenShell + # fail closed when Landlock cannot be applied. `hard_requirement` makes the OpenShell # sandbox startup fail instead of silently degrading if the kernel or workspace # mount cannot enforce these read-only system paths. - compatibility: strict + compatibility: hard_requirement process: run_as_user: sandbox diff --git a/agents/langchain-deepagents-code/start.sh b/agents/langchain-deepagents-code/start.sh index f3328c8e5c3..aa01efdc532 100755 --- a/agents/langchain-deepagents-code/start.sh +++ b/agents/langchain-deepagents-code/start.sh @@ -88,60 +88,9 @@ if ! verify_resource_limits_exact; then fi unset _NEMOCLAW_SANDBOX_RLIMITS -# Invalid state: OpenShell's sandbox-create environment contains the host proxy -# seed, including NO_PROXY=inference.local, so dcode bypasses the managed proxy -# and attempts direct DNS resolution that is not part of the dcode contract. -# Source boundary: that seed remains correct for OpenShell's host-side proxy -# chaining; this agent-owned runtime boundary is the first safe place to replace -# it without changing OpenClaw, Hermes, or global OpenShell route provisioning. -# Source-fix constraint: inference.local is an L7 managed-proxy route, so adding -# sandbox DNS/hosts state or changing the shared seed would widen this fix and -# break the host chaining contract. Direct DNS/hosts resolution is not required. -# Regression: focused tests and the live check cover login-shell, direct dcode, -# and connect paths when the direct DNS/hosts lookup is absent. -# Removal condition: remove this normalization only when OpenShell guarantees -# the managed proxy and normalized NO_PROXY for every sandbox exec/login process, -# or when dcode no longer uses inference.local. -readonly MANAGED_PROXY_HOST_FILE="/usr/local/share/nemoclaw/dcode-proxy-host" -readonly MANAGED_PROXY_PORT_FILE="/usr/local/share/nemoclaw/dcode-proxy-port" -if [ -e /etc/openshell-tls/ca-bundle.pem ] \ - || [ -L /etc/openshell-tls/ca-bundle.pem ]; then - readonly MANAGED_FETCH_CA_BUNDLE_FILE="/etc/openshell-tls/ca-bundle.pem" -else - readonly MANAGED_FETCH_CA_BUNDLE_FILE="/tmp/nemoclaw-managed-startup-ca-bundle.pem" -fi -readonly MANAGED_PROXY_OWNER_UID=0 - -managed_proxy_file_metadata() { - local file="$1" - local metadata - if metadata="$(stat -c '%u:%a' "$file" 2>/dev/null)"; then - printf '%s' "$metadata" - else - stat -f '%u:%Lp' "$file" 2>/dev/null - fi -} - -read_managed_proxy_value() { - local file="$1" - local name="$2" - local metadata - local value - if [ ! -f "$file" ] || [ -L "$file" ] || [ ! -r "$file" ]; then - printf 'Missing or unsafe trusted managed proxy %s file.\n' "$name" >&2 - return 1 - fi - metadata="$(managed_proxy_file_metadata "$file")" || { - printf 'Cannot inspect trusted managed proxy %s file.\n' "$name" >&2 - return 1 - } - if [ "$metadata" != "${MANAGED_PROXY_OWNER_UID}:444" ]; then - printf 'Unsafe ownership or mode on trusted managed proxy %s file.\n' "$name" >&2 - return 1 - fi - value="$(<"$file")" - printf '%s' "$value" -} +readonly MANAGED_FETCH_CA_BUNDLE_FILE="/run/openshell-supervisor-ca/material/ca-bundle.crt" +# OpenShell projects this read-only bundle as the workload identity. +readonly MANAGED_CA_OWNER_UID="$EUID" managed_fetch_ca_bundle_metadata() { local file="$1" @@ -174,18 +123,14 @@ validate_managed_fetch_ca_bundle() { || [[ ! "$owner" =~ ^[0-9]+$ ]] \ || [[ ! "$mode" =~ ^[0-7]{3,4}$ ]] \ || [[ ! "$size" =~ ^[0-9]+$ ]] \ - || [ "$owner" != "$MANAGED_PROXY_OWNER_UID" ] \ + || [ "$owner" != "$MANAGED_CA_OWNER_UID" ] \ || [ "$size" -le 0 ] \ - || (((8#$mode & 0022) != 0)); then + || ((8#$mode != 0444)); then printf '%s\n' 'Unsafe ownership or mode on managed fetch CA bundle file.' >&2 return 1 fi } -# Fail closed if the root-owned image contract is missing. Process-level -# NEMOCLAW_PROXY_* values are not a trusted runtime routing source. -PROXY_HOST="$(read_managed_proxy_value "$MANAGED_PROXY_HOST_FILE" "host")" -PROXY_PORT="$(read_managed_proxy_value "$MANAGED_PROXY_PORT_FILE" "port")" validate_managed_fetch_ca_bundle if [ -e "$MANAGED_FETCH_CA_BUNDLE_FILE" ]; then : "${SSL_CERT_FILE:=$MANAGED_FETCH_CA_BUNDLE_FILE}" @@ -193,54 +138,10 @@ if [ -e "$MANAGED_FETCH_CA_BUNDLE_FILE" ]; then : "${NODE_EXTRA_CA_CERTS:=$MANAGED_FETCH_CA_BUNDLE_FILE}" export SSL_CERT_FILE REQUESTS_CA_BUNDLE NODE_EXTRA_CA_CERTS fi -unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT -# Generic proxy fallbacks are outside the managed dcode contract and may carry -# host credentials even after the scheme-specific proxy values are normalized. -unset ALL_PROXY all_proxy OPENAI_PROXY - -# Keep this validator behavior identical to the host-side TypeScript boundary. -# It is applied only to image-baked values that onboard writes into root-owned -# files at build time; runtime env is explicitly unset above and never reaches -# this check. Underscores remain accepted for controlled internal/container -# aliases such as proxy_name; public DNS hostnames should remain RFC 1123 -# names without them. Schemes, credentials, separators, and whitespace are -# still rejected. -is_valid_proxy_host() { - local value="$1" - [[ "$value" =~ ^[A-Za-z0-9._-]+$ ]] -} - -is_valid_proxy_port() { - local value="$1" - [[ "$value" =~ ^[0-9]{1,5}$ ]] || return 1 - ((10#$value >= 1 && 10#$value <= 65535)) -} - -if ! is_valid_proxy_host "$PROXY_HOST"; then - printf '%s\n' 'Invalid NEMOCLAW_PROXY_HOST for the managed runtime proxy.' >&2 - exit 1 -fi -if ! is_valid_proxy_port "$PROXY_PORT"; then - printf '%s\n' 'Invalid NEMOCLAW_PROXY_PORT for the managed runtime proxy.' >&2 - exit 1 -fi - -_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}" -_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}" -# Deep Agents Code 0.1.87 intentionally ignores environment proxies in -# fetch_url so it can pin direct DNS results against rebinding. OpenShell's -# sandbox instead requires all ordinary egress, including DNS resolution for a -# destination, to stay behind its policy proxy. This explicit variable opts the -# managed package patch into that trusted proxy boundary without teaching the -# upstream tool to trust arbitrary ambient HTTP_PROXY values. It is derived -# only from the root-owned image files validated above. -export DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL="$_PROXY_URL" -export HTTP_PROXY="$_PROXY_URL" -export HTTPS_PROXY="$_PROXY_URL" -export NO_PROXY="$_NO_PROXY_VAL" -export http_proxy="$_PROXY_URL" -export https_proxy="$_PROXY_URL" -export no_proxy="$_NO_PROXY_VAL" +# OpenShell 0.1.3 mediates sockets and DNS. Ambient proxies must not replace that route. +unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy +unset ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY NODE_USE_ENV_PROXY +unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL write_export_if_set() { local name="$1" @@ -251,7 +152,7 @@ write_export_if_set() { prepare_runtime_env() { # Unlike prepare_observability_marker below, this file is intentionally - # volatile: every stateful start rebuilds it from root-owned proxy inputs. + # volatile: every stateful start rebuilds its environment guards. local target=/tmp/nemoclaw-proxy-env.sh local tmp tmp="$(mktemp /tmp/nemoclaw-proxy-env.XXXXXX)" @@ -273,23 +174,16 @@ prepare_runtime_env() { printf '%s\n' 'export LANGCHAIN_TRACING_V2=false' printf '%s\n' 'export DEEPAGENTS_CODE_OFFLINE=1' printf '%s\n' 'export DEEPAGENTS_CODE_RIPGREP_INSTALLER=system' - # Intentionally omit the trusted proxy when unset: its absence signals - # unmanaged mode, where the upstream fetch transport remains authoritative. - write_export_if_set DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL # shellcheck disable=SC2016 printf '%s\n' 'export DEEPAGENTS_CODE_OPENAI_API_KEY="${DEEPAGENTS_CODE_OPENAI_API_KEY:-nemoclaw-managed-inference}"' printf '%s\n' 'unset NEMOCLAW_ATTACHED_PROVIDER_API_KEY' # shellcheck disable=SC2016 printf '%s\n' 'export OPENAI_BASE_URL="${OPENAI_BASE_URL:-https://inference.local/v1}"' - printf '%s\n' 'unset ALL_PROXY all_proxy OPENAI_PROXY' - write_export_if_set HTTP_PROXY - write_export_if_set HTTPS_PROXY - write_export_if_set NO_PROXY - write_export_if_set http_proxy - write_export_if_set https_proxy - write_export_if_set no_proxy + printf '%s\n' 'unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy' + printf '%s\n' 'unset ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY NODE_USE_ENV_PROXY' + printf '%s\n' 'unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL' printf '_nemoclaw_dcode_ca_bundle=%q\n' "$MANAGED_FETCH_CA_BUNDLE_FILE" - printf '_nemoclaw_dcode_ca_owner_uid=%q\n' "$MANAGED_PROXY_OWNER_UID" + printf '_nemoclaw_dcode_ca_owner_uid=%q\n' "$MANAGED_CA_OWNER_UID" cat <<'NEMOCLAW_DCODE_CA_GUARD' if [ -L "$_nemoclaw_dcode_ca_bundle" ]; then printf "%s\n" "Missing or unsafe managed fetch CA bundle file." >&2 @@ -321,7 +215,7 @@ NEMOCLAW_DCODE_CA_METADATA || [ -n "${_nemoclaw_dcode_ca_extra:-}" ] \ || [ "$_nemoclaw_dcode_ca_owner" != "$_nemoclaw_dcode_ca_owner_uid" ] \ || [ "$_nemoclaw_dcode_ca_size" -le 0 ] \ - || [ "$((0$_nemoclaw_dcode_ca_mode & 022))" -ne 0 ]; then + || [ "$((0$_nemoclaw_dcode_ca_mode))" -ne "$((0444))" ]; then printf "%s\n" "Unsafe ownership or mode on managed fetch CA bundle file." >&2 return 1 2>/dev/null || exit 1 fi diff --git a/agents/pi/Dockerfile b/agents/pi/Dockerfile index edd3a40bb9b..6adebc36351 100644 --- a/agents/pi/Dockerfile +++ b/agents/pi/Dockerfile @@ -129,7 +129,7 @@ COPY agents/pi/generate-config.ts agents/pi/check-native-credentials.ts /opt/nem COPY src/lib/core/endpoint-url-safety.ts /src/lib/core/endpoint-url-safety.ts COPY src/lib/inference/native-local/agent-config.ts /src/lib/inference/native-local/agent-config.ts COPY src/lib/inference/native-nvidia/contract.ts /src/lib/inference/native-nvidia/contract.ts -COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/hosted.ts src/lib/inference/native-provider/contract.ts /src/lib/inference/native-provider/ +COPY src/lib/inference/native-provider/agent-config.ts src/lib/inference/native-provider/catalog.ts /src/lib/inference/native-provider/ COPY scripts/lib/entrypoint-env-wrapper.sh /usr/local/lib/nemoclaw/entrypoint-env-wrapper.sh COPY agents/pi/start.sh /usr/local/bin/nemoclaw-start COPY scripts/managed-startup-hold.sh /usr/local/bin/nemoclaw-managed-startup-hold @@ -153,22 +153,12 @@ ARG NEMOCLAW_REASONING= ARG NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION=0 ARG NEMOCLAW_BUILD_ID=default ARG NEMOCLAW_DARWIN_VM_COMPAT=0 -ARG NEMOCLAW_PROXY_HOST=10.200.0.1 -ARG NEMOCLAW_PROXY_PORT=3128 RUN case "$NEMOCLAW_INFERENCE_API" in \ openai-completions) ;; \ *) echo "ERROR: NEMOCLAW_INFERENCE_API must be openai-completions for Pi" >&2; exit 1 ;; \ esac -# The startup script reads the root-owned proxy host and port files instead of -# trusting process-level environment overrides. -RUN install -d -m 0755 /usr/local/share/nemoclaw \ - && printf '%s\n' "$NEMOCLAW_PROXY_HOST" > /usr/local/share/nemoclaw/pi-proxy-host \ - && printf '%s\n' "$NEMOCLAW_PROXY_PORT" > /usr/local/share/nemoclaw/pi-proxy-port \ - && chown root:root /usr/local/share/nemoclaw/pi-proxy-host /usr/local/share/nemoclaw/pi-proxy-port \ - && chmod 0444 /usr/local/share/nemoclaw/pi-proxy-host /usr/local/share/nemoclaw/pi-proxy-port - # hadolint ignore=DL3064 ENV HOME=/sandbox \ PATH="/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin" \ diff --git a/agents/pi/check-native-credentials.ts b/agents/pi/check-native-credentials.ts index 35de072d0ef..b46388d5bec 100644 --- a/agents/pi/check-native-credentials.ts +++ b/agents/pi/check-native-credentials.ts @@ -7,7 +7,7 @@ import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; -import { HOSTED_NATIVE_PROVIDERS } from "../../src/lib/inference/native-provider/hosted.ts"; +import { HOSTED_NATIVE_PROVIDERS } from "../../src/lib/inference/native-provider/catalog.ts"; import { NVIDIA_HOSTED_CREDENTIAL_ENV, NVIDIA_HOSTED_LOGICAL_PROVIDER, diff --git a/agents/pi/generate-config.ts b/agents/pi/generate-config.ts index d51a686ae83..0a71b5b168e 100644 --- a/agents/pi/generate-config.ts +++ b/agents/pi/generate-config.ts @@ -39,6 +39,7 @@ const MANAGED_PROVIDER_ID = "openshell"; const MANAGED_PROVIDER_API_KEY = "nemoclaw-managed-inference"; type Settings = { + nativeLocalCredentialHandle?: string; model: string; baseUrl: string; providerKey: string; @@ -129,6 +130,7 @@ function readSettings(env: NodeJS.ProcessEnv): Settings { "NEMOCLAW_INFERENCE_PROVIDER_ID", ); return { + nativeLocalCredentialHandle: env.NEMOCLAW_LOCAL_INFERENCE_TOKEN, model: normalizeMetadata(readRequiredEnv(env, "NEMOCLAW_MODEL"), "NEMOCLAW_MODEL"), baseUrl: normalizeInferenceBaseUrl( env.NEMOCLAW_INFERENCE_BASE_URL || "https://inference.local/v1", @@ -159,6 +161,7 @@ function buildConfig(settings: Settings): ManagedPiConfig { settings.upstreamProvider, settings.baseUrl, "pi", + settings.nativeLocalCredentialHandle, ); const credentialEnv = settings.baseUrl === NVIDIA_HOSTED_NATIVE_ENDPOINT diff --git a/agents/pi/policy-additions.yaml b/agents/pi/policy-additions.yaml index e8845c5fd9c..0fe925dadfc 100644 --- a/agents/pi/policy-additions.yaml +++ b/agents/pi/policy-additions.yaml @@ -24,9 +24,9 @@ filesystem_policy: landlock: # Pi is a terminal agent, so filesystem policy must fail closed when Landlock # cannot be applied. If the kernel or the workspace mount cannot enforce these - # read-only system paths, `strict` fails the OpenShell sandbox startup instead + # read-only system paths, `hard_requirement` fails the OpenShell sandbox startup instead # of starting with the paths unenforced. - compatibility: strict + compatibility: hard_requirement process: run_as_user: sandbox run_as_group: sandbox diff --git a/agents/pi/start.sh b/agents/pi/start.sh index 155e7a80bad..0d1219684c5 100755 --- a/agents/pi/start.sh +++ b/agents/pi/start.sh @@ -75,70 +75,11 @@ if ! verify_resource_limits_exact; then fi unset _NEMOCLAW_SANDBOX_RLIMITS -readonly MANAGED_PROXY_HOST_FILE="/usr/local/share/nemoclaw/pi-proxy-host" -readonly MANAGED_PROXY_PORT_FILE="/usr/local/share/nemoclaw/pi-proxy-port" -readonly MANAGED_PROXY_OWNER_UID=0 - -read_managed_proxy_value() { - local file="$1" - local name="$2" - local metadata - if [ ! -f "$file" ] || [ -L "$file" ] || [ ! -r "$file" ]; then - printf 'Missing or unsafe trusted managed proxy %s file.\n' "$name" >&2 - return 1 - fi - metadata="$(stat -c '%u:%a' "$file" 2>/dev/null)" || { - printf 'Cannot inspect trusted managed proxy %s file.\n' "$name" >&2 - return 1 - } - if [ "$metadata" != "${MANAGED_PROXY_OWNER_UID}:444" ]; then - printf 'Unsafe ownership or mode on trusted managed proxy %s file.\n' "$name" >&2 - return 1 - fi - printf '%s' "$(<"$file")" -} - -# Fail closed if the root-owned image contract is missing. Process-level -# NEMOCLAW_PROXY_* values are not a trusted runtime routing source. -PROXY_HOST="$(read_managed_proxy_value "$MANAGED_PROXY_HOST_FILE" "host")" -PROXY_PORT="$(read_managed_proxy_value "$MANAGED_PROXY_PORT_FILE" "port")" -unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT -# Generic proxy fallbacks are outside the managed Pi contract and may carry host -# credentials even after the scheme-specific proxy values are normalized. -unset ALL_PROXY all_proxy OPENAI_PROXY - -# These two patterns must match isValidProxyHost and isValidProxyPort in -# src/lib/onboard/dockerfile-patch.ts. They apply only to image-baked values that -# onboard writes into root-owned files at build time; runtime env is explicitly -# unset above and never reaches this check. -is_valid_proxy_host() { - local value="$1" - [[ "$value" =~ ^[A-Za-z0-9._-]+$ ]] -} - -is_valid_proxy_port() { - local value="$1" - [[ "$value" =~ ^[0-9]{1,5}$ ]] || return 1 - ((10#$value >= 1 && 10#$value <= 65535)) -} - -if ! is_valid_proxy_host "$PROXY_HOST"; then - printf '%s\n' 'Invalid NEMOCLAW_PROXY_HOST for the managed runtime proxy.' >&2 - exit 1 -fi -if ! is_valid_proxy_port "$PROXY_PORT"; then - printf '%s\n' 'Invalid NEMOCLAW_PROXY_PORT for the managed runtime proxy.' >&2 - exit 1 -fi - -_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}" -_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}" -export HTTP_PROXY="$_PROXY_URL" -export HTTPS_PROXY="$_PROXY_URL" -export NO_PROXY="$_NO_PROXY_VAL" -export http_proxy="$_PROXY_URL" -export https_proxy="$_PROXY_URL" -export no_proxy="$_NO_PROXY_VAL" +# OpenShell 0.1.3 mediates sandbox sockets. Ambient proxy settings must not +# redirect Pi traffic to the retired application proxy or a host proxy. +unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy \ + ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY NODE_USE_ENV_PROXY \ + NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT # Corporate proxy CA merge (NemoClaw#6210). # OpenShell injects SSL_CERT_FILE for its own L7 proxy CA at runtime. When a @@ -322,7 +263,7 @@ write_export_if_set() { prepare_runtime_env() { # This file is intentionally volatile: it holds no state that must survive a - # restart, so every start rebuilds it from the root-owned proxy files. + # restart, so every start rebuilds it from the sanitized runtime environment. local target=/tmp/nemoclaw-proxy-env.sh local tmp tmp="$(mktemp /tmp/nemoclaw-proxy-env.XXXXXX)" @@ -332,13 +273,10 @@ prepare_runtime_env() { printf '%s\n' 'export PATH="/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin"' printf '%s\n' 'export PI_OFFLINE=1' printf '%s\n' 'export PI_TELEMETRY=0' - printf '%s\n' 'unset ALL_PROXY all_proxy OPENAI_PROXY' - write_export_if_set HTTP_PROXY - write_export_if_set HTTPS_PROXY - write_export_if_set NO_PROXY - write_export_if_set http_proxy - write_export_if_set https_proxy - write_export_if_set no_proxy + printf '%s\n' \ + 'unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy' \ + 'unset ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY NODE_USE_ENV_PROXY' \ + 'unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT' write_export_if_set SSL_CERT_FILE write_export_if_set CURL_CA_BUNDLE write_export_if_set REQUESTS_CA_BUNDLE @@ -347,10 +285,8 @@ prepare_runtime_env() { write_export_if_set NEMOCLAW_SANDBOX_NAME } >"$tmp" # This sandbox-user-owned file is credential-free convenience state for - # independent login and exec shells, not an integrity boundary: the entrypoint - # re-derives trusted proxy values from the root-owned image files. No Pi scan - # currently checks this file's contents; mode 0444 removes write bits so - # ordinary accidental writes fail. + # independent login and exec shells. Mode 0444 prevents accidental writes; + # it does not make sandbox-user-owned content an integrity boundary. chmod 444 "$tmp" mv -f "$tmp" "$target" } diff --git a/ci/e2e-assertion-budget.json b/ci/e2e-assertion-budget.json index 8c9f1d5ff39..fecf60879b7 100644 --- a/ci/e2e-assertion-budget.json +++ b/ci/e2e-assertion-budget.json @@ -1,5 +1,5 @@ { - "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nInitial no-growth baseline for issue #10934 and Epic #10920. The epic reference remains fixed at fe18b2f004e2304c52db33cf7bb5895828197f9a. Current main is lower because #10722 removed the live Shields assertion suites before this ratchet landed. PR #10341 adds the accepted public deferred-install and same-home credential continuation boundary for Hermes and Deep Agents Code; existing per-file budgets remain unchanged. PR #13055 adds five explicitly approved assertions for the native Deep Agents Code OpenRouter request in the existing hosted OpenRouter target. PR #13293 consolidates native-local cases into the existing OpenClaw switching owner. Removing duplicated deterministic metadata and direct mock requests lowers suite totals while retaining real authenticated and sentinel agent turns, lifecycle, model switching, credential isolation and sibling denial. Per-file transitive counts include existing imported helpers; no live files or suite assertion points are added.\n\nPR #13596 adds four assertions to the existing double-onboard target for #13595: the managed OpenClaw state volume exists before `nemoclaw uninstall`, uninstall succeeds, the volume is gone afterwards, and same-name onboarding succeeds.\n\nPR #14000 adds one approved assertion point for owning-gateway connect session detection in concurrent-gateway-ports. The list command exit check replaces the redundant CLI build-file check; the existing --version prerequisite verifies that the compiled CLI runs. Approval: https://github.com/NVIDIA/NemoClaw/pull/14000#issuecomment-6506269922.\n\nPR #13607 adds two live assertions and one read-only configuration probe to the existing balanced OpenClaw egress case for unavailable-search replies. The refreshed delta removes the unrelated public-fetch precondition guard while retaining both live assertions and the probe. Approved delta: https://github.com/NVIDIA/NemoClaw/pull/13607#issuecomment-6327887365.\n\nPR #14122 replaces managed-vLLM binding-shape checks with direct allowed-address and denied-address requests. Source tests retain the default/custom port and authentication permutations; runtime inspection remains in the artifacts.\nThe same PR adds the production bearerless provider-validation proof to the existing inference-routing target. The combined change adds two assertion points and one runtime probe, approved at https://github.com/NVIDIA/NemoClaw/pull/14122#issuecomment-6596327330.", + "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nInitial no-growth baseline for issue #10934 and Epic #10920. The epic reference remains fixed at fe18b2f004e2304c52db33cf7bb5895828197f9a. Current main is lower because #10722 removed the live Shields assertion suites before this ratchet landed. PR #10341 adds the accepted public deferred-install and same-home credential continuation boundary for Hermes and Deep Agents Code; existing per-file budgets remain unchanged. PR #13055 adds five explicitly approved assertions for the native Deep Agents Code OpenRouter request in the existing hosted OpenRouter target. PR #13293 consolidates native-local cases into the existing OpenClaw switching owner. Removing duplicated deterministic metadata and direct mock requests lowers suite totals while retaining real authenticated and sentinel agent turns, lifecycle, model switching, credential isolation and sibling denial. Per-file transitive counts include existing imported helpers; no live files or suite assertion points are added.\n\nPR #13596 adds four assertions to the existing double-onboard target for #13595: the managed OpenClaw state volume exists before `nemoclaw uninstall`, uninstall succeeds, the volume is gone afterwards, and same-name onboarding succeeds.\n\nPR #14000 adds one approved assertion point for owning-gateway connect session detection in concurrent-gateway-ports. The list command exit check replaces the redundant CLI build-file check; the existing --version prerequisite verifies that the compiled CLI runs. Approval: https://github.com/NVIDIA/NemoClaw/pull/14000#issuecomment-6506269922.\n\nPR #13607 adds two live assertions and one read-only configuration probe to the existing balanced OpenClaw egress case for unavailable-search replies. The refreshed delta removes the unrelated public-fetch precondition guard while retaining both live assertions and the probe. Approved delta: https://github.com/NVIDIA/NemoClaw/pull/13607#issuecomment-6327887365.\n\nUser-approved exception on 2026-11-03: PR #12603 repairs the existing MCP credential absence scan. One input guard and five generated probe blocks with eight conditions reject empty credentials, early credential matches, unreadable paths and grep errors. Existing scan call sites and timeouts remain unchanged.\n\nUser-approved assertion-accounting exception on 2026-11-03: PR #12603 makes the existing snapshot residue check fail when the directory cannot be read. The replacement adds one counted generated block, one condition, and one MCP transitive block. The former one-line shell expression was not counted. Runtime deadlines and coverage remain unchanged.\n\nUser-approved exception on 2026-11-04: PR #12603 adds two identity assertions to the existing managed-image activation cleanup. They require a valid sandbox identity and a nonempty cleanup scope before checking runtime-owned control volumes. Existing lifecycle coverage and runtime deadlines remain unchanged.\n\nPR #14122 replaces managed-vLLM binding-shape checks with direct allowed-address and denied-address requests. Source tests retain the default/custom port and authentication permutations; runtime inspection remains in the artifacts.\nThe same PR adds the production bearerless provider-validation proof to the existing inference-routing target. The combined change adds two assertion points and one runtime probe, approved at https://github.com/NVIDIA/NemoClaw/pull/14122#issuecomment-6596327330.", "schemaVersion": 1, "issue": 10934, "epic": 10920, @@ -13,30 +13,30 @@ }, "limits": { "testFileCount": 77, - "liveFileCount": 189, + "liveFileCount": 188, "direct": { - "expectCalls": 1286, - "matcherAssertions": 1269, - "nodeAssertions": 109, - "namedAssertionHelpers": 421, + "expectCalls": 1272, + "matcherAssertions": 1255, + "nodeAssertions": 108, + "namedAssertionHelpers": 415, "failCalls": 0, "throwGuards": 50, - "objectFieldAssertions": 184, - "assertionPoints": 2033, - "generatedProbeBlocks": 90, - "generatedProbeConditions": 213 + "objectFieldAssertions": 182, + "assertionPoints": 2010, + "generatedProbeBlocks": 86, + "generatedProbeConditions": 209 }, "unique": { - "expectCalls": 1714, - "matcherAssertions": 1692, - "nodeAssertions": 132, - "namedAssertionHelpers": 668, + "expectCalls": 1695, + "matcherAssertions": 1673, + "nodeAssertions": 131, + "namedAssertionHelpers": 663, "failCalls": 1, - "throwGuards": 531, - "objectFieldAssertions": 266, - "assertionPoints": 3290, - "generatedProbeBlocks": 219, - "generatedProbeConditions": 725 + "throwGuards": 520, + "objectFieldAssertions": 257, + "assertionPoints": 3245, + "generatedProbeBlocks": 201, + "generatedProbeConditions": 712 }, "fileMetricOrder": [ "directExpectCalls", @@ -47,70 +47,70 @@ ], "files": { "test/e2e/live/agent-turn-latency.test.ts": [15,15,15,26,1], - "test/e2e/live/bedrock-runtime-compatible-anthropic.test.ts": [11,33,12,55,4], - "test/e2e/live/bootstrap-install-smoke.test.ts": [0,0,14,28,3], + "test/e2e/live/bedrock-runtime-compatible-anthropic.test.ts": [11,33,12,54,4], + "test/e2e/live/bootstrap-install-smoke.test.ts": [0,0,10,28,1], "test/e2e/live/brave-search.test.ts": [5,10,5,10,1], "test/e2e/live/brev-workspace-cleanup.test.ts": [0,0,0,1,0], - "test/e2e/live/channels-add-remove.test.ts": [40,90,40,92,0], - "test/e2e/live/channels-stop-start.test.ts": [0,0,26,99,9], + "test/e2e/live/channels-add-remove.test.ts": [40,89,40,91,0], + "test/e2e/live/channels-stop-start.test.ts": [0,0,26,96,9], "test/e2e/live/cloud-onboard.test.ts": [22,24,22,24,1], "test/e2e/live/common-egress-agent.test.ts": [27,63,29,71,6], "test/e2e/live/concurrent-gateway-ports.test.ts": [28,36,28,36,7], "test/e2e/live/cron-preflight-inference-local.test.ts": [3,8,3,8,0], "test/e2e/live/dashboard-remote-bind.test.ts": [23,22,23,24,9], "test/e2e/live/deferred-onboarding.test.ts": [12,14,12,14,0], - "test/e2e/live/dgx-express.test.ts": [36,49,47,68,3], + "test/e2e/live/dgx-express.test.ts": [40,51,48,65,3], "test/e2e/live/double-onboard.test.ts": [46,51,46,51,0], "test/e2e/live/external-gateway-health.test.ts": [4,5,4,11,0], "test/e2e/live/full-e2e.test.ts": [26,29,27,59,8], "test/e2e/live/gpu-double-onboard.test.ts": [21,24,21,27,1], "test/e2e/live/gpu-e2e.test.ts": [36,44,69,90,3], - "test/e2e/live/hermes-discord.test.ts": [9,19,16,58,12], + "test/e2e/live/hermes-discord.test.ts": [9,19,16,55,12], "test/e2e/live/hermes-e2e.test.ts": [82,93,114,129,5], "test/e2e/live/hermes-gpu-startup.test.ts": [19,27,59,84,12], - "test/e2e/live/hermes-inference-switch.test.ts": [37,45,53,84,1], + "test/e2e/live/hermes-inference-switch.test.ts": [38,46,53,84,1], "test/e2e/live/hermes-root-entrypoint-smoke.test.ts": [4,25,4,25,4], "test/e2e/live/hermes-sandbox-secret-boundary.test.ts": [16,23,16,23,3], - "test/e2e/live/hermes-slack-e2e.test.ts": [0,0,29,87,14], - "test/e2e/live/inference-routing-provider-smoke.test.ts": [31,44,42,63,2], - "test/e2e/live/inference-routing.test.ts": [51,73,63,111,16], + "test/e2e/live/hermes-slack-e2e.test.ts": [0,0,29,84,14], + "test/e2e/live/inference-routing-provider-smoke.test.ts": [31,44,39,58,2], + "test/e2e/live/inference-routing.test.ts": [45,66,54,106,7], "test/e2e/live/issue-4434-tui-unreachable-inference.test.ts": [28,32,28,32,0], "test/e2e/live/issue-4462-scope-upgrade-approval.test.ts": [8,18,8,18,0], "test/e2e/live/issue-9880-staging-launchable.test.ts": [0,0,0,0,0], "test/e2e/live/jetson-nvmap-gpu.test.ts": [37,50,37,50,4], "test/e2e/live/launch-readiness-lease-acceptance.test.ts": [6,6,6,10,6], - "test/e2e/live/launchable-smoke.test.ts": [14,28,14,28,3], + "test/e2e/live/launchable-smoke.test.ts": [10,22,10,28,1], "test/e2e/live/llama-cpp-generic-gpu.test.ts": [17,35,33,61,2], - "test/e2e/live/managed-image-activation-e2e.test.ts": [0,0,12,25,0], + "test/e2e/live/managed-image-activation-e2e.test.ts": [0,0,14,26,0], "test/e2e/live/managed-image-multiarch-startup.test.ts": [1,3,1,9,0], - "test/e2e/live/managed-image-protected-runtime.test.ts": [0,0,39,81,16], - "test/e2e/live/mcp-bridge.test.ts": [39,138,193,518,46], - "test/e2e/live/messaging-compatible-endpoint.test.ts": [13,19,13,28,2], - "test/e2e/live/messaging-providers.test.ts": [0,11,6,51,11], + "test/e2e/live/managed-image-protected-runtime.test.ts": [0,0,39,76,15], + "test/e2e/live/mcp-bridge.test.ts": [39,137,190,514,33], + "test/e2e/live/messaging-compatible-endpoint.test.ts": [12,18,12,27,2], + "test/e2e/live/messaging-providers.test.ts": [0,11,6,48,11], "test/e2e/live/model-router-provider-routed-inference.test.ts": [1,2,1,2,0], "test/e2e/live/native-runtime-qualification-case.test.ts": [0,0,9,84,0], "test/e2e/live/network-policy.test.ts": [18,19,18,19,1], "test/e2e/live/ollama-auth-proxy.test.ts": [21,29,21,29,0], - "test/e2e/live/onboard-policy-preset-sequencing.test.ts": [7,7,7,18,1], + "test/e2e/live/onboard-policy-preset-sequencing.test.ts": [7,7,7,17,1], "test/e2e/live/onboard-repair.test.ts": [22,26,22,26,0], - "test/e2e/live/onboard-resume.test.ts": [59,63,59,63,0], - "test/e2e/live/openclaw-discord-pairing.test.ts": [13,20,28,86,10], - "test/e2e/live/openclaw-inference-switch.test.ts": [35,48,62,104,3], + "test/e2e/live/onboard-resume.test.ts": [56,60,56,60,0], + "test/e2e/live/openclaw-discord-pairing.test.ts": [13,20,28,83,10], + "test/e2e/live/openclaw-inference-switch.test.ts": [34,48,58,98,3], "test/e2e/live/openclaw-skill-cli.test.ts": [10,14,10,14,1], - "test/e2e/live/openclaw-slack-pairing.test.ts": [11,18,26,84,10], - "test/e2e/live/openshell-credential-generation-window.test.ts": [42,92,43,112,18], + "test/e2e/live/openclaw-slack-pairing.test.ts": [11,18,26,81,10], + "test/e2e/live/openshell-credential-generation-window.test.ts": [38,86,39,92,2], "test/e2e/live/openshell-gateway-auth-source-contract.test.ts": [0,0,16,28,2], "test/e2e/live/openshell-gateway-upgrade.test.ts": [6,15,6,26,6], "test/e2e/live/overlayfs-autofix.test.ts": [22,25,22,28,2], "test/e2e/live/pi-agent-qualification.test.ts": [39,46,39,62,4], - "test/e2e/live/podman-cpu-lifecycle.test.ts": [35,57,48,92,1], - "test/e2e/live/podman-portable-uninstall.test.ts": [37,36,50,71,3], + "test/e2e/live/podman-cpu-lifecycle.test.ts": [35,57,48,89,1], + "test/e2e/live/podman-portable-uninstall.test.ts": [37,36,50,68,3], "test/e2e/live/portable-cpu-delegation-proof.test.ts": [0,22,0,22,0], - "test/e2e/live/portable-profile-rootless-linux.test.ts": [0,76,0,82,2], - "test/e2e/live/rebuild-hermes.test.ts": [2,8,9,47,6], + "test/e2e/live/portable-profile-rootless-linux.test.ts": [0,75,0,80,2], + "test/e2e/live/rebuild-hermes.test.ts": [2,8,9,44,6], "test/e2e/live/rebuild-openclaw.test.ts": [5,10,5,29,1], "test/e2e/live/registry-targets.test.ts": [2,4,5,24,0], - "test/e2e/live/sandbox-operations.test.ts": [2,11,9,50,6], + "test/e2e/live/sandbox-operations.test.ts": [2,11,9,47,6], "test/e2e/live/sandbox-rlimits-connect.test.ts": [1,1,1,1,0], "test/e2e/live/sandbox-survival.test.ts": [3,8,4,9,0], "test/e2e/live/sessions-agents-cli.test.ts": [19,28,20,31,0], @@ -118,8 +118,8 @@ "test/e2e/live/snapshot-commands.test.ts": [5,5,5,5,0], "test/e2e/live/spark-install.test.ts": [11,13,11,15,6], "test/e2e/live/state-backup-restore.test.ts": [1,5,1,5,0], - "test/e2e/live/telegram-injection.test.ts": [19,28,26,67,12], - "test/e2e/live/token-rotation.test.ts": [28,36,28,36,2], + "test/e2e/live/telegram-injection.test.ts": [19,28,26,64,12], + "test/e2e/live/token-rotation.test.ts": [28,36,28,36,1], "test/e2e/live/tunnel-lifecycle.test.ts": [0,0,8,17,4], "test/e2e/live/whatsapp-qr-compact.test.ts": [29,35,29,35,1], "test/e2e/live/windows-mxc-openclaw-process-container.test.ts": [7,11,7,73,4] diff --git a/ci/e2e-assertion-growth-exceptions.json b/ci/e2e-assertion-growth-exceptions.json index 9be3b21231b..d41defc2631 100644 --- a/ci/e2e-assertion-growth-exceptions.json +++ b/ci/e2e-assertion-growth-exceptions.json @@ -1,5 +1,5 @@ { - "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nMaintainer-approved exception for the exact PR #10341 budget transition. Adds only deferred-onboarding.test.ts with [12,14,12,14,0]; existing per-file budgets remain unchanged. Approval: https://github.com/NVIDIA/NemoClaw/pull/10341#issuecomment-5799263232. Local hooks and candidate CI may use this branch policy. The independent PR check reads only trusted base policy, so its expected growth failure is explicitly waived for #10341. A changed base or candidate budget invalidates this exception. Remove after #10341 merges.\n\nPR #12305 adds 19 live assertion points for the approved stopped-container recovery proof while retaining healthy-rebuild coverage. Exact budget-transition approval: https://github.com/NVIDIA/NemoClaw/pull/12305#issuecomment-5803852010. No other budget transition is accepted by this entry.\n\nThe same 19-assertion helper is also exercised by rebuild-openclaw, independently of the inherited private-MCP failure. Unique assertion counts are unchanged; one helper file is extracted. Revised exact-transition approval: https://github.com/NVIDIA/NemoClaw/pull/12305#issuecomment-5807159038.\n\nProposed exact budget transition for PR #12591: replace a restart progress-message check with a native gateway-owner observation before and after inference switching. Adds one generated probe and five probe conditions; direct and unique expectation counts do not exceed main. The imported helper also appears in the full-e2e transitive inventory. This entry requires maintainer review before landing on main. Remove it after #12591 merges.\n\nPR #12718 is the prerequisite for the PR #12382 budget transition recorded below. The exception matches only PR #12382 and both budget digests. A changed base or candidate budget invalidates it. The independent check reads trusted base policy, so this prerequisite must land on main before #12382 uses the exception. Remove after #12382 merges.\n\nMaintainer Prekshi Vyas (@prekshivyas) approved this exact refreshed PR #12382 transition on 2026-10-08: base budget SHA-256 53c51c4229d387633165771730f8f1a8373f84e9f3303b2144280027a56e3441 to candidate budget SHA-256 8162408384d7a6b97ef0062042c73d4bc9486f54d2dfa6c0c5732e42d1060e2a. Acceptance record: https://github.com/NVIDIA/NemoClaw/pull/12840#issuecomment-6066319444. It includes one additional unique assertion point and one generated probe condition. This refresh preserves the #12608 concurrent-gateway-ports reduction and the previously reviewed runtime assertion changes. The independent check is not waived: the refreshed exception must land on main before enforcement can accept it. A different PR or budget digest remains rejected. Remove after #12382 merges.\n\nPR #12777 adds exactly two approved live assertions for native host-local inference success and sibling provider isolation. Deterministic behavior remains in unit/integration tests. Approval transcription: https://github.com/NVIDIA/NemoClaw/pull/12777#issuecomment-6048385152. The independent check reads trusted base policy; this entry does not disable it. Remove after #12777 merges.\n\nThe #12777 budget digests account for main #12608 removing seven assertions from concurrent-gateway-ports. The candidate preserves that reduction and adds only the same two approved inference-routing assertions. No other file budget increases.\n\nPR #12886 approved the provider-smoke transition extracted by #12955. Approval: https://github.com/NVIDIA/NemoClaw/pull/12886#issuecomment-6083839176. The unchanged change digest binds the same nine assertion points and companion inventory. This entry supports local preliminary validation and the Slice 2 consumer. Independent CI for the prerequisite requires its own maintainer approval record; this entry does not approve that PR.\n\nPR #12958 extracts the same approved provider-smoke budget transition for #12955. Its exact, unedited maintainer approval is https://github.com/NVIDIA/NemoClaw/pull/12958#issuecomment-6093851571. This candidate entry supports static hooks; the independent check continues to verify the PR-specific approval on GitHub. Remove after #12958 merges.\n\nPR #13203 adds the approved 19 assertion points for real authenticated and credentialless DCode native-local requests, credential isolation and sibling denial. Approval: https://github.com/NVIDIA/NemoClaw/pull/13203#issuecomment-6180239679. This exact change digest supports local hooks and candidate CI. Independent enforcement still verifies the PR-specific maintainer approval on GitHub. Remove after #13203 merges.\n\nPR #13201 adds one generated probe block and one counted probe condition in the existing full E2E owner to verify the selected sandbox agent endpoint, API, model and credential reference. Rebecca Sliter approved this exact delta: https://github.com/NVIDIA/NemoClaw/pull/13201#issuecomment-6180464605. This entry supports local validation; the independent check must verify the maintainer record. Remove after #13201 merges.\n\nPR #13293 consolidates OpenClaw native-local scenarios into the existing switching owner and removes duplicated deterministic metadata checks and the redundant mock curl probe. Suite totals fall by 16 expect calls and 10 assertion points without adding live files. Only the owning file transitive allocation grows because it now imports existing live helpers. Maintainer authorization: https://github.com/NVIDIA/NemoClaw/pull/13293#issuecomment-6211036786. This exact transition supports candidate CI; independent enforcement still verifies the maintainer record.\n\nPR #13007 retains the same ten Pi native-local live assertion points approved at https://github.com/NVIDIA/NemoClaw/pull/13007#issuecomment-6175787892. The exact change digest is unchanged against its replacement OpenClaw parent #13293. This entry supports local checks against that stacked base; independent enforcement still checks the authenticated PR approval.\n\nPR #13582 adds eight live assertion points for the WSL2 IPv6 native-local policy boundary. Only openclaw-inference-switch.test.ts increases, by seven direct expectations and the existing onboard-success helper. Deterministic CIDR and receipt coverage remain in source tests. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13582#issuecomment-6308860348. Independent enforcement verifies that PR-specific record. Remove this preliminary local exception after merge.\n\nPR #13645 adds three approved live assertion points for engine-down uninstall diagnostics inside rebuild-openclaw. Exact approval: https://github.com/NVIDIA/NemoClaw/pull/13645#issuecomment-6429899630. This entry enables candidate CI; independent enforcement still verifies the PR-specific maintainer record. Remove after merge.\n\nPR #13771 adds four Hermes OpenRouter rebuild assertions and a bounded diagnostic probe. Exact maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13771#issuecomment-6427647915. The independent PR check verifies that comment. Remove after #13771 merges.\n\nPR #13873 adds two live command assertions for the CLI build and native Hermes Discord qualification when consuming protected evidence from an older trusted producer. Approval: https://github.com/NVIDIA/NemoClaw/pull/13873#issuecomment-6488667052. The parser and identity cases stay in e2e-support tests. Remove after #13873 merges.\n\nPR #13651 moves the native NVIDIA Discord rebuild proof from hermes-discord to hermes-inference-switch, which receives the public endpoint credential. The native switch target also checks status through its installed credential verifier after rebuild. This adds four assertion points for the live status and restored-provider boundary; deterministic rejection and command-failure cases remain in fast tests. This entry permits the approved transition in local and candidate checks; independent enforcement verifies the maintainer record at https://github.com/NVIDIA/NemoClaw/pull/13651#issuecomment-6377201610. The typed helper extraction retains every live predicate and the same unique and transitive totals; the direct census decreases because the phase now lives in the existing helper.\n\nPR #13635 adds the approved protected-runtime deadline companion and five unique assertion points, one generated probe block, and one probe condition. Exact maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13635#issuecomment-6383152261. This entry supports local hooks and candidate CI; independent enforcement verifies the PR-specific GitHub approval. Remove after #13635 merges.\n\nPR #13596 adds four live assertion points to the existing double-onboard target for #13595: the managed OpenClaw state volume exists before `nemoclaw uninstall`, uninstall succeeds, the volume is gone afterwards, and same-name onboarding succeeds. Deterministic removal and failure cases remain in fast tests. This entry permits the approved transition in local and candidate checks; independent enforcement verifies the maintainer record at https://github.com/NVIDIA/NemoClaw/pull/13596#issuecomment-6365742624. Remove after #13596 merges.\n\nPR #13515 extends dashboard-remote-bind with Hermes host-listener coverage. The dashboard remains remotely reachable, API health works on loopback, and the non-loopback API port refuses TCP connections. Approval: https://github.com/NVIDIA/NemoClaw/pull/13515#issuecomment-6295553350. This entry supports local and candidate checks; independent enforcement verifies the authenticated maintainer record.\n\nPR #13515 also checks Hermes loopback API health and non-loopback refusal after stop/start, preserving the onboarding assertions. The combined delta is approved at https://github.com/NVIDIA/NemoClaw/pull/13515#issuecomment-6298021751.\n\nPR #13515 verifies the Hermes API forward identity before health probes after onboarding and restart. These two additional process-ownership assertions are approved at https://github.com/NVIDIA/NemoClaw/pull/13515#issuecomment-6322568985.\n\nPR #13927 reuses the verified public HTTPS fixture transport for two Podman inference targets. Direct and unique assertion counts are unchanged. Each target gains eleven transitive assertion points from existing package and tunnel prerequisites; the extracted Anthropic response helper adds one live companion file. This exact transition supports local validation. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13927#issuecomment-6524030655. Independent enforcement verifies this PR-specific record. Remove after merge.\n\nPR #14062 adds one live Ollama cleanup helper, three existing port-validation guards reached by gpu-double-onboard, and one shared shell probe. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/14062#issuecomment-6582340290. This entry supports local hooks and candidate CI; independent enforcement verifies the PR-specific approval. Remove after merge.\n\nPR #13607 adds two live assertions and one read-only configuration probe to the existing balanced OpenClaw egress case for unavailable-search replies. The refreshed delta removes the unrelated public-fetch precondition guard while retaining both live assertions and the probe. Approved delta: https://github.com/NVIDIA/NemoClaw/pull/13607#issuecomment-6327887365. Remove after #13607 merges.\n\nPR #14000 replaces #12872 and retains one approved assertion point for owning-gateway connect sessions. Exact maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/14000#issuecomment-6506269922. This entry enables candidate CI; independent enforcement verifies the PR-specific approval.\n\nPR #14122 retains the production bearerless provider-validation proof and private-publication proof in both Express targets. Spark and Station each request health on the two allowed addresses and require connection refusal on all discovered unintended addresses. Four added Spark assertions protect its separate physical-host path; deterministic URL construction remains in the support tests. The combined delta adds seven assertion points and one generated probe relative to main. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/14122#issuecomment-6661407269. Independent enforcement verifies this PR-specific approval. Remove after merge.", + "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nMaintainer-approved exception for the exact PR #10341 budget transition. Adds only deferred-onboarding.test.ts with [12,14,12,14,0]; existing per-file budgets remain unchanged. Approval: https://github.com/NVIDIA/NemoClaw/pull/10341#issuecomment-5799263232. Local hooks and candidate CI may use this branch policy. The independent PR check reads only trusted base policy, so its expected growth failure is explicitly waived for #10341. A changed base or candidate budget invalidates this exception. Remove after #10341 merges.\n\nPR #12305 adds 19 live assertion points for the approved stopped-container recovery proof while retaining healthy-rebuild coverage. Exact budget-transition approval: https://github.com/NVIDIA/NemoClaw/pull/12305#issuecomment-5803852010. No other budget transition is accepted by this entry.\n\nThe same 19-assertion helper is also exercised by rebuild-openclaw, independently of the inherited private-MCP failure. Unique assertion counts are unchanged; one helper file is extracted. Revised exact-transition approval: https://github.com/NVIDIA/NemoClaw/pull/12305#issuecomment-5807159038.\n\nProposed exact budget transition for PR #12591: replace a restart progress-message check with a native gateway-owner observation before and after inference switching. Adds one generated probe and five probe conditions; direct and unique expectation counts do not exceed main. The imported helper also appears in the full-e2e transitive inventory. This entry requires maintainer review before landing on main. Remove it after #12591 merges.\n\nPR #12718 is the prerequisite for the PR #12382 budget transition recorded below. The exception matches only PR #12382 and both budget digests. A changed base or candidate budget invalidates it. The independent check reads trusted base policy, so this prerequisite must land on main before #12382 uses the exception. Remove after #12382 merges.\n\nMaintainer Prekshi Vyas (@prekshivyas) approved this exact refreshed PR #12382 transition on 2026-10-08: base budget SHA-256 53c51c4229d387633165771730f8f1a8373f84e9f3303b2144280027a56e3441 to candidate budget SHA-256 8162408384d7a6b97ef0062042c73d4bc9486f54d2dfa6c0c5732e42d1060e2a. Acceptance record: https://github.com/NVIDIA/NemoClaw/pull/12840#issuecomment-6066319444. It includes one additional unique assertion point and one generated probe condition. This refresh preserves the #12608 concurrent-gateway-ports reduction and the previously reviewed runtime assertion changes. The independent check is not waived: the refreshed exception must land on main before enforcement can accept it. A different PR or budget digest remains rejected. Remove after #12382 merges.\n\nPR #12777 adds exactly two approved live assertions for native host-local inference success and sibling provider isolation. Deterministic behavior remains in unit/integration tests. Approval transcription: https://github.com/NVIDIA/NemoClaw/pull/12777#issuecomment-6048385152. The independent check reads trusted base policy; this entry does not disable it. Remove after #12777 merges.\n\nThe #12777 budget digests account for main #12608 removing seven assertions from concurrent-gateway-ports. The candidate preserves that reduction and adds only the same two approved inference-routing assertions. No other file budget increases.\n\nPR #12886 approved the provider-smoke transition extracted by #12955. Approval: https://github.com/NVIDIA/NemoClaw/pull/12886#issuecomment-6083839176. The unchanged change digest binds the same nine assertion points and companion inventory. This entry supports local preliminary validation and the Slice 2 consumer. Independent CI for the prerequisite requires its own maintainer approval record; this entry does not approve that PR.\n\nPR #12958 extracts the same approved provider-smoke budget transition for #12955. Its exact, unedited maintainer approval is https://github.com/NVIDIA/NemoClaw/pull/12958#issuecomment-6093851571. This candidate entry supports static hooks; the independent check continues to verify the PR-specific approval on GitHub. Remove after #12958 merges.\n\nPR #13203 adds the approved 19 assertion points for real authenticated and credentialless DCode native-local requests, credential isolation and sibling denial. Approval: https://github.com/NVIDIA/NemoClaw/pull/13203#issuecomment-6180239679. This exact change digest supports local hooks and candidate CI. Independent enforcement still verifies the PR-specific maintainer approval on GitHub. Remove after #13203 merges.\n\nPR #13201 adds one generated probe block and one counted probe condition in the existing full E2E owner to verify the selected sandbox agent endpoint, API, model and credential reference. Rebecca Sliter approved this exact delta: https://github.com/NVIDIA/NemoClaw/pull/13201#issuecomment-6180464605. This entry supports local validation; the independent check must verify the maintainer record. Remove after #13201 merges.\n\nPR #13293 consolidates OpenClaw native-local scenarios into the existing switching owner and removes duplicated deterministic metadata checks and the redundant mock curl probe. Suite totals fall by 16 expect calls and 10 assertion points without adding live files. Only the owning file transitive allocation grows because it now imports existing live helpers. Maintainer authorization: https://github.com/NVIDIA/NemoClaw/pull/13293#issuecomment-6211036786. This exact transition supports candidate CI; independent enforcement still verifies the maintainer record.\n\nPR #13007 retains the same ten Pi native-local live assertion points approved at https://github.com/NVIDIA/NemoClaw/pull/13007#issuecomment-6175787892. The exact change digest is unchanged against its replacement OpenClaw parent #13293. This entry supports local checks against that stacked base; independent enforcement still checks the authenticated PR approval.\n\nPR #13582 adds eight live assertion points for the WSL2 IPv6 native-local policy boundary. Only openclaw-inference-switch.test.ts increases, by seven direct expectations and the existing onboard-success helper. Deterministic CIDR and receipt coverage remain in source tests. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13582#issuecomment-6308860348. Independent enforcement verifies that PR-specific record. Remove this preliminary local exception after merge.\n\nPR #13645 adds three approved live assertion points for engine-down uninstall diagnostics inside rebuild-openclaw. Exact approval: https://github.com/NVIDIA/NemoClaw/pull/13645#issuecomment-6429899630. This entry enables candidate CI; independent enforcement still verifies the PR-specific maintainer record. Remove after merge.\n\nPR #13771 adds four Hermes OpenRouter rebuild assertions and a bounded diagnostic probe. Exact maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13771#issuecomment-6427647915. The independent PR check verifies that comment. Remove after #13771 merges.\n\nPR #13873 adds two live command assertions for the CLI build and native Hermes Discord qualification when consuming protected evidence from an older trusted producer. Approval: https://github.com/NVIDIA/NemoClaw/pull/13873#issuecomment-6488667052. The parser and identity cases stay in e2e-support tests. Remove after #13873 merges.\n\nPR #13651 moves the native NVIDIA Discord rebuild proof from hermes-discord to hermes-inference-switch, which receives the public endpoint credential. The native switch target also checks status through its installed credential verifier after rebuild. This adds four assertion points for the live status and restored-provider boundary; deterministic rejection and command-failure cases remain in fast tests. This entry permits the approved transition in local and candidate checks; independent enforcement verifies the maintainer record at https://github.com/NVIDIA/NemoClaw/pull/13651#issuecomment-6377201610. The typed helper extraction retains every live predicate and the same unique and transitive totals; the direct census decreases because the phase now lives in the existing helper.\n\nPR #13635 adds the approved protected-runtime deadline companion and five unique assertion points, one generated probe block, and one probe condition. Exact maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13635#issuecomment-6383152261. This entry supports local hooks and candidate CI; independent enforcement verifies the PR-specific GitHub approval. Remove after #13635 merges.\n\nPR #13596 adds four live assertion points to the existing double-onboard target for #13595: the managed OpenClaw state volume exists before `nemoclaw uninstall`, uninstall succeeds, the volume is gone afterwards, and same-name onboarding succeeds. Deterministic removal and failure cases remain in fast tests. This entry permits the approved transition in local and candidate checks; independent enforcement verifies the maintainer record at https://github.com/NVIDIA/NemoClaw/pull/13596#issuecomment-6365742624. Remove after #13596 merges.\n\nPR #13515 extends dashboard-remote-bind with Hermes host-listener coverage. The dashboard remains remotely reachable, API health works on loopback, and the non-loopback API port refuses TCP connections. Approval: https://github.com/NVIDIA/NemoClaw/pull/13515#issuecomment-6295553350. This entry supports local and candidate checks; independent enforcement verifies the authenticated maintainer record.\n\nPR #13515 also checks Hermes loopback API health and non-loopback refusal after stop/start, preserving the onboarding assertions. The combined delta is approved at https://github.com/NVIDIA/NemoClaw/pull/13515#issuecomment-6298021751.\n\nPR #13515 verifies the Hermes API forward identity before health probes after onboarding and restart. These two additional process-ownership assertions are approved at https://github.com/NVIDIA/NemoClaw/pull/13515#issuecomment-6322568985.\n\nPR #13927 reuses the verified public HTTPS fixture transport for two Podman inference targets. Direct and unique assertion counts are unchanged. Each target gains eleven transitive assertion points from existing package and tunnel prerequisites; the extracted Anthropic response helper adds one live companion file. This exact transition supports local validation. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/13927#issuecomment-6524030655. Independent enforcement verifies this PR-specific record. Remove after merge.\n\nPR #14062 adds one live Ollama cleanup helper, three existing port-validation guards reached by gpu-double-onboard, and one shared shell probe. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/14062#issuecomment-6582340290. This entry supports local hooks and candidate CI; independent enforcement verifies the PR-specific approval. Remove after merge.\n\nPR #13607 adds two live assertions and one read-only configuration probe to the existing balanced OpenClaw egress case for unavailable-search replies. The refreshed delta removes the unrelated public-fetch precondition guard while retaining both live assertions and the probe. Approved delta: https://github.com/NVIDIA/NemoClaw/pull/13607#issuecomment-6327887365. Remove after #13607 merges.\n\nPR #14000 replaces #12872 and retains one approved assertion point for owning-gateway connect sessions. Exact maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/14000#issuecomment-6506269922. This entry enables candidate CI; independent enforcement verifies the PR-specific approval.\n\nUser-approved exception on 2026-11-04: PR #12603 adds two identity assertions to managed-image activation cleanup for OpenShell 0.1.7. This change digest includes the existing PR budget differences against main. The entry supports local and candidate checks only; independent enforcement still requires the PR-specific maintainer record. No runtime deadline changes.\n\nPR #14122 retains the production bearerless provider-validation proof and private-publication proof in both Express targets. Spark and Station each request health on the two allowed addresses and require connection refusal on all discovered unintended addresses. Four added Spark assertions protect its separate physical-host path; deterministic URL construction remains in the support tests. The combined delta adds seven assertion points and one generated probe relative to main. Maintainer approval: https://github.com/NVIDIA/NemoClaw/pull/14122#issuecomment-6661407269. Independent enforcement verifies this PR-specific approval. Remove after merge.\n\nUser approved the PR #12603 main-integration assertion exception on 2026-11-05: eleven additional assertion points, one generated probe block and one condition relative to its previous candidate. The refreshed change digest binds the complete transition against main #14122. Existing deadlines remain unchanged. This entry supports local and candidate checks; independent enforcement still requires a PR-specific maintainer record.\n\nUser approved the PR #12603 integration accounting exception on 2026-11-05: named helper calls increase from 661 to 663 while total assertion points remain 3,245. This refreshed change digest binds the complete transition against main #14153. This entry supports local and candidate checks; independent enforcement still requires a PR-specific maintainer record.", "schemaVersion": 1, "exceptions": [ { @@ -119,6 +119,10 @@ "pullRequest": 13607, "changeSha256": "09521e19ebf1ea58baae27cf23f0738de31ac1f950d90ac65385d04e94bf290c" }, + { + "pullRequest": 12603, + "changeSha256": "7849150c7ae63adf6d8454d4f916c4dd7f3ca095d4ccfe8646a6b8ab6d7b75d5" + }, { "pullRequest": 14122, "changeSha256": "53172ca146b03aafdb90d853246436c4778d0bb92747c9768c328033140a84a5" diff --git a/ci/onboard-entry-composition-budget.json b/ci/onboard-entry-composition-budget.json index cbfb489ad98..a8599881401 100644 --- a/ci/onboard-entry-composition-budget.json +++ b/ci/onboard-entry-composition-budget.json @@ -16,7 +16,7 @@ "sandboxCreateIntentResolver.getAgentPolicyPath": 1 }, "provider": { - "handleNimLocalSelection": 32, + "handleNimLocalSelection": 31, "handleRemoteProviderSelection": 81, "handleRemoteProviderSelection.providerExistsInGateway": 1, "handleRemoteProviderSelection.readGatewayProviderMetadata": 1, diff --git a/ci/openshell-upgrade-trust.json b/ci/openshell-upgrade-trust.json index 47dd2035245..2fea06d6cda 100644 --- a/ci/openshell-upgrade-trust.json +++ b/ci/openshell-upgrade-trust.json @@ -1,6 +1,164 @@ { "schemaVersion": 1, "releases": [ + { + "brevTemplateSha256": [ + "c60af9bde1d62f85fce2b6b1b3e285e44a4c6254cdbb9371b5c028d4bf86439f" + ], + "formula": { + "asset": "openshell.rb", + "sha256": "daa268c73d627cc72fb5c36b041e0e8625b1125cc83706f76772d52b5da081d7", + "url": "https://github.com/NVIDIA/OpenShell/releases/download/v0.1.7/openshell.rb" + }, + "installerTemplateSha256": [ + "6f21070fe62f7556fcdb440c70fa0b792d4c1ab71441d2b9dcc74d21dadda252" + ], + "manifests": [ + { + "asset": "openshell-checksums-sha256.txt", + "sha256": "a53cd40267d57969f2a1c917ad93a1b7c280dc13826211e95b276f9bd2c62f7e" + }, + { + "asset": "openshell-gateway-checksums-sha256.txt", + "sha256": "20fb91f0d63581d911ea98b41d0e07175eb859950bd5f032159c168fcdf7dc1a" + }, + { + "asset": "openshell-sandbox-checksums-sha256.txt", + "sha256": "f5070d63a49da60e801813ccadadc08e88cfecb5e781a7eef149d933ce144a9b" + } + ], + "sandboxBuilds": [ + { + "required": false, + "sha256": "b4ae1d60b238c84e290bcd08a962997e5c36ed04237fa59b09b5799b11363feb" + }, + { + "required": false, + "sha256": "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21" + } + ], + "supervisor": { + "image": "ghcr.io/nvidia/openshell/supervisor", + "manifestDigest": "sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06", + "required": false, + "runtimeTemplateSha256": [ + "5b12b5cf2faa010f59ee81695f42ea79782145791165efd84c979cb103d2badd" + ] + }, + "pinLayout": { + "brev": { + "assets": [ + "openshell-x86_64-unknown-linux-musl.tar.gz", + "openshell-aarch64-unknown-linux-musl.tar.gz" + ], + "manifests": [ + "openshell-checksums-sha256.txt" + ] + }, + "installer": { + "assets": [ + "openshell-x86_64-unknown-linux-musl.tar.gz", + "openshell-aarch64-unknown-linux-musl.tar.gz", + "openshell-aarch64-apple-darwin.tar.gz", + "openshell-gateway-x86_64-unknown-linux-gnu.tar.gz", + "openshell-gateway-aarch64-unknown-linux-gnu.tar.gz", + "openshell-gateway-aarch64-apple-darwin.tar.gz", + "openshell-sandbox-x86_64-unknown-linux-musl.tar.gz", + "openshell-sandbox-aarch64-unknown-linux-musl.tar.gz", + "openshell.rb" + ], + "manifests": [ + "openshell-checksums-sha256.txt", + "openshell-gateway-checksums-sha256.txt", + "openshell-sandbox-checksums-sha256.txt" + ] + } + }, + "featureGateTemplateSha256": [ + "5a586061bc3a35911cd920ce71d5f52610e6ab63ad56ba6071d39f05c2d76070" + ], + "version": "0.1.7" + }, + { + "brevTemplateSha256": [ + "5a0cc91a85d32687eefe7e543f9446e0478634e8f9c912e53b87cfb43a746596" + ], + "formula": { + "asset": "openshell.rb", + "sha256": "6b197da850a299774114d517a45cffd3f3259c9aa1a49faae28ec9d8e02a757a", + "url": "https://github.com/NVIDIA/OpenShell/releases/download/v0.1.5/openshell.rb" + }, + "installerTemplateSha256": [ + "3c78eb8993e3ee940057b952639cc197252e58b69e96489ae4ffef90c9fc94e1", + "32226056dbea5585861eddd7b1dd50bfd7dd79345bac04ecbf6194b9ef4e7677", + "fad4b3cde7f698e074ea11cb1373b5eee81564611b480c2562cd8812e920c0c1" + ], + "manifests": [ + { + "asset": "openshell-checksums-sha256.txt", + "sha256": "6023b4ab7302f2a718ae93c3f7e738e9609b161cc6a81b5af138e22235c590c7" + }, + { + "asset": "openshell-gateway-checksums-sha256.txt", + "sha256": "12ab45742c9ce8b95660cc9f8eda47f65710affc8213b04de743f649af0d5361" + }, + { + "asset": "openshell-sandbox-checksums-sha256.txt", + "sha256": "56a7b7a6ed3a6fe62022a6602c7810f54d4a253d980c9c0d737c359845943e15" + } + ], + "sandboxBuilds": [ + { + "required": false, + "sha256": "a048b1fd2acd78f44f50a4cf0e02cdefec691656f3941e92c5ec0e09a69990e2" + }, + { + "required": false, + "sha256": "65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc" + } + ], + "supervisor": { + "image": "ghcr.io/nvidia/openshell/supervisor", + "manifestDigest": "sha256:fabcd01f656e9d23df4386f82e80b4f48b90f3452d7d5c58889007643a069161", + "required": false, + "runtimeTemplateSha256": [ + "b3737aba717db2772fa8b70a3222f904834c262da088ef9bddb92fd504cb6dd5" + ] + }, + "pinLayout": { + "brev": { + "assets": [ + "openshell-x86_64-unknown-linux-musl.tar.gz", + "openshell-aarch64-unknown-linux-musl.tar.gz" + ], + "manifests": [ + "openshell-checksums-sha256.txt" + ] + }, + "installer": { + "assets": [ + "openshell-x86_64-unknown-linux-musl.tar.gz", + "openshell-aarch64-unknown-linux-musl.tar.gz", + "openshell-aarch64-apple-darwin.tar.gz", + "openshell-gateway-x86_64-unknown-linux-gnu.tar.gz", + "openshell-gateway-aarch64-unknown-linux-gnu.tar.gz", + "openshell-gateway-aarch64-apple-darwin.tar.gz", + "openshell-sandbox-x86_64-unknown-linux-musl.tar.gz", + "openshell-sandbox-aarch64-unknown-linux-musl.tar.gz", + "openshell.rb" + ], + "manifests": [ + "openshell-checksums-sha256.txt", + "openshell-gateway-checksums-sha256.txt", + "openshell-sandbox-checksums-sha256.txt" + ] + } + }, + "featureGateTemplateSha256": [ + "38923088393208220121503f8f006c8e8dec3e79a49c50c2ae6907fba9ae4b7f" + ], + "version": "0.1.5" + }, { "brevTemplateSha256": [ "e44a3952cd5f6be9f017a5b69028c2068a746060bff31c3f05eecbb36cd7237c" @@ -11,7 +169,14 @@ "url": "https://github.com/NVIDIA/OpenShell/releases/download/v0.1.3/openshell.rb" }, "installerTemplateSha256": [ - "0b9c65bcc255d97d2fbfef638fbb4b374d7617fb2aaaf6cb76b898ded74e498f" + "0b9c65bcc255d97d2fbfef638fbb4b374d7617fb2aaaf6cb76b898ded74e498f", + "e3bb05b5287f3e1a9091efb9b7105647ac44e4c3ed86bc47d6b9db1ef5a52068", + "8776c0ec750a38156d7371301f298c4645ca6bcf97c8ec7c74166be16ea87b19", + "fed763911e32980302d32362888c51a984d49022f8b2382d29440ebc29f7139d", + "65da7bf3c9a930e11f4028b808d34cdbebc63d1a30672e98e1f2650f9d9a269d", + "4c474fed277ea6c97ec8d49217d9323bd89fc9e129e9ea45ae697be57122837c", + "2b801fac9e584a6b2cb811968c3bd8c6669fff10881452446ec6408b0792ff0d", + "8cd1370203b8ae54bcfe0b1e87d0309f096720dbb81bc28d4308527abc5a6667" ], "manifests": [ { @@ -181,5 +346,12 @@ "version": "0.0.116" } ], - "revokedTemplateSha256": [] + "revokedTemplateSha256": [ + "4c474fed277ea6c97ec8d49217d9323bd89fc9e129e9ea45ae697be57122837c", + "65da7bf3c9a930e11f4028b808d34cdbebc63d1a30672e98e1f2650f9d9a269d", + "fed763911e32980302d32362888c51a984d49022f8b2382d29440ebc29f7139d", + "8776c0ec750a38156d7371301f298c4645ca6bcf97c8ec7c74166be16ea87b19", + "e3bb05b5287f3e1a9091efb9b7105647ac44e4c3ed86bc47d6b9db1ef5a52068", + "0b9c65bcc255d97d2fbfef638fbb4b374d7617fb2aaaf6cb76b898ded74e498f" + ] } diff --git a/ci/pi-agent-qualification-v1-linux-amd64.json b/ci/pi-agent-qualification-v1-linux-amd64.json index 515ba9c874b..ada12d34674 100644 --- a/ci/pi-agent-qualification-v1-linux-amd64.json +++ b/ci/pi-agent-qualification-v1-linux-amd64.json @@ -3,13 +3,13 @@ "agent": "pi", "platform": "linux/amd64", "image": "ghcr.io/nvidia/nemoclaw/pi-sandbox", - "digest": "sha256:f7fc61646d3345290be9b7343a9d798d9bd2865739dd40b966aeb6ba2e769c09", - "reference": "ghcr.io/nvidia/nemoclaw/pi-sandbox@sha256:f7fc61646d3345290be9b7343a9d798d9bd2865739dd40b966aeb6ba2e769c09", + "digest": "sha256:18110ed01e8655c398b43a8bfe7daf55e4d1753c679954ade14cd1d6df9ede62", + "reference": "ghcr.io/nvidia/nemoclaw/pi-sandbox@sha256:18110ed01e8655c398b43a8bfe7daf55e4d1753c679954ade14cd1d6df9ede62", "source": { "repository": "NVIDIA/NemoClaw", - "revision": "92d66bad2eede2ad8f2d9d307792d45d8d2d682a", - "release": "v0.0.141-9-g92d66bad2e", - "cohort": "ghrun-41886659843-5" + "revision": "47194fd432562c33a1371d007d7435aa0c7ff974", + "release": "v0.0.141-345-g47194fd432", + "cohort": "ghrun-41933603090-7" }, "startupProfileContractVersion": 1, "capabilityContractVersion": 1 diff --git a/ci/pi-agent-qualification-v1-linux-arm64.json b/ci/pi-agent-qualification-v1-linux-arm64.json index aa986ddb3a3..db14914d871 100644 --- a/ci/pi-agent-qualification-v1-linux-arm64.json +++ b/ci/pi-agent-qualification-v1-linux-arm64.json @@ -3,13 +3,13 @@ "agent": "pi", "platform": "linux/arm64", "image": "ghcr.io/nvidia/nemoclaw/pi-sandbox", - "digest": "sha256:f06510abe9eb7cb2e016b73dce2be791fb5310bb7bc4f19df43f77173220e3e9", - "reference": "ghcr.io/nvidia/nemoclaw/pi-sandbox@sha256:f06510abe9eb7cb2e016b73dce2be791fb5310bb7bc4f19df43f77173220e3e9", + "digest": "sha256:2de587b77bb4d699fc612c4779d078e36b673680a0a42ddd5994f15770650375", + "reference": "ghcr.io/nvidia/nemoclaw/pi-sandbox@sha256:2de587b77bb4d699fc612c4779d078e36b673680a0a42ddd5994f15770650375", "source": { "repository": "NVIDIA/NemoClaw", - "revision": "92d66bad2eede2ad8f2d9d307792d45d8d2d682a", - "release": "v0.0.141-9-g92d66bad2e", - "cohort": "ghrun-41886659843-5" + "revision": "47194fd432562c33a1371d007d7435aa0c7ff974", + "release": "v0.0.141-345-g47194fd432", + "cohort": "ghrun-41933603090-7" }, "startupProfileContractVersion": 1, "capabilityContractVersion": 1 diff --git a/ci/platform-matrix.json b/ci/platform-matrix.json index 97870e344fc..91244e4c260 100644 --- a/ci/platform-matrix.json +++ b/ci/platform-matrix.json @@ -169,7 +169,7 @@ "name": "Local vLLM (managed install/start)", "status": "caveated", "endpoint_type": "Local OpenAI-compatible", - "notes": "Managed install/start requires Docker on `PATH`; attaching an already-running local vLLM server is a separate path. Appears by default on DGX Spark and qualifying DGX Station GB300 hosts. A labeled Deferred preview appears on hosts that match the N1x identity requirements; N1x remains Deferred until a physical NemoClaw Express E2E test passes. DGX Station is Tested with limitations across qualified profiles on one physical DGX Station GB300. Full Station Express end-to-end qualification for the accepted no-OTA DGX OS `7.6.x` profile is pending. Dual-Station configurations are not yet validated, and dedicated CI coverage is not available. For canonical Station qualification and host preparation, see the Additional Setup page for [OpenClaw](/user-guide/openclaw/get-started/additional-setup/dgx-station-preparation), [Hermes](/user-guide/hermes/get-started/additional-setup/dgx-station-preparation), or [Deep Agents](/user-guide/deepagents/get-started/additional-setup/dgx-station-preparation). Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`, NVIDIA Container Toolkit, and CDI. NemoClaw pins runtime images to immutable digests. New single-Station Express installs default to the experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`; a qualified trusted pair selects distributed Ultra, and `--station-deepseek` selects `deepseek-ai/DeepSeek-V4-Flash`. Direct managed-vLLM defaults are `nvidia/Qwen3.6-35B-A3B-NVFP4` on DGX Spark and the Deferred N1x preview, `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` on DGX Station, and `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8` on generic Linux NVIDIA GPU hosts. The pinned `nvcr.io/nvidia/vllm` images are public, so Docker pulls them without NGC registry login." + "notes": "Managed install/start requires Docker on `PATH`; attaching an already-running local vLLM server is a separate path. Appears by default on DGX Spark and qualifying DGX Station GB300 hosts. A labeled Deferred preview appears on hosts that match the N1x identity requirements; N1x remains Deferred until a physical NemoClaw Express E2E test passes. DGX Station is Tested with limitations across qualified profiles on one physical DGX Station GB300. Full Station Express end-to-end qualification for the accepted no-OTA DGX OS `7.6.x` profile is pending. Dual-Station configurations are not yet validated, and dedicated CI coverage is not available. For canonical Station qualification and host preparation, see the Additional Setup page for [OpenClaw](/user-guide/openclaw/get-started/additional-setup/dgx-station-preparation), [Hermes](/user-guide/hermes/get-started/additional-setup/dgx-station-preparation), or [Deep Agents](/user-guide/deepagents/get-started/additional-setup/dgx-station-preparation). Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`, NVIDIA Container Toolkit, and CDI. NemoClaw pins runtime images to immutable digests. New single-Station Express installs default to the experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`; a qualified trusted pair selects distributed Ultra, and `--station-deepseek` selects `deepseek-ai/DeepSeek-V4-Flash`. Direct managed-vLLM defaults are `nvidia/Qwen3.6-35B-A3B-NVFP4` on DGX Spark and the Deferred N1x preview, experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` on DGX Station, and `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8` on generic Linux NVIDIA GPU hosts. The pinned `nvcr.io/nvidia/vllm` images are public, so Docker pulls them without NGC registry login." } ], @@ -190,7 +190,7 @@ "name": "LangChain Deep Agents Code", "status": "tested", "default": false, - "notes": "Terminal-oriented coding agent runtime with no in-sandbox gateway or dashboard, built on the Deep Agents SDK. Its manifest is `agents/langchain-deepagents-code/manifest.yaml`, and its binary is `dcode`. Onboard with `nemo-deepagents onboard` or `nemoclaw onboard --agent langchain-deepagents-code`, and follow [the quickstart](/user-guide/deepagents/get-started/quickstart). NemoClaw runs it as a managed agent runtime: unmanaged sandbox, Model Context Protocol (MCP) server, and shell overrides are rejected, and credential-bearing proxy URLs are dropped from persisted shell environment values. NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. New Ollama, standalone vLLM, llama.cpp, hosted compatible outside the Personal tier, and qualifying host-local compatible selections use sandbox-attached providers. A fresh Personal-tier hosted compatible selection uses `inference.local` without creating a native attachment. Owned Local NVIDIA NIM, other compatible endpoints without an attachment, and retained shared local selections also use `inference.local`. Live runtime acceptance, broader launch material, and terminal-agent diagnostics are tracked at open issue #4861." + "notes": "Terminal-oriented coding agent runtime with no in-sandbox gateway or dashboard, built on the Deep Agents SDK. Its manifest is `agents/langchain-deepagents-code/manifest.yaml`, and its binary is `dcode`. Onboard with `nemo-deepagents onboard` or `nemoclaw onboard --agent langchain-deepagents-code`, and follow [the quickstart](/user-guide/deepagents/get-started/quickstart). NemoClaw runs it as a managed agent runtime: unmanaged sandbox, Model Context Protocol (MCP) server, and shell overrides are rejected, and credential-bearing proxy URLs are dropped from persisted shell environment values. NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. New Ollama, standalone vLLM, llama.cpp, and qualifying local compatible selections use sandbox-attached providers for qualified host-local endpoints. On OpenShell `0.1.7`, native local onboarding also requires the provider-policy setup and gateway initialization migration in [NemoClaw#13201](https://github.com/NVIDIA/NemoClaw/issues/13201); it is not qualified for this upgrade. Owned Local NVIDIA NIM uses a sandbox-attached provider for its recorded container endpoint. Retained shared custom or local selections require recreation on OpenShell `0.1.7`; legacy data remains available for recovery. Live qualification of the upgrade remains pending. Live runtime acceptance, broader launch material, and terminal-agent diagnostics are tracked at open issue #4861." } ], @@ -208,7 +208,7 @@ { "name": "Managed inference", "status": "tested", - "notes": "Provider-routed model calls through OpenShell, transparent to the agent. NVIDIA Endpoints, supported fixed hosted providers, and new hosted compatible selections outside the Personal tier use sandbox-attached providers. Qualified new local Ollama, standalone vLLM, llama.cpp, and host-local compatible selections also use attached providers where the agent supports them. New OpenClaw Model Router selections use attached providers at the managed host endpoint. A fresh Personal-tier hosted compatible selection, owned Local NVIDIA NIM, compatible endpoints without an attachment, other Model Router selections, and retained shared selections use `inference.local`. Pi native routing remains gated by its activation status; Hermes live qualification for its new local lifecycle remains pending. Bedrock uses a host-local adapter for AWS credentials. Provider credentials remain outside the sandbox. Supports every entry in the Providers table." + "notes": "Provider-routed model calls through OpenShell, transparent to the agent. NVIDIA Endpoints, supported fixed hosted providers, and qualifying custom hosted endpoints use native endpoints through sandbox-attached OpenShell providers. New Deep Agents Code, Hermes, OpenClaw, and Pi selections of Ollama, standalone vLLM, and llama.cpp use sandbox-attached providers for their qualified host-local endpoints. Deep Agents Code, Hermes, OpenClaw, and Pi also use attachments for qualifying host-local compatible endpoints. New Model Router selections for these agents use attached providers at the managed endpoint configured by the routed blueprint. Live Model Router qualification remains pending. Native local onboarding on OpenShell `0.1.7` remains unqualified for this upgrade; Hermes live qualification for its new local lifecycle is pending. Owned Local NVIDIA NIM uses a sandbox-attached provider for its recorded container endpoint. Retained shared custom and local selections use `inference.local` on OpenShell `0.0.x`; they require recreation on `0.1.7`, which also lacks the route removed in `0.1.5`. Legacy data remains available for recovery. Bedrock uses a host-local adapter for AWS credentials. OpenShell keeps provider credentials outside the sandbox. The Providers table remains the required support matrix. New OpenClaw local attachments use the recorded provider endpoint for `status`, `connect`, and `start` health checks." }, { "name": "Declarative network policy", diff --git a/ci/reviewed-npm-audit.json b/ci/reviewed-npm-audit.json index 935018c6353..e57d0b1a632 100644 --- a/ci/reviewed-npm-audit.json +++ b/ci/reviewed-npm-audit.json @@ -12,7 +12,9 @@ "integrity": "sha512-MFnyX2vs47f/yVJRUXKmA7C6wZDH03ZGFFg+R5gZME+yMQDg9WF6G4Kdz/APM7nCUpaoGk2bwNEdUUmR60J54A==", "tarballUrl": "https://npm.pkg.github.com/download/@nvidia/openshell-sdk/0.0.116/3380652847354127ed1bb8010490f8815040ada3" }, - "sourceNestedShrinkwrapPackages": ["@earendil-works/pi-coding-agent@0.80.6"], + "sourceNestedShrinkwrapPackages": [ + "@earendil-works/pi-coding-agent@0.80.6" + ], "sourceRegistryPackagesWithoutIntegrity": [ { "label": "Pi agent core 0.80.6 nested shrinkwrap entry", @@ -286,10 +288,10 @@ } ], "sourceRegistryPackageReplacement": { - "artifactName": "nvidia-openshell-sdk-0.1.5.tgz", - "label": "OpenShell TypeScript SDK 0.1.5", - "packageSpec": "@nvidia/openshell-sdk@0.1.5", - "integrity": "sha512-pqeumlEkIoFko9kL8EmemULuEPGf7dbFz1DXac9mB5H5e0+vGs24T1S+jT/UOtjhYEdmBSR3WjMojBBAyh6lSQ==", - "tarballUrl": "https://npm.pkg.github.com/download/@nvidia/openshell-sdk/0.1.5/141f111b92761575225e0ae977e1cf86d2c6515e" + "artifactName": "nvidia-openshell-sdk-0.1.7.tgz", + "label": "OpenShell TypeScript SDK 0.1.7", + "packageSpec": "@nvidia/openshell-sdk@0.1.7", + "integrity": "sha512-38Zs30o/lvAxHFH+7eUINbXG+QrfFOFUY3rIxoPNXFbURSdWGoo+SNwzCHtP9ulFs9Q0odVsmRr50AImXDM5hA==", + "tarballUrl": "https://npm.pkg.github.com/download/@nvidia/openshell-sdk/0.1.7/e34e85f18bf581cce2e99a74edc3424644cf4ed6" } } diff --git a/ci/source-architecture-budget.json b/ci/source-architecture-budget.json index bcf473fa650..94ff18736ea 100644 --- a/ci/source-architecture-budget.json +++ b/ci/source-architecture-budget.json @@ -6,6 +6,7 @@ "src/lib/actions/sandbox/mcp-bridge-contracts.ts": 24, "src/lib/actions/sandbox/process-recovery.ts": 12, "src/lib/adapters/docker/index.ts": 40, + "src/lib/adapters/fs/regular-file.ts": 21, "src/lib/adapters/openshell/client.ts": 18, "src/lib/adapters/openshell/command-argv.ts": 17, "src/lib/adapters/openshell/resolve.ts": 22, @@ -19,7 +20,8 @@ "src/lib/cli/terminal-style.ts": 37, "src/lib/core/json-types.ts": 32, "src/lib/core/ports.ts": 87, - "src/lib/core/shell-quote.ts": 21, + "src/lib/core/repository-root.ts": 21, + "src/lib/core/shell-quote.ts": 20, "src/lib/core/url-utils.ts": 22, "src/lib/core/wait.ts": 30, "src/lib/credentials/store.ts": 44, @@ -27,7 +29,7 @@ "src/lib/messaging/channels/index.ts": 23, "src/lib/onboard/gateway-binding.ts": 48, "src/lib/platform.ts": 20, - "src/lib/runner.ts": 73, + "src/lib/runner.ts": 72, "src/lib/sandbox-name-contract.ts": 21, "src/lib/security/redact.ts": 50, "src/lib/state/mcp-lifecycle-lock.ts": 17, diff --git a/ci/source-shape-test-budget.json b/ci/source-shape-test-budget.json index 398631bd1d5..223b10b107c 100644 --- a/ci/source-shape-test-budget.json +++ b/ci/source-shape-test-budget.json @@ -423,7 +423,7 @@ }, { "file": "test/install/installer-homebrew-formula-reuse-trust.test.ts", - "test": "accepts only the reviewed OpenShell 0.0.116 installer template", + "test": "accepts only the reviewed OpenShell 0.1.7 installer template", "category": "security" }, { diff --git a/ci/test-file-size-budget.json b/ci/test-file-size-budget.json index 16ca3d5a39e..2d033e1b2c7 100644 --- a/ci/test-file-size-budget.json +++ b/ci/test-file-size-budget.json @@ -4,9 +4,9 @@ "legacyMaxLines": { "src/lib/inference/nim.test.ts": 2063, "src/lib/onboard/preflight.test.ts": 1875, - "test/generation/generate-openclaw-config.test.ts": 1839, + "test/generation/generate-openclaw-config.test.ts": 1777, "test/installer-integration/install-preflight.test.ts": 2995, - "test/agents/openclaw/runtime/nemoclaw-start.test.ts": 3372, + "test/agents/openclaw/runtime/nemoclaw-start.test.ts": 3367, "test/onboarding/onboard-messaging.test.ts": 1960, "test/onboarding/onboard-selection.test.ts": 4123 } diff --git a/docs/about/ecosystem-deepagents.mdx b/docs/about/ecosystem-deepagents.mdx index 27fd471771e..b733f4fe3a8 100644 --- a/docs/about/ecosystem-deepagents.mdx +++ b/docs/about/ecosystem-deepagents.mdx @@ -74,16 +74,24 @@ The following table compares custom OpenShell integration with `nemo-deepagents | Capability | Custom OpenShell + Deep Agents | `nemo-deepagents onboard` | |---|---|---| | Sandbox isolation | You define and apply OpenShell seccomp, Landlock, network namespace isolation, and no-new-privileges policy. | NemoClaw applies the managed Deep Agents policy through the blueprint and requires the runtime checks that the managed agent runtime depends on. | -| Credential handling | You create OpenShell providers manually and decide how `dcode` receives endpoint credentials. | NemoClaw keeps provider credentials outside the sandbox. New NVIDIA Endpoints, supported fixed hosted providers, hosted compatible endpoints outside the Personal tier, Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use sandbox-attached providers. A fresh Personal-tier hosted compatible selection uses `inference.local` without creating a native attachment. Owned Local NVIDIA NIM, other compatible endpoints without an attachment, and retained shared routes also use `inference.local`. | +| Credential handling | You create OpenShell providers manually and decide how `dcode` receives endpoint credentials. | NemoClaw keeps provider credentials outside the sandbox. New NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, Ollama, standalone vLLM, llama.cpp, and qualifying local compatible selections use sandbox-attached native providers. Remaining shared selections retain `inference.local` only on OpenShell `0.0.x`. | | Image hardening | Depends on your base image and install steps. | NemoClaw installs the pinned Deep Agents Code package in `/opt/venv`, uses managed launchers, and rejects unsupported runtime paths such as unmanaged MCP autoloading, nested sandbox providers, and alternate model routes. | | Filesystem policy | You define read-only and read-write paths in policy YAML. | NemoClaw defines a targeted layout: system paths are read-only, `/sandbox` and `/tmp` are writable, and Deep Agents state lives under `/sandbox/.deepagents`. | -| Inference setup | You configure OpenShell inference routing and Deep Agents provider settings manually. | NemoClaw validates the selected provider from the host, configures its sandbox attachment or shared OpenShell route, and writes `/sandbox/.deepagents/config.toml` for the managed OpenAI-compatible path. | +| Inference setup | You configure OpenShell inference routing and Deep Agents provider settings manually. | NemoClaw validates the selected provider from the host, configures the OpenShell route, and writes `/sandbox/.deepagents/config.toml` for the managed OpenAI-compatible path. | | Managed MCP | You decide how MCP servers enter the sandbox and how they receive credentials. | NemoClaw validates and updates the agent-native Deep Agents MCP configuration plus live OpenShell policy, providers, attachments, and credential placeholders through host-side commands. | | Optional integrations | You decide how web search, tracing, and other integrations enter the sandbox. | NemoClaw supports an explicit Tavily policy and credential path plus opt-in bounded trace export to an operator-managed host collector. Native LangSmith tracing and ambient OpenTelemetry exporter configuration remain disabled inside the sandbox. | | Blueprint versioning | No NemoClaw blueprint; your image tag is whatever you built locally. | NemoClaw downloads the blueprint artifact, checks version compatibility, and verifies its digest before applying. | | Lifecycle state | Not included unless you build it. | OpenShell preserves the complete native home. NemoClaw transfers that complete tree during replacement while OpenShell keeps its credential stores outside the agent home. | | Host readiness and operations | You inspect host prerequisites and operate OpenShell resources directly. | NemoClaw provides read-only host readiness reporting, sandbox status and logs, recovery guidance, rebuild, and uninstall workflows. | + +Fresh installations without retained legacy state can install OpenShell `0.1.7` and use sandbox-attached native providers. +The installer refuses to migrate registered OpenShell `0.0.x` sandboxes, including shared `inference.local` routes, and preserves their state. +OpenShell `0.1.5` removed the shared route. +GPU compatibility and native local lifecycle qualification remain incomplete. +Follow the [OpenShell upgrade restrictions](../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before changing an existing installation. + + ## When to Use Which Use the following table to choose NemoClaw or custom OpenShell integration. diff --git a/docs/about/how-it-works.mdx b/docs/about/how-it-works.mdx index 50cf1d0302b..51fe2aa2fcc 100644 --- a/docs/about/how-it-works.mdx +++ b/docs/about/how-it-works.mdx @@ -118,7 +118,7 @@ This separation keeps host orchestration, agent-specific assets, and the sandbox ## Inspect an External OpenShell Gateway NemoClaw packages the experimental `nemoclaw-blueprint-runner` command for infrastructure that already owns an external OpenShell gateway. -This path supports target planning and one credential-free public health request for OpenShell `0.0.116`. +This path supports target planning and one credential-free public health request for OpenShell `0.1.7`. It does not manage the gateway or establish support for another OpenShell release. Create a blueprint directory with a `blueprint.yaml` file that contains one external target. @@ -132,12 +132,12 @@ Neither operation creates or removes either administrator-owned file. ```yaml version: 1.0.0 -min_openshell_version: 0.0.116 -max_openshell_version: 0.0.116 +min_openshell_version: 0.1.7 +max_openshell_version: 0.1.7 openshell_target: endpoint: https://openshell.example.test:8443 workspace: default - expected_release: 0.0.116 + expected_release: 0.1.7 lifecycle: external trust: ca_file: /var/run/openshell-target/ca.pem @@ -166,7 +166,7 @@ It makes one public health request through the official OpenShell SDK. NEMOCLAW_BLUEPRINT_PATH=/absolute/path/to/blueprint nemoclaw-blueprint-runner status --external-target ``` -A successful status result reports `healthy`, release `0.0.116`, and `compatible` in JSON. +A successful status result reports `healthy`, release `0.1.7`, and `compatible` in JSON. The command does not access the authentication file or authenticate. The command supplies the validated CA bundle to the SDK and uses normal TLS hostname verification. @@ -174,7 +174,7 @@ The command supplies the validated CA bundle to the SDK and uses normal TLS host This experimental path does not create, update, delete, or list gateways, workspaces, sandboxes, credentials, or policies. It does not establish workspace readiness or support machine authentication or Kubernetes. A failed status request makes no remote change, so correct the endpoint, CA bundle or its file metadata, gateway health, or release mismatch before you retry. - If status reports that the approved OpenShell SDK is unavailable, restore a NemoClaw installation that includes `@nvidia/openshell-sdk` `0.0.116` before retrying. + If status reports that the approved OpenShell SDK is unavailable, restore a NemoClaw installation that includes `@nvidia/openshell-sdk` `0.1.7` before retrying. ## Readiness and Sandbox Creation @@ -212,17 +212,17 @@ Refer to [Recover and Rebuild Sandboxes](../manage-sandboxes/operate-sandboxes/r New NVIDIA Endpoints, OpenAI, Anthropic, Google Gemini, OpenRouter, and Hermes Provider selections use sandbox-attached OpenShell providers. The agent calls the selected provider API hostname, and OpenShell permits only the approved destination and request paths. -New hosted compatible selections outside the Personal tier use attached providers for OpenClaw, Hermes, and Deep Agents Code, and for Pi when its native route is activated. -New Deep Agents Code and Hermes selections also use native providers for Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible endpoints. -That Personal path creates no native provider attachment and retains `inference.local`. -New OpenClaw Ollama, standalone vLLM, and llama.cpp selections also use sandbox-attached native providers. -Owned Local NVIDIA NIM, compatible endpoints without a native attachment, retained shared local selections, and Model Router retain `inference.local`. +Qualifying custom hosted endpoints use sandbox-attached providers. +New Deep Agents Code, Hermes, OpenClaw, and Pi selections also use sandbox-attached native providers for Ollama, standalone vLLM, Local NVIDIA NIM, llama.cpp, and qualifying host-local compatible endpoints. +Retained shared custom endpoints, unchanged published Pi shared local selections, and retained OpenClaw shared local selections retain `inference.local` on OpenShell `0.0.x`. Hermes live qualification for the new local lifecycle remains pending. -Legacy Deep Agents Code standalone local sandboxes require backup and fresh recreation before native-provider onboarding; NemoClaw blocks their resume and rebuild. An older OpenClaw Ollama, standalone vLLM, or llama.cpp sandbox without an attachment can retain its supported shared-route model switches, but cannot resume or rebuild. Back it up and re-onboard with `--fresh --name --recreate-sandbox`. Owned Local NIM sandboxes retain the shared route and can resume or rebuild without a native attachment. +Legacy Deep Agents Code standalone local sandboxes require backup and fresh recreation before native-provider onboarding; NemoClaw blocks their resume and rebuild. An older OpenClaw Ollama, standalone vLLM, or llama.cpp sandbox without an attachment can retain its supported shared-route model switches, but cannot resume or rebuild. Back it up and re-onboard with `--fresh --name --recreate-sandbox`. New Local NVIDIA NIM sandboxes also use their recorded native attachment for resume and rebuild. Back up and recreate older shared-route NIM sandboxes before upgrading OpenShell. Refer to [About Inference Routing](../inference/about-inference-routing) for endpoints and route inspection. During onboarding, NemoClaw validates the selected provider and model, configures the applicable OpenShell route or provider attachment, and writes the matching model reference into the managed agent configuration. -In both paths, provider credentials stay outside the sandbox and OpenShell mediates the requests. Bedrock AWS credentials stay in the host-local adapter. -When you select the Model Router provider, `inference.local` routes to a host-side router that chooses from a configured model pool available through NVIDIA Endpoints for each request. +In both paths, provider credentials stay outside the sandbox and OpenShell mediates the requests. +Bedrock AWS credentials stay in the host-local adapter. +When you select Model Router, OpenShell attaches a native provider to the sandbox for the host router endpoint. +The router chooses a model from the configured NVIDIA Endpoints pool for each request. For Hermes, `$$nemoclaw inference set` updates `/sandbox/.hermes/config.yaml` at runtime without diff --git a/docs/about/overview.mdx b/docs/about/overview.mdx index 0ccd7e1aedb..f43c37dba6f 100644 --- a/docs/about/overview.mdx +++ b/docs/about/overview.mdx @@ -37,7 +37,7 @@ NemoClaw groups its supported capabilities around running, connecting, controlli | Area | Capability | |---|---| | Run supported agents | Onboards OpenClaw, Hermes, or LangChain Deep Agents Code into an agent-specific sandbox image and configuration. | -| Route inference | Connects supported hosted and local inference providers through OpenShell-managed routes. NVIDIA Endpoints, fixed hosted providers, and new hosted compatible selections outside the Personal tier use sandbox-attached providers. Qualified new local selections also use attached providers where the agent supports them. A fresh Personal-tier hosted compatible selection, owned Local NVIDIA NIM, custom endpoints without an attachment, and retained shared selections use `inference.local`. Bedrock uses a host-local adapter for AWS credentials. Hermes local lifecycle and Pi native routing still have qualification limits. Provider credentials remain outside the sandbox. | +| Route inference | Connects supported hosted and local inference providers through OpenShell-managed routes. NVIDIA Endpoints, supported fixed hosted providers, and qualifying custom hosted endpoints use sandbox-attached native providers. New Deep Agents Code, Hermes, OpenClaw, and Pi selections also use native providers for Ollama, standalone vLLM, Local NVIDIA NIM, llama.cpp, and qualifying host-local compatible endpoints. Retained shared custom endpoints and retained shared local routes use `inference.local` on OpenShell `0.0.x`. Hermes live qualification for the new local lifecycle remains pending. Provider credentials remain outside the sandbox; Bedrock AWS credentials stay in the host-local adapter. | | Control sandbox access | Applies network policy, filesystem policy, process controls, and managed inference through OpenShell. | | Operate the lifecycle | Reports host readiness, creates or updates sandboxes, reports status and logs, and rebuilds or recovers sandboxes while preserving the native agent home. | | Connect Model Context Protocol (MCP) servers | Manages authenticated HTTPS Streamable HTTP MCP servers for all supported agent runtimes. OpenShell replaces credential placeholders at approved egress boundaries. | diff --git a/docs/changelog/2026-11-03.mdx b/docs/changelog/2026-11-03.mdx index 2a2f7161bbf..e3d5aa19743 100644 --- a/docs/changelog/2026-11-03.mdx +++ b/docs/changelog/2026-11-03.mdx @@ -6,7 +6,7 @@ SPDX-License-Identifier: Apache-2.0 ## v0.0.141 NemoClaw v0.0.141 adds adaptive routing configuration for managed Hermes and improves sandbox recovery, inference validation, and credential handling. -Eligible installer and CLI operations now send operation telemetry to NVIDIA's Production receiver by default. +Eligible installer and CLI operations now attempt to send operation telemetry to NVIDIA's Production receiver by default. ### Hermes and Inference diff --git a/docs/deployment/deploy-to-headless-server.mdx b/docs/deployment/deploy-to-headless-server.mdx index 742ba732935..9f9244172b4 100644 --- a/docs/deployment/deploy-to-headless-server.mdx +++ b/docs/deployment/deploy-to-headless-server.mdx @@ -220,8 +220,7 @@ $$nemoclaw headless-agent connect --probe-only On a normal read, `$$nemoclaw headless-agent status` exits nonzero when the sandbox, gateway, local container, or authoritative inference route is not verified, except for a degraded native NVIDIA result. When gateway and sandbox state are verified and probes are not suppressed, its main `Inference` line checks an available selected provider path from inside the sandbox. Shared providers use `https://inference.local/v1/models`, then send an inference request over the same route when that probe reports the route reachable. -NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible endpoints outside the Personal tier, and qualified new local selections verify the recorded sandbox-attached OpenShell provider with an inference request. -A fresh Personal-tier hosted compatible selection verifies the shared route and has no native attachment. +NVIDIA Endpoints, supported fixed hosted providers, qualifying hosted compatible selections, and qualified local selections verify the recorded sandbox-attached OpenShell provider with an inference request. Status repeats inference requests up to three total attempts when they return a transient gateway status; refer to the [CLI commands reference](../reference/commands) for the retried statuses and their token cost. The line reports `healthy` when the route served the request and `unauthorized` when it rejected the request with HTTP `401` or `403`. Other final HTTP failures or invalid 2xx responses report `unhealthy`. @@ -315,13 +314,19 @@ unset TOKEN -Deep Agents Code does not expose a dashboard URL or gateway token. NVIDIA Endpoints, supported fixed hosted providers, and new Ollama, standalone vLLM, llama.cpp, hosted compatible outside the Personal tier, and qualifying host-local compatible selections use native endpoints through sandbox-attached OpenShell providers. A fresh Personal-tier hosted compatible selection uses the OpenShell-managed `inference.local` route without creating a native attachment. Owned Local NVIDIA NIM, retained legacy routes, and other custom routes without an attachment also use that shared route. +Deep Agents Code does not expose a dashboard URL or gateway token. +NVIDIA Endpoints, supported fixed hosted providers, and new Ollama, standalone vLLM, Local NVIDIA NIM, llama.cpp, and qualifying host-local compatible selections use sandbox-attached OpenShell providers. +Supported new custom hosted selections also use native providers bound to the selected endpoint and API. +On OpenShell `0.0.x`, retained legacy shared routes use `inference.local` until the sandbox is safely recreated. -New OpenClaw Ollama, standalone vLLM, llama.cpp, and hosted compatible selections outside the Personal tier use a sandbox-attached provider. Local NVIDIA NIM, a fresh Personal-tier hosted compatible selection, and custom endpoints without an attachment use `inference.local`; an older local selection without an attachment may retain it until a safe fresh recreation. +New OpenClaw Ollama, standalone vLLM, Local NVIDIA NIM, and llama.cpp selections use a sandbox-attached native provider. +Supported new OpenAI-compatible and Anthropic-compatible selections also use a native provider bound to the selected endpoint and API. +On OpenShell `0.0.x`, existing shared routes retain `inference.local` until the sandbox is safely recreated. +For endpoint requirements, refer to [Switch Inference Providers](../inference/manage-inference/switch-providers). OpenClaw generates a new gateway token when the sandbox container starts with mutable configuration. Retrieve the dashboard URL or token again after the container starts or a replacement sandbox is created. diff --git a/docs/deployment/set-up-mcp-bridge.mdx b/docs/deployment/set-up-mcp-bridge.mdx index 99c29e18ddf..a81f00020be 100644 --- a/docs/deployment/set-up-mcp-bridge.mdx +++ b/docs/deployment/set-up-mcp-bridge.mdx @@ -29,7 +29,7 @@ The integration has three parts: This integration depends on the OpenShell MCP/JSON-RPC L7 policy support from [NVIDIA/OpenShell#1865](https://github.com/NVIDIA/OpenShell/pull/1865). -Current NemoClaw builds require the pinned stable OpenShell `0.0.116` release, which exposes native `protocol: mcp` policy handling and endpoint-bound credential replacement. +Current NemoClaw builds require the pinned stable OpenShell `0.1.7` release, which exposes native `protocol: mcp` policy handling and endpoint-bound credential replacement. Development builds are rejected because they do not match the reviewed credential boundary for the supported managed runtime. NemoClaw accepts Streamable HTTP MCP endpoints only. @@ -63,8 +63,9 @@ The agent stores only an OpenShell resolver placeholder for the recorded credent OpenShell keeps the raw credential in its provider store and combines credential replacement with generated MCP policy at egress. -Hermes includes the stable credential handle from the readiness check in this placeholder. -Registration, inspection, rollback, and lifecycle reconciliation preserve that handle while OpenShell still reports it. +Hermes uses the credential placeholder observed by the readiness check. +Ordinary static credentials use revisioned placeholders; refresh-managed credentials can use stable handles. +Registration, inspection, rollback, and lifecycle reconciliation preserve the placeholder while OpenShell still reports it. For the normal MCP client path, OpenShell evaluates the effective policy for the destination host and port, adapter binary, literal endpoint path, and MCP method before replacing placeholders in allowed HTTP request headers. @@ -74,31 +75,34 @@ NemoClaw accepts canonical HTTPS MCP URLs and writes the credential placeholder ### DNS Pinning Contract -The MCP integration pins its OpenShell DNS enforcement contract to [`NVIDIA/OpenShell@d1155aa70042d3e2ee49dbfa15346b108b7c1d92`](https://github.com/NVIDIA/OpenShell/tree/d1155aa70042d3e2ee49dbfa15346b108b7c1d92). +The MCP integration pins its OpenShell DNS enforcement contract to [`NVIDIA/OpenShell@6648bd0c290efbc41ba131ee9831ee45cd431f94`](https://github.com/NVIDIA/OpenShell/tree/6648bd0c290efbc41ba131ee9831ee45cd431f94). With `proxy_connect_by_hostname` disabled, OpenShell uses one socket-address list throughout resolution, validation, and connection: -- [`crates/openshell-supervisor-network/src/proxy.rs:3857-3886`](https://github.com/NVIDIA/OpenShell/blob/d1155aa70042d3e2ee49dbfa15346b108b7c1d92/crates/openshell-supervisor-network/src/proxy.rs#L3857-L3886) produces the list. -- [`crates/openshell-supervisor-network/src/proxy.rs:3911-3951`](https://github.com/NVIDIA/OpenShell/blob/d1155aa70042d3e2ee49dbfa15346b108b7c1d92/crates/openshell-supervisor-network/src/proxy.rs#L3911-L3951) validates every address. -- [`crates/openshell-supervisor-network/src/proxy.rs:3983-4043`](https://github.com/NVIDIA/OpenShell/blob/d1155aa70042d3e2ee49dbfa15346b108b7c1d92/crates/openshell-supervisor-network/src/proxy.rs#L3983-L4043) connects through the validated list. +- [`resolve_socket_addrs`](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/crates/openshell-supervisor-network/src/proxy.rs#L4201-L4230) produces the list. +- [`validate_allowed_ips_for_resolved_addrs`](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/crates/openshell-supervisor-network/src/proxy.rs#L4255-L4295) validates every address against the policy pins. +- [`dial_upstream`](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/crates/openshell-supervisor-network/src/proxy.rs#L4327-L4388) connects through the validated list. The CONNECT path passes that validated connector to the direct or operator-configured upstream dial path. -By default, direct connections and corporate-proxy CONNECT requests use the validated address list at [`crates/openshell-supervisor-network/src/proxy.rs:3983-4043`](https://github.com/NVIDIA/OpenShell/blob/d1155aa70042d3e2ee49dbfa15346b108b7c1d92/crates/openshell-supervisor-network/src/proxy.rs#L3983-L4043). -The explicit HTTP-forward path validates the destination before it connects at [`crates/openshell-supervisor-network/src/proxy.rs:5613-5655`](https://github.com/NVIDIA/OpenShell/blob/d1155aa70042d3e2ee49dbfa15346b108b7c1d92/crates/openshell-supervisor-network/src/proxy.rs#L5613-L5655). +The [explicit HTTP-forward path](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/crates/openshell-supervisor-network/src/proxy.rs#L5967-L6037) also validates the destination before it connects. The exception is OpenShell's operator-controlled `proxy_connect_by_hostname` option. -When enabled, OpenShell sends the requested hostname to the corporate proxy instead of the validated address, which reopens proxy-side DNS resolution; the upstream source records that residual risk at [`crates/openshell-supervisor-network/src/proxy.rs:3998-4031`](https://github.com/NVIDIA/OpenShell/blob/d1155aa70042d3e2ee49dbfa15346b108b7c1d92/crates/openshell-supervisor-network/src/proxy.rs#L3998-L4031). +When enabled, OpenShell sends the requested hostname to the corporate proxy instead of the validated address, which reopens proxy-side DNS resolution. +The [`dial_upstream` source](https://github.com/NVIDIA/OpenShell/blob/6648bd0c290efbc41ba131ee9831ee45cd431f94/crates/openshell-supervisor-network/src/proxy.rs#L4342-L4375) records that risk. NemoClaw does not enable or prohibit that host-level OpenShell option, so operators who require the pinned-address guarantee must leave it disabled and ensure their gateway configuration does not enable it. -### Stable OpenShell 0.0.116 Limitations + -OpenShell v0.0.116 attributes network policy with `/proc//exe` and process ancestors, so script-based adapters require Node or Python interpreter grants rather than immutable package-entrypoint identities. -NemoClaw compensates by restricting each server to its HTTPS destination, path, MCP method profile, DNS pins, and unique least-privilege credential. +### Stable OpenShell 0.1.7 Limitations -Remove the interpreter grants when OpenShell exposes stable script or package entrypoint attribution. +OpenShell v0.1.7 attributes network policy with `/proc//exe`, verified process ancestors, and command-line-derived script paths. +NemoClaw's generated MCP policies grant Node or Python interpreter paths; those grants do not identify immutable package entrypoints. +NemoClaw compensates by restricting each server to its HTTPS destination, path, MCP method profile, DNS pins, and unique least-privilege credential. -OpenShell v0.0.116 requires a static credential provider to have a complete endpoint binding before the supervisor projects its stable credential handle. -NemoClaw imports the endpointless `nemoclaw-mcp-v1` profile, creates a dedicated provider for each server, and binds that provider to one generated MCP endpoint with `credential_binding.provider`. +OpenShell v0.1.7 requires a static credential provider to have a complete endpoint binding before the supervisor projects a usable credential placeholder. +NemoClaw imports an endpointless profile that declares the credential name supplied through `--env`. +The profile identity depends on that name, not the credential value. +NemoClaw creates a dedicated provider for each server and binds it to one generated MCP endpoint with `credential_binding.provider`. OpenShell withholds the credential before the binding is active and outside the bound host, port, and path. Credential keys must still be unique across providers attached to the same sandbox. @@ -106,14 +110,17 @@ NemoClaw rejects managed MCP credential-key reuse and requires a unique least-pr It also rejects credential-bound definitions for the same normalized endpoint URL when their credential bindings differ, because OpenShell cannot choose between credentials for otherwise indistinguishable requests. Entries that intentionally share one credential binding are not ambiguous; otherwise, keep one managed definition for that URL or use distinct endpoint URLs. -A managed MCP provider created by an earlier NemoClaw build can have the profile-less `generic` type and cannot satisfy this binding contract. -NemoClaw refuses to restart or rebuild that registration. -Remove the MCP server, export its credential again, and add the server so NemoClaw recreates it with the endpointless profile. +Earlier managed MCP providers can use `generic` or the `nemoclaw-mcp-v1` profile, which does not declare a credential. +NemoClaw accepts matching legacy providers only for explicit cleanup and refuses to restart or rebuild their registrations. +Remove the MCP server, export its recorded credential variable again, and add the server with the same endpoint and options. +Removal preserves the provider object for separate inspection and cleanup. +Refer to [Rotate a Credential](manage-mcp-servers#rotate-a-credential) for the provider-cleanup procedure. -OpenShell v0.0.116 updates, attaches, detaches, and deletes providers by mutable name instead of an atomic immutable identity. -NemoClaw compensates with randomized provider names, the per-sandbox lifecycle lock, and immediate ownership checks against the recorded provider ID and credential-key metadata plus the expected provider type before mutations. +OpenShell v0.1.7 supports resource-version checks, but its `provider update` and `provider delete` CLI commands still target mutable provider names. +NemoClaw cannot bind those CLI mutations atomically to the provider identity and version from its own inspection. +It uses randomized provider names, the per-sandbox lifecycle lock, and immediate ownership checks against the recorded provider ID, credential-key metadata, and expected provider type. -NemoClaw fails closed and preserves retryable state when those checks do not match, but the checks do not provide compare-and-swap behavior against another OpenShell client. +NemoClaw fails closed and preserves retryable state when those checks do not match, but its inspection and mutation remain separate operations. Do not concurrently replace or mutate a managed provider through another OpenShell client while an MCP lifecycle command is running. Use an MCP service you trust with the credential it receives. @@ -143,11 +150,11 @@ mcp_servers: resources: true prompts: true headers: - Authorization: Bearer openshell:resolve:env:s0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef_GITHUB_MCP_TOKEN + Authorization: Bearer openshell:resolve:env:v7_GITHUB_MCP_TOKEN ``` -The `s…_` prefix is an illustrative OpenShell stable credential handle. -Hermes records the handle that the readiness check proves for the live provider. +The `v7_` prefix is an illustrative OpenShell credential revision. +Hermes records the placeholder that the readiness check proves for the live provider. ### Verify the Hermes Reload diff --git a/docs/get-started/quickstart-hermes.mdx b/docs/get-started/quickstart-hermes.mdx index 3565881082f..e15a2d5f674 100644 --- a/docs/get-started/quickstart-hermes.mdx +++ b/docs/get-started/quickstart-hermes.mdx @@ -395,7 +395,8 @@ Use these details when your first-run path needs more control. After you select a provider and model, review the summary and apply the configuration. NemoClaw writes Hermes configuration into `/sandbox/.hermes` and starts the Hermes gateway inside the sandbox. NVIDIA Endpoints, OpenAI, Anthropic, Gemini, OpenRouter, and Hermes Provider use native endpoints through sandbox-attached OpenShell providers. - Custom and local providers retain the shared `inference.local` route. + Qualifying custom hosted endpoints also use sandbox-attached providers. + Hermes local selections and remaining custom shared selections retain `inference.local` on OpenShell `0.0.x`. OpenShell keeps upstream credentials outside the sandbox in both cases. The Hermes image includes runtime dependencies for supported NemoClaw messaging integrations, the API service, and its health endpoint. The base image does not include unsupported Hermes integrations. @@ -504,7 +505,7 @@ Use these details when your first-run path needs more control. The restart can interrupt active agent requests. If the restart fails, the route and configuration remain committed. Run `nemohermes gateway restart` to apply the switch. - Custom and local providers retain the shared OpenShell route. + Hermes local selections and remaining custom shared selections retain the shared OpenShell route on `0.0.x`. ```bash nemohermes inference set --model --provider diff --git a/docs/get-started/quickstart-langchain-deepagents-code.mdx b/docs/get-started/quickstart-langchain-deepagents-code.mdx index 514ba2913ee..820b83762ca 100644 --- a/docs/get-started/quickstart-langchain-deepagents-code.mdx +++ b/docs/get-started/quickstart-langchain-deepagents-code.mdx @@ -179,11 +179,9 @@ NemoClaw writes `/sandbox/.deepagents/config.toml` with an OpenAI-compatible pro NVIDIA Endpoints uses `https://integrate.api.nvidia.com/v1` through a least-privilege OpenShell provider attached to the sandbox. Supported fixed hosted providers also use their native endpoints through sandbox-attached providers. New Ollama, standalone vLLM, llama.cpp, and qualifying local compatible selections use their qualified host-local endpoint through a sandbox-attached OpenShell provider. -New hosted compatible endpoints normally use their recorded endpoint through a sandbox-attached provider. -A fresh Personal-tier hosted compatible selection instead uses `https://inference.local/v1` without creating a native provider attachment. -Other retained shared selections and owned Local NVIDIA NIM also use `https://inference.local/v1`. -Each route puts a scoped placeholder in the agent config instead of the real provider key. - +Qualifying custom hosted endpoints also use sandbox-attached providers. +Remaining custom shared selections retain `https://inference.local/v1` on OpenShell `0.0.x`. +Both paths put a scoped placeholder in the agent config instead of the real provider key. When onboarding records a reasoning effort on a `compatible-endpoint` route that uses `openai-completions`, managed startup writes that value to a root-owned file and Deep Agents Code model requests carry it as an `extra_body.reasoning_effort` request parameter. Leave `NEMOCLAW_REASONING_EFFORT` unset to keep the endpoint's own default. Deep Agents Code has no runtime `inference set` path, so re-onboard the sandbox with `nemo-deepagents onboard --fresh --name --recreate-sandbox` to change the recorded effort. @@ -192,8 +190,8 @@ When you use NVIDIA Endpoints without selecting another model, new Deep Agents C For this model, the managed image maps the OpenAI-compatible route to Deep Agents `0.7.29`'s native Nemotron 3 Ultra harness profile, including model-specific tool-calling, filesystem, retry, context, and final-answer safeguards. Rebuild existing Deep Agents Code sandboxes after upgrading NemoClaw so their image includes the pinned runtime and profile. This agent-specific default does not change the shared Nemotron 3.5 Super VL default for OpenClaw and Hermes. - -The image launcher normalizes the runtime proxy environment for interactive, login-shell, and direct-exec paths and removes inherited proxy credentials and bypass entries before `dcode` starts. +The image launcher clears inherited HTTP proxy variables, proxy credentials, and bypass entries for interactive, login-shell, and direct-exec paths before `dcode` starts. +OpenShell `0.1.7` mediates network requests through its destination policy. Managed interactive sessions pre-complete Deep Agents Code's optional first-run onboarding, skip its dependency and model selection screens, then open the TUI with the model selected during NemoClaw onboarding. The image includes `ripgrep` and `dos2unix`, and ordinary sessions suppress the optional Tavily warning unless web search is configured or invoked. @@ -308,14 +306,14 @@ nemo-deepagents rebuild If you upgrade from a release that persisted LangSmith environment values, rebuild each existing Deep Agents Code sandbox so its image includes the corrected `start.sh`. If an existing sandbox displays `Choose a Recommended Model`, rebuild it so its image includes the managed startup behavior. -`status` reports the selected harness as a terminal runtime and prints the interactive/headless command shape. If `status` reports `Runtime health: degraded` with an OOM kill count, rebuild the sandbox to restore the terminal runtime. Proxy launchers and startup scripts are baked into the sandbox image. After upgrading from a release that used `inference.local` for NVIDIA Endpoints, recreate each NVIDIA sandbox during onboarding. NemoClaw does not add the required native provider attachment to an existing beta sandbox. NemoClaw v0.0.78 and newer clients fail closed when a pre-v0.0.78 sandbox image lacks the trusted `/usr/local/lib/nemoclaw/dcode-managed-exec` route-probe helper, even when the installed Deep Agents Code version still matches the managed manifest. Rebuild the sandbox to install that image-owned helper before retrying `status`, `doctor`, `connect`, or onboarding recovery. There is no dashboard port or long-running gateway process for this harness. +`status` reports the selected harness as a terminal runtime and prints the interactive/headless command shape. If `status` reports `Runtime health: degraded` with an OOM kill count, rebuild the sandbox to restore the terminal runtime. Managed launchers and startup scripts are baked into the sandbox image. After upgrading from a release that used `inference.local` for NVIDIA Endpoints, recreate each NVIDIA sandbox during onboarding. NemoClaw does not add the required native provider attachment to an existing beta sandbox. NemoClaw v0.0.78 and newer clients fail closed when a pre-v0.0.78 sandbox image lacks the trusted `/usr/local/lib/nemoclaw/dcode-managed-exec` route-probe helper, even when the installed Deep Agents Code version still matches the managed manifest. Rebuild the sandbox to install that image-owned helper before retrying `status`, `doctor`, `connect`, or onboarding recovery. There is no dashboard port or long-running gateway process for this harness. Older Deep Agents Code sandboxes with a standalone local shared route have no native provider attachment. Back up the sandbox before upgrading OpenShell, then recreate it with `nemo-deepagents onboard --fresh --name --recreate-sandbox`. Expect downtime while the replacement is created. Resume and rebuild stop if the old standalone local route has no native attachment; NemoClaw does not migrate it automatically. This requirement applies to Ollama, standalone vLLM, llama.cpp, and qualifying local compatible endpoints. -Owned Local NVIDIA NIM keeps its shared route and can resume without a native attachment. +Local NVIDIA NIM also requires its recorded native attachment for resume; back up and recreate older shared-route sandboxes. ## Next Steps diff --git a/docs/get-started/quickstart-pi.mdx b/docs/get-started/quickstart-pi.mdx index c86d5b8416f..82005a3770e 100644 --- a/docs/get-started/quickstart-pi.mdx +++ b/docs/get-started/quickstart-pi.mdx @@ -53,14 +53,15 @@ For an existing installation, run the canonical onboarding command. nemoclaw onboard --agent pi --name my-pi ``` -Choose a provider and model that use the OpenAI Chat Completions API. -NemoClaw keeps upstream credentials on the host. -NVIDIA Endpoints, supported fixed hosted providers, new hosted custom endpoints outside the Personal policy tier, and new qualified local selections use sandbox-attached OpenShell providers. -OpenShell supplies the provider credential or adapter token at egress; Pi's configuration contains only a non-secret placeholder or environment reference. -New Personal-tier hosted custom selections, Local NVIDIA NIM, and unchanged published Pi shared-route selections use `inference.local`. -A failed native selection stops without falling back to the shared route. -For direct providers, OpenShell retains the credential across rebuild and sandbox destruction. -When no sandbox needs that provider, remove it explicitly with `nemoclaw credentials reset --yes`. +Choose a provider and model that use the OpenAI Chat Completions API. NemoClaw keeps upstream credentials outside the sandbox. NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. New Pi selections for Ollama, standalone vLLM, llama.cpp, and qualified host-local compatible endpoints use sandbox-attached providers. +Qualifying custom hosted endpoints also use sandbox-attached providers. +OpenShell supplies the direct provider credential or Bedrock adapter token at egress. +Pi configuration contains a non-secret placeholder or environment reference. +Bedrock AWS credentials stay in its host-local adapter. +Remaining custom shared selections and unchanged published Pi shared-route selections retain `inference.local`. +Those shared routes require OpenShell `0.0.x`; OpenShell `0.1.5` removed `inference.local`. +Pi native local onboarding on `0.1.7` remains unqualified for this upgrade. +A failed native selection stops without falling back to the shared route. For direct provider routes, the credential remains in OpenShell provider state across rebuild and sandbox destruction. When no sandbox needs that provider, remove it explicitly with `nemoclaw credentials reset --yes`. Stock onboarding selects a Pi managed-image digest. It does not build the Pi Dockerfile on the host. The reviewed Dockerfile is a trusted CI image-build source, not a user onboarding mechanism. A missing, incomplete, mutable, or inconsistent image contract fails before sandbox creation. diff --git a/docs/get-started/quickstart.mdx b/docs/get-started/quickstart.mdx index ae35ed2f7d1..9b8bc69152f 100644 --- a/docs/get-started/quickstart.mdx +++ b/docs/get-started/quickstart.mdx @@ -390,9 +390,11 @@ Use these details when your first-run path needs more control. After confirmation, NemoClaw registers the selected provider with OpenShell. - NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. - New Ollama, standalone vLLM, and llama.cpp selections also use sandbox-attached providers for their host-local endpoints. - Local NVIDIA NIM, custom endpoints, and earlier local selections without a sandbox-attached provider retain the shared `inference.local` route. + NVIDIA Endpoints, supported fixed hosted providers, and qualifying custom hosted endpoints use native endpoints through sandbox-attached OpenShell providers. + New OpenClaw Ollama, standalone vLLM, and llama.cpp selections also use attached providers. + For these new local attachments, `status`, `connect`, and `start` probe the recorded provider endpoint. + Remaining custom shared selections retain `inference.local` on OpenShell `0.0.x`. + Native local onboarding on `0.1.7` remains unqualified for this upgrade. The wizard asks whether to enable web search and offers Brave Search or Tavily Search. Provide `BRAVE_API_KEY` for Brave Search or `TAVILY_API_KEY` for Tavily Search when prompted. NemoClaw validates the selected key before it creates the sandbox, registers a sandbox-scoped OpenShell provider, and writes only an OpenShell resolver placeholder into the OpenClaw configuration. diff --git a/docs/inference/choose-inference-provider.mdx b/docs/inference/choose-inference-provider.mdx index 205481e2d0d..f3682c01289 100644 --- a/docs/inference/choose-inference-provider.mdx +++ b/docs/inference/choose-inference-provider.mdx @@ -45,7 +45,7 @@ Use this status table to distinguish validated provider integrations from adapte | Local llama.cpp (managed) | Experimental | Local OpenAI-compatible | Uses repository-owned YAML recipes and exact GGUF verification. The selected runtime-provider boundary owns lifecycle; all currently qualified profiles require Docker. Qualified DGX Spark hosts offer NVIDIA Nemotron 3 Nano 30B-A3B. Qualifying Windows WSL N1x and Station GB300 hosts automatically use managed llama.cpp during Windows Express or non-interactive onboarding when no provider is requested or recorded and `NEMOCLAW_MODEL` is unset. An unset recipe selects Qwen 3.6 35B-A3B; a compatible explicit recipe selects that managed recipe. The WSL profiles remain pending physical agent qualification. | | Local NVIDIA NIM | Experimental | Local OpenAI-compatible | Requires `NEMOCLAW_EXPERIMENTAL=1` and a NIM-capable NVIDIA GPU. Host must have the NVIDIA Container Toolkit installed and a healthy CDI spec. Onboarding evaluates the canonical `host.gpu.nvidia_available`, `host.gpu.container_toolkit_available`, and `host.gpu.cdi_healthy` readiness capabilities before gateway, image, or sandbox lifecycle effects. NIM images pull from `nvcr.io` and require NGC registry login. NemoClaw gates this path behind the experimental flag. Onboarding filters the validated image list by usable memory. In non-interactive mode, onboarding selects the first compatible model unless the operator sets `NEMOCLAW_MODEL`. Model selection uses detected free GPU memory. If free memory is unavailable, NemoClaw uses total GPU memory. On DGX Spark, NemoClaw caps usable memory at 50 percent of total memory to match NIM's reported unified-memory limit. The Nemotron 3 Super catalog minimum includes NIM's reported runtime allocation. Local NVIDIA NIM is unavailable on N1x. NemoClaw omits the provider from onboarding and rejects `NEMOCLAW_PROVIDER=nim-local` on N1x. Select another standard provider or use the Deferred managed-vLLM preview on N1x. On Linux arm64 DGX Spark and DGX Station hosts, onboarding warns that some NIM images may not publish a `linux/arm64` manifest; the warning is advisory, and the selected image pull can still fail when the registry has no matching platform manifest. Managed vLLM has host-specific default models and is not gated on the same boxes. Catalogued Local NIM images in `bin/lib/nim-images.json` include `nvidia/nemotron-3-super-120b-a12b`, `nvidia/nemotron-3-nano-30b-a3b`, and `nvidia/llama-3.3-nemotron-super-49b-v1`. | | Local vLLM (already running) | Tested with limitations | Local OpenAI-compatible | On a qualified N1x host, this route requires explicit standard-onboarding intent and remains unvalidated. On other hosts, no explicit intent is required. It appears in the onboarding menu when NemoClaw detects a server on `localhost:${NEMOCLAW_VLLM_PORT:-8000}`. The model is whatever the existing server serves. | -| Local vLLM (managed install/start) | Tested with limitations | Local OpenAI-compatible | Managed install/start requires Docker on `PATH`; attaching an already-running local vLLM server is a separate path. Appears by default on DGX Spark and qualifying DGX Station GB300 hosts. A labeled Deferred preview appears on hosts that match the N1x identity requirements; N1x remains Deferred until a physical NemoClaw Express E2E test passes. DGX Station is Tested with limitations across qualified profiles on one physical DGX Station GB300. Full Station Express end-to-end qualification for the accepted no-OTA DGX OS `7.6.x` profile is pending. Dual-Station configurations are not yet validated, and dedicated CI coverage is not available. For canonical Station qualification and host preparation, see the Additional Setup page for [OpenClaw](/user-guide/openclaw/get-started/additional-setup/dgx-station-preparation), [Hermes](/user-guide/hermes/get-started/additional-setup/dgx-station-preparation), or [Deep Agents](/user-guide/deepagents/get-started/additional-setup/dgx-station-preparation). Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`, NVIDIA Container Toolkit, and CDI. NemoClaw pins runtime images to immutable digests. New single-Station Express installs default to the experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`; a qualified trusted pair selects distributed Ultra, and `--station-deepseek` selects `deepseek-ai/DeepSeek-V4-Flash`. Direct managed-vLLM defaults are `nvidia/Qwen3.6-35B-A3B-NVFP4` on DGX Spark and the Deferred N1x preview, `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` on DGX Station, and `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8` on generic Linux NVIDIA GPU hosts. The pinned `nvcr.io/nvidia/vllm` images are public, so Docker pulls them without NGC registry login. | +| Local vLLM (managed install/start) | Tested with limitations | Local OpenAI-compatible | Managed install/start requires Docker on `PATH`; attaching an already-running local vLLM server is a separate path. Appears by default on DGX Spark and qualifying DGX Station GB300 hosts. A labeled Deferred preview appears on hosts that match the N1x identity requirements; N1x remains Deferred until a physical NemoClaw Express E2E test passes. DGX Station is Tested with limitations across qualified profiles on one physical DGX Station GB300. Full Station Express end-to-end qualification for the accepted no-OTA DGX OS `7.6.x` profile is pending. Dual-Station configurations are not yet validated, and dedicated CI coverage is not available. For canonical Station qualification and host preparation, see the Additional Setup page for [OpenClaw](/user-guide/openclaw/get-started/additional-setup/dgx-station-preparation), [Hermes](/user-guide/hermes/get-started/additional-setup/dgx-station-preparation), or [Deep Agents](/user-guide/deepagents/get-started/additional-setup/dgx-station-preparation). Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`, NVIDIA Container Toolkit, and CDI. NemoClaw pins runtime images to immutable digests. New single-Station Express installs default to the experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`; a qualified trusted pair selects distributed Ultra, and `--station-deepseek` selects `deepseek-ai/DeepSeek-V4-Flash`. Direct managed-vLLM defaults are `nvidia/Qwen3.6-35B-A3B-NVFP4` on DGX Spark and the Deferred N1x preview, experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` on DGX Station, and `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8` on generic Linux NVIDIA GPU hosts. The pinned `nvcr.io/nvidia/vllm` images are public, so Docker pulls them without NGC registry login. | {/* provider-status:end */} ## Hosted Providers @@ -104,7 +104,8 @@ The documented support applies to the NemoClaw adapter rather than every compati ## Routed Inference Use [Model Router](../hosted-inference/set-up-model-router) when you want a host-side router to select from a configured model pool. -The router registers as an OpenAI-compatible provider while the sandbox remains on `inference.local`. +New Model Router selections use a sandbox-attached OpenAI-compatible provider. +The agent calls the configured router endpoint, normally `http://host.openshell.internal:4000/v1`. ## Next Step diff --git a/docs/inference/choose-local-inference-server.mdx b/docs/inference/choose-local-inference-server.mdx index 4e500ee8152..2f833ee37b1 100644 --- a/docs/inference/choose-local-inference-server.mdx +++ b/docs/inference/choose-local-inference-server.mdx @@ -12,13 +12,17 @@ content: NemoClaw supports operator-run servers and NemoClaw-managed runtime profiles for local inference. Choose the option that matches your host, model, and operational needs. -New Deep Agents Code, Hermes, OpenClaw, and Pi sandboxes use sandbox-attached OpenShell providers for Ollama, standalone vLLM, and llama.cpp. -Deep Agents Code, Hermes, and Pi also use native providers for qualifying host-local compatible endpoints. -Local NVIDIA NIM continues to use `inference.local`. -Unchanged published Pi shared local selections retain their recorded route during resume and rebuild. -Existing OpenClaw Ollama, standalone vLLM, and llama.cpp sandboxes without an attachment retain supported shared-route model switches, but cannot resume or rebuild. +New Deep Agents Code, Hermes, OpenClaw, and Pi sandboxes use sandbox-attached OpenShell providers for Ollama, standalone vLLM, Local NVIDIA NIM, and llama.cpp. +Deep Agents Code, Hermes, OpenClaw, and Pi also use attachments for qualifying host-local compatible endpoints. +Other retained shared routes use `inference.local` on OpenShell `0.0.x`. +Unchanged published Pi shared local selections retain their recorded route during resume and rebuild on `0.0.x`. +OpenShell `0.1.5` removed `inference.local`; migration of those selections is incomplete. +Native local onboarding on `0.1.7` remains unqualified for this upgrade. + +On OpenShell `0.0.x`, existing OpenClaw Ollama, standalone vLLM, and llama.cpp sandboxes without an attachment retain supported shared-route model switches. +They cannot resume or rebuild. Back them up and re-onboard with `--fresh --name --recreate-sandbox`. -Owned Local NVIDIA NIM sandboxes can resume or rebuild on their shared route. +New Local NVIDIA NIM sandboxes use their recorded native attachment for resume and rebuild. Back up and recreate older NIM sandboxes before upgrading OpenShell. Hermes live qualification for the new local lifecycle remains pending. @@ -131,17 +135,20 @@ The profile rejects a model that does not resolve to that recipe and all `NEMOCL It also rejects `NEMOCLAW_MANAGED_CLUSTER_PEERS`. If you set `NEMOCLAW_SERVING_PRESET`, it must match the profile's catalog preset. Set `NEMOCLAW_VLLM_PORT` before installation to use another host listener port. -Before installation, ensure no other process or container owns `${NEMOCLAW_VLLM_PORT:-8000}` on `127.0.0.1` or on the private IPv4 gateway of `openshell-docker`. +Before installation, ensure no other process or container owns `${NEMOCLAW_VLLM_PORT:-8000}` on `127.0.0.1`. The profile performs these actions: - Uses the pinned runtime image and fixed serving command from the catalog. - Stores model files in the host Hugging Face cache. -- Publishes the authenticated server on `127.0.0.1:` and the same port on the private IPv4 gateway of the `openshell-docker` bridge, such as `172.18.0.1:`. The default is `8000`. +- Publishes the authenticated server only on `127.0.0.1:`. + The default port is `8000`. - Never publishes the Docker host port on `0.0.0.0` or `::`. - Reuses the owner-only host-global managed-vLLM API key. -After the runtime passes its readiness check, Deep Agents Code, Hermes, and OpenClaw onboarding attach a provider for the qualified host-local endpoint. OpenShell reaches the authenticated server through the private bridge binding, while host-side readiness and recovery use the loopback binding. +After the runtime passes its readiness check, Deep Agents Code, Hermes, and OpenClaw onboarding attach a provider for the qualified host-local endpoint. +The native OpenShell supervisor, host readiness checks, and recovery use the same loopback listener. +OpenShell supplies the sandbox route through `host.openshell.internal`; the runtime does not require an `openshell-docker` bridge. Verify the sandbox route: @@ -157,17 +164,13 @@ Verify the bounded host publication: ```bash VLLM_HOST_PORT="${NEMOCLAW_VLLM_PORT:-8000}" -docker network inspect \ - --format '{{json .IPAM.Config}}' \ - openshell-docker docker container inspect \ --format '{{json (index .NetworkSettings.Ports "8000/tcp")}}' \ nemoclaw-vllm ``` -The bridge result must contain exactly one private IPv4 `Gateway` value. Docker indexes these bindings under the recipe-owned container port, `8000/tcp`, even when `VLLM_HOST_PORT` selects another host port. -The container result must contain exactly two bindings with `HostPort` equal to `VLLM_HOST_PORT`: one `HostIp` value of `127.0.0.1` and one that exactly matches the private bridge gateway. +The container result must contain exactly one binding with `HostIp` equal to `127.0.0.1` and `HostPort` equal to `VLLM_HOST_PORT`. Reject the runtime if a binding uses `0.0.0.0`, `::`, an empty address, another host address, or an additional entry. If installation stops after a model download, rerun the same installer command. diff --git a/docs/inference/custom-endpoint-security.mdx b/docs/inference/custom-endpoint-security.mdx index 3cd878da819..16bb49a0533 100644 --- a/docs/inference/custom-endpoint-security.mdx +++ b/docs/inference/custom-endpoint-security.mdx @@ -14,20 +14,13 @@ NemoClaw keeps provider credentials on the host and validates explicit custom en ## Protect Provider Credentials -New direct hosted custom selections outside the Personal policy tier use the recorded endpoint through a sandbox-attached OpenShell provider. -Fresh Personal-tier selections and retained shared routes use `inference.local`. +Qualifying custom hosted endpoints use a sandbox-attached OpenShell provider. +Remaining custom shared selections use `inference.local` on OpenShell `0.0.x`. - -New Deep Agents Code sandboxes normally use the recorded endpoint through a sandbox-attached OpenShell provider for direct hosted compatible endpoints and qualified host-local endpoints. -Bedrock selections use a host-local adapter endpoint instead. -A fresh Personal-tier hosted compatible selection instead uses `inference.local` without creating a native attachment. -Other retained shared routes and owned Local NVIDIA NIM also use `inference.local`. -Refer to [How Inference Routing Works](../about-inference-routing) for the qualification boundary. - - -New Hermes sandboxes use the recorded endpoint through a sandbox-attached OpenShell provider for direct hosted compatible endpoints outside the Personal policy tier and qualified host-local endpoints. -Bedrock selections use a host-local adapter endpoint instead. -Fresh Personal-tier hosted selections and retained custom routes without an attachment use `inference.local`. + +New Deep Agents Code and Hermes sandboxes use the recorded endpoint through a sandbox-attached OpenShell provider when the host-local selection qualifies. +Qualifying custom hosted endpoints also use sandbox-attached providers. +Remaining custom shared selections retain `inference.local` on OpenShell `0.0.x`. Refer to [How Inference Routing Works](../about-inference-routing) for the qualification boundary. @@ -161,19 +154,17 @@ Hermes Provider setup, Hermes host-side `config set`, and a direct blueprint run -## Use the Sandbox Host Alias - -NemoClaw accepts `http://host.openshell.internal:` only when the URL includes an explicit port from `1024` through `65535`. -This narrow exception supports NemoClaw-managed sandbox-to-host inference routes and Hermes host-side `config set` writes to supported provider `baseUrl` fields that already use the OpenShell host bridge. -Other Hermes `config set` keys continue to reject the bridge host. -Accepting this URL shape does not by itself make an arbitrary local service, embedding provider, config key, or network route a supported NemoClaw integration. +## Use a Private Host Endpoint -For example, a containerized gateway can commonly reach a local server at `http://host.openshell.internal:8000/v1`. -Because this name is a sandbox-internal alias, onboarding skips the host-side endpoint probe. -Verify the runtime route after onboarding when you use it. +NemoClaw rejects an unverified custom route that uses `host.openshell.internal` because the alias and port do not prove which host process owns the destination. +Saved custom hosted provider bindings also reject loopback or OpenShell-managed hostnames as their upstream endpoint. +AWS Bedrock retains its separately validated adapter binding. +Qualified OpenAI-compatible native-local routes can retain their recorded host bridge endpoint. +NemoClaw also uses the alias for routes created from recorded runtime authority, such as its managed credential and HTTPS pin adapters. -The exception permits only HTTP, the literal `host.openshell.internal` name, and an explicit unprivileged port. -NemoClaw still rejects HTTPS, privileged ports, subdomains, loopback names, private IP literals, userinfo, query strings, and fragments through this path. +For an operator-owned private service, use its reachable hostname or IP address and add that exact host to `NEMOCLAW_TRUSTED_PRIVATE_HOSTS` before onboarding. +NemoClaw still resolves and pins every admitted address. +It rejects wildcards, link-local metadata addresses, and other reserved destinations. ## Related Topics diff --git a/docs/inference/how-inference-routing-works.mdx b/docs/inference/how-inference-routing-works.mdx index e457612aad4..e6a73617fea 100644 --- a/docs/inference/how-inference-routing-works.mdx +++ b/docs/inference/how-inference-routing-works.mdx @@ -29,7 +29,7 @@ Sandbox agent -> provider API hostname -> sandbox-attached OpenShell provider -> | Anthropic | `https://api.anthropic.com` | | Google Gemini | `https://generativelanguage.googleapis.com/v1beta/openai/` | | OpenRouter | `https://openrouter.ai/api/v1` | -| Hermes Provider | `https://inference-api.nousresearch.com/v1`, or the endpoint returned by authenticated login | +| Hermes Provider | `https://inference-api.nousresearch.com/v1`, the endpoint returned by authenticated login, or a validated endpoint retained from an existing selection | Each sandbox records its selected provider attachment. Changing one sandbox's native selection does not change another sandbox's route. OpenRouter attribution headers remain in the managed agent configuration. @@ -38,23 +38,40 @@ A different destination cannot gain access through a later DNS change. Existing sandboxes without a native attachment retain their recorded shared route. Legacy local routes have the recovery restrictions described under [Credential Boundary](#credential-boundary). -New Deep Agents Code, Hermes, OpenClaw, and Pi selections for Ollama, standalone vLLM, and llama.cpp use a sandbox-attached provider for the qualified host-local endpoint. -New hosted OpenAI-compatible and Anthropic-compatible selections for OpenClaw, Deep Agents Code, Hermes, and Pi use an endpoint-bound attached provider outside the Personal policy tier. -For a direct custom endpoint, the agent uses the recorded endpoint and a scoped credential reference. +New Deep Agents Code, Hermes, OpenClaw, and Pi selections for Ollama, standalone vLLM, Local NVIDIA NIM, and llama.cpp use sandbox-attached providers for qualified host-local endpoints. +Deep Agents Code, Hermes, OpenClaw, and Pi also use attached providers for qualifying host-local compatible endpoints. +OpenClaw and Hermes compatible attachments preserve the selected Chat Completions or Responses API. +Qualifying custom hosted endpoints use sandbox-attached providers. +For a direct endpoint, the agent uses the recorded endpoint and a scoped credential reference. Bedrock uses the host-local adapter endpoint and an OpenShell adapter token. -A fresh Personal-tier hosted compatible selection instead stays on the shared `inference.local` route and creates no native attachment. -Deep Agents Code, Hermes, and Pi use attached providers for qualified host-local compatible endpoints. -Custom endpoints without an attachment, Local NVIDIA NIM, Model Router, and unchanged published shared selections retain the shared route. +New Model Router selections use sandbox-attached providers for the configured router endpoint, normally `http://host.openshell.internal:4000/v1`. +OpenShell authorizes model-list and Chat Completions requests and supplies the provider credential; the sandbox receives only an endpoint-bound credential reference. +NemoClaw verifies the recorded router process, port, and credential identity before publishing the selection. +Remaining shared custom selections and unchanged published Pi shared local selections retain the shared route. +Shared `inference.local` routes require OpenShell `0.0.x`. +Fresh installations without retained legacy state can install OpenShell `0.1.7` and use sandbox-attached native providers. +The installer refuses to migrate registered OpenShell `0.0.x` sandboxes, including shared `inference.local` routes, and preserves their state. +OpenShell `0.1.5` removed the shared route. +GPU compatibility and native local lifecycle qualification remain incomplete. +Follow the [OpenShell upgrade restrictions](../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before changing an existing installation. Hermes live qualification for the new local lifecycle remains pending. ```text Sandbox agent -> inference.local -> shared OpenShell route -> selected provider and model ``` -Use `$$nemoclaw inference get` to inspect the selected sandbox's recorded route. +Use `$$nemoclaw inference get` to inspect the named sandbox's recorded route. `status`, `doctor`, and startup checks verify its recorded native provider identity before using a native attachment. Missing or conflicting ownership stops the operation instead of falling back to the shared route. + +Native local provider profiles include the pinned image's resolved Python interpreter, `/usr/bin/python3.13`. +An older imported profile that lacks this executable can conflict with the current profile definition. +NemoClaw stops and leaves an incompatible existing profile unchanged. +Preserve the profile and provider while a NemoClaw maintainer reviews their ownership and attached sandboxes and chooses an explicit replacement plan. +Retry only after the profile conflict is resolved. + + ## Credential Boundary Provider credentials stay on the host in OpenShell provider state or, for AWS Bedrock Runtime, in its host-local adapter. @@ -63,13 +80,27 @@ For attached direct custom endpoints, OpenShell binds the credential environment The Bedrock adapter uses a host-local endpoint and supplies OpenShell an adapter token instead of direct endpoint IP pins. Removing a private host from the ambient allowlist does not revoke an already verified direct endpoint binding. +Legacy NVIDIA Endpoints sandboxes recorded as `nvidia-nim` use a shared route. +Resume requires a confirmed OpenShell `0.0.x` runtime; otherwise, NemoClaw stops before changing the provider or route. +Back up the sandbox before re-onboarding with `--fresh --name --recreate-sandbox` to attach a native provider. + Local Ollama and local vLLM routes do not require the host `OPENAI_API_KEY`. NemoClaw uses provider-specific local tokens for those routes. For retained shared local routes, rebuilds migrate away from stale OpenAI credential requirements. -Legacy Deep Agents Code local sandboxes without an attachment require a backup and fresh recreation. Existing OpenClaw Ollama, standalone vLLM, and llama.cpp sandboxes without an attachment retain supported model switches on their shared route, but cannot resume or rebuild. Back one up and re-onboard with `--fresh --name --recreate-sandbox`. Owned Local NVIDIA NIM sandboxes retain the shared route and can resume or rebuild. +Legacy Deep Agents Code local sandboxes without an attachment require a backup and fresh recreation. +On OpenShell `0.0.x`, existing OpenClaw Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible sandboxes without an attachment retain supported shared-route model switches. +They cannot resume or rebuild. +Back one up and re-onboard with `--fresh --name --recreate-sandbox`. +Legacy Model Router sandboxes without a native attachment require a backup and fresh recreation with `--fresh --name --recreate-sandbox`. +NemoClaw preserves their legacy volumes and does not automatically copy the data. +Native Model Router resume and model changes verify the existing provider and router ownership; uncertain ownership stops the operation and preserves recovery state. +See [Set Up Model Router](hosted-inference/set-up-model-router) for its model pool and process lifecycle. +New Local NVIDIA NIM sandboxes use their recorded native attachment for resume and rebuild. +For older NIM sandboxes without a native attachment, preserve backups, the existing installation, and sandbox data. +Follow the [OpenShell upgrade restrictions](../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path). Legacy Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible routes without a native attachment cannot resume or rebuild. Back up required sandbox state before running `nemohermes onboard --fresh --name --recreate-sandbox` to replace the sandbox with a native attachment. -Owned Local NVIDIA NIM and nonqualifying custom routes retain their shared route. +Remaining custom shared selections retain their shared route on OpenShell `0.0.x`. For an attached Hermes route, startup refreshes the credential reference before starting the gateway. Rebuild refreshes only the restored credential references, preserving the selected model and provider settings. When a rebuild reuses an automatically bridged compatible-endpoint route without a host API key, NemoClaw reapplies the config-only bridge rewrite without reading or passing the credential stored in OpenShell. @@ -78,12 +109,9 @@ When a rebuild reuses an automatically bridged compatible-endpoint route without ## Deep Agents Configuration -For Deep Agents, NemoClaw writes `/sandbox/.deepagents/config.toml` with a managed OpenAI-compatible provider, a scoped placeholder API key, and `use_responses_api = false`. -Supported fixed native hosted selections use the endpoints above. -Direct hosted compatible selections use their recorded endpoint, while Bedrock selections use the host-local adapter endpoint. -A fresh Personal-tier hosted compatible selection uses `https://inference.local/v1` without a native attachment. -New Ollama, standalone vLLM, llama.cpp, and qualifying local compatible selections use their qualified host-local endpoint through an attached provider. -Owned Local NVIDIA NIM and retained shared-route selections use `https://inference.local/v1`. +For Deep Agents, NemoClaw writes `/sandbox/.deepagents/config.toml` with a managed OpenAI-compatible provider, a scoped placeholder API key, and `use_responses_api = false`. Supported native hosted selections use the endpoints above. +Direct hosted compatible selections use their recorded endpoint; Bedrock selections use the host-local adapter endpoint. +New Ollama, vLLM, Local NVIDIA NIM, llama.cpp, Model Router, and qualifying local compatible selections use their qualified host-local endpoint through an attached provider. Other shared-route selections retain `https://inference.local/v1`. The managed `dcode` runtime uses Chat Completions through OpenShell even when a compatible endpoint also supports the Responses API. diff --git a/docs/inference/set-up-anthropic-compatible-endpoint.mdx b/docs/inference/set-up-anthropic-compatible-endpoint.mdx index 7182e3a9f16..bd8790f1516 100644 --- a/docs/inference/set-up-anthropic-compatible-endpoint.mdx +++ b/docs/inference/set-up-anthropic-compatible-endpoint.mdx @@ -31,11 +31,23 @@ $$nemoclaw onboard Select **Other Anthropic-compatible endpoint**. Enter the endpoint base URL, model ID, and API key when prompted. Use any non-empty placeholder such as `dummy` when the endpoint does not require authentication. -For a new hosted endpoint outside the Personal policy tier, NemoClaw attaches an endpoint-bound OpenShell provider to the sandbox. -A new Personal-tier selection and an existing sandbox without a native attachment keep the shared `inference.local` route. For a direct custom endpoint, OpenShell holds the upstream credential and the sandbox receives a scoped `openshell:resolve:env:...` reference instead of the raw key. For AWS Bedrock Runtime, a host-local adapter holds the AWS credentials and OpenShell receives only the adapter token. +## Understand Native Endpoint Selection + +New qualifying custom endpoints use a sandbox-attached OpenShell provider. +Use a public endpoint or an explicitly trusted private hostname or IP address. +NemoClaw rejects an Anthropic-compatible route that uses `host.openshell.internal` because the alias does not prove which process owns the selected port. +Follow the [custom endpoint security requirements](custom-endpoint-security) for private destinations and address pinning. + +The native route binds the endpoint, API, and credential source to the selected sandbox. +OpenShell supplies the host credential at egress; the agent receives a credential reference. +Existing sandboxes without a native attachment retain their recorded shared route on OpenShell `0.0.x`. +For retained shared selections, follow the [resume and rebuild restrictions](../../manage-sandboxes/operate-sandboxes/recover-and-rebuild-sandboxes#resolve-rebuild-preflight-stops). +OpenShell `0.1.5` removed `inference.local`; this repair does not complete migration of every shared route. +Native local qualification remains pending for this upgrade. + ## Understand OpenClaw Validation @@ -62,12 +74,12 @@ Refer to [Onboarding rejects an Anthropic-compatible tool call](../../reference/ ## Understand Hermes Routing -Hermes uses the OpenAI Chat Completions frontend for this provider. -NemoClaw validates `/v1/chat/completions`. -For a new hosted selection outside the Personal tier, NemoClaw attaches an endpoint-bound OpenAI-compatible provider. -Hermes uses the direct custom endpoint's OpenAI-compatible `/v1` base URL or, for Bedrock, the host-local adapter URL in its model configuration. -That configuration uses `${COMPATIBLE_ANTHROPIC_API_KEY}`; the sandbox environment supplies the scoped OpenShell credential reference. -For a new Personal-tier selection or a retained shared route, Hermes uses `https://inference.local/v1` while NemoClaw registers the upstream with OpenShell as `type=openai` using `OPENAI_BASE_URL`. +Hermes uses OpenAI Chat Completions for this provider. +For a qualifying native selection, Hermes calls the recorded endpoint through its sandbox-attached OpenShell provider. +NemoClaw validates `/v1/chat/completions` and configures the same API for inference. +The route keeps `COMPATIBLE_ANTHROPIC_API_KEY` as its host credential binding. +Hermes configuration uses `${COMPATIBLE_ANTHROPIC_API_KEY}`; its sandbox environment supplies a scoped OpenShell credential reference instead of the API key. +For Bedrock, the configured endpoint is the host-local adapter URL. The OpenClaw-only native Anthropic `emit_ok` streaming check does not run on this route. This path avoids duplicate Anthropic SSE `message_start` events. @@ -85,11 +97,10 @@ AWS Bedrock routes keep their existing OpenAI-compatible adapter behavior. Deep Agents Code supports only OpenAI-compatible inference for this provider. NemoClaw validates `/v1/chat/completions` and selects `openai-completions`. -For a new hosted selection outside the Personal tier, `/sandbox/.deepagents/config.toml` uses the direct custom endpoint's OpenAI-compatible `/v1` surface or, for Bedrock, the host-local adapter URL. +For a qualifying native selection, `/sandbox/.deepagents/config.toml` records the direct endpoint or, for Bedrock, the host-local adapter URL. The file sets `api_key_env = "COMPATIBLE_ANTHROPIC_API_KEY"`. The sandbox environment supplies the scoped OpenShell credential reference for that name. -Personal-tier and retained shared selections use `https://inference.local/v1` in that file. -For direct custom endpoints, OpenShell keeps the `COMPATIBLE_ANTHROPIC_API_KEY` credential on the host for either route. +Retained shared selections use `https://inference.local/v1` on OpenShell `0.0.x`. The OpenClaw-only native Anthropic `emit_ok` streaming check does not run on this route. If the endpoint implements only Anthropic Messages, onboarding stops instead of creating a Deep Agents sandbox with an unusable runtime route. @@ -99,10 +110,12 @@ If the endpoint implements only Anthropic Messages, onboarding stops instead of ## Run Non-Interactive Onboarding Set `NEMOCLAW_PROVIDER=anthropicCompatible` and provide the endpoint URL, model, and credential. +Replace the example endpoint and model with your server values. +Use `dummy` only when that server does not require authentication. ```bash NEMOCLAW_PROVIDER=anthropicCompatible \ - NEMOCLAW_ENDPOINT_URL=http://localhost:8080 \ + NEMOCLAW_ENDPOINT_URL=https://inference.example.com \ NEMOCLAW_MODEL=my-model \ COMPATIBLE_ANTHROPIC_API_KEY=dummy \ $$nemoclaw onboard --non-interactive diff --git a/docs/inference/set-up-llama-cpp.mdx b/docs/inference/set-up-llama-cpp.mdx index 98457f4927f..7b3ea8137e0 100644 --- a/docs/inference/set-up-llama-cpp.mdx +++ b/docs/inference/set-up-llama-cpp.mdx @@ -20,14 +20,17 @@ content: NemoClaw provides two first-class llama.cpp paths. You can attach an authenticated server that you operate, or let NemoClaw select and materialize a compatible repository-owned profile. Both paths are experimental and use OpenAI Chat Completions. +OpenShell `0.1.7` GPU qualification remains pending, including Docker Desktop and Windows WSL validation of the managed path. - -Agent requests use `https://inference.local/v1`. - - -New Deep Agents Code and Hermes sandboxes use the recorded local endpoint through a sandbox-attached OpenShell provider. + +New OpenClaw, Hermes, Deep Agents Code, and Pi llama.cpp selections use the recorded local endpoint through a sandbox-attached OpenShell provider. OpenShell supplies the credential at egress; the sandbox receives only a credential reference. -Refer to [How Inference Routing Works](../about-inference-routing) for the qualification and existing-sandbox boundaries. +Refer to [How Inference Routing Works](../about-inference-routing) for qualification and existing-sandbox boundaries. +For Pi's release-candidate and platform restrictions, refer to [Pi Support and Security](/user-guide/pi/reference/pi-support). +Pi native local onboarding on OpenShell `0.1.7` remains unqualified. + + +Older OpenClaw sandboxes without a native attachment retain their recorded shared route, which requires OpenShell `0.0.x`. Live qualification of the Hermes native local lifecycle is pending. @@ -59,11 +62,20 @@ The server must satisfy all of these requirements: - Report a stable, non-path served model alias. - Return native llama.cpp model metadata without conflicting model entries. -On native Docker, publishing only `-p 127.0.0.1:8081:8081` can leave the sandbox route unreachable. -Keep the loopback publication and also publish `-p :8081:8081` for the OpenShell network. -For a host process, configure its listener or forwarding so both addresses work. -Retain native API-key authentication and restrict non-loopback access to the sandbox network. -Onboarding stops when the existing network probe confirms that the attached server cannot be reached. +On native Linux Docker with OpenShell `0.1.7` and its default supervisor network mode, publish the server with `-p 127.0.0.1:8081:8081`, or bind the host process to `127.0.0.1:8081`. +The native supervisor reaches this loopback listener through `host.openshell.internal`; a second publication on a Docker bridge is unnecessary. +Retain native API-key authentication. +NemoClaw checks TCP reachability from a restricted probe container on the supervisor's host network. +The probe does not depend on a retained OpenShell Docker bridge. + +Docker Desktop, the portable profile, and native Podman retain their existing gateway-address probes. +For these paths, preserve host-loopback access and make the authenticated server reachable through the selected runtime's host gateway. +Restrict any non-loopback listener to the required runtime network. +These probe changes do not establish OpenShell `0.1.7` platform qualification. + +A confirmed connection failure stops onboarding. +If the probe cannot run, exceeds its execution deadline, or cannot resolve the hostname, this preflight remains inconclusive and continues. +An inconclusive result does not prove reachability; subsequent inference-route validation still applies. Configure the server with its own supported mechanism before you run NemoClaw. Start the server with an explicit served model alias. @@ -317,51 +329,17 @@ Refer to [CLI Commands](../../reference/commands) for complete command behavior. During managed installation, NemoClaw first checks internal runtime readiness and host-loopback health. It repeats these checks when onboarding resumes. -It then runs the authoritative OpenShell Docker bridge probe on fixed port `8081`. -On Docker Desktop in WSL, NemoClaw runs the loopback health probe from the WSL CLI process because a host-network container cannot reach the WSL loopback listener. -If the bridge probe cannot connect or times out after the earlier checks pass, onboarding stops without changing UFW. -On Docker Desktop in WSL, this failure reports the private sandbox bridge probe result. -Before retrying onboarding, confirm these conditions: - -- Docker Desktop is running. -- Its WSL integration is enabled for this distribution. -- The OpenShell Docker network and host gateway are available. - -The WSL path does not print a UFW command. - -On native Linux, when the bridge reports a valid narrow subnet and a gateway IP address inside that subnet, the error reports these values: - -- The detected Docker network. -- The source subnet. -- The gateway IP address. -- The fixed port `8081`. -- The narrow UFW command. - -On native Linux, if the bridge topology is missing or invalid, onboarding fails closed without printing topology or a UFW command. -Inspect the OpenShell Docker bridge configuration before you retry onboarding. - - - On native Linux, the printed command changes UFW. - Confirm that the reported Docker network, source subnet, and gateway IP address belong to the OpenShell Docker bridge before you run it. - The rule allows TCP traffic only from that subnet to port `8081` on that gateway IP address. - - -On native Linux, run the command from the onboarding error. -It has this form: - -```bash -sudo ufw allow from to port 8081 proto tcp -``` - -On native Linux, after you apply the rule, rerun the same onboarding selection. - -On native Linux, remove the rule if onboarding still reports the bridge failure or the managed runtime no longer needs it: - -```bash -sudo ufw --force delete allow from to port 8081 proto tcp -``` - -On native Linux, use the same subnet and gateway IP address that appeared in the original command. +On native Linux, it then probes `host.openshell.internal:8081` from a restricted probe container using the native supervisor's host network. +The alias resolves to `127.0.0.1` for this probe. +The model container remains on its internal Docker network with no published ports. +The authenticated host bridge listens only on `127.0.0.1:8081`. + +On Docker Desktop in WSL, NemoClaw runs the loopback health probe from the WSL CLI process. +It then checks `host.openshell.internal:8081` from a bridge-network probe container using Docker's host gateway. +If either probe fails, onboarding stops and reports the failed check without changing UFW or suggesting a firewall rule. +Check that Docker is running and that the managed host bridge is listening on port `8081`. +On Docker Desktop in WSL, also check that WSL integration is enabled for the distribution and that Docker's host gateway is reachable. +Correct the reported failure, then rerun the same onboarding selection. If resumed onboarding reports that it cannot reserve host-local inference, run `$$nemoclaw doctor`. If the diagnostic identifies the former WSL Qwen image, resume is unavailable. diff --git a/docs/inference/set-up-model-router.mdx b/docs/inference/set-up-model-router.mdx index 4d1e4376763..92e3a35157a 100644 --- a/docs/inference/set-up-model-router.mdx +++ b/docs/inference/set-up-model-router.mdx @@ -10,13 +10,14 @@ content: type: "how_to" --- Model Router runs on the host and selects a model from a configured pool for each request. -OpenShell registers it as an OpenAI-compatible provider while the sandbox remains on `inference.local`. +OpenShell attaches an OpenAI-compatible provider to each sandbox for the configured Model Router endpoint. ## How the Router Fits The Model Router option uses the `routed` inference profile in `nemoclaw-blueprint/blueprint.yaml`. -During onboarding, NemoClaw starts the router proxy on host port `4000`, waits for its health endpoint, and registers the `nvidia-router` provider with OpenShell. -The sandbox does not call port `4000` directly. +During onboarding, NemoClaw starts the router proxy on host port `4000`, waits for its health endpoint, and prepares the sandbox-owned native provider for the `nvidia-router` selection. +The agent addresses `http://host.openshell.internal:4000/v1` through the OpenShell supervisor. +The supervisor permits the selected provider’s model-list and Chat Completions requests. On a host without cached routing-model files, Model Router downloads and loads its routing model before the health endpoint responds. NemoClaw gives a running router up to 10 minutes to pass its health check and stops onboarding sooner if the process exits. Routed OpenClaw setup then waits for the selected sandbox gateway to accept connections before requesting its native restart. @@ -29,8 +30,8 @@ NemoClaw does not truncate or rotate the file. If output capture fails, router startup continues without captured output. NemoClaw sends one semantic health request at a time so slow upstream checks do not overlap. Each request uses only the time that remains in the 10-minute startup limit. -If a semantic health response completed before startup failed, the error identifies the last request outcome, HTTP status, duration, and captured body size. -When a completed response returns an unhealthy-endpoint error, the startup error includes a redacted excerpt. +If startup fails after a health check, the error identifies the last request outcome, HTTP status, duration, and captured body size. +When the last completed response contains an unhealthy-endpoint error, the startup error includes a redacted excerpt, even if a later request times out. When output capture succeeded, the startup error also includes the log path. When the log contains readable output from this run, the error includes a redacted excerpt. The owner-only log contains unredacted router output. @@ -71,8 +72,16 @@ Review the uninstall scope before you use it for log removal. Sandbox agent -> OpenShell -> Model Router on port 4000 -> NVIDIA API ``` -Credentials flow through the OpenShell provider system. -The sandbox never sees raw API keys. +The host router uses the configured NVIDIA credential for its upstream model pool. +OpenShell stores the provider credential and gives the sandbox an endpoint-bound resolver reference. +The sandbox never receives the raw API key. +Before provider registration, NemoClaw verifies the recorded router process, port, and credential identity. +It stops without publishing the selection if ownership cannot be verified. + +Existing sandboxes that use the retired shared `inference.local` route require explicit recreation. +Back up the sandbox before re-onboarding with `--fresh --name --recreate-sandbox`. +NemoClaw preserves legacy volumes and does not automatically copy their contents. +A native Model Router resume revalidates the router and provider ownership before publishing the selection. ## Configure the Model Pool diff --git a/docs/inference/set-up-nvidia-nim.mdx b/docs/inference/set-up-nvidia-nim.mdx index f0af8d8247d..08d4f06ac11 100644 --- a/docs/inference/set-up-nvidia-nim.mdx +++ b/docs/inference/set-up-nvidia-nim.mdx @@ -12,6 +12,10 @@ content: NemoClaw can pull, start, and manage a local NVIDIA NIM container on hosts with a NIM-capable NVIDIA GPU. This provider is experimental and requires an explicit opt-in. +New Local NVIDIA NIM selections use a native OpenShell provider attached to the sandbox. +Live qualification on OpenShell `0.1.7` remains pending. +For older NIM sandboxes without a native attachment, preserve backups, the existing installation, and sandbox data. +Follow the [OpenShell upgrade restrictions](../../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path). Local NVIDIA NIM is unavailable on N1x. NemoClaw omits this provider from onboarding and rejects `NEMOCLAW_PROVIDER=nim-local` on N1x. Use the [Deferred managed-vLLM preview](set-up-vllm#use-n1x-express) instead. @@ -59,6 +63,9 @@ Non-interactive onboarding cannot prompt for registry credentials. Run `docker login nvcr.io` before starting non-interactive onboarding. When `NGC_API_KEY` or `NVIDIA_INFERENCE_API_KEY` is already exported, NemoClaw passes it into the managed NIM container through the process environment instead of command-line arguments. +The NGC key stays with Docker and the NIM container. +The native inference provider stores a nonsecret sentinel for the existing unauthenticated NIM API. +OpenShell limits the attached provider to the configured NIM port and the models and Chat Completions paths. ## Understand Model Detection diff --git a/docs/inference/set-up-ollama.mdx b/docs/inference/set-up-ollama.mdx index 840c6d088ac..cd4b03b612f 100644 --- a/docs/inference/set-up-ollama.mdx +++ b/docs/inference/set-up-ollama.mdx @@ -26,6 +26,20 @@ NemoClaw detects Ollama on the host and can install, start, or upgrade it on sup Fresh Hermes Portable onboarding does not use the host Ollama setup described in the remaining sections. It creates a receipt-owned Ollama runner through the current user's rootless Podman authority. +On OpenShell `0.1.7`, fresh onboarding attaches a native provider for `http://host.openshell.internal:11434/v1` to the Hermes sandbox. +The supervisor supplies a credential reference through `NEMOCLAW_LOCAL_INFERENCE_TOKEN`; its stored value is the nonsecret Ollama sentinel. +The private publication journal binds the transaction to the provider name, endpoint profile, immutable provider ID, and resource version. +Recovery and cleanup stop when that authority changes. +Existing shared-route journals are preserved for their recorded cleanup and are not adopted as native providers. +Back up an existing sandbox before retiring its old runtime and creating a native replacement. +The OpenShell `0.1.7` upgrade remains unqualified until the complete runtime E2E suite passes. + +Portable CLI operations require exactly OpenShell `0.1.7`. +A retained lifecycle receipt that records a `0.1.5` executable is rejected without rewriting the receipt or deleting its recorded resources. +There is no automatic migration of that receipt. +Preserve the receipt and sandbox data, and follow the [OpenShell upgrade restrictions](../../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before planning a replacement. +Do not edit the recorded version to bypass the refusal. + Run the fresh Portable command in the [Hermes quickstart](../../get-started/quickstart#use-portable-ollama) with `NEMOCLAW_PROVIDER=ollama` and an explicit `NEMOCLAW_MODEL`. This experimental profile requires exactly Podman `5.7.0` for both the client and the local rootless engine. @@ -158,8 +172,10 @@ The menu identifies a reachable daemon as running and labels an installed but st NemoClaw lists installed models or offers starter models when none are installed. For registry-known models, each menu entry shows the download size and approximate VRAM requirement. -The menu shows currently available GPU memory when the host reports it, including `0`, or total GPU memory when available memory is unknown. -A `0` reading means no GPU memory is currently free, so each known starter model is shown as exceeding available memory. +The menu shows currently available GPU memory when the host reports it, or total GPU memory when available memory is unknown. +A reported value of `0 MiB` means no available GPU memory; NemoClaw does not substitute total memory for that reading. +Each known starter model then appears as exceeding available memory. +Free GPU memory before retrying a model that does not fit. Among installed models that fit, NemoClaw selects the registry-known model with the highest approximate VRAM requirement. Unregistered tags follow registry-known tags and retain Ollama's list order. diff --git a/docs/inference/set-up-openai-compatible-endpoint.mdx b/docs/inference/set-up-openai-compatible-endpoint.mdx index 970bbe54edd..eda9b9b4ab2 100644 --- a/docs/inference/set-up-openai-compatible-endpoint.mdx +++ b/docs/inference/set-up-openai-compatible-endpoint.mdx @@ -13,26 +13,24 @@ content: Use the custom OpenAI-compatible provider for servers that implement `/v1/chat/completions` or a compatible `/v1/responses` API. Examples include vLLM, TensorRT-LLM, llama.cpp, LocalAI, and other compatible servers. +Qualifying custom hosted endpoints use sandbox-attached OpenShell providers. + -New hosted custom selections outside the Personal policy tier use the recorded endpoint through a sandbox-attached OpenShell provider. -Fresh Personal-tier selections and retained shared routes use `inference.local`. -OpenShell holds the upstream credential and supplies it at egress. - - -New Deep Agents Code sandboxes normally use sandbox-attached OpenShell providers for hosted compatible endpoints and qualified host-local endpoints. -Fresh onboarding keeps a hosted compatible endpoint on `inference.local` when you select the Personal policy tier. -For an attached route, the agent uses the recorded endpoint and a credential reference; OpenShell supplies the credential at egress. -Retained shared routes and owned Local NVIDIA NIM use `inference.local`. -Refer to [How Inference Routing Works](../about-inference-routing) for the qualification boundary. +New OpenClaw sandboxes use attached providers for qualified host-local compatible endpoints. +The attachment preserves the selected Chat Completions or Responses API and restricts requests to that API, model discovery, and the recorded endpoint. +The sandbox receives a credential reference; OpenShell supplies the host-managed credential at egress. +Remaining custom shared selections use `inference.local` on OpenShell `0.0.x`. +OpenShell forwards that traffic to the endpoint configured during onboarding. - -New Hermes sandboxes use sandbox-attached OpenShell providers for hosted compatible endpoints outside the Personal policy tier and qualified host-local endpoints. -Fresh Personal-tier hosted selections stay on `inference.local` without a native attachment. -For an attached route, the agent uses the recorded endpoint and a credential reference; OpenShell supplies the credential at egress. -Retained shared routes and owned Local NVIDIA NIM use `inference.local`. -Refer to [How Inference Routing Works](../about-inference-routing) for the qualification boundary. + +New Deep Agents Code, Hermes, and Pi sandboxes use sandbox-attached OpenShell providers for qualified host-local endpoints. +The agent uses the recorded endpoint and a credential reference; OpenShell supplies the credential at egress. +Remaining custom shared selections retain `inference.local` on OpenShell `0.0.x`. +For retained shared selections, follow the [resume and rebuild restrictions](../../manage-sandboxes/operate-sandboxes/recover-and-rebuild-sandboxes#resolve-rebuild-preflight-stops). +Refer to [How Inference Routing Works](../about-inference-routing) for qualification and existing-sandbox boundaries. +The native attachment preserves the selected Chat Completions or Responses API through model changes and rebuilds. Live qualification of the Hermes native local lifecycle is pending. @@ -90,28 +88,21 @@ Use a host-routable URL such as `http://localhost:8000/v1` when you want onboard To qualify for automatic rewriting, an HTTP endpoint URL must use the loopback host `localhost`, `127.0.0.1`, or `[::1]`. Automatic rewriting is limited to the port selected by `NEMOCLAW_VLLM_PORT` (`8000` by default) and ports `11434` and `11435`. NemoClaw validates the entered URL from the host. -Shared selections use `https://inference.local/v1` inside the sandbox. -Attached selections use the recorded endpoint; qualified host-local selections may use `host.openshell.internal:` to reach the host. +For eligible loopback endpoints, it registers the OpenShell gateway route through `host.openshell.internal:` for sandbox traffic. +Other qualifying custom endpoints retain their entered endpoint. +For an authenticated backend already reachable at `http://host.openshell.internal:/v1`, native attachments preserve the accepted unprivileged port. +NemoClaw rejects ports reserved for its gateways, Model Router, dashboards, and credential proxies. +Automatic loopback rewriting remains limited to the ports listed above. +For qualifying custom hosted endpoints, the attached provider profile authorizes the selected endpoint and API. -Shared OpenClaw compatible selections use the base `inference.local` policy. -New hosted selections outside the Personal tier use their endpoint-bound attached provider profile. -Neither route requires adding the `local-inference` preset. +For qualified host-local endpoints, the attached provider profile authorizes the recorded endpoint, selected API, and OpenClaw runtime binaries. +Remaining custom shared selections use the base `inference.local` policy on OpenShell `0.0.x`. - -For qualified host-local endpoints, the attached provider profile authorizes the recorded endpoint and the managed DCode runtime binaries. -Hosted compatible endpoints normally use their recorded endpoint through an attached provider. -Fresh Personal-tier onboarding keeps a hosted compatible endpoint on the base `inference.local` policy and does not create a native provider attachment. -NemoClaw refuses Personal for an existing sandbox that already has a custom native provider attachment; create a new sandbox instead. -Retained shared routes and owned Local NVIDIA NIM use the base `inference.local` policy. -These routes do not require the `local-inference` preset. -Refer to [Policy Tiers](../../reference/network-policies#policy-tiers) for the Personal-tier limitation and recovery steps. - - -For qualified host-local endpoints, the attached provider profile authorizes the recorded endpoint and the Hermes runtime binaries. -New hosted compatible endpoints outside the Personal tier use an endpoint-bound attached provider. -Fresh Personal-tier and retained shared selections use the base `inference.local` policy. -These routes do not require adding the `local-inference` preset. + +For qualified host-local endpoints, the attached provider profile authorizes the recorded endpoint and the selected agent runtime binaries. +Remaining custom shared selections use the base `inference.local` policy on OpenShell `0.0.x`. +These managed paths do not require adding the `local-inference` preset. For authenticated endpoints, NemoClaw leaves URLs without an explicit port, URLs on `:80` or another privileged port, and URLs on unsupported ports unchanged. Those authenticated URLs require a separately compatible runtime topology and network policy. This rewrite depends on an OpenShell topology that resolves `host.openshell.internal` inside the sandbox; if that bridge is unavailable, onboarding can still validate the host URL, but `$$nemoclaw status` is the authoritative runtime check. diff --git a/docs/inference/set-up-vllm.mdx b/docs/inference/set-up-vllm.mdx index 4fcf0ba2e5d..38cdba12ec7 100644 --- a/docs/inference/set-up-vllm.mdx +++ b/docs/inference/set-up-vllm.mdx @@ -12,6 +12,7 @@ content: NemoClaw can connect to an existing vLLM server or, when Docker is available on `PATH`, install and start a managed vLLM container on supported NVIDIA GPU hosts. It also provides a Deferred N1x preview pending a physical NemoClaw Express E2E run. +OpenShell `0.1.7` GPU qualification remains pending; the network changes below do not establish Docker Desktop, Windows WSL, or Jetson/Tegra qualification. NemoClaw forces the Chat Completions API path for vLLM. @@ -20,32 +21,35 @@ The vLLM `/v1/responses` endpoint does not run the configured tool-call parser, Local vLLM does not require authentication by default. -Existing-server and single-host managed-vLLM paths use `NEMOCLAW_VLLM_PORT`, which defaults to `8000`, on host loopback for validation and on the OpenShell Docker bridge for sandbox traffic. +Existing-server and single-host managed-vLLM paths use `NEMOCLAW_VLLM_PORT`, which defaults to `8000`, for host-loopback validation. +With OpenShell `0.1.7` and its default Docker supervisor network mode, the supervisor reaches a managed server through the same loopback listener. After onboarding, `status` and `doctor` probe an authenticated managed single-host vLLM container on its published loopback port. For a bearerless server, they use the port in the sandbox's recorded `http://host.openshell.internal:/v1` route. When that route is valid, no later port environment override is required. For vLLM, `connect` runs these host probes only when it repairs the route. A missing or invalid recorded route makes a bearerless probe fall back to `NEMOCLAW_VLLM_PORT`, which defaults to `8000`. Use a host firewall with default-deny inbound rules. -Allow the configured TCP port only from the OpenShell Docker subnet to its gateway address, keep loopback access, and deny the port on every other interface. +For an operator-managed server, restrict the configured TCP port to the required local inference path and preserve loopback access. Do not expose the port to your LAN or the internet. -A new managed single-host container publishes the configured host port on exactly two host addresses: `127.0.0.1` and the private IPv4 gateway of the `openshell-docker` bridge. +A new managed single-host container publishes the configured host port only on `127.0.0.1`, independently of bearer authentication. The container continues to listen on its recipe-owned internal port `8000`. It never publishes that host port on `0.0.0.0` or `::`. -OpenShell reaches the server through the private bridge binding. - -The provider route remains `https://inference.local` inside the sandbox. - - -New Deep Agents Code and Hermes standalone vLLM selections use the recorded local endpoint through a sandbox-attached OpenShell provider. +The native OpenShell supervisor reaches the server through `host.openshell.internal`, which resolves to loopback for the NemoClaw-managed gateway. + +New OpenClaw, Hermes, Deep Agents Code, and Pi standalone vLLM selections use the recorded local endpoint through a sandbox-attached OpenShell provider. OpenShell supplies the credential at egress; the sandbox receives only a credential reference. Refer to [How Inference Routing Works](../about-inference-routing) for qualification and existing-sandbox boundaries. +For Pi's release-candidate and platform restrictions, refer to [Pi Support and Security](/user-guide/pi/reference/pi-support). +Pi native local onboarding on OpenShell `0.1.7` remains unqualified. + + +Older OpenClaw sandboxes without a native attachment retain their recorded shared route, which requires OpenShell `0.0.x`. Live qualification of the Hermes native local lifecycle is pending. -The managed Muse Glimmer profile uses the same bearer-protected listener and two-address publication boundary. +The managed Muse Glimmer profile uses the same bearer-protected loopback listener. The qualified two-DGX Spark and two-DGX Station paths use bearer-protected `/v1` routes and Docker host networking on both runtime containers; network isolation remains required for other endpoints and distributed traffic. Follow [Set Up vLLM on Two DGX Sparks](set-up-vllm-on-two-dgx-sparks) or [Set Up vLLM on Two DGX Stations](set-up-vllm-on-two-dgx-stations) for their isolation requirements. @@ -250,10 +254,10 @@ In an interactive run, the managed path lists the supported model registry for y Press **Enter** to use the profile default or select another validated model with its registered `vllm serve` arguments. The single-host path pulls the vLLM image, downloads model weights into `~/.cache/huggingface`, and starts the `nemoclaw-vllm` container with the configured host port (`8000` by default). -For every single-host profile, Docker publishes that port only on `127.0.0.1` and the private IPv4 gateway of the `openshell-docker` bridge. +For every single-host profile, Docker publishes that port only on `127.0.0.1`. Bearerless profiles do not require NemoClaw's managed bearer key. Existing containers keep their published bindings until they are recreated. -Managed route recovery rejects owned bearerless containers whose bindings do not match the required pair. +Managed route recovery rejects owned bearerless containers whose bindings are not limited to the required loopback address and port. To recreate the container, follow [Handle a Running vLLM Server](#handle-a-running-vllm-server) with the original model and port settings. The distributed DGX pages document their separate container names, endpoints, and lifecycle checks. The container carries NemoClaw ownership and immutable catalog labels so later onboarding, recovery, and uninstall runs can distinguish it from a bring-your-own server without trusting the container name alone. @@ -264,7 +268,7 @@ On hosts other than N1x, if another vLLM server already occupies the configured On N1x, NemoClaw stops managed onboarding and requires you to stop that server before trying the Deferred preview again. Except for the interrupted managed install described below, an occupied single-host port stops installation before NemoClaw persists a bearer key or records the selected model. For an interrupted install, NemoClaw can remove and replace only a running `nemoclaw-vllm` container with the ownership label and matching port mappings through the selected local Docker connection. -Managed route recovery accepts an owned bearerless container only when container port `8000` maps to the requested host port on exactly `127.0.0.1` and the private IPv4 gateway of `openshell-docker`. +Managed route recovery accepts an owned bearerless container only when container port `8000` maps to the requested host port on `127.0.0.1` alone. Route recovery rejects other bindings, including `0.0.0.0` and `::`. If route recovery rejects the container, follow [Handle a Running vLLM Server](#handle-a-running-vllm-server) to recreate it with the original model and port settings. It refuses an unlabeled, stopped, distributed, ambiguous, differently mapped, or differently addressed container. @@ -739,7 +743,7 @@ On N1x, that setting is explicit preview intent and waives only the pending phys Add `NEMOCLAW_EXPERIMENTAL=1` on a generic Linux NVIDIA GPU host. Non-interactive runs use the profile default unless you set `NEMOCLAW_VLLM_MODEL`. -The commands above invoke `$$nemoclaw onboard` directly, so a DGX Station run with no model or peer selects the `nemotron-3.5-super-vl-120b` catalog default. +The commands above invoke `$$nemoclaw onboard` directly, so a DGX Station run with no model or peer selects the experimental `nemotron-3.5-super-vl-120b` catalog default. Supplying `NEMOCLAW_PROVIDER=install-vllm` to the shell installer enters the Station host-preparation boundary while retaining that profile default. For paired non-interactive setup, follow [Set Up vLLM on Two DGX Stations](set-up-vllm-on-two-dgx-stations). diff --git a/docs/inference/switch-providers.mdx b/docs/inference/switch-providers.mdx index e51f136efb6..22dd2695723 100644 --- a/docs/inference/switch-providers.mdx +++ b/docs/inference/switch-providers.mdx @@ -10,12 +10,13 @@ content: type: "how_to" --- Move a sandbox to another provider family while keeping the selected inference path, agent configuration, and host registry aligned. -Use onboarding first when the target is neither `compatible-endpoint` nor `compatible-anthropic-endpoint` and is not registered. +Runtime switching accepts catalog providers and qualifying providers already registered on the selected sandbox's OpenShell gateway. +Use onboarding to configure a provider that is supported by onboarding but unavailable for runtime switching. The compatible endpoint workflow below can register an absent custom provider from complete route metadata. -## Find a Registered Provider +## Choose a Supported Provider List the configured provider credentials when you need the provider ID. @@ -27,13 +28,58 @@ $$nemoclaw credentials list It accepts recognized underscore spellings of canonical provider IDs and hyphenated aliases, such as `ollama_local`, `nvidia_prod`, and `open_router`. NemoClaw normalizes an accepted alias or underscore spelling to its canonical OpenShell provider ID and records the canonical name in the sandbox registry. -If the requested provider is unsupported, NemoClaw lists selectable supported provider IDs registered on the selected gateway. -If it cannot read the gateway list, it lists every supported provider ID instead. +The legacy hosted alias `nvidia-nim`, including `nvidia_nim`, selects NVIDIA Endpoints and normalizes to `nvidia-prod`. +The installer names `nim`, `nim-local`, and `nim_local` select Local NVIDIA NIM through `vllm-local`. +This local selection requires the NIM container and endpoint recorded for the sandbox. +When switching back from another provider, an onboarding session matching the sandbox, provider, and model can supply that metadata. +The endpoint must match the NIM runtime, and that runtime must serve the requested model. +Missing or inconsistent NIM state stops the switch before mutation; it does not select hosted NVIDIA Endpoints or standalone vLLM. +Rerun onboarding for the affected sandbox with `NEMOCLAW_PROVIDER=nim-local` to repair its recorded selection, then retry the switch. +Endpoint, API-family, and credential overrides do not apply to this local selection. -If OpenShell cannot find a requested provider other than `compatible-endpoint` or `compatible-anthropic-endpoint`, NemoClaw leaves the route and sandbox state unchanged. +If a provider is outside the catalog, NemoClaw checks for its exact name on the selected sandbox's verified gateway. +An unavailable inventory or an absent provider stops the switch before mutation. +Messaging providers cannot be selected for inference. + +If OpenShell cannot find a requested catalog provider other than `compatible-endpoint` or `compatible-anthropic-endpoint`, NemoClaw leaves the route and sandbox state unchanged. Run `$$nemoclaw onboard` to register the provider, then retry the switch. For a compatible custom endpoint, pass the complete route metadata described below so NemoClaw can register the provider. +### Select an Existing Registered Provider + +Use the existing provider's name with the runtime switch command below. +Provide its endpoint, API family, and credential environment variable through the native agent configuration or the corresponding command options. +Use `--endpoint-url`, `--inference-api`, and `--credential-env` only for metadata that agrees with the existing configuration. +The endpoint must use HTTPS, including operator-managed private endpoints. +NemoClaw rejects HTTP before attaching the provider or reading its credential handle. +Retained HTTP records remain readable for inspection and removal of NemoClaw-created attachments, but NemoClaw refuses to resume credential use through those endpoints. +Every profile endpoint matching the selected host and port must use `protocol: rest` and explicit `enforcement: enforce`. +Each matching endpoint's `tls` field must be omitted or an empty string to retain automatic TLS inspection. +Its `allow_uninspected_credentials` field must be omitted or `false`. +NemoClaw refuses selection and credential use when these checks fail. +Identity inspection and removal of NemoClaw-created attachments remain available, subject to identity and ownership checks. +The profile's existing rules, access mode, and IP restrictions remain operator-controlled. +NemoClaw verifies the provider identity and its declared profile against the selected endpoint and authentication method before attaching it to the sandbox. +Missing or conflicting metadata stops the switch. + +Before changing a registered provider attachment, NemoClaw verifies the sandbox's recorded lifecycle identity against the live sandbox. +Missing lifecycle metadata, a changed identity, or an unsuccessful identity check stops the operation. +These checks also apply when removing a new attachment or restoring the previous attachment after failure. +If identity verification fails, NemoClaw stops further attachment changes and preserves the remaining resources. +Inspect the reported sandbox and resolve the identity failure before retrying. + +The credential value remains with OpenShell. +NemoClaw reads the selected sandbox's OpenShell credential handle, verifies the attachment again, and writes that handle into the agent configuration. +It does not import raw credentials or accept authentication-header overrides from the native configuration. +The existing provider and its credentials remain operator-owned; switching does not update or delete them. + +The switch uses the native endpoint without changing the shared inference route. +After switching, confirm the provider and endpoint with `$$nemoclaw inference get`. +Run `$$nemoclaw status` to verify inference health. +If verification fails before the selection is committed, NemoClaw attempts to remove only its newly created attachment and restore the previous selection. +A target attachment that existed before the switch is retained. +If cleanup or restoration cannot be confirmed, resolve the reported failure before retrying. + @@ -48,7 +94,7 @@ Target a non-default sandbox by name. $$nemoclaw inference set --provider --model --sandbox ``` -For native NVIDIA and fixed hosted providers, NemoClaw updates only the selected sandbox attachment and verifies a request through its native endpoint. A hosted compatible endpoint uses a native provider during a runtime switch only when the sandbox already records a valid native attachment; a shared-route sandbox keeps its shared route. OpenClaw also keeps model switches on a recorded native Ollama, standalone vLLM, or llama.cpp attachment within that sandbox. +For native NVIDIA and fixed hosted providers, NemoClaw updates only the selected sandbox attachment and verifies a request through its native endpoint. A hosted compatible endpoint uses a native provider during a runtime switch only when the sandbox already records a valid native attachment; a retained shared-route sandbox keeps its route only on OpenShell `0.0.x`. OpenClaw also keeps model switches on a recorded native Ollama, standalone vLLM, or llama.cpp attachment within that sandbox. OpenRouter requests include its attribution headers. If sandbox verification fails, NemoClaw tries to restore the previous inference selection and does not commit the agent configuration. @@ -60,13 +106,19 @@ For OpenClaw, NemoClaw updates the provider namespace and selected model in the It adds or updates the requested model entry without removing other model entries in that provider. For NVIDIA Endpoints, native hosted providers, and native local providers, it writes the attached provider's scoped OpenShell credential reference into the native configuration. This reference is not a raw provider credential. -A switch to a local provider without a matching sandbox-owned attachment stops before changing the shared route. Back up the sandbox and create a fresh one with the local selection. Existing shared-route local sandboxes can still change models through their recorded shared route where its owner supports that operation. +When switching to a different Ollama, standalone vLLM, or llama.cpp provider, NemoClaw prepares and attaches a provider for the selected local runtime. +The switch verifies existing attachment ownership and leaves the shared route unchanged. +Existing shared-route local sandboxes that keep the same provider can still use supported model switches on OpenShell `0.0.x`. +Older Ollama, standalone vLLM, and llama.cpp sandboxes without an attachment cannot resume or rebuild. +Back them up and re-onboard with `--fresh --name --recreate-sandbox`. After any changed configuration sync, NemoClaw restarts only the OpenClaw gateway and verifies its health. The sandbox remains running, but agent requests are briefly interrupted. After every changed synchronized route, NemoClaw verifies that the local CLI device has the managed gateway's required pairing scopes before it reports success. If the restart fails, the route and configuration remain committed. Run `$$nemoclaw gateway restart` to apply the committed configuration. If pairing does not converge, the route and configuration remain committed. +For `initial-state-unavailable` or `final-state-unavailable`, use the observation failure category in the error and audit entry to diagnose the failed check. +The category does not include raw observer output, pairing state, or credentials. Run `$$nemoclaw doctor --fix`, then retry the agent turn. @@ -74,8 +126,9 @@ Run `$$nemoclaw doctor --fix`, then retry the agent turn. For Hermes, NemoClaw updates the native `/sandbox/.hermes/config.yaml`, including the model, base URL, API-family mode, and managed API-key value. -A hosted compatible selection outside the Personal tier uses an attached provider during a runtime switch only when this sandbox already records a valid native attachment. Qualified Ollama, standalone vLLM, llama.cpp, and host-local compatible selections use sandbox-attached providers without changing the shared route. -Owned Local NVIDIA NIM and nonqualifying custom endpoints retain the shared route. +Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use sandbox-attached providers without changing the shared route. +Qualifying custom hosted endpoints also use sandbox-attached providers. +Remaining custom shared selections retain `inference.local` on OpenShell `0.0.x`. A legacy local sandbox without a valid Hermes attachment must be recreated before a native local switch: back up required state, then run `nemohermes onboard --fresh --name --recreate-sandbox`. Live qualification of the Hermes native local lifecycle is pending. `inference.local` uses the OpenShell proxy rewrite sentinel. @@ -100,18 +153,21 @@ A rebuild restores the previous native `openclaw.json`, so it does not repair th -If the in-sandbox configuration sync fails after the gateway route changes, NemoClaw keeps the gateway and host registry aligned and prints a rebuild hint. -Run the rebuild before relying on the running agent. +If the in-sandbox configuration sync fails after the gateway route changes, NemoClaw keeps the gateway and host registry aligned. +For an existing registered provider, retry the same `inference set` command to finish configuration sync. +For other providers, follow the printed rebuild hint before relying on the running agent. Use `--no-verify` only when OpenShell cannot verify the target provider at switch time and you have already confirmed its provider and credential. This flag does not bypass shared-gateway compatibility checks. -When you explicitly supply a direct compatible endpoint at `http://host.openshell.internal:`, NemoClaw skips OpenShell's host-side provider probe because that hostname resolves only inside the sandbox network. -It then sends a validation request from the target sandbox before persisting the route in NemoClaw state. +NemoClaw rejects an unverified custom route that uses `host.openshell.internal` because the alias and port do not prove which host process owns the destination. +Qualified OpenAI-compatible native-local routes and managed adapter routes retain their recorded host bridge endpoint. +Use a public endpoint or an explicitly trusted private hostname or IP address. The output limit is 16 tokens for ordinary providers and 256 tokens for Google Gemini. -For sandbox-only custom endpoints, NemoClaw waits 6 seconds for OpenShell's route cache after changing the provider or model, then verifies the new route from inside the sandbox. +Before sandbox verification, NemoClaw waits 6 seconds when the provider, model, or provider attachment changes. +This wait also applies when a custom endpoint change replaces the attachment but keeps the provider and model names. When the switch also changes the API family and that request returns HTTP `400` or `404`, NemoClaw retries up to two times after delays of 2 and 4 seconds. NemoClaw uses the same retry schedule when validation fails before it receives an HTTP status, such as while the gateway route is reloading. Each retry uses the same output limit for the selected provider. diff --git a/docs/inference/understand-provider-validation.mdx b/docs/inference/understand-provider-validation.mdx index 050d223455a..659b081c7e7 100644 --- a/docs/inference/understand-provider-validation.mdx +++ b/docs/inference/understand-provider-validation.mdx @@ -19,6 +19,9 @@ When both streams are available, choose `retry` to open the secure API-key promp If either stream is not an interactive terminal, NemoClaw exits with a nonzero status. Rerun onboarding with both streams attached to retry credential entry. NemoClaw retries transient upstream failures before it reports a provider failure. +For NVIDIA Endpoints, an HTTP `500` response permits an initial request and up to three retries, with delays of 5, 15, and 30 seconds. +If the fourth response is HTTP `500`, that retry sequence fails without replay through another transport. +Other transport and credential recovery behavior is unchanged. After a fatal provider or inference-validation failure before sandbox creation, NemoClaw attempts to release an unowned NemoClaw-managed OpenShell gateway and remove its registration so provider credentials do not remain in a live process. It preserves a gateway that another registered sandbox uses, an external supervisor owns, or whose teardown authority cannot be proven. If listener release cannot be confirmed, NemoClaw keeps the gateway registration for recovery and reports the remaining listener. @@ -27,6 +30,14 @@ The `nvapi-` prefix check applies only to `NVIDIA_INFERENCE_API_KEY`. OpenRouter keys must be non-empty and begin with `sk-or-`. Other provider keys use provider-aware validation during the retry flow. +## Local Server Readiness + +Local vLLM and NIM probes that use `/v1/models` require HTTP `200` and a JSON body with a `data` array. +An empty array passes this availability check; model selection and inference validation remain separate checks. +Redirects and other HTTP statuses fail readiness even when the body contains a model list. +Managed NIM also requires its recorded container to be running. +If NemoClaw cannot inspect that container, it stops the health wait. + ## Native Local Provider Profile Verification During native local endpoint onboarding, NemoClaw retains the expected OpenShell provider profile after a successful inspection and checks it again after attaching the provider. @@ -56,25 +67,30 @@ This applies to local runtimes such as Ollama and vLLM. Onboarding continues only when a retry returns a structured tool call. If the last retry fails or times out, validation stops without another Chat Completions attempt. + +Fresh installations without retained legacy state can install OpenShell `0.1.7` and use sandbox-attached native providers. +The installer refuses to migrate registered OpenShell `0.0.x` sandboxes, including shared `inference.local` routes, and preserves their state. +OpenShell `0.1.5` removed the shared route. +GPU compatibility and native local lifecycle qualification remain incomplete. +Follow the [OpenShell upgrade restrictions](../../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before changing an existing installation. + + The managed Deep Agents runtime keeps `use_responses_api = false` and uses Chat Completions. -NVIDIA Endpoints, supported fixed hosted providers, and new Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use native endpoints through sandbox-attached OpenShell providers. -Hosted compatible endpoints normally use the same native path. -Fresh Personal-tier onboarding instead keeps a hosted compatible endpoint on `inference.local` and creates no native provider attachment. -NemoClaw refuses Personal for an existing sandbox that already has a custom native provider attachment. -Owned Local NVIDIA NIM and retained legacy shared routes use `inference.local`. +NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, and new Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use native endpoints through sandbox-attached OpenShell providers. +Remaining shared selections use `inference.local` only on OpenShell `0.0.x`. `NEMOCLAW_PREFERRED_API` does not change that runtime selection. -New OpenClaw Ollama, standalone vLLM, llama.cpp, and hosted compatible selections outside the Personal tier use their recorded sandbox-attached provider for validation. -Local NVIDIA NIM, new Personal-tier hosted compatible selections, and retained shared routes use `inference.local`. -An older OpenClaw Ollama, standalone vLLM, or llama.cpp sandbox without a native attachment can use supported shared-route model switches, but cannot resume or rebuild. +New OpenClaw Ollama, standalone vLLM, llama.cpp, and qualifying custom hosted selections use their recorded sandbox-attached native provider for validation. +Remaining shared selections use `inference.local` only on OpenShell `0.0.x`. +On OpenShell `0.0.x`, an older OpenClaw Ollama, standalone vLLM, or llama.cpp sandbox without a native attachment can use supported shared-route model switches, but cannot resume or rebuild. Back it up and re-onboard with `--fresh --name --recreate-sandbox`. -Owned Local NVIDIA NIM sandboxes can resume or rebuild on the shared route. +New Local NVIDIA NIM sandboxes use their recorded native attachment for resume and rebuild. Back up and recreate older NIM sandboxes before upgrading OpenShell. @@ -130,7 +146,9 @@ Verify that route from inside the sandbox after onboarding. -For a shared OpenClaw `compatible-endpoint` selection, the compatible endpoint check validates `inference.local` from inside the sandbox. +For OpenClaw, the `compatible-endpoint` check validates the selected route from inside the sandbox. +Qualifying custom hosted selections use their recorded native provider attachment. +Remaining shared selections use `inference.local` only on OpenShell `0.0.x`. The check runs whether or not you select a messaging channel and stops onboarding if the route fails. diff --git a/docs/inference/use-hermes-provider.mdx b/docs/inference/use-hermes-provider.mdx index 238879faccc..11c144dcfdf 100644 --- a/docs/inference/use-hermes-provider.mdx +++ b/docs/inference/use-hermes-provider.mdx @@ -19,6 +19,16 @@ NemoClaw binds the native provider to the endpoint returned by Hermes authentica The agent calls that endpoint while OpenShell retains and injects its credential. The sandbox does not receive raw provider credentials. +When recovery or a model change finds a previously recorded Hermes endpoint, NemoClaw registers an attached provider for that exact destination. +Public destinations retain validated public IP addresses; the reserved `host.openshell.internal` alias uses the runtime-owned host address. +HTTP endpoints retain HTTP, and the provider allows only model discovery and Chat Completions beneath the recorded path. +A private destination, protected host port, or conflicting provider identity stops recovery before credentials are registered. +Authentication-returned replacement endpoints still require HTTPS. +If no owned native credential remains, resume onboarding and authenticate again before retrying the model change. + +The onboarding wizard selects the managed Hermes endpoint. +To configure a new custom endpoint, use [OpenAI-Compatible Endpoint](../custom-endpoints/set-up-openai-compatible-endpoint). + ## Model Choices The Hermes Provider catalog includes curated models such as these examples. diff --git a/docs/inference/use-nvidia-endpoints.mdx b/docs/inference/use-nvidia-endpoints.mdx index 908528ad3fa..2154c140be1 100644 --- a/docs/inference/use-nvidia-endpoints.mdx +++ b/docs/inference/use-nvidia-endpoints.mdx @@ -45,6 +45,8 @@ If the live catalog is unavailable or does not contain safe model IDs, the wizar Sandboxes created by an earlier release used the shared `inference.local` route and do not have the attached NVIDIA provider receipt. NemoClaw does not migrate these beta sandboxes automatically. +For a retained `nvidia-nim` record, `rebuild` stops before backup or deletion unless OpenShell reports a complete `0.0.x` version. +An unknown OpenShell version also blocks this legacy rebuild path. For these sandboxes, `inference set` refuses a same-provider NVIDIA model change with exit status `2`. @@ -54,6 +56,7 @@ Recreate the sandbox during onboarding to use the native provider path. ```bash $$nemoclaw onboard --fresh --name --recreate-sandbox ``` + ## Validation diff --git a/docs/inference/use-shared-gateway-routes.mdx b/docs/inference/use-shared-gateway-routes.mdx index 1c48b9fa8ab..ae9def0ae1c 100644 --- a/docs/inference/use-shared-gateway-routes.mdx +++ b/docs/inference/use-shared-gateway-routes.mdx @@ -10,12 +10,18 @@ content: type: "how_to" --- -These rules apply to providers that still use the shared route, including compatible endpoints without an attachment, Local NVIDIA NIM, and retained shared local selections. -NVIDIA Endpoints, fixed hosted providers, new hosted compatible selections outside the Personal tier, new Deep Agents Code, Hermes, and Pi qualified local selections, and new OpenClaw Ollama, standalone vLLM, and llama.cpp selections use sandbox-attached providers. +These rules apply to providers that still use the shared route, including retained custom shared routes and retained shared local selections. +They require an OpenShell `0.0.x` gateway. +OpenShell `0.1.5` removed `inference.local`, and `0.1.7` also lacks that route. +Do not use these procedures with either runtime. +NVIDIA Endpoints, fixed hosted providers, and qualifying custom hosted endpoints use sandbox-attached providers. +New Ollama, standalone vLLM, llama.cpp, and Local NVIDIA NIM selections use these attachments for Deep Agents Code, Hermes, OpenClaw, and Pi. +Deep Agents Code, Hermes, OpenClaw, and Pi also use attachments for qualifying host-local compatible endpoints. Their selected model or provider does not re-point the shared route. -Unchanged published Pi local selections retain their recorded shared route during resume and rebuild. -Fresh Pi local selections require a native provider attachment. +Unchanged published Pi local selections without native attachments retain their recorded shared route during resume and rebuild on OpenShell `0.0.x`. +These retained routes include Local NVIDIA NIM. +Fresh Pi selections for Ollama, standalone vLLM, llama.cpp, Local NVIDIA NIM, and qualifying host-local compatible endpoints require a native provider attachment. OpenShell exposes one shared inference route per gateway. NemoClaw records each sandbox's intended provider and model so compatible sandboxes can take turns using that route. @@ -44,10 +50,11 @@ $$nemoclaw inference set --provider --model ``` For a shared-route selection, when only the provider or model differs, the command warns immediately before it re-points the shared route. -An OpenClaw model switch on its recorded native local attachment keeps the shared route unchanged. The warning names every affected registered sandbox and its recorded provider and model. Those sandboxes use the new live route until `connect`, onboarding, or another `inference set` command re-points it. +An OpenClaw model switch on its recorded native local attachment keeps the shared route unchanged. + For the same provider name, custom endpoint, API-family, and credential environment-variable values must remain compatible. Incomplete routes and invalid gateway bindings also stop the command before route, agent config, or registry mutation. diff --git a/docs/inference/verify-inference-route.mdx b/docs/inference/verify-inference-route.mdx index 96b87330460..b3abc00f17d 100644 --- a/docs/inference/verify-inference-route.mdx +++ b/docs/inference/verify-inference-route.mdx @@ -4,7 +4,7 @@ title: "Verify the Sandbox Inference Route" sidebar-title: "Verify the Inference Route" description: "Verify that a NemoClaw sandbox can reach its configured model through its selected inference provider." -description-agent: "Verifies the in-sandbox inference path for native NVIDIA and fixed hosted providers and shared inference routes." +description-agent: "Verifies the in-sandbox inference path for native provider attachments and retained shared inference routes." keywords: ["verify inference provider", "nemoclaw inference route", "sandbox inference health"] content: type: "how_to" @@ -20,7 +20,10 @@ Read the selected inference path first. $$nemoclaw inference get ``` -For a selected sandbox with a valid native NVIDIA or fixed hosted attachment, the command returns that sandbox's recorded provider, model, and native endpoint. +For a selected sandbox with a valid native NVIDIA, fixed hosted, or native local attachment, the command returns the recorded provider, model, and native endpoint. + +It validates the recorded selection without querying the shared route or authenticating a model request. + For shared-route providers, the command reads the live shared gateway route. Confirm that the provider, model, and endpoint match the path you intended to configure. @@ -40,7 +43,7 @@ For NVIDIA Endpoints, the `Inference` row verifies the recorded sandbox attachme It then sends a native Chat Completions request to `https://integrate.api.nvidia.com/v1`. It does not probe `https://inference.local/v1/models` for this provider. -For fixed hosted providers, the `Inference` row verifies the recorded attachment and sends a model request to the native endpoint. +For fixed hosted and native local providers, the `Inference` row verifies the recorded attachment and sends a model request to the native endpoint. A healthy shared route cannot satisfy this check. For shared-route providers, the `Inference` row first checks the sandbox's `inference.local` path. @@ -50,6 +53,12 @@ When the live provider differs, `status` does not carry the recorded API family The row reports `healthy` only when the route returns a structurally valid result for the selected API family. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. Status diagnostics do not include the response body. + +If the check reports that the native inference credential handle is `missing` or `invalid`, it stopped before sending the model request. +OpenShell supplies this handle; do not copy a provider API key into the sandbox to replace it. +Check the recorded provider with `$$nemoclaw inference get` and retain the diagnostic for investigation. +The diagnostic does not include the credential value. + An ordinary HTTP `404` from `/v1/models` reports an unhealthy route because the model catalog is absent. A legacy shared-route `openrouter-api` selection uses NemoClaw's Chat Completions-only OpenRouter adapter. Its models route returns `404` by design, so `status` requires a successful inference request before it reports the route as healthy. @@ -69,11 +78,17 @@ Restore the configured endpoint or proxy, run `$$nemoclaw onboard --resume` to c ## Understand Local Provider Post-Ready Checks -For local Ollama, local vLLM, and local NVIDIA NIM on Docker GPU sandboxes using the compatibility route, onboarding performs an additional check after the sandbox becomes ready. +New Local NVIDIA NIM selections use the recorded sandbox-attached provider endpoint. +Final verification requires a successful native model request. +Live qualification on OpenShell `0.1.7` remains pending. +For older NIM sandboxes without a native attachment, follow the [OpenShell upgrade restrictions](../../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path). + +The retained Docker GPU compatibility route has an additional post-ready check for local Ollama, local vLLM, and Local NVIDIA NIM. +Local NVIDIA NIM uses the `vllm-local` provider identifier, so this compatibility check also applies to it. The check follows the selected inference route from inside the sandbox: - For a sandbox-attached local provider, it requests `/models` at the recorded native endpoint with the supervisor-issued scoped credential reference. -- For local NVIDIA NIM and retained shared routes, it requests `https://inference.local/v1/models`. +- For retained shared routes, it requests `https://inference.local/v1/models`. When the check runs, the request must succeed with a 2xx response before onboarding commits the GPU runtime and registers the sandbox. If the sandbox lacks curl, onboarding warns and skips this check. @@ -90,8 +105,14 @@ For those routes, continue to the final route check, then use the status command ## Understand Final Route Checks -For NVIDIA Endpoints and fixed hosted providers, onboarding verifies the sandbox-attached provider through its native endpoint. -It does not request the shared `inference.local` models route. +For NVIDIA Endpoints, fixed hosted providers, and native local providers including Local NVIDIA NIM, final verification requires a successful model request through the sandbox-attached provider. +This final check does not request the shared `inference.local` models route. + + +When onboarding resumes with a recorded registered-provider selection, final verification checks the provider identity, profile, and sandbox attachment. +It requires a successful model request to the recorded native endpoint. +It does not use the shared `inference.local` models route. + For shared-route providers, onboarding first requests `https://inference.local/v1/models` from inside the sandbox after policy and process recovery. @@ -111,6 +132,16 @@ Onboarding retries any other 404 within its startup budget, then reports the mis Confirm the provider and model configuration and restore the endpoint's `/v1/models` route. Run `$$nemoclaw onboard --resume` to verify the retained session again without rebuilding a healthy sandbox. +For native local inference smoke checks, NemoClaw verifies that direct access to each resolved host address is unavailable. +Each address must refuse the connection, report network or host unreachable, or return the expected OpenShell policy denial. +A transport result establishes direct-route unavailability but does not identify which control blocked access. +An HTTP `403` alone does not prove that OpenShell denied the forbidden request. +The denial must match the requested route and its provider policy. +Address-resolution failures, timeouts, and unexpected responses fail verification. +If a model request fails after bounded retries, the transport diagnostic reports the exception type and numeric error code when available. +It omits raw exception text to avoid exposing credentials. +Verification remains failed; preserve the diagnostic for investigation instead of bypassing the check. + Provider setup still performs its own model, credential, and endpoint validation before this final route check. Use the status command and a short agent request after onboarding to verify ongoing availability and model responses. diff --git a/docs/inference/view-active-inference-route.mdx b/docs/inference/view-active-inference-route.mdx index 648d7b6231b..8478c3a59f5 100644 --- a/docs/inference/view-active-inference-route.mdx +++ b/docs/inference/view-active-inference-route.mdx @@ -13,25 +13,18 @@ content: Use the NemoClaw CLI to read the selected inference provider and model. When the selected sandbox has a valid native provider attachment, the command reports the sandbox's recorded provider, model, and native endpoint without reading the shared gateway route. -For providers that still use the shared route, the command reads the live OpenShell inference route. +For providers that still use the shared route on OpenShell `0.0.x`, the command reads the live OpenShell inference route. For a compatible custom provider on a shared route, the command also reports its persisted endpoint URL when the registry provider and model match the live route. -Fixed native hosted providers include OpenAI, Anthropic, Gemini, OpenRouter, and Hermes Provider. +Native hosted providers include OpenAI, Anthropic, Gemini, OpenRouter, and Hermes Provider. +Qualifying custom hosted endpoints also report their sandbox-attached native provider. New OpenClaw selections for Ollama, standalone vLLM, and llama.cpp also report their sandbox-attached local provider. -New hosted compatible selections outside the Personal tier report their endpoint-bound attachment. -Local NVIDIA NIM, new Personal-tier hosted compatible selections, and retained shared routes keep shared-route readback. +Remaining custom shared selections keep shared-route readback on OpenShell `0.0.x`. - -Deep Agents Code normally reports a recorded native attachment for hosted compatible endpoints and qualified Ollama, vLLM, llama.cpp, and host-local compatible endpoints. -A fresh Personal-tier hosted compatible selection instead reports the shared `inference.local` route and has no native attachment. + +Deep Agents Code and Hermes also report a recorded native local attachment for qualified Ollama, vLLM, llama.cpp, and host-local compatible endpoints. The displayed endpoint is the host-reachable address recorded for that sandbox. -Legacy selections without an attachment and owned Local NVIDIA NIM retain shared-route readback. - - -Hermes also reports a recorded native attachment for hosted compatible endpoints outside the Personal tier and qualified Ollama, vLLM, llama.cpp, and host-local compatible endpoints. -Fresh Personal-tier hosted compatible selections keep shared-route readback. -The displayed endpoint is the host-reachable address recorded for that sandbox. -Legacy selections without an attachment and owned Local NVIDIA NIM retain shared-route readback. +Remaining custom shared selections retain shared-route readback on OpenShell `0.0.x`. An invalid recorded native attachment produces an error instead of falling back to the shared route. This command reports configuration state and does not authenticate a model request. diff --git a/docs/manage-sandboxes/add-mcp-server.mdx b/docs/manage-sandboxes/add-mcp-server.mdx index cc8520cb44f..fb2f5f45565 100644 --- a/docs/manage-sandboxes/add-mcp-server.mdx +++ b/docs/manage-sandboxes/add-mcp-server.mdx @@ -21,7 +21,9 @@ Deep Agents Code `mcp add` and `mcp restart` require native MCP capability v3. A legacy image stops with migration guidance before it changes a live provider, policy, or agent configuration. -Hermes stores MCP servers in its native `config.yaml`. Registration and inspection preserve the stable credential handle from the readiness check in the OpenShell resolver placeholder without comparing it with host-side desired state. +Hermes stores MCP servers in its native `config.yaml`. +Registration and inspection use the credential placeholder observed by the readiness check. +Ordinary static credentials use revisioned placeholders; refresh-managed credentials can use stable handles. Before an MCP add or restart mutation, NemoClaw requires the Hermes helper to report reconcile-finality capability version 1. An add or restart with an older helper stops with rebuild guidance and leaves the provider, policy, attachment, and native configuration unchanged. @@ -107,6 +109,9 @@ Before registration, configure an HTTPS endpoint that meets these requirements: If the endpoint certificate chains to a private CA, provide that CA with `NEMOCLAW_CORPORATE_CA_BUNDLE` before onboarding. Managed-image rebuilds preserve the recorded CA profile and do not import a newly exported bundle. +With OpenShell `0.1.7`, NemoClaw also supplies the validated CA to the gateway's network supervisors for destination TLS verification. +This trust applies across the gateway; changing it does not automatically update existing sandboxes. +Private-CA MCP runtime qualification remains pending; refer to [Supervisor Destination Trust](../../security/configure-corporate-ca-trust#understand-supervisor-destination-trust). For source validation and custom-image requirements, refer to [Configure Corporate CA Trust](../../security/configure-corporate-ca-trust). For a host-local MCP process, keep the process bound to loopback when practical. @@ -169,12 +174,12 @@ Do not reuse OpenShell Google Cloud compatibility names such as `GCP_PROJECT_ID` NemoClaw rejects them because OpenShell exposes those non-secret configuration names as child-process values. It also rejects `GCE_METADATA_HOST`, `GCE_METADATA_IP`, and `METADATA_SERVER_DETECTION`, which OpenShell rewrites for its metadata emulator. -NemoClaw pins the child-visible compatibility list to OpenShell `v0.0.116` commit `d1155aa70042d3e2ee49dbfa15346b108b7c1d92`. +NemoClaw pins the child-visible compatibility list to OpenShell `v0.1.7` commit `5d25211b25a8fc887e8d5c606d57d6ab22f1d76c`. Review this list with every OpenShell version change. OpenShell reserves credential names matching `v[0-9]+_[A-Za-z0-9_]+` for revisioned placeholders. NemoClaw rejects names such as `v10_GITHUB_TOKEN` because OpenShell skips them instead of attaching a credential resolver. -OpenShell `0.0.116` also reserves names matching `s<64-lowercase-hex>_[A-Za-z0-9_]+` for stable credential handles. +OpenShell `0.1.7` also reserves names matching `s<64-lowercase-hex>_[A-Za-z0-9_]+` for stable credential handles. NemoClaw rejects host subprocess controls such as `PATH`, proxy or TLS variables, and `OPENSHELL_*`, `GRPC_*`, `LC_*`, or `XDG_*` keys. Loader, shell, language, and agent runtime controls such as `LD_PRELOAD`, `BASH_ENV`, `NODE_OPTIONS`, `PYTHONHOME`, `NEMOCLAW_*`, and `OPENCLAW_*` are rejected as well. @@ -221,7 +226,7 @@ A named tunnel can stabilize the hostname without stabilizing its address set. Use stable private DNS and a routed private HTTPS endpoint when you operate the MCP server locally. Authenticated MCP rejects `host.openshell.internal`, `host.docker.internal`, and `host.containers.internal`. -OpenShell `v0.0.116` has a trusted-gateway branch for a narrow driver-provided topology, but it does not expose an attested driver gateway address that NemoClaw can pin for this flow. +NemoClaw's managed MCP flow cannot pin an attested driver gateway address for these aliases. Use a routed HTTPS endpoint on a stable private address instead. ## Understand the Generated Method Profile diff --git a/docs/manage-sandboxes/backup-restore.mdx b/docs/manage-sandboxes/backup-restore.mdx index 2f7dadcdfd3..94f3b0051d0 100644 --- a/docs/manage-sandboxes/backup-restore.mdx +++ b/docs/manage-sandboxes/backup-restore.mdx @@ -52,7 +52,11 @@ After restore, NemoClaw replaces these references with a current OpenShell provi Rebuild also [repairs an already-stripped route](https://github.com/NVIDIA/NemoClaw/pull/14153) when all three fields contain exactly `[STRIPPED_BY_MIGRATION]`. -Recovery requires a registered Hermes `nvidia-prod` route, matching endpoint and model metadata, and verified sandbox identity and provider binding. +Recovery requires a registered Hermes `nvidia-prod` route, matching endpoint and model metadata, and verified sandbox identity and native NVIDIA provider attachment. +The stripped value identifies damage; it does not authorize recovery. +NemoClaw verifies the source configuration digest before replacing only the three credential fields. +Other configuration must remain unchanged. +Ordinary configuration writes and readiness checks continue to reject stripped credentials. Partial damage, other providers, missing bindings, and ambiguous routes remain rejected. Before recovery, update NemoClaw and use its matching managed Hermes image. @@ -68,7 +72,7 @@ $$nemoclaw status --json Verify that status reports `provider: "nvidia-prod"` and `inferenceHealth` with both `ok` and `probed` set to `true`. Then confirm that Hermes answers an inference request and that your user settings and Discord configuration remain present. Retained Discord configuration does not establish successful Discord message delivery. -Recovery acquires a fresh OpenShell provider-bound handle; do not insert a raw API key or edit the stripped fields manually. +Recovery acquires a current OpenShell provider-bound credential reference; do not insert a raw API key or edit the stripped fields manually. If restore or validation fails, NemoClaw reports failure and retains the backup. Preserve that backup and inspect the reported cause before another mutation; repeating a failed rebuild is not a general recovery procedure. @@ -105,6 +109,18 @@ Backups created by older NemoClaw versions from a selective per-agent path inven `$$nemoclaw destroy` removes NemoClaw-owned managed state volumes and, for Deep Agents Code, clears the complete retained `/sandbox` root before asking OpenShell to delete the sandbox. Download files you need before destroy. To replace software while preserving the complete native state, use `$$nemoclaw rebuild`. + + +## Legacy Managed State Volumes + +Managed OpenClaw and Hermes state volumes require OpenShell attachment approval for the selected workspace before onboarding or recreation. +Missing, partial, conflicting, or workspace-mismatched approval labels stop the operation before sandbox deletion or changes to the existing volume. +NemoClaw preserves the volume without copying its data or changing its labels. +Keep the existing sandbox and volume available until an approved migration path is available. +Back up required state before using `destroy` or uninstall, which retain their documented data-removal behavior. + + + ## Related Topics - [Recover and Rebuild Sandboxes](../operate-sandboxes/recover-and-rebuild-sandboxes) diff --git a/docs/manage-sandboxes/gateway-lifecycle-control.mdx b/docs/manage-sandboxes/gateway-lifecycle-control.mdx index b472e7d2f06..6b752935e84 100644 --- a/docs/manage-sandboxes/gateway-lifecycle-control.mdx +++ b/docs/manage-sandboxes/gateway-lifecycle-control.mdx @@ -46,12 +46,11 @@ A healthy gateway is not restarted. If the OpenClaw gateway is stopped, `recover` reports it without relaunching it. -If the existing Hermes supervisor is waiting for authenticated gated recovery after a clean exit or exit status `79`, `recover` requests a supervised gateway relaunch before verification. -Otherwise, `recover` reports a stopped gateway without relaunching it. +If the Hermes gateway is stopped, `recover` reports it without relaunching it. Separately, `recover` validates and releases a NemoClaw cron restore gate left by an interrupted rebuild. -For other stopped gateways or a forced runtime reload, use `gateway restart`. +For a stopped gateway or a forced runtime reload, use `gateway restart`. If the native command cannot run, restart the sandbox through OpenShell. diff --git a/docs/manage-sandboxes/manage-mcp-servers.mdx b/docs/manage-sandboxes/manage-mcp-servers.mdx index bfbee1704c3..82929315c0a 100644 --- a/docs/manage-sandboxes/manage-mcp-servers.mdx +++ b/docs/manage-sandboxes/manage-mcp-servers.mdx @@ -121,7 +121,7 @@ Correct the reported precondition, runtime, connection, authentication, protocol Provider presence and metadata cannot prove that OpenShell rewrites the recorded resolver placeholder when a request leaves the sandbox. `mcp status ` requests a differential wire-level credential-resolution probe by default. -Before sending probe traffic, NemoClaw verifies exact generated policy, expected provider attachment, provider ID, `nemoclaw-mcp-v1` type, valid resource version, and exactly one matching credential key. If readiness does not match, it reports `ok: null` with `probe skipped` and sends no request. +Before sending probe traffic, NemoClaw verifies exact generated policy, expected provider attachment, provider ID, profile type derived from the recorded credential name, valid resource version, and exactly one matching credential key. If readiness does not match, it reports `ok: null` with `probe skipped` and sends no request. When readiness passes, the probe sends the same idempotent MCP `initialize` request twice from inside the sandbox through the adapter runtime. One request carries the real placeholder header, and the control request carries a deliberately unresolvable literal bearer. diff --git a/docs/manage-sandboxes/manage-messaging-channels.mdx b/docs/manage-sandboxes/manage-messaging-channels.mdx index 5f78ca82d5e..46d16f08b8c 100644 --- a/docs/manage-sandboxes/manage-messaging-channels.mdx +++ b/docs/manage-sandboxes/manage-messaging-channels.mdx @@ -37,6 +37,21 @@ For Google Chat, re-add the channel and paste the replacement service-account JS For detailed token rotation procedures, refer to [Credential Rotation](../../security/credential-rotation). +### Shared Messaging Credential Profile + +The shared messaging profile is `nemoclaw-messaging-v1`. +It declares `TELEGRAM_BOT_TOKEN`, `DISCORD_BOT_TOKEN`, `WECHAT_BOT_TOKEN`, `SLACK_BOT_TOKEN`, `SLACK_APP_TOKEN`, and `MSTEAMS_APP_PASSWORD` for the existing channel bindings. +Each declaration is optional in the profile because individual providers hold the credentials selected for their channel; creating a provider still requires its primary credential. +The profile grants no endpoints, binaries, files, or inference access. +Channel policy and provider-binding checks control where each credential can be used. +Channels with their own provider profiles, including Hermes Discord and Google Chat, retain those profiles. + +An older shared messaging provider can have the `nemoclaw-mcp-v1` type. +That type does not match the new shared messaging profile, and NemoClaw does not relabel the existing provider automatically. +If an operation reports that the named provider does not match the required endpointless credential binding, inspect that provider and its sandbox attachments. +To replace the binding for the intended sandbox, rerun `$$nemoclaw channels add ` with the channel's complete credentials. +The replacement remains subject to the attachment restrictions and failure recovery above. + ## Remove a Channel Remove the channel and clear its stored credentials: diff --git a/docs/manage-sandboxes/recover-rebuild-sandboxes.mdx b/docs/manage-sandboxes/recover-rebuild-sandboxes.mdx index 51a6bd4795e..941255dfb37 100644 --- a/docs/manage-sandboxes/recover-rebuild-sandboxes.mdx +++ b/docs/manage-sandboxes/recover-rebuild-sandboxes.mdx @@ -72,6 +72,24 @@ It returns success only after OpenShell reports the sandbox ready, the agent run If a check fails, the command exits nonzero, identifies the failure, and prints recovery guidance before you retry `start`. +### Stop Reports a Terminal Phase + +If `stop` reports `Completed` or `Error` followed by `Stopped was not confirmed`, NemoClaw exits nonzero immediately. +It requires OpenShell to confirm `Stopped` for the same sandbox before recording a successful intentional stop. +OpenShell can preserve a terminal process result when the sandbox's compute resource is missing, even after accepting the stop request. +The message alone does not prove that the compute resource is absent. + +Inspect the sandbox before another lifecycle operation: + +```bash +$$nemoclaw status +$$nemoclaw logs --tail 100 +``` + +Preserve the reported phase and error, then correct the reported OpenShell or provider condition. +If the container is missing, follow the [state-preservation limits for rebuild](#rebuild-while-preserving-state) before replacing it. +Do not edit the intentional-stop record or treat `Completed` or `Error` as confirmation of `Stopped`. + ## Recover the Agent Runtime @@ -107,6 +125,11 @@ $$nemoclaw gateway restart The restart command invokes OpenClaw's native lifecycle command through the selected OpenShell sandbox. NemoClaw requires `/readyz` to return HTTP `200` with JSON `{"ready":true}` before it checks or recovers host-side dashboard, messaging, and agent forwards. +If restart readiness fails, preserve the `OpenClaw restart readiness:` JSON line when reporting the failure. +It records the native restart exit status, total probe count, and the last 32 probe observations at most. +Each observation includes the command status, transport error category, HTTP status, JSON parsing result, and readiness decision when available. +The summary reports how many older observations were dropped and excludes response bodies, credentials, and configuration values. +These diagnostics use the existing probes and leave the restart failure unchanged. Refer to [`$$nemoclaw recover`](../../reference/commands#$$nemoclaw-name-recover) and [`$$nemoclaw gateway restart`](../../reference/commands#$$nemoclaw-name-gateway-restart) for details. @@ -120,8 +143,7 @@ $$nemoclaw recover ``` The command verifies the native gateway and re-establishes the dashboard port forward. -If the existing Hermes supervisor is waiting for authenticated gated recovery after a clean exit or exit status `79`, `recover` requests a supervised gateway relaunch before verification. -Otherwise, `recover` reports a stopped gateway without relaunching it; use `gateway restart` to request a native restart. +If the Hermes gateway is stopped, `recover` reports it without relaunching it; use `gateway restart` to request a native restart. If the host forward is already active, recovery accepts it only after OpenShell ownership is reconciled and the local endpoint is reachable. If a listener that NemoClaw cannot attribute to the sandbox's OpenShell forward holds the port, `recover` reports it and leaves it running. It then exits nonzero and does not restore the forward. @@ -209,20 +231,26 @@ Every configured route must serve a valid response to a bounded request for the Shared-route providers use `inference.local`. Their models probe must return HTTP 2xx, except that a retained legacy `openrouter-api` shared route can return HTTP 404 for `GET /v1/models` when the bounded model request succeeds. NVIDIA Endpoints, supported fixed hosted selections, and new OpenClaw Ollama, standalone vLLM, and llama.cpp selections verify their recorded sandbox-attached provider and run the bounded model request at its native endpoint without the shared-route models probe. -For Hermes, new Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use the same native attachment check. Owned Local NVIDIA NIM and retained legacy shared routes continue to use `inference.local`. -Retained OpenClaw local selections with a valid shared route continue to use `inference.local` for supported model switches. An older OpenClaw Ollama, standalone vLLM, or llama.cpp sandbox without a native attachment cannot resume or rebuild. Back it up and re-onboard it with `--fresh --name --recreate-sandbox`. Owned Local NVIDIA NIM sandboxes retain the shared route and can resume or rebuild without a native attachment. +For Hermes, new Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use the same native attachment check. +New Local NVIDIA NIM selections also use the native attachment check for both agents. +Retained legacy shared routes continue to use `inference.local`. +Retained OpenClaw local selections with a valid shared route continue to use `inference.local` for supported model switches. An older OpenClaw Ollama, standalone vLLM, or llama.cpp sandbox without a native attachment cannot resume or rebuild. Back it up and re-onboard it with `--fresh --name --recreate-sandbox`. New Local NVIDIA NIM sandboxes also use their recorded native attachment for resume and rebuild. Back up and recreate older shared-route NIM sandboxes before upgrading OpenShell. A failed required probe or model request rejects launch readiness. This check is stricter than the HTTP 200–499 reachability diagnostic used by ordinary `connect`. Every configured route must serve a valid response to a bounded request for the recorded model before NemoClaw accepts or publishes launch readiness. -For new Deep Agents Code standalone local selections, NemoClaw verifies the recorded sandbox-attached provider and requests the model through its qualified host-local endpoint. Older Deep Agents Code standalone local shared routes without an attachment stop before resume or rebuild; back up and recreate the sandbox with `nemo-deepagents onboard --fresh --name --recreate-sandbox`. Owned Local NVIDIA NIM retains its shared route and does not require a native attachment to resume or rebuild. For retained legacy and custom shared routes, the models probe must return HTTP 2xx, except that a legacy `openrouter-api` route can return HTTP 404 for `GET /v1/models` when the model request succeeds. NVIDIA Endpoints and supported fixed hosted selections instead verify the recorded native attachment and request the model through its native endpoint. +For new Deep Agents Code standalone local selections, NemoClaw verifies the recorded sandbox-attached provider and requests the model through its qualified host-local endpoint. Older Deep Agents Code standalone local shared routes without an attachment stop before resume or rebuild; back up and recreate the sandbox with `nemo-deepagents onboard --fresh --name --recreate-sandbox`. Local NVIDIA NIM also requires its recorded native attachment for resume and rebuild. For retained legacy and custom shared routes, the models probe must return HTTP 2xx, except that a legacy `openrouter-api` route can return HTTP 404 for `GET /v1/models` when the model request succeeds. NVIDIA Endpoints and supported fixed hosted selections instead verify the recorded native attachment and request the model through its native endpoint. A failed request rejects launch readiness and identifies the inference request as the failed check. For OpenClaw, `connect --probe-only` settles the existing allowlisted pairing flow before it publishes a credential-free pairing qualification with the lease. The readiness evidence binds the OpenClaw version and trusted registry and agent manifest configuration. Its pairing qualification binds the canonical CLI client, exact paired device identity, required operator role and scopes, owning OpenShell gateway, sandbox lifecycle identity, and fixed lease epoch. Before lease acceptance, `launch` makes a bounded, read-only observation of the current OpenClaw-owned pairing state through the owning OpenShell gateway. It skips the complete pairing approval pass only when the qualification still matches exactly and no relevant allowlisted request is pending. Any missing, unreadable, malformed, ambiguous, or changed observation runs the complete pairing approval pass. A relevant allowlisted pending request also runs that complete path. +If publication fails with `reason=pairing-observation-failed`, preserve the `pairing=` category from the `Readiness evidence:` line when reporting the failure. +The category distinguishes failures such as an observation deadline, changing state, or incompatible state without exposing pairing records or credentials. +The probe still exits nonzero because it could not publish readiness evidence. + Hermes and LangChain Deep Agents Code retain their existing session setup on the lease-accepted path. When those checks pass, it can skip duplicate recovery, readiness polling, and inference-route repair. @@ -275,6 +303,9 @@ The OpenShell CLI has no command that starts a gateway, and `openshell status` o The procedure requires an explicit provider attachment. Repeat the attachment after rebuilding if the sandbox still needs Tavily. +Rebuild requires the OpenSSH `sftp` client on the host. +If it is unavailable, install the OpenSSH client tools before running rebuild. + ```bash $$nemoclaw rebuild ``` @@ -300,9 +331,9 @@ When NemoClaw can identify the exact stopped Docker container and prove its immu When that provider-owned capture is not available, rebuild asks OpenShell to start the exact stopped sandbox for the duration of the pre-recreate state backup and asks OpenShell to return it to `Stopped` afterward, the same lifecycle path `$$nemoclaw backup-all` uses. If OpenShell cannot start the sandbox, or the retried backup still fails, rebuild aborts without recreating the sandbox; it reports an incomplete-backup notice only when a partial backup was retained. If the backup succeeds but OpenShell cannot return the sandbox to `Stopped`, rebuild also aborts and reports the failed transition. If neither path can establish stopped-source authority, repair the OpenShell lifecycle state or recover the data manually before retrying; NemoClaw does not fall back to a selective directory copy. - -On OpenShell 0.0.116, `start` cannot restart a sandbox in `Phase: Error`. - +OpenShell 0.1.7 can restart a sandbox in `Phase: Error` when it records a failed main process and its exit code. +It also permits retry after a provisioning timeout once compute cleanup completes. +Other `Error` states remain rejected; inspect the reported failure instead of assuming every error is restartable. If OpenShell still reports the sandbox in `Phase: Error` but its Docker-driver container is missing, rebuild cannot back up its live workspace and stops before replacement. Recover the workspace from @@ -335,6 +366,14 @@ Correct the reported configuration or route error, then rerun `$$nemoclaw ]` means verification could not establish release; `` names the error category, such as `timeout` or `transport`. +Neither classification identifies the listener's owner or the original failure's cause. +Preserve the sandbox registry entry and resolve the reported verification failure before retrying the operation. + ## Run Multiple Sandboxes diff --git a/docs/manage-sandboxes/runtime-controls.mdx b/docs/manage-sandboxes/runtime-controls.mdx index 8d8498196f2..1b25a466904 100644 --- a/docs/manage-sandboxes/runtime-controls.mdx +++ b/docs/manage-sandboxes/runtime-controls.mdx @@ -117,12 +117,16 @@ Valid changes made by Hermes or the sandbox user are authoritative. Startup and Use supported host config and inference commands for settings that NemoClaw owns so its registry stays aligned with the runtime. Those commands can report a registry mismatch for the operation they manage, but generic startup, restart, recovery, health, probe, and connect paths do not enforce registry equality. -Hermes host-side config writes run as a sealed transaction. -NemoClaw binds the write to the SHA-256 digest of the matching read, temporarily seals the mutable config paths, atomically installs fresh config inodes, refreshes the strict and compatibility hashes, and then restores the mutable paths. -The root-only mutation lock stays held through every host-side Hermes config write. - -If another host mutation is active, the command reports `Hermes config mutation is already in progress`. If another lifecycle request owns the supervisor, it reports `SUPERVISOR_BUSY`. Both errors are retryable. - +Hermes host-side `config set` writes run as the sandbox user through the sandbox's recorded OpenShell gateway. +NemoClaw binds the write to the SHA-256 digest of the matching read and holds the native configuration transaction lock while it writes, refreshes the integrity hashes, and reloads the Hermes gateway. +The transaction refuses changed source content, unsafe paths, and changes to MCP server definitions. +Use the host-side `mcp` commands to change those definitions. +If a failure occurs after a verified write, the transaction attempts to restore the previous configuration and hashes and reload the gateway; it refuses to overwrite a concurrent change during rollback. +A failed configuration transaction does not confirm either the requested configuration or a successful rollback. +Check gateway health with `$$nemoclaw status` and resolve the reported failure before retrying. +A retry reads the current configuration through the recorded runtime target before it attempts another write. + +Legacy root-owned lifecycle transactions can still report `Hermes config mutation is already in progress` or `SUPERVISOR_BUSY`. Let the active command finish, then retry instead of editing lock or seal files manually. diff --git a/docs/manage-sandboxes/update-sandboxes.mdx b/docs/manage-sandboxes/update-sandboxes.mdx index 8659cde9b6c..adcc92c28a8 100644 --- a/docs/manage-sandboxes/update-sandboxes.mdx +++ b/docs/manage-sandboxes/update-sandboxes.mdx @@ -43,7 +43,16 @@ The `langchain-deepagents-code` agent manifest declares the expected Deep Agents Run the maintained installer to update the host CLI and reconcile registered sandboxes. -The maintained OpenShell 0.0.116 pin limits a routed sandbox name to 19 characters and rejects consecutive hyphens. +When installing OpenShell `0.1.x` with registered sandboxes, the installed OpenShell version must already be within the target release's supported range. +This release requires `0.1.7`; any other installed OpenShell version stops the installer. +After sandbox-name validation passes, the installer lists the registered sandbox names and exits before backup or gateway retirement. +It leaves the existing gateway and sandboxes unchanged. +Keep the existing installation available until you can recreate those sandboxes through an approved migration path. +`NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE=1` and `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` do not bypass this stop. + + + +NemoClaw limits a routed sandbox name to 19 characters and rejects consecutive hyphens. A NemoClaw sandbox name must contain 1 to 19 characters. It must start with a lowercase letter, use lowercase letters, numbers, and single internal hyphens, and end with a letter or number. Before the OpenShell upgrade, the installer checks registered sandboxes for the selected gateway. @@ -52,7 +61,7 @@ If the installer finds incompatible names, it lists each name and exits before i - Prepare the NemoClaw CLI for the upgrade. - Start a backup of the registered sandboxes. - Retire the selected OpenShell gateway. -- Install OpenShell 0.0.116. +- Install OpenShell 0.1.7. - Recreate a sandbox. NemoClaw does not automatically truncate or change the sandbox name. @@ -85,74 +94,84 @@ If a support workflow asks you to pass the maintained tag explicitly, clear any curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_REF= NEMOCLAW_INSTALL_TAG=lkg bash ``` -During the automatic upgrade path, the installer prepares the current NemoClaw CLI without replacing OpenShell and requires a fresh backup of every registered sandbox. +The backup and reconciliation steps below run only after the sandbox-name and installed-version checks pass. +For this release, registered sandboxes must already use OpenShell `0.1.7`. +When sandboxes are registered, an out-of-range installation stops before CLI preparation, backup, forward retirement, or gateway retirement. +The installer does not automatically migrate registered `0.0.x` sandboxes to `0.1.7`. + +After those checks pass, the automatic path prepares the current NemoClaw CLI and requires a fresh backup of every registered sandbox. If any sandbox is skipped or fails, the installer exits before it changes the gateway. For a failed required pre-upgrade backup, NemoClaw removes the incomplete timestamped backup directory so it cannot appear as a recovery point. If that removal fails, the installer reports the retained path; preserve it and do not use it for recovery. -After backup, the installer requires any existing OpenShell executable it will use to report a version, then compares that version with the supported range declared by the prepared current source. -Before it retires an out-of-range gateway, the installer retires every applicable registered legacy forward through its exact sandbox, bind, port, process, gateway, and workspace identity, including Hermes dashboard, active messaging, and declared agent forwards. -If it cannot prove that retirement, it exits before retiring the gateway and preserves every completed backup. -It retires the running gateway before replacing an out-of-range OpenShell installation, keeps the gateway when the installed version is supported, and stops without retiring it when the installed version or supported range cannot be validated. -If OpenShell lifecycle commands cannot retire the gateway, the installer can use verified service or PID-file evidence on Linux and macOS. -For the default gateway on port `8080`, Linux first checks verified systemd service evidence for `nemoclaw-openshell-gateway.service`, while macOS first checks verified Homebrew gateway service evidence. -On either platform, the installer can then use a verified NemoClaw-managed gateway PID file for the configured gateway port. -If macOS cannot obtain a conclusive `lsof` listener observation for a stale PID, the installer stops retirement and preserves the PID file, OpenShell registration, and every sandbox backup. -After the service or PID-file check confirms the gateway process is stopped, the installer tries to remove the selected OpenShell registration. -If registration removal fails, the installer warns that onboarding will replace the stale registration. -If neither fallback can verify and stop the process, the installer stops after backup, preserves every sandbox backup, and reports the failed operation. -After the automatic path retires an out-of-range gateway, it forces installation of the OpenShell version pinned by the prepared source before sandbox recovery. -This mandatory installation applies to source and managed install modes and cannot remain deferred after gateway retirement. -After the installer replaces the packaged gateway, it waits up to 30 seconds for the gateway port to bind before sandbox recovery begins. -If the gateway does not bind, the installer stops and preserves the backups. -If the forced installation fails, the installer does not stage a gateway service or start sandbox recovery, preserves the backups, and reports how to retry. +The installer keeps the running gateway when the installed OpenShell version is supported. +Reusing OpenShell also requires the installed components to pass identity and capability checks. +Before replacing components, the installer requires a complete empty sandbox inventory from every registered OpenShell gateway. +Preserved NemoClaw sandbox records also stop component replacement, including when the installed CLI reports `0.1.x` or gateway registrations are missing. +When retained registry entries have valid identities, the refusal lists their sandbox names and the path to `sandboxes.json`. +Preserve that file and the named sandboxes while resolving the refusal; do not delete registry entries to bypass it. +If a gateway registration entry is a symbolic link or is not a directory, the component installer stops before downloading replacements. +The component installer also refuses a `sandboxes.json` or legacy gateway `metadata.json` file larger than 16 MiB. +Preserve an oversized file; do not truncate or delete it to bypass the check. +An empty runtime inventory does not prove that retained sandbox data was retired. +If that check fails, it stops before downloading replacement components. +Preserve the installed CLI, gateway registrations, NemoClaw sandbox records, and sandbox data while resolving the reported failure. +Gateway retirement and forced component replacement are not an automatic recovery path for registered out-of-range sandboxes in this release. +Keep those sandboxes and the existing installation available until an approved migration path exists. +If a separately prepared installation passes every guard but component installation fails, preserve its backups and follow the reported recovery guidance. + + +If an existing managed state volume lacks OpenShell attachment approval for the selected workspace, recreation stops before sandbox deletion or changes to that volume. +Keep the existing sandbox and volume available; refer to [Legacy Managed State Volumes](../state-and-backups/create-and-restore-snapshots#legacy-managed-state-volumes). + + ## Resume a Manually Prepared Upgrade -Use this path only with the current release CLI. -If the installer has not already reported that it prepared the current CLI, start the supported automatic flow first: + +`NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` does not enable migration of registered sandboxes from an out-of-range OpenShell version to `0.1.7`. +The installer checks the installed OpenShell version before it accepts the flag. +When the selected gateway has registered sandboxes, an out-of-range, missing, or unreadable OpenShell version stops the installer before it accepts prepared state. +Passing this version check does not permit component replacement while NemoClaw sandbox records remain. +Do not retire the legacy gateway, delete its registration, or remove sandbox records to bypass this check. +Keep the existing installation and sandbox data available until an approved migration path exists. + -```bash -curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE=1 bash -``` +Use the retry command below only for an eligible upgrade whose strict backup, forward retirement, and gateway retirement already completed. +Do not retire a gateway to make it eligible for this command. +Preserve the completed backups. -That flow prepares the current CLI before it replaces OpenShell. -If it completes, do not run the manual steps below. -Continue manually only when the automatic flow stops after preparing the CLI or a support workflow directs you to this recovery path. -Set the prepared-upgrade state only after strict backup succeeds, every applicable registered legacy forward is retired, and the gateway is retired. -Strict backup can report a confirmed stranded registry sandbox without creating a backup for it because both the selected gateway and the container provider prove that no sandbox runtime remains. -That record requires the cleanup or recreation action printed by `$$nemoclaw backup-all`, but it does not block a prepared upgrade after the strict command succeeds. -Run the backup and forward-retirement step with the current prepared NemoClaw CLI before you retire the gateway: +Before retrying, inspect the installed version: ```bash -selected_port=8080 -gateway_name=nemoclaw -NEMOCLAW_GATEWAY_PORT="$selected_port" NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS=1 $$nemoclaw backup-all --retire-legacy-forwards && \ - openshell gateway destroy -g "$gateway_name" +openshell --version ``` -For a non-default gateway, replace `8080` with its port and set `gateway_name` to `nemoclaw-`. -If an older OpenShell release rejects the named destroy command for the default `nemoclaw` gateway, run `openshell gateway destroy` instead. -Do not substitute `openshell gateway remove`: removal can unregister a gateway without stopping its process. -Set `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` only after the command pipeline exits successfully. -Confirm that the strict backup summary reports `0 failed, 0 skipped` and that the `Legacy dashboard forwards:` summary appears before gateway destruction. -A confirmed stranded record has no backup to reuse; complete the reported cleanup or recreation action during recovery. -If either command fails or cannot prove a retirement, resolve that failure while the completed backups remain intact. -Set `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` on the `bash` side of the install pipeline when you rerun the installer. +For registered sandboxes on the selected gateway, the installed version must be verifiable and within the target release's supported range. +This release requires `0.1.7`. +Component replacement still stops if any NemoClaw sandbox registry retains records. +All preparation steps must have completed successfully. +A successful manual backup alone does not establish those conditions. +This flag does not change runtime, GPU, or inference qualification requirements. + +Preserve the gateway port used during preparation. +Set `selected_port` to `8080` for the default gateway, or to the port of the selected gateway you prepared: ```bash +selected_port=8080 curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_GATEWAY_PORT="$selected_port" NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1 bash ``` -Set `NEMOCLAW_GATEWAY_PORT=8080` when you prepared the default gateway. -Omitting it selects port `8080` only when no automatic alternate-port marker exists, so an explicit value prevents a previously recorded alternate environment from receiving the prepared upgrade. -`NEMOCLAW_OPENSHELL_UPGRADE_PREPARED` asserts that you completed those steps, so the installer skips the repeated backup and gateway-retirement phase before it checks whether OpenShell is installed or whether its version is in range. -It reuses the latest backups, forces installation of the OpenShell version pinned by the prepared source, and starts recovery only after that installation succeeds. -If the installation fails, rerun the same install-pipeline command to preserve `NEMOCLAW_GATEWAY_PORT` and `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED`. +An explicit port prevents a recorded alternate-port environment from receiving the prepared upgrade. +After the version and registry checks pass, the installer skips repeated backup and gateway retirement. +It requests installation of the OpenShell version pinned by the prepared source. +The component installer must independently accept the retained state before recovery can reuse the latest backups. +If installation fails again, preserve the backups and follow the reported recovery guidance. +Do not clear the registry or replace the installed executable merely to bypass the version check. Outside these forced paths, an in-range source checkout preserves an existing OpenShell installation and installs OpenShell only when it is missing. ## Reconcile Registered Sandboxes -After the host CLI and OpenShell update, the installer runs `$$nemoclaw upgrade-sandboxes --auto` to reconcile the existing sandboxes. +Only after the installer passes the version, registry, and component checks and completes the host update does it run `$$nemoclaw upgrade-sandboxes --auto` to reconcile the existing sandboxes. During installer-driven recovery, every sandbox selected for recovery requires a validated latest backup. The backup's sandbox and agent identities must match the registry. diff --git a/docs/manage-sandboxes/workspace-files.mdx b/docs/manage-sandboxes/workspace-files.mdx index 79084e3b72f..94e8cc903c8 100644 --- a/docs/manage-sandboxes/workspace-files.mdx +++ b/docs/manage-sandboxes/workspace-files.mdx @@ -201,7 +201,6 @@ It does not continue with a partial backup. ### Deleted During Sandbox Destroy Running `$$nemoclaw destroy` deletes the sandbox and its persistent state volume. -OpenShell removes the native home and workspace with the sandbox. Back up your workspace files before running `$$nemoclaw destroy`. @@ -283,7 +282,7 @@ Deep Agents Code normally creates the `/sandbox/.deepagents/agent/` paths below | Path | Purpose | |---|---| -| `/sandbox/.deepagents/config.toml` | NemoClaw-generated model and provider configuration. New NVIDIA Endpoints, supported fixed hosted providers, hosted compatible endpoints outside the Personal tier, Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use sandbox-attached providers with scoped credential references. A fresh Personal-tier hosted compatible selection uses `inference.local` without a native attachment. Owned Local NVIDIA NIM, other compatible endpoints without an attachment, and retained shared routes also use `inference.local`. | +| `/sandbox/.deepagents/config.toml` | NemoClaw-generated model and provider configuration. New NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, Ollama, standalone vLLM, llama.cpp, and qualifying local compatible selections use native endpoints with scoped credential references. Remaining shared selections use `inference.local` only on OpenShell `0.0.x`. | | `/sandbox/.deepagents/.state/` | Deep Agents Code runtime state, including persisted session and MCP-related state. | | `/sandbox/.deepagents/agent/AGENTS.md` | Global memory file for the default Deep Agents Code agent, loaded at session start. | | `/sandbox/.deepagents/agent/memories/` | Topic-specific markdown memories that Deep Agents Code can read and update across sessions. | @@ -295,6 +294,14 @@ Deep Agents Code normally creates the `/sandbox/.deepagents/agent/` paths below | `/sandbox/.deepagents/.state/chatgpt-auth.json` | Upstream ChatGPT auth state. The managed launchers refuse to start when this file exists. | | `/sandbox/.deepagents/.env` | User-managed Deep Agents Code environment file. It persists as agent-written native-home state; do not place raw credentials here when an OpenShell credential binding is available. | + +Fresh installations without retained legacy state can install OpenShell `0.1.7` and use sandbox-attached native providers. +The installer refuses to migrate registered OpenShell `0.0.x` sandboxes, including shared `inference.local` routes, and preserves their state. +OpenShell `0.1.5` removed the shared route. +GPU compatibility and native local lifecycle qualification remain incomplete. +Follow the [OpenShell upgrade restrictions](../operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before changing an existing installation. + + Project-level Deep Agents files can also exist inside the working repository, such as `.deepagents/AGENTS.md`, `.deepagents/skills/`, and `.deepagents/.mcp.json`. Those files are ordinary project files. They persist when they live under your sandbox workspace because the entire native home/workspace is OpenShell-owned storage. diff --git a/docs/monitoring/monitor-sandbox-activity.mdx b/docs/monitoring/monitor-sandbox-activity.mdx index a438ab3f4c9..e5b57bfae2e 100644 --- a/docs/monitoring/monitor-sandbox-activity.mdx +++ b/docs/monitoring/monitor-sandbox-activity.mdx @@ -27,10 +27,16 @@ Run the status command to view sandbox state, gateway health, and the active inf $$nemoclaw status ``` -On a normal read with verified gateway and sandbox state, `$$nemoclaw status` checks an available selected provider path from inside the sandbox as the authoritative inference check. Shared providers use `https://inference.local/v1/models`. NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible endpoints outside the Personal tier, and qualified new local selections verify the recorded sandbox-attached OpenShell provider with an inference request to its native endpoint. A fresh Personal-tier hosted compatible selection verifies the shared route and has no native attachment. +On a normal read with verified gateway and sandbox state, `$$nemoclaw status` checks an available selected provider path from inside the sandbox as the authoritative inference check. +Shared providers use `https://inference.local/v1/models`. +NVIDIA Endpoints, supported fixed hosted providers, qualifying hosted compatible endpoints, and qualified local selections verify the recorded sandbox-attached OpenShell provider with an inference request to its native endpoint. For local Ollama and local vLLM routes, it also prints labeled host-side backend diagnostics that help identify which hop failed. -During ordinary lifecycle contention, status can return a read-only snapshot only when receipt and registry checks confirm that no conflicting lifecycle authority exists before and after the read. The snapshot shows `Inference: not probed (another lifecycle operation is in progress)` and `inferenceProbeSuppressed: true` in JSON. If a conflicting authority exists, cannot be checked, or appears during the read, status rejects the snapshot and reports lifecycle contention. A snapshot can still observe a shared route, but it does not verify inference health. Wait for the lifecycle operation to finish and rerun status before using the result as inference evidence. +During ordinary lifecycle contention, status can return a read-only snapshot only when receipt and registry checks confirm that no Portable lifecycle authority exists before and after the read. +The snapshot shows `Inference: not probed (another lifecycle operation is in progress)` and `inferenceProbeSuppressed: true` in JSON. +If Portable lifecycle authority exists, cannot be checked, or appears during the read, status rejects the snapshot and reports lifecycle contention. +A snapshot can still observe a shared route, but it does not verify inference health. +Wait for the lifecycle operation to finish and rerun status before using the result as inference evidence. Review these output fields. diff --git a/docs/reference/architecture.mdx b/docs/reference/architecture.mdx index 2b7ef2ac582..1aeecd0fe1f 100644 --- a/docs/reference/architecture.mdx +++ b/docs/reference/architecture.mdx @@ -29,8 +29,8 @@ graph LR USER(["👤 User"]):::user subgraph EXTERNAL["External Services"] - SHARED_INFERENCE["Shared-route inference providers
Other custom and local routes
Local NVIDIA NIM · Model Router"]:::external - NVIDIA_INFERENCE["Native inference providers
NVIDIA · fixed hosted providers
Deep Agents and Hermes local endpoints"]:::external + SHARED_INFERENCE["Retained shared-route providers
OpenShell 0.0.x selections"]:::external + NATIVE_INFERENCE["Native inference providers
Hosted and qualified local endpoints
Model Router · Local NVIDIA NIM"]:::external INTEGRATIONS["Integration APIs
MCP · GitHub · package indexes"]:::external INTERNET["Internet
PyPI · npm · GitHub · APIs"]:::external end @@ -67,7 +67,7 @@ graph LR AGENT -->|"Shared-route requests
inference.local · placeholder credentials"| GW GW -->|"Shared-route request with
credential injected"| SHARED_INFERENCE - AGENT -->|"Attached native provider
restricted request · placeholder credential"| NVIDIA_INFERENCE + AGENT -->|"Attached native provider
restricted request · placeholder credential"| NATIVE_INFERENCE INTEGRATIONS -->|"External APIs"| INTEG INTEG -->|"Policy-gated access"| AGENT @@ -89,7 +89,7 @@ When systemd returns the service identity, NemoClaw requires trusted package pat It then checks the effective binary's version against the blueprint range (`min_openshell_version` through `max_openshell_version`). This check prevents a supported user-local OpenShell install from hiding an out-of-range binary selected by the package unit's absolute `ExecStart`. NemoClaw rejects a known version below the minimum, above the maximum, or identified as a development build. -Managed installations require exact stable OpenShell 0.0.116; the development channel is not eligible. +Managed installations require exact stable OpenShell 0.1.7; the development channel is not eligible. An undetermined version or an unsupported version stops package-unit adoption. When the systemd identity query reports a recognized unavailable user manager, NemoClaw checks the standard user unit paths before standalone fallback. It checks `.wants`, `.requires`, and `.upholds` activation links for both gateway service names. @@ -175,14 +175,14 @@ graph TB end end - SHARED_INFER["Shared-route inference provider
Other custom and local routes
Local NVIDIA NIM · Model Router"]:::external - NVIDIA_INFER["Native inference providers
NVIDIA · fixed hosted providers
Deep Agents and Hermes local endpoints"]:::external + SHARED_INFER["Retained shared-route provider
OpenShell 0.0.x selections"]:::external + NATIVE_INFER["Native inference providers
Hosted and qualified local endpoints
Model Router · Local NVIDIA NIM"]:::external CLI -->|"openshell CLI
(orchestrates)"| GW GW -->|"creates/recreates
Docker-driver sandbox"| SANDBOX SANDBOX -->|"shared-route requests
inference.local · placeholder credentials"| GW GW -->|"shared-route egress
credential injected at L7 proxy"| SHARED_INFER - SANDBOX -->|"attached native provider
restricted egress · placeholder rewritten"| NVIDIA_INFER + SANDBOX -->|"attached native provider
restricted egress · placeholder rewritten"| NATIVE_INFER class HOST host class DOCKER docker @@ -199,12 +199,13 @@ Layering from top to bottom: | Docker daemon | Host service | Runs the Docker-driver sandbox container and, on affected Linux hosts, the optional gateway compatibility container. | | Sandbox container | Docker container | Runs the selected supported agent runtime and NemoClaw integration under Landlock + seccomp + netns. | | OpenShell L7 proxy | Gateway process | Intercepts shared-route agent egress and rewrites `Authorization` headers (Bearer/Bot) and URL-path segments to inject the real credential at the network boundary. | -| OpenShell attached native provider | Provider profile attached to the sandbox | Restricts NVIDIA, fixed hosted, and qualified Deep Agents and Hermes local provider egress by host, method, path, and binary, then rewrites the placeholder credential at the egress boundary. | +| OpenShell attached native provider | Provider profile attached to the sandbox | Restricts qualified hosted and local provider egress, including Model Router and local NVIDIA NIM, by host, method, path, and binary, then rewrites the placeholder credential at the egress boundary. | NemoClaw never gives the sandbox a raw provider key. At onboard time it registers credentials with OpenShell's provider and placeholder system. -Shared routes use the gateway L7 proxy to substitute the real value at egress. -NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible selections outside the Personal tier, and qualified local endpoints use sandbox-attached provider profiles. +Retained OpenShell 0.0.x shared routes use the gateway L7 proxy to substitute the real value at egress. +NVIDIA Endpoints, supported fixed hosted providers, and qualified custom and local endpoints use sandbox-attached provider profiles. +Model Router and local NVIDIA NIM use this native path. These profiles restrict requests and replace placeholder credentials at egress. The CLI helper `isInferenceRouteReady` (in `src/lib/onboard.ts`) is a host-side readiness check used by the resume flow to decide whether the active route already covers the chosen provider and model. It is not a runtime component. @@ -383,13 +384,16 @@ Inside the sandbox: ## Inference Routing Agent inference follows one of two OpenShell-managed paths. -Owned Local NVIDIA NIM, compatible endpoints without a native attachment, and retained shared routes use `inference.local` through the OpenShell gateway: +Retained shared custom endpoints, unchanged published Pi shared local selections, and retained legacy shared local routes use `inference.local` through an OpenShell `0.0.x` gateway: ```text Supported agent runtime (sandbox) ──▶ inference.local ──▶ OpenShell gateway ──▶ Provider endpoint ``` -NVIDIA Endpoints, supported fixed hosted providers, and new hosted compatible selections outside the Personal tier use restricted providers attached to the sandbox. Qualified new local selections also use this path where the agent supports them. Pi native routing remains gated by its activation status. For example, NVIDIA Endpoints uses this path: +NVIDIA Endpoints, supported fixed hosted providers, and new qualified custom and local selections use restricted providers attached to the sandbox. +The local selections include Ollama, standalone vLLM, llama.cpp, Model Router, and local NVIDIA NIM. +Pi native routing remains gated by its activation status. +For example, NVIDIA Endpoints uses this path: ```text Supported agent runtime (sandbox) ──▶ integrate.api.nvidia.com ──▶ NVIDIA Endpoints @@ -400,8 +404,9 @@ Hermes live qualification for the new local lifecycle remains pending. The NVIDIA request does not use the shared `inference.local` route. The attached OpenShell profile allows only the configured endpoint, methods, paths, and binaries, and it replaces the placeholder credential at egress. -When you select the Model Router provider, the OpenShell gateway routes to a host-side router process instead of a single upstream model. +When you select Model Router, the sandbox-attached provider permits requests to the configured host-side router endpoint. The router selects from the configured pool, then calls the upstream NVIDIA endpoint with the credential held outside the sandbox. +Local NVIDIA NIM follows the same native-provider path to its local service endpoint. Some model and provider combinations need agent-specific compatibility setup. NemoClaw keeps those declarations under `nemoclaw-blueprint/model-specific-setup//` so fixes for each supported agent can be tested and reviewed independently. diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 2bacc453b27..918ee333bb5 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -787,7 +787,9 @@ Recreation without an explicit flag or environment value preserves the recorded Choose the managed Deep Agents Code auto-approval capability during onboarding. A new sandbox defaults to `disabled`. Omitting the flag preserves the sandbox registry’s recorded capability, including a previously disabled capability, during resume or recreation. `thread-opt-in` permits explicit activation inside Deep Agents Code; onboarding does not activate an approval mode or run `dcode -y`. -NemoClaw rejects `thread-opt-in` for other agents and custom images before it starts onboarding. Pass `--dcode-auto-approval disabled` to clear a recorded capability when switching agents. +NemoClaw rejects `thread-opt-in` for other agents and custom images. +Pass `--dcode-auto-approval disabled` to clear a recorded capability when switching agents. +When flags, environment, or the saved session establish that incompatibility, NemoClaw stops before starting onboarding. Before sandbox registration, resume preserves the explicit choice saved in the onboarding session. An explicit capability change for an existing sandbox requires recreation through the existing onboarding lifecycle. See [Configure Thread Auto-Approval](../manage-sandboxes/operate-sandboxes/run-deep-agents-code#configure-thread-auto-approval) for activation, status, and revocation. @@ -856,21 +858,21 @@ The installer detects existing sandbox sessions before onboarding and prints a w curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_SINGLE_SESSION=1 bash ``` -When existing sandboxes were created with OpenShell earlier than `0.0.37`, the installer prompts before running the automatic gateway upgrade path. -For scripted installs, set `NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE=1` to allow the automatic path to prepare the current CLI without replacing OpenShell, back up every registered sandbox with the current state manifest, retire an installed gateway whose OpenShell version is outside the current release's supported range, install the supported OpenShell release, and recover the existing sandboxes. -The installer reads that supported range from the prepared current source and stops without retiring the gateway if the installed version is unknown or the range is missing or invalid. -When the installed OpenShell version is already supported, the installer keeps the running gateway through the host update. -On Linux and macOS, if installed OpenShell lifecycle commands cannot retire the gateway, the installer uses verified NemoClaw-managed service or PID-file evidence for the supported recovery path. -For the default gateway on port `8080`, Linux recovery first checks a verified active `nemoclaw-openshell-gateway.service`, while macOS recovery first checks a verified active Homebrew gateway service. -Both paths then check a verified NemoClaw-managed gateway PID file for any configured gateway port. -When a macOS PID file names a process that is no longer running, the installer removes the stale PID file only after `lsof` reports no listener and no diagnostic output. -If `lsof` is unavailable, reports permission or other diagnostics, or produces inconsistent listener results, the installer stops before retiring the gateway and preserves the PID file, OpenShell registration, and sandbox backups. -After either fallback confirms the gateway process is stopped, the installer tries to remove the selected OpenShell registration and warns if onboarding must replace a stale registration. -If neither fallback can verify and stop the process, the installer stops after backup with every sandbox backup preserved. +When installing OpenShell `0.1.x` with registered sandboxes, the installed OpenShell version must already be within the target release's supported range. +This release requires `0.1.7`; any other installed OpenShell version stops the installer. +After name validation, the installer reports the affected sandbox names. +It leaves the gateway and sandboxes unchanged before backup or gateway retirement; neither `NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE=1` nor `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` bypasses the stop. +Refer to [Update Sandboxes](../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before upgrading registered sandboxes. +The automatic backup and reconciliation path runs only after sandbox-name and installed-version checks pass. +For this release, registered sandboxes must already use OpenShell `0.1.7`. +The installer prepares the current CLI, requires fresh backups, and keeps the supported running gateway through the host update. If any registered sandbox cannot be backed up, the installer aborts before it changes the gateway. -After the automatic path retires an out-of-range gateway, it forces installation of the OpenShell version pinned by the prepared source before recovery. -This mandatory installation applies to source and managed install modes and cannot remain deferred after gateway retirement. -If the forced installation fails, the installer does not stage a gateway service or start recovery, preserves the backups, and tells you to rerun with `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1`. +When sandboxes are registered, an out-of-range installation stops before CLI preparation, backup, forward retirement, or gateway retirement. +The installer does not automatically migrate registered `0.0.x` sandboxes to `0.1.7`. +Neither the experimental-upgrade flag nor a completed backup authorizes that cutover. +Retained NemoClaw sandbox records block component replacement even with a `0.1.x` CLI or an empty runtime inventory. +If the installer reports retained records, stop and preserve those records, gateway registrations, sandbox data, and backups. +Use `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` only for previously prepared recovery that passes every guard, as described below. On macOS, if the verified Homebrew installation does not resolve executable OpenShell CLI and gateway binaries, the installer also stops before recovery and preserves the backups and prepared recovery state. It does not fall back to a stale user-local OpenShell binary. When the registry contains a pre-fingerprint OpenClaw or Hermes entry with no recorded custom-image evidence, an interactive install asks you to confirm that the listed sandbox used a NemoClaw-managed image. @@ -878,43 +880,15 @@ For a non-interactive install, set `NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE` to The confirmation permits those legacy entries to recover onto the current managed image, but it does not override recorded custom-image evidence. After confirmed recovery, the installer skips generic onboarding unless DGX Station reconciliation remains necessary. Required DGX Station reconciliation still passes host admission before onboarding. -For any registered-sandbox upgrade that you prepare manually, use the current release CLI. -If the installer has not already reported that it prepared the current CLI, start the supported automatic flow first: - -```bash -curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE=1 bash -``` - -That flow prepares the current CLI before it replaces OpenShell. -If it completes, do not run the manual steps below. -Continue manually only when the automatic flow stops after preparing the CLI or a support workflow directs you to this recovery path. -Use the prepared current CLI to require strict backup and retire all applicable registered legacy forwards before you retire the old gateway. -Strict backup can report a confirmed stranded registry sandbox without creating a backup for it because both the selected gateway and the container provider prove that no sandbox runtime remains. -That record requires the cleanup or recreation action printed by `$$nemoclaw backup-all`, but it does not block a prepared upgrade after the strict command succeeds: - -```bash -selected_port=8080 -gateway_name=nemoclaw -retire_legacy_forwards_flag=--retire-legacy-forwards -NEMOCLAW_GATEWAY_PORT="$selected_port" NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS=1 $$nemoclaw backup-all "$retire_legacy_forwards_flag" && \ - openshell gateway destroy -g "$gateway_name" -``` - -For a non-default gateway, replace `8080` with its port and set `gateway_name` to `nemoclaw-`. -If an older OpenShell release rejects the named destroy command for the default `nemoclaw` gateway, run `openshell gateway destroy` instead. -Do not substitute `openshell gateway remove`: removal can unregister a gateway without stopping its process. -Set `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` only after the command pipeline exits successfully. -Confirm that the strict backup summary reports `0 failed, 0 skipped` and that the `Legacy dashboard forwards:` summary appears before gateway destruction. -A confirmed stranded record has no backup to reuse; complete the reported cleanup or recreation action during recovery. -If backup, forward retirement, or gateway retirement fails, resolve the failure while the completed backups remain intact. -This environment variable asserts that strict backup, legacy-forward retirement, and gateway retirement are complete, so the installer skips that repeated preparation phase before it checks whether OpenShell is installed or whether its version is in range. -For a non-default gateway, preserve the selected port on the `bash` side of the install pipeline. - -```bash -curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_GATEWAY_PORT="$selected_port" NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1 bash -``` - -It reuses the latest backups, forces the pinned OpenShell installation, and starts recovery only after that installation succeeds. If the installation fails, rerun the same install-pipeline command to preserve `NEMOCLAW_GATEWAY_PORT` and `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED`. +Use `NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1` only after the current release CLI completes strict backup and forward retirement, and the selected gateway is retired. +Preparation can be completed manually or by the installer. +The installer checks the installed OpenShell version before it accepts this flag. +When the selected gateway has registered sandboxes, an out-of-range, missing, or unreadable OpenShell version stops the upgrade even when backups exist. +Passing the version check does not bypass the component installer's retained-record check. +Do not retire a legacy gateway, delete its registration, or remove sandbox records to bypass that stop. +Preserve the existing installation, sandbox data, and completed backups. +Follow [Resume a Manually Prepared Upgrade](../manage-sandboxes/operate-sandboxes/update-sandboxes#resume-a-manually-prepared-upgrade) for the version check and retry command. +This recovery flag does not change the runtime, GPU, or inference qualification requirements. The wizard prompts for a provider first, then collects the provider credential if needed. Supported non-experimental choices include NVIDIA Endpoints, OpenRouter, OpenAI, Anthropic, Google Gemini, and compatible OpenAI or Anthropic endpoints. Credentials are registered with the OpenShell gateway and never persisted to host disk. Refer to [Credential Storage](../security/credential-storage) for details on inspection, rotation, and migration from earlier releases. The legacy `$$nemoclaw setup` command is deprecated; use `$$nemoclaw onboard` instead. @@ -1240,6 +1214,9 @@ Resume retains the recorded digest and requires the same `--from-image` value, o Rebuild consumes the same digest without building an image. Cleanup retains the published image, and upgrades leave it pinned and publisher-managed. +For Hermes, onboarding applies the selected inference provider and model to the sandbox configuration after image setup. +It then restarts the Hermes gateway to load that configuration. + #### `--name ` @@ -1342,8 +1319,8 @@ During that lease, another `launch` still verifies these conditions: - The recorded inference selection matches its provider path, and every configured route serves a valid response to a bounded request for the recorded model. - NVIDIA Endpoints, supported fixed hosted selections, new hosted compatible selections outside the Personal tier, and qualified new local selections verify their recorded sandbox-attached provider and request the model at its native endpoint without a shared-route models probe. - Local NVIDIA NIM, fresh Personal-tier hosted compatible selections, custom endpoints without an attachment, and retained shared selections use `inference.local`. + NVIDIA Endpoints, supported fixed hosted selections, qualifying custom hosted endpoints, and new OpenClaw Ollama, standalone vLLM, and llama.cpp selections verify their recorded sandbox-attached provider and request the model at its native endpoint without a shared-route models probe. + Retained shared custom endpoints and retained shared local selections use `inference.local`. Their models probe must return HTTP 2xx, except that a retained legacy `openrouter-api` shared route can return HTTP 404 for `GET /v1/models` when the bounded model request succeeds. A failed model request rejects readiness before the agent starts. This check is stricter than the HTTP 200–499 reachability diagnostic used by ordinary `connect`. @@ -1352,8 +1329,8 @@ During that lease, another `launch` still verifies these conditions: - The recorded inference selection matches its provider path, and every configured route serves a valid response to a bounded request for the recorded model. - NVIDIA Endpoints, supported fixed hosted selections, new hosted compatible selections outside the Personal tier, and qualifying new local selections verify their recorded sandbox-attached provider and request the model at its native endpoint without a shared-route models probe. - Owned Local NVIDIA NIM, retained legacy routes, and other custom routes without an attachment use `inference.local`. + NVIDIA Endpoints, supported fixed hosted selections, qualifying custom hosted endpoints, and new Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections verify their recorded sandbox-attached provider and request the model at its native endpoint without a shared-route models probe. + Retained legacy and other custom shared routes use `inference.local`. Their models probe must return HTTP 2xx, except that a retained legacy `openrouter-api` shared route can return HTTP 404 for `GET /v1/models` when the bounded model request succeeds. A failed model request rejects readiness before the agent starts. @@ -1427,15 +1404,16 @@ A recognized broken shared-route result lets `connect` attempt DNS or route repa When the initial shared-route probe cannot return a trusted result, `connect` fails closed before health-driven repair and before opening SSH. It prints a bounded, redacted last-probe detail and points you to `$$nemoclaw doctor`. -For NVIDIA Endpoints, supported fixed hosted selections, new hosted compatible selections outside the Personal tier, and qualified new local selections with an attachment, `connect` does not inspect or change the unrelated shared route. +For NVIDIA Endpoints, supported fixed hosted selections, qualifying custom hosted endpoints, new Deep Agents Code and Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections, and new OpenClaw Ollama, standalone vLLM, and llama.cpp selections, `connect` does not inspect or change the unrelated shared route. It verifies the recorded sandbox-attached provider and sends a native inference request for the selected model. For a verified native NVIDIA attachment, HTTP `429`, `500`, `502`, `503`, or `504`, or a confirmed request timeout, lets interactive `connect` open a diagnostic shell with a degraded warning after gateway checks pass. The failed request leaves model availability unverified; retry after the upstream service recovers. `connect --probe-only` still requires a successful inference request before it can publish launch-readiness evidence. Other providers and terminal credential, route, policy, and malformed-response failures stop `connect` before SSH opens. -After route discovery succeeds, every configured Deep Agents Code route must also serve a valid response to a bounded request for the recorded model. -A failed model request or missing provider or model metadata stops `connect` before SSH opens. +After route discovery succeeds, Deep Agents Code sends a bounded request for the recorded model. +The native NVIDIA transient-failure exception above also applies to interactive Deep Agents Code connections. +Other failed model requests or missing provider or model metadata stop `connect` before SSH opens. Run `$$nemoclaw doctor` to diagnose a failed model request. If the sandbox is registered locally but missing from a healthy gateway, `connect` preserves the registry entry and points you to `rebuild --yes`, `onboard`, or `destroy` instead of deleting the metadata needed for recovery. @@ -1617,7 +1595,7 @@ curl -sN http://127.0.0.1:/v1/chat/completions \ -For Deep Agents sandboxes, `agent` forwards to the manifest-declared terminal command. Bare invocations run `dcode`, and `--help` runs `dcode --help`. Use `dcode -n` for explicit headless automation when you are already connected to the sandbox, or use `$$nemoclaw agent -n ""` from the host. Add `--json` to either form for one managed, versioned JSON envelope on stdout. The host wrapper forwards the flag to `dcode`. For the schema, status and exit behavior, and 1 MiB output limit, refer to [Run Deep Agents Code](/user-guide/deepagents/manage-sandboxes/operate-sandboxes/run-deep-agents-code). The host wrapper keeps `HOME=/sandbox`, the managed proxy environment, and the manifest-declared Deep Agents config path aligned with `connect`. Interactive `$$nemoclaw agent` launches the same terminal TUI as `dcode`. Headless `$$nemoclaw agent -n ""` uses the managed headless boundary, where non-shell tools can auto-run without the interactive approval UI. +For Deep Agents sandboxes, `agent` forwards to the manifest-declared terminal command. Bare invocations run `dcode`, and `--help` runs `dcode --help`. Use `dcode -n` for explicit headless automation when you are already connected to the sandbox, or use `$$nemoclaw agent -n ""` from the host. Add `--json` to either form for one managed, versioned JSON envelope on stdout. The host wrapper forwards the flag to `dcode`. For the schema, status and exit behavior, and 1 MiB output limit, refer to [Run Deep Agents Code](/user-guide/deepagents/manage-sandboxes/operate-sandboxes/run-deep-agents-code). The host wrapper keeps `HOME=/sandbox` and the manifest-declared Deep Agents config path aligned with `connect`; the managed launcher clears inherited proxy variables. Interactive `$$nemoclaw agent` launches the same terminal TUI as `dcode`. Headless `$$nemoclaw agent -n ""` uses the managed headless boundary, where non-shell tools can auto-run without the interactive approval UI. @@ -1672,6 +1650,14 @@ Use `$$nemoclaw inference set`, credential commands, and policy commands when th Write one value into the agent configuration in a sandbox. The command validates every HTTP and HTTPS URL in the value, including URLs nested inside JSON objects or arrays. It pins an HTTP host to the validated IP address. +The configuration reads and write use the sandbox's recorded OpenShell runtime target, including its gateway, workspace, and local TLS configuration. +A missing or invalid recorded target stops the operation before a configuration read or write. +If the target changes during validation, the command stops before writing; retry against the current registration. +The write runs as the sandbox user, checks that the source configuration has not changed, refreshes the integrity hashes, and reloads the Hermes gateway. +The transaction preserves MCP server definitions; use the host-side `mcp` commands to change them. +If the transaction fails, check `$$nemoclaw status` and resolve the reported failure before retrying because rollback might not have completed. +A retry reads the current configuration through the recorded runtime target. + ```bash $$nemoclaw my-assistant config set --key model.default --value openai:nvidia/nemotron $$nemoclaw my-assistant config set --key agent.max_turns --value 100 --restart @@ -1681,7 +1667,7 @@ $$nemoclaw my-assistant config set --key agent.max_turns --value 100 --restart | -------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `--key ` | Dotpath to update in the config. Required | | `--value ` | Value to write. The command parses a JSON value when it can, and otherwise writes the text as a string. Required | -| `--restart` | Restart the Hermes gateway after writing | +| `--restart` | Request an additional verified Hermes gateway restart after the transaction reload | | `--config-accept-new-path` | Write a dotpath that does not already exist in the config | The command treats a dotpath that does not already exist in the config as a possible typo. An interactive run asks for confirmation before writing the new dotpath. A run without a TTY, or a run with `NEMOCLAW_NON_INTERACTIVE=1`, refuses the write. Pass `--config-accept-new-path`, or set `NEMOCLAW_CONFIG_ACCEPT_NEW_PATH=1`, to write the new dotpath without the confirmation. If the confirmation reaches the end of input, for example when you press `Ctrl-D` or run the command from a harness that closes stdin, the command exits non-zero without writing and repeats the same guidance. @@ -1691,7 +1677,18 @@ The command refuses to write `gateway` or any dotpath under `gateway.`, which ho -For Hermes, the command validates the complete candidate configuration with the bundled Hermes schema before it writes configuration or integrity metadata. An incomplete structural object is rejected without changing the configuration or hashes. Private URLs remain rejected unless the existing Hermes configuration explicitly sets `security.allow_private_urls: true`. That opt-in allows private URLs for any Hermes configuration value. Public hostnames still receive DNS validation and pinning. Hermes can restart its gateway when it applies a configuration change. Use `--restart` when the command must request and verify that restart. `config set` changes `config.yaml` only. Hermes startup and rebuild can update managed `.env` keys and their integrity hashes without printing their values. +For Hermes, the command validates the complete candidate configuration with the bundled Hermes schema before it writes configuration or integrity metadata. +An incomplete structural object is rejected without changing the configuration or hashes. +Private URLs remain rejected unless the existing Hermes configuration explicitly sets `security.allow_private_urls: true`. +That opt-in allows private URLs for any Hermes configuration value. +Public hostnames still receive DNS validation and pinning. +The configuration transaction reloads the Hermes gateway even without `--restart`. +Pass `--restart` to request an additional explicit gateway restart after the write. +`config set` changes `config.yaml` only. +Hermes startup and rebuild can update managed `.env` keys and their integrity hashes without printing their values. + +If the additional `--restart` fails after the transaction verified its reload, the command reports that configuration was applied and the additional restart was not confirmed. +Run `$$nemoclaw gateway restart` to retry that restart. @@ -1733,8 +1730,7 @@ If the OpenClaw gateway is stopped, `recover` reports it without relaunching it. -If the existing Hermes supervisor is waiting for authenticated gated recovery after a clean exit or exit status `79`, `recover` requests a supervised gateway relaunch before verification. -Otherwise, `recover` reports a stopped gateway without relaunching it. +If the Hermes gateway is stopped, `recover` reports it without relaunching it. @@ -1820,7 +1816,12 @@ Hermes can also report secret-boundary refusal when the native command reports a -For Hermes, `HERMES_RUNTIME_PREPARATION_FAILED stage=` identifies a supported input that still failed after five bounded preparation attempts. Correct the named input through its supported host-side configuration or credential flow, then run `$$nemoclaw stop` followed by `$$nemoclaw start`. If startup layout repair reports the maximum log-entry count or repair depth, archive or remove old retained logs from a trusted host-side recovery environment, then stop and start the sandbox. For symbolic links, hard links, unsafe entry types, or paths that changed during repair, restore reviewed files from an independent trusted backup into a recreated sandbox or recreate from host-side onboarding configuration. Rebuild for an integrity mismatch or when the input or sandbox state cannot be repaired safely. +For Hermes, `HERMES_RUNTIME_PREPARATION_FAILED stage=` identifies a supported input that still failed after five bounded preparation attempts. +Correct the named input through its supported host-side configuration or credential flow, then run `$$nemoclaw stop` followed by `$$nemoclaw start`. +If startup layout repair reports the maximum log-entry count or repair depth, archive or remove old retained logs from a trusted host-side recovery environment, then stop and start the sandbox. +For symbolic links, hard links, unsafe entry types, or paths that changed during repair, restore reviewed files from an independent trusted backup into a recreated sandbox or recreate from host-side onboarding configuration. +For `Hermes integrity metadata is not ready`, follow the backup and trusted recovery instructions in [Explain Network Policy to Agents](../network-policy/explain-network-policy-to-agents). +For other failures, rebuild when the input or sandbox state cannot be repaired safely. @@ -1884,7 +1885,21 @@ If the gateway remains stopped, follow the [native gateway restart procedure](.. When that deadline expires, `start` keeps the existing container, exits non-zero, and prints the `NEMOCLAW_CONNECT_TIMEOUT` value to use on the next run. -After runtime verification and any required host-forward checks pass, `start` sends one inference request through the sandbox's selected provider path using its recorded provider and model. NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible selections outside the Personal tier, and qualified new local selections use native endpoints through sandbox-attached OpenShell providers. Owned Local NVIDIA NIM, fresh Personal-tier hosted compatible selections, custom endpoints without an attachment, and retained shared routes use `inference.local`. After an OpenClaw sandbox and gateway become Ready, a verified native NVIDIA attachment can start with degraded inference when the request returns HTTP `429`, `500`, `502`, `503`, or `504`, or a confirmed timeout. The command clears the stop record and exits `0`, but the failed request leaves model availability unverified. A terminal credential, route, policy, or malformed-response failure still exits non-zero. Other providers also exit non-zero when the inference request fails. For terminal failures, the command prints the probe result, including any HTTP status, and points you to the sandbox doctor command. Ordinary providers request 16 reply tokens; Gemini requests 256 reply tokens so reasoning has room before visible content. The request allows up to 90 seconds, and the host wrapper allows 95 seconds for bounded process startup and cleanup. Each run consumes provider tokens on a hosted path. When the sandbox records no provider or no model, `start` skips the request and exits `0`. For a shared gateway route, `doctor` still classifies an HTTP `401` or `403` response as reachable. The native NVIDIA doctor probe instead requires a 2xx response from the sandbox-attached provider. Correct the provider credential when `start` reports either status. +After runtime verification and any required host-forward checks pass, `start` sends one inference request through the sandbox's selected provider path using its recorded provider and model. +NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, and new Deep Agents Code and Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections, plus new OpenClaw Ollama, standalone vLLM, and llama.cpp selections, use native endpoints through sandbox-attached OpenShell providers. +Retained shared custom endpoints, unchanged published Pi shared local selections, and retained shared local selections use `inference.local` on OpenShell `0.0.x`. +After an OpenClaw sandbox and gateway become Ready, a verified native NVIDIA attachment can start with degraded inference when the request returns HTTP `429`, `500`, `502`, `503`, or `504`, or a confirmed timeout. +The command clears the stop record and exits `0`, but the failed request leaves model availability unverified. +A terminal credential, route, policy, or malformed-response failure still exits non-zero. +Other providers also exit non-zero when the inference request fails. +For terminal failures, the command prints the probe result, including any HTTP status, and points you to the sandbox doctor command. +Ordinary providers request 16 reply tokens; Gemini requests 256 reply tokens so reasoning has room before visible content. +The request allows up to 90 seconds, and the host wrapper allows 95 seconds for bounded process startup and cleanup. +Each run consumes provider tokens on a hosted path. +When the sandbox records no provider or no model, `start` skips the request and exits `0`. +For a shared gateway route, `doctor` still classifies an HTTP `401` or `403` response as reachable. +The native NVIDIA doctor probe instead requires a 2xx response from the sandbox-attached provider. +Correct the provider credential when `start` reports either status. ### `$$nemoclaw status` @@ -1954,7 +1969,11 @@ $$nemoclaw my-assistant status --json $$nemoclaw sandbox status my-assistant --json ``` -On a normal status read, the command probes the selected provider path from inside the sandbox, and when that probe reports the path reachable it sends an inference request over the same path. NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible selections outside the Personal tier, and qualified new local selections use native endpoints through sandbox-attached OpenShell providers. Owned Local NVIDIA NIM, fresh Personal-tier hosted compatible selections, custom endpoints without an attachment, and retained shared routes use `inference.local`. The inference request verifies the selected provider path, and both checks exercise the path that agent traffic uses. For provider-path results, the main `Inference` line uses these states: +On a normal status read, the command probes the selected provider path from inside the sandbox, and when that probe reports the path reachable it sends an inference request over the same path. +NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, and new Deep Agents Code and Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections, plus new OpenClaw Ollama, standalone vLLM, and llama.cpp selections, use native endpoints through sandbox-attached OpenShell providers. +Retained shared custom endpoints, unchanged published Pi shared local selections, and retained shared local selections use `inference.local` on OpenShell `0.0.x`. +The inference request verifies the selected provider path, and both checks exercise the path that agent traffic uses. +For provider-path results, the main `Inference` line uses these states: | State | Meaning | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -2126,7 +2145,7 @@ The `Registered gateway binding` result depends on the sandbox registry: - For a registered sandbox with an invalid binding, the check fails, and `doctor` does not select, probe, or recover a gateway from that binding. - For an unregistered sandbox name, `doctor` uses the gateway selected by `NEMOCLAW_GATEWAY_PORT` for its other gateway checks and omits the `Registered gateway binding` check. -For inference health, `doctor` treats the selected provider path inside the sandbox as authoritative. Providers that still use the shared route probe the shared gateway route at `https://inference.local/v1/models`; responses from `200` through `499`, including `401` and `403`, pass that reachability check. NVIDIA Endpoints instead probes its sandbox-attached provider at `https://integrate.api.nvidia.com/v1/models` and requires a 2xx response. For the shared route, HTTP `500` through `599`, interim `100` through `199`, transport failures with status `000`, invalid status values, and an unavailable authoritative probe fail the check. For native NVIDIA, any non-2xx response or unavailable probe fails it. Fixed hosted providers, new hosted compatible selections outside the Personal tier, and qualified new local selections verify their recorded attachment and make an inference request to their native endpoint. +For inference health, `doctor` treats the selected provider path inside the sandbox as authoritative. Providers that still use the shared route probe the shared gateway route at `https://inference.local/v1/models`; responses from `200` through `499`, including `401` and `403`, pass that reachability check. NVIDIA Endpoints instead probes its sandbox-attached provider at `https://integrate.api.nvidia.com/v1/models` and requires a 2xx response. For the shared route, HTTP `500` through `599`, interim `100` through `199`, transport failures with status `000`, invalid status values, and an unavailable authoritative probe fail the check. For native NVIDIA, any non-2xx response or unavailable probe fails it. Fixed hosted providers, qualifying custom hosted endpoints, and qualified Deep Agents Code, Hermes, and OpenClaw local selections verify their recorded attachment and make an inference request to their native endpoint. A healthy shared route cannot satisfy that check. Host-side upstream probes use the same authenticated model-invocation checks as status and remain diagnostic only, so their failure does not fail `doctor` when the authoritative in-sandbox path is reachable. For gateway runtimes, `doctor` also reports an informational `Serving process: not checked` result because its fresh sandbox probes do not attest the long-running gateway process. This result does not fail the readiness check. Terminal runtimes omit it because they have no long-running gateway process. The `Inference` `Route` check warns when either the provider or model is unknown. After the gateway is healthy, run `$$nemoclaw status` to refresh the route information. Warnings do not make the command fail. Failed checks, including a failed or unavailable authoritative inference route, exit non-zero so scripts can use `doctor` as a readiness gate. Use `--json` for machine-readable output. For a `compatible-endpoint` route that uses `openai-completions`, the JSON report includes an informational `Inference` check labeled `Reasoning effort`. The check reports `low`, `medium`, `high`, or `endpoint-default` and never includes credentials. Because the check has `info` status, it does not change the command's exit status. @@ -2348,7 +2367,7 @@ The sandbox must be running for `nemohermes my-assistant gateway-token --quiet` -`gateway-token` is not applicable to Deep Agents sandboxes because there is no OpenClaw gateway token. NVIDIA Endpoints, supported fixed hosted providers, and new Ollama, standalone vLLM, llama.cpp, hosted compatible outside the Personal tier, and qualifying host-local compatible selections use native endpoints through sandbox-attached OpenShell providers. Owned Local NVIDIA NIM, retained legacy routes, and other custom routes without an attachment use `inference.local`. +`gateway-token` is not applicable to Deep Agents sandboxes because there is no OpenClaw gateway token. NVIDIA Endpoints, supported fixed hosted providers, and new Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use native endpoints through sandbox-attached OpenShell providers. Retained legacy and other custom shared routes use `inference.local`. @@ -2404,19 +2423,27 @@ It preserves runtime resources and shared providers; cleanup on the original run A conflicting active session for the same name, sandbox-scoped providers, or additional resource ownership still blocks this operation. The command prints the private archive path and keeps the archive after recovery. -Before changing a Docker-backed sandbox, NemoClaw inspects every container with the requested `openshell.ai/sandbox-name` label. The command continues when Docker returns no matching containers. For one matching container, the command continues only when all these labels have the required values: +Before changing a Docker-backed sandbox, NemoClaw inspects every container with the requested `openshell.ai/sandbox-name` label. +The command continues when Docker returns no matching containers. +A matching workload container must have these labels: - `openshell.ai/managed-by=openshell` - A nonempty `openshell.ai/sandbox-workspace` - A nonempty `openshell.ai/sandbox-id` -In an ordinary destroy flow, if the initial inspection cannot complete, more than one container matches, a matching container has conflicting or incomplete labels, or Docker returns malformed identity data, `destroy` exits before changing sandbox resources. -Retained-sandbox recovery accepts multiple managed containers only when every immutable sandbox ID matches the retained recovery fingerprint. +For OpenShell `0.1.7`, the workload also requires `openshell.ai/isolation-role=sandbox` and a nonempty `openshell.ai/sandbox-namespace`. +NemoClaw accepts that workload alone or a pair containing that workload and one OpenShell-managed container with `openshell.ai/isolation-role=supervisor`. +The pair must have distinct container IDs, the same sandbox ID and namespace, and no supervisor workspace label. +Any retained recovery fingerprint must match the sandbox ID. + +For older containers without isolation-role labels, an ordinary destroy requires one complete managed container identity. +Retained-sandbox recovery can accept multiple such containers only when they share one workspace and every sandbox ID matches the retained recovery fingerprint. +If the initial inspection fails, returns malformed identity data, or finds conflicting, incomplete, or unexpected containers, `destroy` exits before changing sandbox resources. The identity checks still apply with `--force`, `--yes`, or `NEMOCLAW_NON_INTERACTIVE=1`; those controls authorize confirmation but do not authorize an unproven container identity. -NemoClaw rechecks the identity after read-only preflight, before provider cleanup, and synchronously at the sandbox-deletion boundary. +NemoClaw rechecks container IDs, ownership labels, sandbox IDs, workspaces, namespaces, and isolation roles after read-only preflight, before provider cleanup, and at the sandbox-deletion boundary. If a later recheck detects drift or fails, `destroy` refuses sandbox deletion, preserves local ownership state, and reports any earlier cleanup already performed. -If OpenShell reports the sandbox absent after preflight captured one matching Docker container, `destroy` rechecks and removes only that container ID. -If Docker reports another OpenShell-managed container, removal fails, or NemoClaw cannot confirm removal, `destroy` exits nonzero and preserves the registry entry. +If OpenShell reports the sandbox absent, `destroy` rechecks and removes only residual container IDs captured by preflight. +If Docker reports an unexpected container, removal fails, or NemoClaw cannot confirm removal, `destroy` exits nonzero and preserves the registry entry. Correct the reported Docker state, then rerun `destroy`. If Docker cannot complete the inspection, correct the reported Docker error before you rerun `destroy`. For common recovery steps, refer to [Docker is not running](troubleshooting#docker-is-not-running) and [Docker permission denied on Linux](troubleshooting#docker-permission-denied-on-linux). @@ -2429,7 +2456,7 @@ If `destroy` reports conflicting, incomplete, or malformed identity data, inspec ```bash docker ps -a --no-trunc \ --filter "label=openshell.ai/sandbox-name=my-assistant" \ - --format 'table {{.ID}}\t{{.Label "openshell.ai/managed-by"}}\t{{.Label "openshell.ai/sandbox-workspace"}}\t{{.Label "openshell.ai/sandbox-id"}}' + --format 'table {{.ID}}\t{{.Label "openshell.ai/managed-by"}}\t{{.Label "openshell.ai/sandbox-workspace"}}\t{{.Label "openshell.ai/sandbox-id"}}\t{{.Label "openshell.ai/sandbox-namespace"}}\t{{.Label "openshell.ai/isolation-role"}}' ``` The labels show what each container claims. They do not prove container ownership. @@ -2933,7 +2960,7 @@ $$nemoclaw my-assistant mcp list [--json] ### `$$nemoclaw mcp add` -Add an MCP Streamable HTTP server to a sandbox. Pass `--url` for the MCP endpoint and the required single `--env KEY` bearer credential for the sandbox-side MCP client. Pass repeatable `--deny-tool ` options to block matching `tools/call` requests at the OpenShell proxy. The generated deny rules match tool names only and do not inspect arguments. Pass a repeatable `--trusted-private-host ` option to admit an exact RFC1918, CGNAT, or IPv6 unique local destination for the current command. The declaration must equal the normalized host from `--url`. For managed MCP, use a DNS hostname for an IPv6 unique local address because NemoClaw has not qualified direct IPv6-literal MCP URLs. You can supply exact hosts through `NEMOCLAW_TRUSTED_PRIVATE_HOSTS` instead, and NemoClaw combines the variable with command options. NemoClaw records the resulting exact trust intent and address pins, so restart, rebuild, and restore do not depend on the ambient environment. NemoClaw registers that credential in an OpenShell provider, installs a generated OpenShell `protocol: mcp` policy for the target endpoint, attaches the provider to the running sandbox, and writes only an OpenShell resolver placeholder for the recorded key into the agent configuration. Inline `--env KEY=VALUE` is rejected because it would expose the value in NemoClaw process arguments. Load the variable from a secret manager or masked prompt, export it without recording the value in shell history, and pass only `--env KEY`. All endpoints must use HTTPS. The full URL and path are persisted and displayed, so URLs cannot contain userinfo, query strings, fragments, known secret-shaped path material, percent-escaped or glob-style paths, or port zero. Server names must start with a letter and contain at most 64 letters, digits, hyphens, or underscores, and endpoint hostnames must use canonical lowercase DNS labels. NemoClaw rejects invalid names and endpoints before it writes lifecycle state or changes OpenShell resources. NemoClaw generates a narrow `protocol: mcp` policy for the destination, literal path, adapter binaries, pinned addresses, explicit MCP methods, and a 131,072-byte request-body limit. OpenShell `0.0.116` evaluates that policy before replacing the attached provider placeholder in the allowed request header. NemoClaw imports the endpointless `nemoclaw-mcp-v1` profile and binds the dedicated provider to that endpoint with `credential_binding.provider`. OpenShell withholds the credential before the binding is active and outside the bound host, port, and path. The sandbox client connects directly through OpenShell's existing egress path, and NemoClaw does not run a host-side MCP data-plane bridge, proxy, relay, or listener. After the add commits, NemoClaw freshly verifies the exact generated policy, expected provider attachment, recorded provider ID, `nemoclaw-mcp-v1` type, valid resource version, and exactly one credential key matching the recorded key. If those readiness checks pass, it sends a differential pair of wire-level MCP `initialize` requests from inside the sandbox — one with the placeholder header and one with an unresolvable control bearer — to verify that OpenShell resolves the credential on egress; otherwise it reports an inconclusive `probe skipped` result and sends no request. Neither outcome fails the committed add, and `--no-probe` skips this check. For full setup details, see [Add an MCP Server](../manage-sandboxes/mcp-servers/add-an-mcp-server). +Add an MCP Streamable HTTP server to a sandbox. Pass `--url` for the MCP endpoint and the required single `--env KEY` bearer credential for the sandbox-side MCP client. Pass repeatable `--deny-tool ` options to block matching `tools/call` requests at the OpenShell proxy. The generated deny rules match tool names only and do not inspect arguments. Pass a repeatable `--trusted-private-host ` option to admit an exact RFC1918, CGNAT, or IPv6 unique local destination for the current command. The declaration must equal the normalized host from `--url`. For managed MCP, use a DNS hostname for an IPv6 unique local address because NemoClaw has not qualified direct IPv6-literal MCP URLs. You can supply exact hosts through `NEMOCLAW_TRUSTED_PRIVATE_HOSTS` instead, and NemoClaw combines the variable with command options. NemoClaw records the resulting exact trust intent and address pins, so restart, rebuild, and restore do not depend on the ambient environment. NemoClaw registers that credential in an OpenShell provider, installs a generated OpenShell `protocol: mcp` policy for the target endpoint, attaches the provider to the running sandbox, and writes only an OpenShell resolver placeholder for the recorded key into the agent configuration. Inline `--env KEY=VALUE` is rejected because it would expose the value in NemoClaw process arguments. Load the variable from a secret manager or masked prompt, export it without recording the value in shell history, and pass only `--env KEY`. All endpoints must use HTTPS. The full URL and path are persisted and displayed, so URLs cannot contain userinfo, query strings, fragments, known secret-shaped path material, percent-escaped or glob-style paths, or port zero. Server names must start with a letter and contain at most 64 letters, digits, hyphens, or underscores, and endpoint hostnames must use canonical lowercase DNS labels. NemoClaw rejects invalid names and endpoints before it writes lifecycle state or changes OpenShell resources. NemoClaw generates a narrow `protocol: mcp` policy for the destination, literal path, adapter binaries, pinned addresses, explicit MCP methods, and a 131,072-byte request-body limit. OpenShell 0.1.7 evaluates that policy before replacing the attached provider placeholder in the allowed request header. NemoClaw imports an endpointless profile that declares the selected credential name and binds the dedicated provider to that endpoint with `credential_binding.provider`. OpenShell withholds the credential before the binding is active and outside the bound host, port, and path. The sandbox client connects directly through OpenShell's existing egress path, and NemoClaw does not run a host-side MCP data-plane bridge, proxy, relay, or listener. After the add commits, NemoClaw freshly verifies the exact generated policy, expected provider attachment, recorded provider ID, profile type derived from the recorded credential name, valid resource version, and exactly one credential key matching the recorded key. If those readiness checks pass, it sends a differential pair of wire-level MCP `initialize` requests from inside the sandbox — one with the placeholder header and one with an unresolvable control bearer — to verify that OpenShell resolves the credential on egress; otherwise it reports an inconclusive `probe skipped` result and sends no request. Neither outcome fails the committed add, and `--no-probe` skips this check. For full setup details, see [Add an MCP Server](../manage-sandboxes/mcp-servers/add-an-mcp-server). Credential-bound managed definitions may share a normalized endpoint only when they intentionally share one credential binding. NemoClaw rejects the same endpoint with different credential bindings; use a distinct endpoint URL when different credentials are required. @@ -3021,7 +3048,7 @@ Restart, rebuild, and restore do not refresh pins. ### `$$nemoclaw mcp status` -Inspect MCP server state for one server or for all servers discovered from the agent-native configuration. Status joins that source with the live OpenShell policy, provider, attachment, and credential-key shape; no MCP registry row is consulted or created. Status also compares current DNS answers with recorded pins without changing the policy. Text output reports `public address pins:` or `private address pins:` with `match`, `drift`, or `unresolved`; public targets that resolve to private or special-use addresses report `rejected`. JSON output reports the state, recorded pins, and validated current pins when available in `publicTarget` or `trustedPrivateTarget`. Public drift directs you to `mcp update --refresh-public-pins`, which changes only the live endpoint address pins. Trusted-private drift still requires review and remove-and-add approval. When a single server is named, status requests a differential wire-level credential-resolution probe. It sends no probe traffic unless the policy binding is present, the expected provider attachment is confirmed, and the live provider has the `nemoclaw-mcp-v1` type, a valid resource version, and exactly one credential key matching the source placeholder; a readiness failure reports `unknown` with a `probe skipped` detail. When ready, the same MCP `initialize` is sent from inside the sandbox once with the recorded resolver placeholder header and once with a deliberately-unresolvable control bearer. Classification uses the two HTTP status codes plus curl exit codes for transport, timeout, and policy-denial outcomes; response bodies are never captured or printed. A `verified` verdict requires the placeholder request to be accepted (HTTP 2xx) while the control is rejected — the only outcome that proves a valid credential was on the wire. Identical HTTP 400, 401, or 403 rejections raise a warning that names the hypotheses — the placeholder forwarded verbatim, an expired or revoked credential that resolved correctly, or (for HTTP 400) endpoint request validation — and tells you to verify the stored credential first. For HTTP 401 or 403, a confirmed-valid credential means the host is not rewriting placeholders and agent runtimes receive the same auth failure and skip the server; HTTP 400 remains inconclusive because the endpoint may reject the probe request itself. Every other outcome — differing rejections (an endpoint may reject two different literal bearers differently), endpoints that skip authentication, endpoint outages, policy denials, and unreachable sandboxes — reports as `unknown` rather than blaming the credential rewrite, and a source URL that fails the current authenticated-endpoint boundary is never probed. +Inspect MCP server state for one server or for all servers discovered from the agent-native configuration. Status joins that source with the live OpenShell policy, provider, attachment, and credential-key shape; no MCP registry row is consulted or created. Status also compares current DNS answers with recorded pins without changing the policy. Text output reports `public address pins:` or `private address pins:` with `match`, `drift`, or `unresolved`; public targets that resolve to private or special-use addresses report `rejected`. JSON output reports the state, recorded pins, and validated current pins when available in `publicTarget` or `trustedPrivateTarget`. Public drift directs you to `mcp update --refresh-public-pins`, which changes only the live endpoint address pins. Trusted-private drift still requires review and remove-and-add approval. When a single server is named, status requests a differential wire-level credential-resolution probe. It sends no probe traffic unless the policy binding is present, the expected provider attachment is confirmed, and the live provider has the profile type derived from the recorded credential name, a valid resource version, and exactly one credential key matching the source placeholder; a readiness failure reports `unknown` with a `probe skipped` detail. When ready, the same MCP `initialize` is sent from inside the sandbox once with the recorded resolver placeholder header and once with a deliberately-unresolvable control bearer. Classification uses the two HTTP status codes plus curl exit codes for transport, timeout, and policy-denial outcomes; response bodies are never captured or printed. A `verified` verdict requires the placeholder request to be accepted (HTTP 2xx) while the control is rejected — the only outcome that proves a valid credential was on the wire. Identical HTTP 400, 401, or 403 rejections raise a warning that names the hypotheses — the placeholder forwarded verbatim, an expired or revoked credential that resolved correctly, or (for HTTP 400) endpoint request validation — and tells you to verify the stored credential first. For HTTP 401 or 403, a confirmed-valid credential means the host is not rewriting placeholders and agent runtimes receive the same auth failure and skip the server; HTTP 400 remains inconclusive because the endpoint may reject the probe request itself. Every other outcome — differing rejections (an endpoint may reject two different literal bearers differently), endpoints that skip authentication, endpoint outages, policy denials, and unreachable sandboxes — reports as `unknown` rather than blaming the credential rewrite, and a source URL that fails the current authenticated-endpoint boundary is never probed. `mcp status` exits with status `2` when the managed MCP projection is a symbolic link, FIFO, or @@ -3603,20 +3630,24 @@ After the rebuild succeeds, retrieve the replacement token with `nemohermes my-a After restore, the command starts Deep Agents with the complete source native home, including `config.toml`, hooks, skills, memories, histories, packages, and unrelated native entries. Before changing the sandbox, rebuild verifies that the recorded provider path is still reachable and that the target provider, model, reasoning settings, web search selection, base image, current agent MCP source, and current live OpenShell policy can be captured. NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. -Deep Agents also uses attached providers for hosted compatible endpoints outside the Personal tier and qualified local endpoints; a fresh Personal-tier hosted selection stays on the shared route. -Local NVIDIA NIM, other agents' shared local selections, and custom endpoints without an attachment retain the shared `inference.local` route. If those checks fail after backup, NemoClaw keeps the existing sandbox intact. Use rebuild after a failed Deep Agents version check or as the apply step for legacy MCP migration. +Deep Agents also uses attached providers for qualifying custom hosted endpoints and qualified local endpoints. +Other agents' retained shared local selections and remaining custom shared selections retain the `inference.local` route. If those checks fail after backup, NemoClaw keeps the existing sandbox intact. Use rebuild after a failed Deep Agents version check or as the apply step for legacy MCP migration. To enable Tavily egress, apply the `tavily` preset, register the credential on the host, and explicitly attach the provider. Follow [Optional Tavily Egress](/user-guide/deepagents/get-started/quickstart#optional-tavily-egress). Repeat the attachment after rebuilding if Tavily access is still needed; rebuild omits the provider unless a saved enabled selection exists. Deep Agents Code has no NemoClaw-managed web-search selection that makes this attachment durable. -Legacy Deep Agents Code local sandboxes without an attachment stop before rebuild. Local NVIDIA NIM retains its shared rebuild path. +Legacy Deep Agents Code local sandboxes without an attachment stop before rebuild. Local NVIDIA NIM also requires its recorded native attachment before rebuild. Back up the sandbox data before running `nemo-deepagents onboard --fresh --name --recreate-sandbox`. -Legacy OpenClaw local sandboxes without a native attachment stop before rebuild. Back up the sandbox data before running `$$nemoclaw onboard --fresh --name --recreate-sandbox`. +Older OpenClaw Ollama, standalone vLLM, and llama.cpp sandboxes without a native attachment stop before rebuild. +Back up the sandbox data before running `$$nemoclaw onboard --fresh --name --recreate-sandbox`. +New Local NVIDIA NIM sandboxes use their recorded native attachment during rebuild. +For older shared-route NIM sandboxes, preserve backups, the existing installation, and sandbox data. +Follow the [OpenShell upgrade restrictions](../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path). @@ -3693,7 +3724,13 @@ A readiness, backup, or return failure marks that sandbox as failed, and `backup $$nemoclaw backup-all ``` -Before an OpenShell upgrade, the installer prepares the current release CLI and uses it to run `backup-all` in strict mode. Strict mode requires every registered sandbox that still has a runtime to produce a fresh backup and aborts before gateway changes if an ordinary sandbox is skipped or fails. When strict mode reports an ordinary skipped sandbox, start that sandbox or its container and rerun the installer or `$$nemoclaw backup-all`. When the output says that Docker or Podman did not respond, start that container engine first and then rerun the printed command, because the sandbox cannot start while its engine is down. A registry record that the selected gateway and container provider both prove has no remaining runtime is reported as confirmed stranded, produces no backup, and does not fail strict mode. Follow the reported cleanup or recreation action in [Resume a Manually Prepared Upgrade](../manage-sandboxes/operate-sandboxes/update-sandboxes#resume-a-manually-prepared-upgrade) before recovery. +Before an OpenShell upgrade, the installer prepares the current release CLI and uses it to run `backup-all` in strict mode. +Strict mode requires every registered sandbox that still has a runtime to produce a fresh backup and aborts before gateway changes if an ordinary sandbox is skipped or fails. +When strict mode reports an ordinary skipped sandbox, start that sandbox or its container and rerun the installer or `$$nemoclaw backup-all`. +If the output says that Docker or Podman did not respond, start that container engine before rerunning the printed command. +The sandbox cannot start while its engine is down. +A registry record that the selected gateway and container provider both prove has no remaining runtime is reported as confirmed stranded, produces no backup, and does not fail strict mode. +Follow the reported cleanup or recreation action in [Interpret Automatic Recovery Results](../manage-sandboxes/operate-sandboxes/update-sandboxes#interpret-automatic-recovery-results) before recovery. During strict pre-upgrade backup, a running sandbox whose in-sandbox SSH endpoint does not answer fails its backup and stops the upgrade before gateway changes. During a standalone `$$nemoclaw backup-all` run, the failure is recorded and the command continues with later sandboxes. Set `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1` exactly to skip such sandboxes during a standalone run instead of recording the failure. Other values such as `true`, `yes`, or `0` are not accepted. This variable does not weaken the installer's strict pre-upgrade requirement: in strict mode the sandbox is reported as a failed backup, and the output says that the variable applies only to a standalone run. A skipped sandbox's uncommitted state is not included in its last successful backup. @@ -3922,15 +3959,15 @@ $$nemoclaw my-assistant inference get -Switch the active inference provider or model for a NemoClaw-managed OpenClaw sandbox. For NVIDIA Endpoints, OpenAI, Anthropic, Gemini, OpenRouter, and Hermes Provider, the command attaches or updates the selected sandbox's native provider. A hosted compatible endpoint outside the Personal tier uses a native provider during `inference set` only when this sandbox already records a valid native attachment. The command can then attach or update the selected custom provider. A sandbox created with a Personal-tier hosted selection, Local NVIDIA NIM, custom endpoints without an attachment, and retained shared selections use the shared OpenShell route. It then patches the selected running agent config to match the selected inference path, recomputes the config hash, and updates the NemoClaw registry. It is also available in sandbox-first form as `$$nemoclaw inference set --provider --model `. For OpenClaw, the patch updates the OpenClaw config provider namespace and selected model. After any changed config sync, NemoClaw uses the managed supervisor to restart only the OpenClaw gateway and verify its health and forwards. The sandbox remains running, but agent requests are briefly interrupted. If the restart fails, the selected inference path and config remain committed; run `$$nemoclaw gateway restart` to finish applying the switch. After every changed synchronized inference selection, NemoClaw also verifies that the local CLI device has the managed gateway's required pairing scopes before it reports success. If pairing does not converge, the selected inference path and config remain committed. Run `$$nemoclaw doctor --fix`, then retry the agent turn. +Switch the active inference provider or model for a NemoClaw-managed OpenClaw sandbox. For NVIDIA Endpoints, OpenAI, Anthropic, Gemini, OpenRouter, and Hermes Provider, the command attaches or updates the selected sandbox's native provider. Retained shared custom endpoints and unmigrated local selections retain their existing shared OpenShell route. It then patches the selected running agent config to match the selected inference path, recomputes the config hash, and updates the NemoClaw registry. It is also available in sandbox-first form as `$$nemoclaw inference set --provider --model `. For OpenClaw, the patch updates the OpenClaw config provider namespace and selected model. After any changed config sync, NemoClaw uses the managed supervisor to restart only the OpenClaw gateway and verify its health and forwards. The sandbox remains running, but agent requests are briefly interrupted. If the restart fails, the selected inference path and config remain committed; run `$$nemoclaw gateway restart` to finish applying the switch. After every changed synchronized inference selection, NemoClaw also verifies that the local CLI device has the managed gateway's required pairing scopes before it reports success. If pairing does not converge, the selected inference path and config remain committed. Run `$$nemoclaw doctor --fix`, then retry the agent turn. Switch the active inference provider or model for a NemoClaw-managed Hermes sandbox. -For NVIDIA Endpoints, OpenAI, Anthropic, Gemini, OpenRouter, and Hermes Provider, the command attaches or updates the selected sandbox's native provider. A hosted compatible endpoint outside the Personal tier uses a native provider during `inference set` only when this sandbox already records a valid native attachment. The command can then attach or update the selected custom provider. +For NVIDIA Endpoints, OpenAI, Anthropic, Gemini, OpenRouter, and Hermes Provider, the command attaches or updates the selected sandbox's native provider. A qualifying custom endpoint uses a native provider during `inference set` only when this sandbox already records a valid native attachment. The command can then attach or update the selected custom provider. New Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections also use sandbox-attached providers. -Owned Local NVIDIA NIM, nonqualifying custom endpoints, and retained legacy shared routes keep their shared OpenShell route. +Remaining custom shared selections and retained legacy shared routes keep their shared OpenShell route on `0.0.x`. It then patches the shared native `/sandbox/.hermes/config.yaml`, recomputes the config hash, and updates the NemoClaw registry. Changes between shared routes do not rebuild or restart Hermes. If the current or selected route uses a native hosted or local provider, NemoClaw restarts the Hermes gateway after configuration sync. @@ -3950,7 +3987,13 @@ The command is also available in sandbox-first form as `$$nemoclaw infere By default, the command syncs the default registered sandbox. -During onboarding, NVIDIA Endpoints, the five fixed hosted providers, and new hosted compatible selections outside the Personal tier use sandbox-attached providers without changing the shared route. Qualified new local selections also receive native attachments during onboarding. At runtime, `inference set` can attach a custom native provider when the sandbox already records a valid native attachment; a retained shared-route sandbox stays on the shared route. Hermes also supports native local switches through `inference set`. For Deep Agents Code provider or model changes, follow [Validate a Different Deep Agents Selection](/user-guide/deepagents/inference/manage-inference/switch-providers#validate-a-different-deep-agents-selection). The following route-conflict rules apply whenever the command changes the shared gateway route, including owned Local NVIDIA NIM, retained legacy and other custom shared routes, and Pi local routes. +NVIDIA Endpoints, the five fixed hosted providers, and qualifying custom hosted endpoints use sandbox-attached providers without changing the shared route. +New Deep Agents Code and Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections receive native attachments during onboarding. +New OpenClaw Ollama, standalone vLLM, and llama.cpp selections also receive native attachments during onboarding. +Hermes also supports native local switches through `inference set`. +For Deep Agents Code provider or model changes, follow [Validate a Different Deep Agents Selection](/user-guide/deepagents/inference/manage-inference/switch-providers#validate-a-different-deep-agents-selection). +The following route-conflict rules apply whenever the command changes the shared gateway route, including retained legacy shared routes, retained custom shared endpoints, and unchanged published Pi shared local routes. +These shared routes require OpenShell `0.0.x`. Before changing the route, NemoClaw compares it with every same-gateway registry entry, including stopped sandboxes. When only the provider or model differs, the command names the affected sandboxes and warns immediately before it re-points the shared route. Those sandboxes use the new live route until `connect`, onboarding, or another `inference set` command re-points it. @@ -3973,7 +4016,17 @@ You can also name the sandbox in sandbox-first position instead of passing `--sa $$nemoclaw my-assistant inference set --provider nvidia-prod --model nvidia/nemotron-3-super-120b-a12b ``` -Pass both `--provider` and `--model` to select the provider and model for the target sandbox. For NVIDIA Endpoints and the five fixed hosted providers, NemoClaw attaches or updates the selected sandbox's native provider and syncs its agent config without changing the shared route. Hermes qualified local switches use native attachments. OpenClaw model switches on recorded native Ollama, standalone vLLM, and llama.cpp attachments also leave the shared route unchanged; selecting a different local provider without an attachment requires a backup and fresh sandbox. Nonqualifying custom endpoints, Local NVIDIA NIM, and retained shared local selections retain their existing shared-route workflow. Use onboarding to create a new Deep Agents Code native local selection. NemoClaw resolves the OpenShell gateway from the target sandbox's recorded gateway binding, including non-default `NEMOCLAW_GATEWAY_PORT` deployments. Do not run `openshell inference set` directly on a shared NemoClaw gateway because that bypasses registry compatibility checks and can break other sandboxes. When either flag is missing, `$$nemoclaw inference set` reports both required flags without suggesting a raw OpenShell command. The command updates the host registry after the selected inference path changes. +Pass both `--provider` and `--model` to select the provider and model for the target sandbox. +For NVIDIA Endpoints and the five fixed hosted providers, NemoClaw attaches or updates the selected sandbox's native provider and syncs its agent config without changing the shared route. +Hermes qualified local switches use native attachments. +OpenClaw model switches on recorded native Ollama, standalone vLLM, and llama.cpp attachments also leave the shared route unchanged. +When switching to a different provider among those local options, NemoClaw prepares and attaches a provider for the selected runtime without changing the shared route. +Retained shared custom endpoints and unmigrated local selections retain their existing shared-route workflow on OpenShell `0.0.x`. +Use onboarding to create a new Deep Agents Code native local selection. +NemoClaw resolves the OpenShell gateway from the target sandbox's recorded gateway binding, including non-default `NEMOCLAW_GATEWAY_PORT` deployments. +Do not run `openshell inference set` directly on a shared NemoClaw gateway because that bypasses registry compatibility checks and can break other sandboxes. +When either flag is missing, `$$nemoclaw inference set` reports both required flags without suggesting a raw OpenShell command. +The command updates the host registry after the selected inference path changes. @@ -3990,22 +4043,27 @@ A rebuild restores the previous native `openclaw.json`, so it does not repair th If the in-sandbox config write or integrity hash update fails, the selected inference path and NemoClaw registry remain committed, but the command exits with status `1`. -For a compatibility-hash refusal, follow the printed backup and trusted recovery guidance. +If integrity metadata is invalid, back up needed data and restore matching configuration and integrity files from a trusted backup, or create a fresh Hermes sandbox. A rebuild preserves the native sandbox home and may retain the mismatch. +For an existing registered provider, retry the same `inference set` command to finish configuration sync after resolving any integrity metadata error. For an older Hermes sandbox with an OpenClaw `POLICY.md` file, follow [Explain Network Policy to Agents](../network-policy/explain-network-policy-to-agents). -Canonical supported provider IDs are `nvidia-prod`, `nvidia-nim`, `nvidia-router`, `openai-api`, `openrouter-api`, `anthropic-prod`, `compatible-anthropic-endpoint`, `gemini-api`, `compatible-endpoint`, `hermes-provider`, `llama-cpp-local`, `ollama-local`, and `vllm-local`. +Canonical supported provider IDs are `nvidia-prod`, `nvidia-router`, `openai-api`, `openrouter-api`, `anthropic-prod`, `compatible-anthropic-endpoint`, `gemini-api`, `compatible-endpoint`, `hermes-provider`, `llama-cpp-local`, `ollama-local`, and `vllm-local`. The installer alias `llama-cpp` is accepted and normalized to `llama-cpp-local`. +The legacy hosted alias `nvidia-nim`, including `nvidia_nim`, normalizes to `nvidia-prod` for NVIDIA Endpoints. +The installer names `nim`, `nim-local`, and `nim_local` select Local NVIDIA NIM through `vllm-local` and require matching recorded NIM state. +See [Choose a Supported Provider](../inference/manage-inference/switch-providers#choose-a-supported-provider) for NIM prerequisites and recovery when that state is missing or inconsistent. Recognized underscore spellings of canonical provider IDs and hyphenated installer aliases are also accepted. For example, `ollama_local` normalizes to `ollama-local`, `nvidia_prod` normalizes to `nvidia-prod`, and `open_router` normalizes to `openrouter-api`. NemoClaw preserves an unrecognized underscore spelling for normal provider validation. Use `--no-verify` only when OpenShell cannot verify the provider at switch time but you have already confirmed the provider and credential. -When you explicitly supply a direct compatible endpoint at `http://host.openshell.internal:`, NemoClaw skips OpenShell's host-side provider probe because that hostname resolves only inside the sandbox network. -Before it persists the route in the NemoClaw registry or agent config, the command sends a validation request from the target sandbox. +NemoClaw rejects an unverified custom route that uses `host.openshell.internal` because the alias and port do not prove which host process owns the destination. +Qualified OpenAI-compatible native-local routes and managed adapter routes retain their recorded host bridge endpoint. +Use a public endpoint or an explicitly trusted private hostname or IP address. The output limit is 16 tokens for ordinary providers and 256 tokens for Google Gemini. When the switch changes the API family and that request returns HTTP `400` or `404`, NemoClaw retries up to two times after delays of 2 and 4 seconds. NemoClaw uses the same retry schedule when validation fails before it receives an HTTP status, such as while the gateway route is reloading. @@ -4036,7 +4094,7 @@ When you supply endpoint options, missing or `inference set` provenance and ever For public HTTP URLs, NemoClaw stores the validated IP address to prevent DNS rebinding. For a DNS-backed HTTPS URL, NemoClaw routes the endpoint through a local HTTPS Pin Runtime adapter that terminates a pinned, SNI-correct outbound connection to the real upstream hostname; the sandbox and the persisted registry only ever see a local `host.openshell.internal` route, never the real hostname. HTTPS IP-literal URLs remain supported and do not need the adapter. -NemoClaw accepts `http://host.openshell.internal:` only with an explicit port from `1024` through `65535`; this narrow exception supports NemoClaw's sandbox-to-host inference routes and is not a general private-endpoint bypass. +NemoClaw reserves `host.openshell.internal` for qualified OpenAI-compatible native-local routes and routes created from recorded runtime authority, such as managed credential and HTTPS pin adapters. `--credential-env` may also be supplied for compatible provider metadata; supported `--inference-api` values are `openai-completions`, `anthropic-messages`, and `openai-responses`. @@ -4144,16 +4202,22 @@ $$nemoclaw credentials list ### `$$nemoclaw credentials add ` -Register a provider credential with the OpenShell gateway by name and type. Each `--credential` takes the env variable name whose value the gateway should read; export the value first so it is not placed in argv. Pass either repeatable `--credential ` or `--from-existing`, but do not combine them. With `--from-existing`, NemoClaw inspects the provider profile and rejects registration when one of its credential keys is reserved by a managed MCP server. After the gateway accepts a provider other than Tavily, rebuild each sandbox that should use it. For Tavily, follow the selection and attachment requirements below. +Register a provider credential with the OpenShell gateway by name and type. Each `--credential` takes the env variable name whose value the gateway should read; export the value first so it is not placed in argv. Pass either repeatable `--credential ` or `--from-existing`, but do not combine them. With `--from-existing`, NemoClaw inspects the provider profile and rejects registration when one of its credential keys is reserved by a managed MCP server. After a successful provider creation or attachment, NemoClaw compares a previously verified profile with its current gateway definition. If it warns that verification did not complete or the profile changed, the successful operation remains successful. Ask the gateway operator to inspect the profile before relying on its credential boundary. -Registered providers are gateway-wide and can attach during later sandbox builds or rebuilds. +Provider registration is gateway-wide. +Native NVIDIA Endpoints and fixed hosted providers attach when a sandbox selects or restores that inference provider. +After registering one of these providers, follow the command's `inference set` guidance. +Ordinary extra providers other than Tavily attach during subsequent sandbox builds and rebuilds after reconciliation. +Rebuild each existing sandbox that should use one of those ordinary extra providers. Tavily providers attach during builds or rebuilds only when the sandbox has an enabled Tavily selection. Removing the Tavily preset preserves that exclusion through rebuilds and keeps the gateway credential available to other sandboxes. -`nemoclaw credentials reset ` removes the provider from the gateway and detaches it from every sandbox currently using it; it cannot limit the provider to selected sandboxes. -To replace a credential, reset the provider and register the replacement. -Rebuild each detached sandbox, or explicitly reattach Tavily when no enabled Tavily selection is saved. +Registration alone does not change a sandbox's inference selection. +Credential reset refuses removal while a sandbox records that provider as its registered inference selection. +For providers eligible for reset, the command can detach current sandbox attachments before deleting the provider. +After registering a replacement, rebuild each sandbox reported as detached, or explicitly reattach Tavily when no enabled Tavily selection is saved. +For reset prerequisites and inference key rotation, refer to [Rotate or Remove a Stored Credential](../security/credential-storage#rotate-or-remove-a-stored-credential). Tavily requires a provider profile that permits the intended agent runtime. Use `--type tavily --agent hermes` to select the versioned `tavily-hermes-v1` profile for Hermes Python. @@ -4202,6 +4266,10 @@ For a manually registered Tavily provider, register the replacement on the host Follow [Optional Tavily Egress](/user-guide/deepagents/get-started/quickstart#optional-tavily-egress) for the Deep Agents registration and attachment procedure. Run `$$nemoclaw credentials list` first if you are not sure of the provider name. +If a sandbox records the provider as its registered inference selection, reset stops without changing the provider or its attachments. +Switch each affected sandbox to another provider before resetting it. +Alternatively, preserve required sandbox state and destroy those sandboxes first. + When a sandbox records an attachment to a native NVIDIA or supported fixed hosted provider, reset stops without detaching or deleting it. Rotate its credential in place by supplying the selected provider's credential through the host environment and rerunning `$$nemoclaw onboard --name `. Use `NVIDIA_INFERENCE_API_KEY` for NVIDIA Endpoints and the provider-specific variable documented under `onboard` for another selection. To remove it completely, preserve required sandbox state, destroy every attached sandbox, and then retry reset. @@ -4673,8 +4741,8 @@ The following variables let you tune onboarding without editing the Dockerfile o | `NEMOCLAW_OLLAMA_REQUIRE_TOOLS` | `0` to disable, anything else to keep the default | When set to `0`, skips the Ollama tool-calling capability check during local-inference onboarding. | | `NEMOCLAW_OLLAMA_INSTALL_MODE` | `system`, `user`, or empty/unset | Pins the Linux Ollama install location. Refer to the Linux Ollama install mode details below. | | `NEMOCLAW_OLLAMA_PROXY_SKIP_BIND_PROBE` | exactly `1` to enable | Starts the Ollama auth proxy even when the selected Ollama daemon or unauthenticated compatible endpoint answers on a non-loopback interface. The proxy otherwise refuses to start, because any host that can reach that backend can bypass the proxy's token. Binding the selected backend to loopback is the supported fix; use this override only when you accept that exposure. Export the variable rather than setting it for one command: a later NemoClaw command that respawns the proxy without it refuses to start again. | -| `NEMOCLAW_PROXY_HOST` | hostname or IP | Overrides the sandbox-side outbound HTTP proxy host. Defaults to `10.200.0.1`. | -| `NEMOCLAW_PROXY_PORT` | integer port | Overrides the sandbox-side outbound HTTP proxy port. Defaults to `3128`. | +| `NEMOCLAW_PROXY_HOST` | hostname or IP | Legacy proxy host metadata. Does not change managed network routing on OpenShell `0.1.7`; managed startup clears this input from the runtime environment. | +| `NEMOCLAW_PROXY_PORT` | integer port | Legacy proxy port metadata. Does not change managed network routing on OpenShell `0.1.7`; managed startup clears this input from the runtime environment. | | `NEMOCLAW_INFERENCE_INPUTS` | comma-separated list of `text` and/or `image` | OpenClaw only. @@ -4805,7 +4873,13 @@ export TELEGRAM_BOT_TOKEN_AGENT_B= $$nemoclaw onboard --agent hermes ``` -For each entry, NemoClaw registers an OpenShell provider with the endpointless `nemoclaw-mcp-v1` profile. OpenShell resolves the named credential placeholder to the operator-supplied value at egress. The Hermes profile `.env` files are operator-owned: write `${TELEGRAM_BOT_TOKEN_AGENT_A}` (or the matching placeholder for each entry) into the per-profile `.env` so the in-sandbox Hermes process inherits the OpenShell placeholder instead of a raw token. NemoClaw never reads, writes, or rewrites these `.env` files; verify after onboarding that each profile's `.env` references the placeholder and that no raw bot token value sits on disk. +NemoClaw adds each accepted extension to its canonical provider's credential set. +The provider uses an immutable profile type derived from its base profile and credential names. +That profile declares the selected keys and preserves the base profile's endpoint and executable permissions. +Credential values do not determine the profile type. +OpenShell resolves the named credential placeholder to the operator-supplied value at egress. +The Hermes profile `.env` files are operator-owned: write `${TELEGRAM_BOT_TOKEN_AGENT_A}` (or the matching placeholder for each entry) into the per-profile `.env` so the in-sandbox Hermes process inherits the OpenShell placeholder instead of a raw token. +NemoClaw never reads, writes, or rewrites these `.env` files; verify after onboarding that each profile's `.env` references the placeholder and that no raw bot token value sits on disk. Entries are split on whitespace and commas and must match `^[A-Z][A-Z0-9_]{0,127}$`. Each entry must extend a canonical channel envKey with a non-empty `_` (for example `TELEGRAM_BOT_TOKEN_AGENT_A`); the canonical envKeys are `TELEGRAM_BOT_TOKEN`, `DISCORD_BOT_TOKEN`, `SLACK_BOT_TOKEN`, `SLACK_APP_TOKEN`, `WECHAT_BOT_TOKEN`, `BRAVE_API_KEY`, and `TAVILY_API_KEY`. Bare canonical envKeys, the control env itself, and arbitrary host secret names (`GITHUB_TOKEN`, `AWS_SECRET_ACCESS_KEY`, `KUBECONFIG`, and similar) are refused so they cannot leak into the sandbox provider gateway. Duplicates are dropped silently. The list is capped at 32 entries per sandbox. Offending tokens emit one warning each and are skipped. @@ -4814,6 +4888,10 @@ The primary credential is required to create that provider. An extension already stored in an existing provider remains available until that provider is replaced or removed. If the extension credential is absent from the credential store, export it before onboarding. +Changing the credential names can require replacing the provider with a different profile type. +NemoClaw refuses that replacement if any previously stored credential value is missing or the provider is attached to another sandbox. +Preserve the existing provider until you can supply the complete credentials and resolve its attachment restrictions for an explicit replacement. + @@ -4930,7 +5008,7 @@ The following flags toggle optional behaviors during onboarding. Set them before | `NEMOCLAW_SANDBOX_BASE_IMAGE_REFRESH` | `1`, `true`, `yes`, or `on` to enable | Bypasses recorded sandbox base-image resolution metadata during onboarding, recreation, and rebuild. NemoClaw reruns candidate resolution but can still use a compatible image from Docker's local image store. Versioned release candidates that exist locally but fail validation are refreshed from the registry once during normal resolution. This setting does not discard onboarding session state. | | `NEMOCLAW_SANDBOX_BASE_LOCAL_BUILD` | unset or `auto` (default); `1`, `true`, `yes`, or `on` to enable; `0`, `false`, `no`, or `off` to disable | Controls whether base-image resolution may build a compatible image locally. The default allows builds during normal CLI runs and disables them when `NODE_ENV=test` or `VITEST=true`. When source inputs or a missing/incompatible release-version base require a fresh build, disabling local builds makes resolution fail instead of using an unproven image. | | `NEMOCLAW_DOCKER_GPU_PATCH` | unset, `auto`, `fallback`, `1`, or `0`; other legacy nonzero values remain accepted through `v0.0.x` and will be removed in `v0.1.0` | Selects Linux Docker-driver GPU routing. Unset, `auto`, or `0` uses native OpenShell GPU injection on ordinary native Linux. `fallback` explicitly opts into one native attempt followed by one bounded compatibility retry when trusted host evidence identifies a GPU-routing failure. `1` and legacy nonzero values select the compatibility patch from the outset. Docker Desktop WSL and Jetson/Tegra use the compatibility path by default; Docker Desktop WSL ignores `0`, while Jetson/Tegra accepts `0` only as a troubleshooting override that bypasses device-group propagation. | -| `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH` | `1` to enable; disabled by default | This setting explicitly opts into the Linux gateway compatibility container for an older host ABI or a diagnostic run; use it only on a trusted local host because it uses host networking and mounts the Docker socket read-only even though the socket still exposes the privileged Docker API; prefer OpenShell 0.0.116's directly supported glibc 2.39+ path; see [Gateway Compatibility Container](/user-guide/openclaw/security/security-controls/gateway-authentication-controls#gateway-compatibility-container) for the container boundary and removal conditions. | +| `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH` | `1` to enable; disabled by default | This setting explicitly opts into the Linux gateway compatibility container for an older host ABI or a diagnostic run; use it only on a trusted local host because it uses host networking and mounts the Docker socket read-only even though the socket still exposes the privileged Docker API; prefer direct startup when the host meets the installed OpenShell gateway binary's glibc requirements; see [Gateway Compatibility Container](/user-guide/openclaw/security/security-controls/gateway-authentication-controls#gateway-compatibility-container) for the container boundary and removal conditions. | | `NEMOCLAW_OPENSHELL_GATEWAY_BIN` | path | Advanced override for the `openshell-gateway` binary used by Linux Docker-driver startup. For the default port, the installer accepts the binary under an absolute `XDG_BIN_HOME` when set, otherwise `~/.local/bin/openshell-gateway`; it also accepts `/usr/local/bin/openshell-gateway` or `/usr/bin/openshell-gateway`. Another path fails service staging. The macOS Homebrew service uses the formula's binary. Defaults to the binary next to `openshell`, then common install paths. | | `NEMOCLAW_OPENSHELL_SANDBOX_BIN` | path | Advanced override for the `openshell-sandbox` binary used by Linux Docker-driver startup. The macOS Homebrew service uses the formula's driver layout. Defaults to the binary next to `openshell`, then common install paths. | | `NEMOCLAW_OPENSHELL_GATEWAY_STATE_DIR` | path | Advanced override for the Linux Docker-driver gateway SQLite state directory and standalone-fallback PID file. Use a dedicated absolute directory; onboarding rejects relative paths, the shared NemoClaw state root or its parents, and existing nonempty directories without valid managed gateway configuration. Let onboarding create it when possible; its containing directory must be owned by the current user or root without group or world write access, and a pre-created state directory must be owner-controlled, non-symbolic, and mode `0700`. For an externally supervised gateway, use the `stateDir` declared by `NEMOCLAW_GATEWAY_MANAGEMENT`; if this variable is also set, it must resolve to that same directory. Pass the original resolved absolute value with `NEMOCLAW_GATEWAY_PORT` when uninstalling that gateway. Port `8080` defaults to `~/.local/state/nemoclaw/openshell-docker-gateway`; non-default ports default to `~/.local/state/nemoclaw/openshell-docker-gateway-`. | @@ -5024,8 +5102,8 @@ The following variables tune how long internal probes wait before giving up. Def | Variable | Default | Effect | | -------------------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NEMOCLAW_MCP_PROVIDER_SYNC_TIMEOUT_SECONDS` | `30` | Maximum time to wait for an OpenShell MCP provider credential handle to become active or fully revoked inside the sandbox. Integer seconds; raise only when provider synchronization is unusually slow. | -| `NEMOCLAW_SANDBOX_EXEC_TIMEOUT_MS` | per call site (typically `15000`) | Overrides the default timeout for `openshell sandbox exec` calls issued by recovery and lifecycle helpers. Integer milliseconds up to `2147483647`; other values fall back to the per-call-site default. | -| `NEMOCLAW_STATUS_PROBE_TIMEOUT_MS` | built-in default | Overrides the timeout for the OpenShell status probe used by `$$nemoclaw status`. Integer milliseconds up to `2147483647`; other values fall back to the default. | +| `NEMOCLAW_SANDBOX_EXEC_TIMEOUT_MS` | per call site (typically `15000`) | Overrides the default timeout for `openshell sandbox exec` calls issued by recovery and lifecycle helpers. Positive integer milliseconds up to `2147483647`; other values fall back to the per-call-site default. | +| `NEMOCLAW_STATUS_PROBE_TIMEOUT_MS` | built-in default | Overrides the timeout for the OpenShell status probe used by `$$nemoclaw status`. Positive integer milliseconds up to `2147483647`; other values fall back to the default. | | `NEMOCLAW_WSL_GPU_PROOF_TIMEOUT_MS` | `180000` | Maximum time for the selected provider's bounded CUDA workload on an eligible ARM64 Linux host. A positive integer overrides the default and is capped at `900000` (15 minutes). Invalid, infinite, zero, and negative values use the default. | ### Onboard and Sandbox Readiness Timeouts @@ -5106,6 +5184,7 @@ The following flags change defaults for commands that manage existing sandboxes. | `NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE` | JSON array of sandbox names | Confirms to the installer that the listed set of pre-fingerprint OpenClaw or Hermes sandboxes used NemoClaw-managed images, allowing recovery onto the current managed image. The normalized names must exactly match the installer's printed array. Set it only after verifying every named sandbox. Recorded custom-image evidence remains blocked. | | `NEMOCLAW_DISABLE_INFERENCE_ROUTE_REPAIR` | `1` to enable | Skips automatic DNS-proxy mutation for stale `inference.local` routes during `$$nemoclaw connect` and `$$nemoclaw connect --probe-only`. The command still probes the route and exits non-zero when the provider-specific requirement fails. An `ollama-local` route still requires a healthy authenticated proxy and HTTP 2xx from `inference.local/v1/models`. Use only as a troubleshooting escape hatch. | +| `NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS` | Exactly `1` to enable; unset by default | Enables strict `$$nemoclaw backup-all` checks. A failed backup or an ordinary skipped sandbox makes the command exit nonzero. A record confirmed absent from both the selected gateway and container provider produces no backup and does not fail strict mode. The installer sets this variable for pre-upgrade backups; it does not bypass the OpenShell version check. | | `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP` | Exactly `1` to opt in (`true`, `yes`, `0` are not accepted) | Applies to standalone `$$nemoclaw backup-all` runs. Skips running sandboxes whose in-sandbox SSH endpoint does not answer. It does not relax the installer's strict pre-upgrade backup, which still aborts if any registered sandbox is skipped or fails. Any uncommitted state since the last successful backup is not included in the skipped backup. | | `NEMOCLAW_UNINSTALL_ALL_GATEWAY_PORTS` | `1` to opt in | Makes `$$nemoclaw uninstall` remove every gateway port on the host instead of only the port `NEMOCLAW_GATEWAY_PORT` selects. Equivalent to passing the `--all-gateway-ports` flag; the whole-host `Proceed?` confirmation still applies unless `--yes` is also passed. Each port runs as its own uninstall, and the variable is dropped from those runs so the sweep cannot re-enter itself. | | `NEMOCLAW_UNINSTALL_DESTROY_USER_DATA` | `1` to opt in | Acknowledges data loss during `$$nemoclaw uninstall`, skips eligible fresh sandbox backups, and removes the otherwise-preserved entries (`rebuild-backups/`, `backups/`, `sandboxes.json`) in the selected gateway's state root. It does not select the explicit `--destroy-user-data` CLI-shim removal path; shim handling follows the ordinary selected-gateway scope. The global `Proceed?` confirmation still applies unless `--yes` is also passed. | diff --git a/docs/reference/configure-runtime-identity.mdx b/docs/reference/configure-runtime-identity.mdx index 66a3c125053..946122eda80 100644 --- a/docs/reference/configure-runtime-identity.mdx +++ b/docs/reference/configure-runtime-identity.mdx @@ -14,6 +14,12 @@ content: The direct OpenClaw blueprint runner can attach a provider-neutral runtime identity to a sandbox. The shipped blueprint and normal `nemoclaw onboard` do not enable or configure this experimental capability. + +The direct runner's identity apply path remains unqualified for OpenShell 0.1.7 until its protected conformance tests pass. +The runner now uses sandbox-attached native inference providers. +Keep this experimental flow disabled until the inference compatibility work in [NemoClaw#12558](https://github.com/NVIDIA/NemoClaw/issues/12558) and its conformance tests are complete. + + The bundled Okta and Microsoft Entra profiles are data-only implementations of the component. The blueprint schema describes the OpenShell provider binding and OAuth refresh inputs without using the identity provider as a schema discriminator. @@ -35,7 +41,7 @@ Runtime identity profiles may use DNS-backed HTTPS only within a provider type's NemoClaw resolves every destination before import and rejects private or internal addresses. OpenShell performs connect-time server-side request forgery (SSRF) and L7 enforcement before it injects the provider credential. -OpenShell 0.0.116 verifies the original hostname's certificate but does not pin the address that NemoClaw resolved during import. +OpenShell 0.1.7's gateway refresh client verifies the HTTPS hostname but does not pin the address that NemoClaw resolved during import. The 0.1.7 identity apply path remains unqualified. Only identity-platform-controlled DNS namespaces such as `okta.com`, `login.microsoftonline.com`, and `graph.microsoft.com` belong in this trust table. An attacker-controlled or customer-controlled suffix requires a pinning-capable upstream refresh boundary and new conformance evidence. @@ -45,7 +51,7 @@ An attacker-controlled or customer-controlled suffix requires a pinning-capable The deterministic `TC-INF-12` and `TC-INF-13` protected end-to-end (E2E) scenarios are the conformance gates for this boundary. Both run the real blueprint runner against a real OpenShell gateway and sandbox. -The scenarios enable provider-derived policy and restore its prior setting. +The scenarios require attached-provider policy composition without an active global policy override. They exchange a refresh token through a public HTTPS OAuth endpoint and prove that the sandbox sees only opaque placeholders. They prove that the existing placeholder resolves to the refreshed bearer after credential rotation. They prove that OpenShell rejects unreviewed credential delivery before the bearer reaches the protected resource. @@ -54,21 +60,20 @@ They also check that `apply` output contains no OAuth secret material and that t It proves that a `GET /v1.0/users` request is rejected before its bearer reaches the fixture. The deterministic fixture proves the complete Entra-shaped runtime path without storing a third-party tenant credential in pull-request CI; tenant acceptance remains a maintainer-run check against Microsoft Entra and Graph. -Focused tests cover malformed profiles, endpoint and DNS rejection, subprocess scoping, same-name resources, partial-apply compensation, and retryable rollback receipts. +Focused tests cover malformed profiles, endpoint and DNS rejection, subprocess scoping, same-name resources, partial-apply recovery receipts, and refusal of unsafe rollback. The gateway refresh client's DNS-pinning limitation is the accepted residual boundary. Expanding the trusted hostname policy requires upstream pinning and new conformance evidence. ## Enable Provider-Derived Policy -Enable provider-derived policy on the target gateway before you apply the blueprint. +OpenShell `0.1.7` composes attached-provider policies automatically when no active global policy overrides them. +The `providers_v2_enabled` setting no longer exists. +Use a gateway without an active global policy override for this experimental flow. +Ask the gateway operator to review an existing override before retrying; changing global policy can affect other sandboxes. -```bash -openshell settings set --global --key providers_v2_enabled --value true --yes -``` - -Apply reads the gateway-global setting before identity mutation and again before attachment. -It stops unless the JSON value is exactly `true`. -This check prevents an attachment whose provider-derived network policy and credential injection are inactive. +Apply inspects global policy history and the current global policy before identity mutation and again before attachment. +It stops if an override is active or the response cannot be verified. +Apply does not change gateway settings. ## Configure the Okta Reference Profile @@ -138,7 +143,7 @@ Obtain the refresh token through an authorized OAuth bootstrap flow for the same The host process can read `OKTA_REFRESH_TOKEN` and `OKTA_CLIENT_SECRET`, and the runner forwards them only to the scoped refresh-configuration subprocess. These environment variables remain available until you unset them or the host process exits. After apply succeeds, run `unset OKTA_REFRESH_TOKEN OKTA_CLIENT_SECRET` in the host shell. -OpenShell retains the refresh material in the gateway credential store until rollback or provider deletion removes the owned provider. +OpenShell retains the refresh material in the gateway credential store until an authorized gateway operator removes the owned provider. ## Configure the Microsoft Entra Reference Profile @@ -175,7 +180,7 @@ NemoClaw does not package the authorization redirect, consent UI, PKCE exchange, The host process can read `ENTRA_REFRESH_TOKEN` and `ENTRA_CLIENT_SECRET`, and the runner forwards them only to the scoped refresh-configuration subprocess. These environment variables remain available until you unset them or the host process exits. After apply succeeds, run `unset ENTRA_REFRESH_TOKEN ENTRA_CLIENT_SECRET` in the host shell. -OpenShell retains the refresh material in the gateway credential store until rollback or provider deletion removes the owned provider. +OpenShell retains the refresh material in the gateway credential store until an authorized gateway operator removes the owned provider. Acquire the initial refresh token outside NemoClaw using Microsoft-supported identity tooling, then export the application client ID, refresh token, and optional client secret into the host process that runs the direct blueprint runner. @@ -188,7 +193,7 @@ A maintainer-run tenant acceptance must establish all of the following criteria: - The protected-resource request below returns the signed-in delegated user. - OpenShell rejects a request to another Graph path before bearer delivery. - Credential refresh gives a new child the same stable handle, and `GET /v1.0/me` succeeds with the refreshed credential. The handle changes only after an authorization-binding epoch change. -- Rollback removes the owned attachment and provider while preserving any reused sandbox. +- Rollback refuses cleanup through mutable resource names and preserves the sandbox, providers, and ownership receipt for operator recovery. Do not save the Graph response, refresh token, access token, or client secret in a terminal transcript or CI artifact. Run the protected-resource acceptance request from a newly launched sandbox child, where `ENTRA_ACCESS_TOKEN` is the child-scoped OpenShell placeholder. @@ -205,17 +210,34 @@ curl -fsS \ During plan, the runner reports only the provider type, provider name, and credential key. It does not report secret material. -During apply, the runner inspects the target sandbox before identity mutation. -It requires either the sandbox name in the `Ready` phase or an explicit sandbox-not-found result. -If sandbox creation races with another creator, the runner repeats the same inspection. -It inspects the sandbox again after inference routing and immediately before identity attachment. -If the sandbox changed, the runner compensates without attachment. - -The runner always inspects the configured inference provider before identity mutation. -It reuses a provider only when its name, type, and required non-secret key shape match the blueprint. -It permits creation only after an explicit provider-not-found result. -If another actor creates the provider concurrently, the runner repeats the binding inspection before it treats the provider as unowned reusable state. -Any other inspection failure stops apply. +During apply, the runner verifies the gateway policy and inspects the target sandbox before identity mutation. +A fresh sandbox receives a unique creation label, and the runner records its immutable identity before configuring inference. +The runner preserves the blueprint's sandbox name, image, provider name, and forwarded ports. +It validates the sandbox policy, creates and attaches a native inference provider, and configures OpenClaw before attaching the runtime identity. +A creation race stops apply; matching names alone do not establish ownership. + +Reusing a sandbox requires a matching native inference selection recorded by NemoClaw or a completed earlier blueprint run. +The runner checks the sandbox identity, provider profile, provider identity, attachment, model, API, and any supplied endpoint. +It repeats the identity checks around later operations. +The runner marks native configuration publication complete only after the write succeeds and resource identities still match. +An incomplete or invalid matching run receipt stops reuse even when another complete receipt exists. + +Immediately before writing native inference configuration or changing its timeout, the runner rechecks the recorded sandbox identity. +If that check detects a replacement or cannot establish the identity, the runner stops before the write and preserves the resources and receipt for recovery. +Retain the receipt and ask the gateway operator to compare it with the live sandbox before retrying. +These checks do not lock the gateway against changes by another privileged client. + +A supplied inference credential requires an existing credential binding during reuse; apply does not rotate that credential. +The optional `timeout_secs` value updates the selected OpenClaw provider's request timeout without changing the agent turn timeout. +Fresh authenticated inference uses a sandbox-scoped OpenShell credential handle. +An admitted credential-free endpoint receives a provider with no credentials and does not inherit a host credential. + +The direct runner's existing inference endpoint restrictions still apply. +Private and local addresses and DNS-backed HTTPS endpoints are rejected. +For inference providers created by the direct runner, credentials require HTTPS. +Creation and saved blueprint receipt validation reject credential-bearing HTTP before changing resources or configuration. +Credential-free HTTP remains allowed only for an admitted public literal IP address. +These restrictions are separate from the reviewed runtime identity destination rules above. The runner also inspects the requested runtime identity provider name. Any same-name provider stops apply. @@ -235,19 +257,13 @@ NemoClaw cannot confirm which definition OpenShell used at bind time. If either export fails or times out, the runner warns that verification did not complete and asks the gateway operator to inspect the profile. These warnings contain no credential values and do not roll back setup. -Before attachment, the runner creates or reuses the configured inference provider. -For a reused sandbox and provider, it preserves the live route only when the provider, model, and requested timeout match the blueprint. -An absent or different route must pass `openshell inference set`. -The runner attaches the runtime identity and applies policy additions only after OpenShell confirms the route. -It performs the initial token rotation after attachment. +Before identity attachment and token refresh, the runner rechecks the native inference selection and the sandbox identity. +It stops if either changed. Each successful apply creates and owns the runtime identity provider whose refresh state it mutates. -The persisted plan records ownership of the runtime identity provider, sandbox attachment, inference provider, and sandbox. +The persisted plan records acquired resource identities and any incomplete native creation intent. The runner writes the receipt as it acquires each resource. -The receipt preserves a recovery path when automatic compensation fails. - -If a later step fails, the runner reverses only resources that apply created. -It detaches the runtime identity provider, deletes created providers, and removes a created sandbox. +If a later step fails, it preserves the resources and receipt for operator recovery because cleanup through mutable names cannot prove that it would remove the same resources. The runner never places the refresh token or optional client secret in command arguments or persisted plans. It passes them to `openshell provider refresh configure` through a scoped subprocess environment. @@ -259,6 +275,28 @@ The OpenShell L7 proxy substitutes the corresponding access token only for admit After token refresh, launch a new child process and confirm that it receives the same stable handle while resolution uses the refreshed token. An earlier child keeps that handle as well; a different handle is expected only after an authorization-binding epoch change. +## Resume Native Inference Configuration + +If apply stopped while publishing native inference configuration, retain its run receipt. +The direct runner can resume that write only when the receipt records the sandbox identity, creation label, and attached native provider. +An uncertain sandbox creation or incomplete provider attachment requires gateway-operator recovery. + +Use the direct runner's `reconcile --run-id ` action with the failed run ID. +Before writing configuration, the runner verifies the recorded OpenShell gateway, sandbox identity, creation label, provider profile, provider identity, attachment, and credential binding. +Missing, changed, or ambiguous evidence stops recovery and preserves the receipt and resources. +The runner uses the existing provider and credential binding without creating, attaching, or deleting resources or rotating credentials. + +After the write, the runner rechecks the resource identities before marking native configuration publication complete in the receipt. +Confirm that reconcile reports `native inference configuration is complete` for the failed run ID. +This result does not complete runtime identity setup or establish port forwards. +Rerun apply with the same blueprint to continue runtime identity setup after native configuration recovery succeeds. +Reusing the recovered sandbox does not recreate port forwards. +Apply still refuses a same-name runtime identity provider and requires gateway-operator recovery when an earlier attempt already created it. + +If reconcile fails, retain the run receipt and ask the gateway operator to compare its immutable identities with the live resources. +Do not delete resources based only on their names. +The rollback refusal below still applies. + ## Inspect Runtime Identity State Check gateway-side state without printing credential values. @@ -270,20 +308,27 @@ openshell sandbox provider list The direct runner `status` action reports the non-secret identity ownership receipt. -## Roll Back Runtime Identity +### Compare Credential Operation Times + +During apply, the runner logs UTC start and completion timestamps for identity configuration, attachment, and explicit rotation. +These timing messages contain only the operation, its start or completion, and the timestamp. +A missing completion message leaves that operation's outcome unverified. -The `rollback` action verifies the current provider binding before it changes resources. -It detaches the provider only when the apply created the sandbox attachment. -It deletes the provider only when the apply created it. +The protected E2E OAuth fixture also records each token request's `receivedAt` timestamp and `sequence` in its existing `*-oauth-token-requests.json` artifact. +Compare those records with the apply timing messages to locate token requests relative to the runner operations. +Timing alone does not identify the gateway caller or its refresh reason, establish causation, or prove that duplicate issuance is fixed. +The conformance assertions and experimental qualification requirement still apply. + +## Roll Back Runtime Identity -Rollback removes a sandbox only when the persisted plan proves that apply created it. -It preserves reused sandboxes and legacy plans with unknown sandbox ownership. -After an owned sandbox is removed, rollback deletes an inference provider only when the receipt proves that apply created it. -The same provider rule applies immediately when the sandbox was reused. +The `rollback` action refuses cleanup when the run contains a runtime identity receipt, native inference creation intent, or resources recorded as created by apply. +OpenShell exposes cleanup through mutable sandbox and provider names, so the runner cannot prove that deletion would target the recorded resource identities. +The refusal preserves resources and the run receipt and does not write a rollback completion marker. +It also applies when sandbox creation has an uncertain outcome and no created-resource flag was recorded yet. -If removal of an apply-owned sandbox fails, rollback returns the bounded error without writing its completion marker. -The ownership receipt remains available for retry. -Rollback does not mutate a same-name provider when its binding no longer matches the receipt. +Retain the receipt and ask the gateway operator to compare its immutable resource identities with the live state before recovery. +A same-name resource is not proof of ownership. +Legacy plans that record no owned resources leave the sandbox untouched and can be marked rolled back. ## Review the Scope Limits diff --git a/docs/reference/enterprise-readiness.mdx b/docs/reference/enterprise-readiness.mdx index ce47e0ed149..0c7a7ac7b75 100644 --- a/docs/reference/enterprise-readiness.mdx +++ b/docs/reference/enterprise-readiness.mdx @@ -82,7 +82,7 @@ Each row links to deeper documentation and, when a concrete fix is in progress, | Audit and session records | Supported with caveats | OpenClaw stores per-session JSONL event logs, and Hermes stores its own runtime state. Deep Agents Code has no equivalent session-log export, but it can emit bounded-content traces when you explicitly enable observability. Export is manual per sandbox. | | Resource quotas | Supported with caveats | The entrypoint applies best-effort process and file-descriptor limits (`ulimit -u 512`, `ulimit -n 65536`). Set hard limits through the container runtime for fail-closed enforcement. Refer to [Process Controls](../security/security-controls/process-controls). | | Cost and spend controls | Platform or partner-owned | Deny-by-default egress and routed inference reduce exfiltration and stray endpoints, but NemoClaw does not enforce per-token spend budgets. Set spend limits with your inference provider and monitor unattended agents. | -| Credential isolation | Supported | Inference credentials remain outside the sandbox. NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible selections outside the Personal tier, and qualified new local selections use sandbox-attached providers where supported. Fresh Personal-tier hosted compatible selections, owned Local NVIDIA NIM, custom endpoints without an attachment, and retained shared selections use `inference.local`. Bedrock AWS credentials stay in a host-local adapter. Hermes local lifecycle and Pi native routing retain qualification limits. Refer to [How Inference Routing Works](../inference/about-inference-routing) for those boundaries. CLI output redaction adds defense in depth, and OpenClaw sandboxes also run a memory secret scanner. Refer to [Credential Storage](../security/credential-storage) and [Security Best Practices](../security/best-practices). | +| Credential isolation | Supported | Inference credentials remain outside the sandbox. Bedrock AWS credentials stay in its host-local adapter. NVIDIA Endpoints, supported fixed hosted providers, qualifying hosted compatible endpoints, and new Deep Agents Code and Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections, plus new OpenClaw Ollama, standalone vLLM, and llama.cpp selections use native endpoints through sandbox-attached OpenShell providers. Retained shared custom endpoints, unchanged published Pi shared local selections, and retained shared local selections use `inference.local` on OpenShell `0.0.x`. Hermes live qualification for the new local lifecycle remains pending. Refer to [How Inference Routing Works](../inference/about-inference-routing) for qualification and existing-sandbox boundaries. CLI output redaction adds defense in depth, and OpenClaw sandboxes also run a memory secret scanner. Refer to [Credential Storage](../security/credential-storage) and [Security Best Practices](../security/best-practices). | | Upgrades and lifecycle | Supported with caveats | Upgrade NemoClaw, then run `$$nemoclaw rebuild` to recreate the sandbox with the current image while backing up and restoring its complete native home. Do not update the agent runtime in place inside the sandbox because the agent is image-pinned. Version attachment flexibility remains tracked in [#2217](https://github.com/NVIDIA/NemoClaw/issues/2217). | | State persistence | Supported | OpenShell preserves the native home/workspace across routine lifecycle operations. NemoClaw transfers the complete native tree during rebuild and recreation. Refer to [Persist Sandbox State](../manage-sandboxes/state-and-backups/create-and-restore-snapshots). | | Remote and cloud deployment | Supported with caveats | Provision a tested Linux host, connect over SSH, run the installer, and run `$$nemoclaw onboard`. For OpenClaw or Hermes, keep the agent dashboard bound to loopback and access it through SSH port forwarding. Automatic recovery after a host reboot is not guaranteed; follow the manual recovery flow in [Deploy to a Headless Server](../deployment/deploy-to-headless-server). | @@ -132,7 +132,7 @@ Use the following phrasing to describe NemoClaw accurately in customer and field - Describe NemoClaw as an open-source reference stack for running agents more safely inside OpenShell, in active development, rather than a finished enterprise control plane. - State that egress is deny-by-default and that an operator approves new endpoints, so the agent cannot reach arbitrary hosts. -- Explain that inference credentials stay outside the sandbox. New hosted compatible selections outside the Personal tier use sandbox-attached providers; fresh Personal-tier selections, endpoints without an attachment, and retained shared selections use `inference.local`. Bedrock AWS credentials stay in a host-local adapter. Hermes local lifecycle and Pi native routing retain qualification limits. Refer to [How Inference Routing Works](../inference/about-inference-routing) for the full routing and qualification boundaries. +- Explain that inference credentials remain outside the sandbox. Bedrock AWS credentials stay in its host-local adapter. NVIDIA Endpoints, supported fixed hosted providers, qualifying hosted compatible endpoints, and new Deep Agents Code and Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections, plus new OpenClaw Ollama, standalone vLLM, and llama.cpp selections use native endpoints through sandbox-attached OpenShell providers. Retained shared custom endpoints, unchanged published Pi shared local selections, and retained shared local selections use `inference.local` on OpenShell `0.0.x`. Hermes live qualification for the new local lifecycle remains pending. Refer to [How Inference Routing Works](../inference/about-inference-routing) for qualification and existing-sandbox boundaries. - Frame centralized fleet management, operator RBAC, enterprise SSO, and multi-tenant isolation as roadmap-only or out of scope today, and avoid presenting them as current capabilities. - Position cost controls and spend limits as provider-owned, and recommend setting provider-side limits for unattended agents. - When a customer hits a known limitation, point to the workaround in this guide and the tracked issue rather than promising a fix date. diff --git a/docs/reference/headless-lifecycle-package.mdx b/docs/reference/headless-lifecycle-package.mdx index f12497b05fa..c024360fd52 100644 --- a/docs/reference/headless-lifecycle-package.mdx +++ b/docs/reference/headless-lifecycle-package.mdx @@ -12,7 +12,9 @@ content: --- Use `nemoclaw/lifecycle` when a service needs deterministic Hermes planning and read-only observation without invoking the NemoClaw command-line interface (CLI). -The first API version supports Hermes `0.21.3` with OpenShell `0.0.116`. +The first API version supports Hermes `0.21.3` with OpenShell `0.1.5`. +The Hermes Portable CLI and `nemoclaw-acp` require OpenShell `0.1.7`. +Their runtime pin is independent of this API definition; upgrading the CLI does not change the injected observer contract. ## Understand the Boundary @@ -75,4 +77,4 @@ When the capability reports that the resource is missing, the result has `state: This API does not define an image reference, entrypoint, provider, network policy, checkpoint, ownership record, or mutation authority. It does not create, stop, start, replace, delete, or clean up a sandbox. -An OpenShell release after `0.0.116` requires separate compatibility evidence before a later API definition can support it. +An OpenShell release after `0.1.5` requires separate compatibility evidence before a later API definition can support it. diff --git a/docs/reference/network-policies.mdx b/docs/reference/network-policies.mdx index 7e621eccfd7..df16766c5e9 100644 --- a/docs/reference/network-policies.mdx +++ b/docs/reference/network-policies.mdx @@ -163,10 +163,12 @@ To narrow it, explicitly remove `brew` and apply `brew-balanced`. Resuming onboarding reapplies the saved selection; skipping optional setup preserves existing grants. When Personal is selected or carried forward, the `personal-open-internet` preset is mandatory for every agent and every onboarding entry point. Interactive choices, `NEMOCLAW_POLICY_MODE=custom`, and `NEMOCLAW_POLICY_MODE=skip` control only additional presets; they cannot deselect, skip, or replace Personal's required web authority. -Fresh onboarding keeps a hosted custom endpoint on the shared `inference.local` route when you select Personal. -NemoClaw refuses Personal for an existing sandbox that already has a custom native provider attachment. -Create a new sandbox, enter the custom endpoint, and choose Personal when prompted for the policy tier. -For non-interactive onboarding, set `NEMOCLAW_POLICY_TIER=personal` before onboarding. + +With OpenShell `0.1.7`, NemoClaw refuses Personal when an attached custom provider uses port `80` or `443`. +The provider's restricted address list conflicts with Personal's broader address list on those ports. +Choose another tier for these endpoints. +Custom endpoints on other ports retain their provider restrictions and do not trigger this refusal when their recorded attachment is valid. +NemoClaw also refuses Personal if it cannot validate the recorded custom attachment. The Personal tier applies the `personal-open-internet` policy preset with a hostless L4 endpoint on destination ports `80` and `443`. @@ -241,35 +243,36 @@ Unset, blank, or whitespace-only `NEMOCLAW_POLICY_TIER` values use the `balanced ### Inference -NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. +NVIDIA Endpoints, supported fixed hosted providers, and qualifying custom hosted endpoints use native endpoints through sandbox-attached OpenShell providers. OpenShell mediates inference traffic and keeps provider credentials outside the sandbox. -New hosted custom endpoints outside the Personal tier use sandbox-attached providers. A fresh Personal-tier hosted selection, custom endpoints without an attachment, owned Local NVIDIA NIM, and retained shared selections use `inference.local`. Qualified new Ollama, standalone vLLM, and llama.cpp selections also use attached providers. +Qualifying OpenClaw hosted compatible selections and new Ollama, standalone vLLM, and llama.cpp selections use sandbox-attached providers. +Remaining shared custom and local selections retain `inference.local` on OpenShell `0.0.x`. +Refer to [How Inference Routing Works](../inference/about-inference-routing) for qualification and existing-sandbox boundaries. -New Hermes hosted compatible selections outside the Personal tier, Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use sandbox-attached providers. +Qualifying Hermes hosted compatible selections, Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use sandbox-attached providers. The profile restricts the recorded host, port, methods, paths, and Hermes runtime binaries. OpenShell replaces the scoped credential reference at the allowed endpoint; the upstream credential remains outside the sandbox. -Owned Local NVIDIA NIM, nonqualifying custom endpoints, and retained legacy local routes continue to use `inference.local`. +Remaining custom shared selections and retained legacy local routes use `inference.local` on OpenShell `0.0.x`. Hermes live qualification for the new local lifecycle remains pending. Refer to [How Inference Routing Works](../inference/about-inference-routing) for the route boundary. -New Deep Agents Code sandboxes use attached providers for hosted compatible endpoints outside the Personal tier. -A fresh Personal-tier hosted compatible selection instead stays on `inference.local` without creating a native attachment. -Qualified Ollama, standalone vLLM, llama.cpp, and host-local compatible endpoints also use native providers. +Qualifying Deep Agents Code hosted compatible selections use sandbox-attached providers. +New Deep Agents Code sandboxes use native providers for qualified Ollama, standalone vLLM, llama.cpp, and host-local compatible endpoints. The local attached profile permits only `GET /v1/models` and `POST /v1/chat/completions` on the recorded host and port, using the listed Python and curl binaries. These local HTTP endpoints do not use TLS termination on port `443`. OpenShell replaces the sandbox's credential placeholder at the allowed endpoint. For a credentialless upstream, the protected local proxy removes the sentinel authorization header before forwarding the request. -Custom endpoints without a native attachment and owned Local NVIDIA NIM retain the shared `inference.local` route. +Remaining custom shared selections retain `inference.local` on OpenShell `0.0.x`. Existing DCode local sandboxes without native authority require explicit recreation; NemoClaw does not migrate them automatically. Before recreation, download the state you need to an independently managed location and verify that the download is complete. Follow [Transfer State Manually](../manage-sandboxes/state-and-backups/transfer-state-manually) before deleting the existing sandbox. diff --git a/docs/reference/pi-support.mdx b/docs/reference/pi-support.mdx index 2946fb40727..b227b2054d8 100644 --- a/docs/reference/pi-support.mdx +++ b/docs/reference/pi-support.mdx @@ -33,8 +33,8 @@ Pi is a NemoClaw agent runtime. Docker is the initial compute runtime. Pi-specif | Host platform | Linux AMD64 or Linux ARM64 | | Compute runtime | Docker | | Managed API | OpenAI Chat Completions | -| Sandbox route | Attached provider for NVIDIA, fixed hosted providers, new hosted custom endpoints outside Personal, and new qualified local selections; shared route for new Personal-tier hosted custom endpoints, Local NVIDIA NIM, and published legacy selections | -| Qualification model | `nvidia/nemotron-3.5-super-vl-120b-a12b` | +| Sandbox route | Native endpoint for NVIDIA, fixed hosted providers, qualifying custom hosted endpoints, and new qualified local selections; shared route for remaining custom shared selections and published legacy selections | +| Qualification model | `nvidia/nemotron-3.5-super-vl-120b-a12b` | | Interactive command | `pi` | | Headless command | `pi --no-approve --print` | @@ -67,15 +67,19 @@ If the settings update fails, NemoClaw restores the previous catalog or removes When NemoClaw applies a Pi configuration, existing `models.json` and `settings.json` paths must be regular files. A symbolic link at either path stops that update before the Pi generator runs. -For direct provider routes, the upstream credential remains in OpenShell provider state. -It survives rebuild and sandbox destruction until an operator runs `nemoclaw credentials reset --yes` after all dependent sandboxes are gone. -NVIDIA Endpoints, supported fixed hosted providers, new hosted custom endpoints outside the Personal policy tier, and new qualified local selections use sandbox-attached providers. -For direct hosted custom endpoints, OpenShell binds the recorded endpoint, API, credential environment name, and admitted destination addresses to that provider. -The AWS Bedrock adapter keeps AWS credentials in its host process and gives OpenShell an adapter token instead of direct endpoint IP pins. -For an attached hosted custom selection, Pi's generated `models.json` reads `${COMPATIBLE_API_KEY}` or `${COMPATIBLE_ANTHROPIC_API_KEY}` from its sandbox environment; the environment value is a scoped `openshell:resolve:env:...` reference. -Shared selections use a non-secret managed-route placeholder in `models.json`. -OpenShell supplies the issued credential at egress without putting the upstream key in Pi's configuration. -New Personal-tier hosted custom selections, Local NVIDIA NIM, and unchanged published Pi shared-route selections use `inference.local`. +For direct provider routes, the upstream credential remains in OpenShell provider state. It survives rebuild and sandbox destruction until an operator runs `nemoclaw credentials reset --yes` after all dependent sandboxes are gone. NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. New Pi selections for Ollama, standalone vLLM, llama.cpp, and qualified host-local compatible endpoints use sandbox-attached providers. +Qualifying custom hosted endpoints also use sandbox-attached providers. +For an attached hosted custom selection, `models.json` reads `${COMPATIBLE_API_KEY}` or `${COMPATIBLE_ANTHROPIC_API_KEY}` from the sandbox environment. +That environment value is a scoped `openshell:resolve:env:...` reference. +OpenShell supplies the direct provider credential or Bedrock adapter token at egress. +Pi configuration contains a non-secret placeholder or environment reference. +Bedrock AWS credentials stay in its host-local adapter. +Remaining custom shared selections and unchanged published Pi shared-route selections retain `inference.local`. +Those shared routes require OpenShell `0.0.x`; OpenShell `0.1.5` removed `inference.local`. +On OpenShell `0.1.7`, NemoClaw refuses to resume or rebuild a retained Pi local selection without a native attachment. +Back up required state before recreating the sandbox with `nemoclaw onboard --agent pi --name --fresh --recreate-sandbox`. +NemoClaw preserves legacy volumes and does not automatically copy their data. +Pi native local onboarding on `0.1.7` remains unqualified for this upgrade. A failed native selection stops without falling back to the shared route. Direct unmanaged provider access is denied. NemoClaw rejects an unsupported API family, empty model, credential-bearing base URL, or malformed model tuning before Pi starts. For OpenRouter, Pi uses `https://openrouter.ai/api/v1` through its recorded native provider. Readiness requires a successful bounded inference request for the selected model. @@ -95,6 +99,10 @@ A rebuild replays the recorded values. Pi does not support `NEMOCLAW_REASONING_E The baseline network policy permits only the managed inference endpoint and the required Chat Completions routes. Network capability is limited to the root-owned Pi and Node runtime binaries. +For an OpenShell `0.1.7` native provider route, Pi startup and managed shells clear inherited proxy and bypass variables. +OpenShell mediates DNS and applies destination policy. +`NEMOCLAW_PROXY_HOST` and `NEMOCLAW_PROXY_PORT` do not configure this Pi route. + Direct access to provider endpoints, GitHub, npm, host control, and container-runtime sockets is denied. Operators must select an explicit maintained policy before additional services become reachable. Pi runs as the unprivileged `sandbox` user. Landlock strict mode and the OpenShell filesystem policy protect the image and host boundary. Writable paths are limited to `/sandbox`, `/sandbox/.pi`, `/tmp`, and `/dev/null`. diff --git a/docs/reference/platform-support.mdx b/docs/reference/platform-support.mdx index 3edf74d27e8..7dcba59e9d6 100644 --- a/docs/reference/platform-support.mdx +++ b/docs/reference/platform-support.mdx @@ -65,7 +65,7 @@ Pick the matching onboarding entry point for each agent. |-------|--------|---------|-------| | OpenClaw | Tested | Yes | Default agent runtime. Onboard with `nemoclaw onboard` (no `--agent` flag required). | | Hermes | Tested | No | First-class agent with dedicated CLI (`nemohermes`), Dockerfile, manifest, docs, and Hermes E2E coverage in `.github/workflows/e2e.yaml` (`hermes-e2e`, `hermes-*`, and Hermes rebuild/switch jobs). Onboard with `nemohermes onboard` or `nemoclaw onboard --agent hermes`. Unlocks the Hermes Provider inference route. Known structural gaps: model-provider compatibility registry is empty (backfilled after failures, see `nemoclaw-blueprint/model-specific-setup/hermes/README.md`); no Hermes-specific unit tests in `nemoclaw/src/`; macOS and WSL CI suites do not differentiate agents. Suitable for evaluation and the documented onboarding paths; production parity with OpenClaw is not yet asserted. | -| LangChain Deep Agents Code | Tested | No | Terminal-oriented coding agent runtime with no in-sandbox gateway or dashboard, built on the Deep Agents SDK. Its manifest is `agents/langchain-deepagents-code/manifest.yaml`, and its binary is `dcode`. Onboard with `nemo-deepagents onboard` or `nemoclaw onboard --agent langchain-deepagents-code`, and follow [the quickstart](/user-guide/deepagents/get-started/quickstart). NemoClaw runs it as a managed agent runtime: unmanaged sandbox, Model Context Protocol (MCP) server, and shell overrides are rejected, and credential-bearing proxy URLs are dropped from persisted shell environment values. NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. New Ollama, standalone vLLM, llama.cpp, hosted compatible outside the Personal tier, and qualifying host-local compatible selections use sandbox-attached providers. A fresh Personal-tier hosted compatible selection uses `inference.local` without creating a native attachment. Owned Local NVIDIA NIM, other compatible endpoints without an attachment, and retained shared local selections also use `inference.local`. Live runtime acceptance, broader launch material, and terminal-agent diagnostics are tracked at open issue #4861. | +| LangChain Deep Agents Code | Tested | No | Terminal-oriented coding agent runtime with no in-sandbox gateway or dashboard, built on the Deep Agents SDK. Its manifest is `agents/langchain-deepagents-code/manifest.yaml`, and its binary is `dcode`. Onboard with `nemo-deepagents onboard` or `nemoclaw onboard --agent langchain-deepagents-code`, and follow [the quickstart](/user-guide/deepagents/get-started/quickstart). NemoClaw runs it as a managed agent runtime: unmanaged sandbox, Model Context Protocol (MCP) server, and shell overrides are rejected, and credential-bearing proxy URLs are dropped from persisted shell environment values. NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. New Ollama, standalone vLLM, llama.cpp, and qualifying local compatible selections use sandbox-attached providers for qualified host-local endpoints. On OpenShell `0.1.7`, native local onboarding also requires the provider-policy setup and gateway initialization migration in [NemoClaw#13201](https://github.com/NVIDIA/NemoClaw/issues/13201); it is not qualified for this upgrade. Owned Local NVIDIA NIM uses a sandbox-attached provider for its recorded container endpoint. Retained shared custom or local selections require recreation on OpenShell `0.1.7`; legacy data remains available for recovery. Live qualification of the upgrade remains pending. Live runtime acceptance, broader launch material, and terminal-agent diagnostics are tracked at open issue #4861. | {/* agent-status:end */} ## Platforms @@ -107,7 +107,7 @@ NemoClaw routes inference through the OpenShell gateway. Each row below is a pro | Local llama.cpp (managed) | Experimental | Local OpenAI-compatible | Uses repository-owned YAML recipes and exact GGUF verification. The selected runtime-provider boundary owns lifecycle; all currently qualified profiles require Docker. Qualified DGX Spark hosts offer NVIDIA Nemotron 3 Nano 30B-A3B. Qualifying Windows WSL N1x and Station GB300 hosts automatically use managed llama.cpp during Windows Express or non-interactive onboarding when no provider is requested or recorded and `NEMOCLAW_MODEL` is unset. An unset recipe selects Qwen 3.6 35B-A3B; a compatible explicit recipe selects that managed recipe. The WSL profiles remain pending physical agent qualification. | | Local NVIDIA NIM | Experimental | Local OpenAI-compatible | Requires `NEMOCLAW_EXPERIMENTAL=1` and a NIM-capable NVIDIA GPU. Host must have the NVIDIA Container Toolkit installed and a healthy CDI spec. Onboarding evaluates the canonical `host.gpu.nvidia_available`, `host.gpu.container_toolkit_available`, and `host.gpu.cdi_healthy` readiness capabilities before gateway, image, or sandbox lifecycle effects. NIM images pull from `nvcr.io` and require NGC registry login. NemoClaw gates this path behind the experimental flag. Onboarding filters the validated image list by usable memory. In non-interactive mode, onboarding selects the first compatible model unless the operator sets `NEMOCLAW_MODEL`. Model selection uses detected free GPU memory. If free memory is unavailable, NemoClaw uses total GPU memory. On DGX Spark, NemoClaw caps usable memory at 50 percent of total memory to match NIM's reported unified-memory limit. The Nemotron 3 Super catalog minimum includes NIM's reported runtime allocation. Local NVIDIA NIM is unavailable on N1x. NemoClaw omits the provider from onboarding and rejects `NEMOCLAW_PROVIDER=nim-local` on N1x. Select another standard provider or use the Deferred managed-vLLM preview on N1x. On Linux arm64 DGX Spark and DGX Station hosts, onboarding warns that some NIM images may not publish a `linux/arm64` manifest; the warning is advisory, and the selected image pull can still fail when the registry has no matching platform manifest. Managed vLLM has host-specific default models and is not gated on the same boxes. Catalogued Local NIM images in `bin/lib/nim-images.json` include `nvidia/nemotron-3-super-120b-a12b`, `nvidia/nemotron-3-nano-30b-a3b`, and `nvidia/llama-3.3-nemotron-super-49b-v1`. | | Local vLLM (already running) | Tested with limitations | Local OpenAI-compatible | On a qualified N1x host, this route requires explicit standard-onboarding intent and remains unvalidated. On other hosts, no explicit intent is required. It appears in the onboarding menu when NemoClaw detects a server on `localhost:${NEMOCLAW_VLLM_PORT:-8000}`. The model is whatever the existing server serves. | -| Local vLLM (managed install/start) | Tested with limitations | Local OpenAI-compatible | Managed install/start requires Docker on `PATH`; attaching an already-running local vLLM server is a separate path. Appears by default on DGX Spark and qualifying DGX Station GB300 hosts. A labeled Deferred preview appears on hosts that match the N1x identity requirements; N1x remains Deferred until a physical NemoClaw Express E2E test passes. DGX Station is Tested with limitations across qualified profiles on one physical DGX Station GB300. Full Station Express end-to-end qualification for the accepted no-OTA DGX OS `7.6.x` profile is pending. Dual-Station configurations are not yet validated, and dedicated CI coverage is not available. For canonical Station qualification and host preparation, see the Additional Setup page for [OpenClaw](/user-guide/openclaw/get-started/additional-setup/dgx-station-preparation), [Hermes](/user-guide/hermes/get-started/additional-setup/dgx-station-preparation), or [Deep Agents](/user-guide/deepagents/get-started/additional-setup/dgx-station-preparation). Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`, NVIDIA Container Toolkit, and CDI. NemoClaw pins runtime images to immutable digests. New single-Station Express installs default to the experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`; a qualified trusted pair selects distributed Ultra, and `--station-deepseek` selects `deepseek-ai/DeepSeek-V4-Flash`. Direct managed-vLLM defaults are `nvidia/Qwen3.6-35B-A3B-NVFP4` on DGX Spark and the Deferred N1x preview, `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` on DGX Station, and `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8` on generic Linux NVIDIA GPU hosts. The pinned `nvcr.io/nvidia/vllm` images are public, so Docker pulls them without NGC registry login. | +| Local vLLM (managed install/start) | Tested with limitations | Local OpenAI-compatible | Managed install/start requires Docker on `PATH`; attaching an already-running local vLLM server is a separate path. Appears by default on DGX Spark and qualifying DGX Station GB300 hosts. A labeled Deferred preview appears on hosts that match the N1x identity requirements; N1x remains Deferred until a physical NemoClaw Express E2E test passes. DGX Station is Tested with limitations across qualified profiles on one physical DGX Station GB300. Full Station Express end-to-end qualification for the accepted no-OTA DGX OS `7.6.x` profile is pending. Dual-Station configurations are not yet validated, and dedicated CI coverage is not available. For canonical Station qualification and host preparation, see the Additional Setup page for [OpenClaw](/user-guide/openclaw/get-started/additional-setup/dgx-station-preparation), [Hermes](/user-guide/hermes/get-started/additional-setup/dgx-station-preparation), or [Deep Agents](/user-guide/deepagents/get-started/additional-setup/dgx-station-preparation). Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`, NVIDIA Container Toolkit, and CDI. NemoClaw pins runtime images to immutable digests. New single-Station Express installs default to the experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`; a qualified trusted pair selects distributed Ultra, and `--station-deepseek` selects `deepseek-ai/DeepSeek-V4-Flash`. Direct managed-vLLM defaults are `nvidia/Qwen3.6-35B-A3B-NVFP4` on DGX Spark and the Deferred N1x preview, experimental `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` on DGX Station, and `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8` on generic Linux NVIDIA GPU hosts. The pinned `nvcr.io/nvidia/vllm` images are public, so Docker pulls them without NGC registry login. | {/* provider-status-full:end */} ## Messaging Integrations @@ -135,7 +135,7 @@ Each row below is a launch-facing capability claim that NemoClaw makes in docs, |------------|--------|-------| | Guided onboarding | Tested | Single-command interactive wizard (`$$nemoclaw onboard`) that walks the user through inference provider selection, credential setup, and sandbox creation or update. It opens the agent dashboard when the selected runtime provides one. Non-interactive mode is supported with `--non-interactive` and `NEMOCLAW_*` environment variables for CI and scripted installs. | | Sandboxed execution | Tested with limitations | Landlock, seccomp, network namespace isolation, no-new-privileges, privilege dropping, and process limits (nproc 512 at `scripts/lib/sandbox-rlimits.sh:8`) are tested and on by default. OpenShell owns capability enforcement for managed entrypoints, exec processes, and connect shells; those paths do not call NemoClaw capability handling. Direct container runtimes that explicitly override the image user to root retain the existing compatibility drop and required `setpriv` transition. Direct container launches must configure capabilities at the container runtime. | -| Managed inference | Tested | Provider-routed model calls through OpenShell, transparent to the agent. NVIDIA Endpoints, supported fixed hosted providers, and new hosted compatible selections outside the Personal tier use sandbox-attached providers. Qualified new local Ollama, standalone vLLM, llama.cpp, and host-local compatible selections also use attached providers where the agent supports them. New OpenClaw Model Router selections use attached providers at the managed host endpoint. A fresh Personal-tier hosted compatible selection, owned Local NVIDIA NIM, compatible endpoints without an attachment, other Model Router selections, and retained shared selections use `inference.local`. Pi native routing remains gated by its activation status; Hermes live qualification for its new local lifecycle remains pending. Bedrock uses a host-local adapter for AWS credentials. Provider credentials remain outside the sandbox. Supports every entry in the Providers table. | +| Managed inference | Tested | Provider-routed model calls through OpenShell, transparent to the agent. NVIDIA Endpoints, supported fixed hosted providers, and qualifying custom hosted endpoints use native endpoints through sandbox-attached OpenShell providers. New Deep Agents Code, Hermes, OpenClaw, and Pi selections of Ollama, standalone vLLM, and llama.cpp use sandbox-attached providers for their qualified host-local endpoints. Deep Agents Code, Hermes, OpenClaw, and Pi also use attachments for qualifying host-local compatible endpoints. New Model Router selections for these agents use attached providers at the managed endpoint configured by the routed blueprint. Live Model Router qualification remains pending. Native local onboarding on OpenShell `0.1.7` remains unqualified for this upgrade; Hermes live qualification for its new local lifecycle is pending. Owned Local NVIDIA NIM uses a sandbox-attached provider for its recorded container endpoint. Retained shared custom and local selections use `inference.local` on OpenShell `0.0.x`; they require recreation on `0.1.7`, which also lacks the route removed in `0.1.5`. Legacy data remains available for recovery. Bedrock uses a host-local adapter for AWS credentials. OpenShell keeps provider credentials outside the sandbox. The Providers table remains the required support matrix. New OpenClaw local attachments use the recorded provider endpoint for `status`, `connect`, and `start` health checks. | | Declarative network policy | Tested | YAML-defined egress with policy presets. Presets include `slack`, `discord`, `telegram`, `weather`, `openclaw-pricing`, `huggingface`, `npm`, `pypi`, `brew`, and others. Hot-reloadable at runtime with `$$nemoclaw policy add`. | | State persistence and rebuild transfer | Tested | OpenShell preserves the native agent home during routine lifecycle operations. Before destructive replacement, NemoClaw archives and integrity-checks the complete native home/workspace, then restores it into the replacement sandbox. | | Agent skills | Tested | Packaged agent skills are discoverable by Cursor, Claude Code, and other coding assistants under `.agents/skills/`. Skills also install into the sandbox with `$$nemoclaw skill install`. | diff --git a/docs/reference/troubleshoot-mcp-servers.mdx b/docs/reference/troubleshoot-mcp-servers.mdx index 1fc1e709520..ec58dc6031b 100644 --- a/docs/reference/troubleshoot-mcp-servers.mdx +++ b/docs/reference/troubleshoot-mcp-servers.mdx @@ -55,8 +55,13 @@ If restart reports a missing provider, remove the native registration and re-add ## Provider Uses the Legacy Generic Profile -If restart or rebuild reports that the provider uses the profile-less legacy `generic` type, remove that server and add it again with the recorded credential variable exported. -OpenShell cannot bind that provider to an MCP endpoint, and NemoClaw accepts it only for cleanup. +The legacy `generic` type and the older `nemoclaw-mcp-v1` profile do not declare the credential required by OpenShell 0.1.7. +If restart or rebuild reports either type, remove the server and add it again with the recorded credential variable exported. +Preserve its URL, denied-tool selectors, and trusted-private declaration. +NemoClaw accepts a matching legacy provider only for cleanup and preserves its provider object. +Follow [Rotate a Credential](../manage-sandboxes/mcp-servers/manage-mcp-servers#rotate-a-credential) before deleting a retained provider. +After re-adding the server, run `$$nemoclaw mcp status --json`. +Verify that the provider and adapter are ready; treat an inconclusive credential-resolution probe as unverified. ## Rebuild Cannot Prove a Credential Handle @@ -409,7 +414,11 @@ For a host-local server, keep the MCP process on loopback when practical. Place an operator-managed HTTPS reverse proxy on a stable private address that the OpenShell gateway can route to. The certificate must match the original endpoint hostname and chain to a trust root available to the managed runtime. For a private CA, set `NEMOCLAW_CORPORATE_CA_BUNDLE` before onboarding a new sandbox. -The managed image installs that CA for both the OpenShell upstream proxy and sandbox TLS clients. +The managed image installs the CA for sandbox TLS clients. +With OpenShell `0.1.7`, NemoClaw also configures the gateway's network supervisors with the validated CA for destination TLS verification. +An existing supervisor can retain an earlier bundle, so a gateway configuration change alone does not establish recovery. +Follow [Supervisor Destination Trust](../security/configure-corporate-ca-trust#understand-supervisor-destination-trust) for recreation and verification requirements. +Private-CA MCP runtime qualification remains pending. Restrict the proxy listener to the OpenShell gateway or required deployment subnet with host firewall rules. `--trusted-private-host` permits a private destination; it does not install a CA or disable certificate validation. diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index 5656a2179c3..34c0d0a0d4d 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -488,7 +488,10 @@ Use `NEMOCLAW_GATEWAY_BIND_ADDRESS=0.0.0.0` only on supported gateway modes and ### Older-glibc gateway compatibility container -OpenShell 0.0.116 directly supports Linux hosts with glibc 2.39 or newer. On an older trusted host, `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=1` explicitly opts into NemoClaw's compatibility container. Leave it unset on supported hosts. +NemoClaw compares the installed OpenShell gateway binary's required glibc version with the Linux host's glibc version. +If the host's version is older, startup stops and requires an explicit compatibility choice. +On a trusted host, `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=1` opts into NemoClaw's compatibility container. +Leave it unset when the host meets the installed binary's requirements. The compatibility container uses host networking and mounts the host Docker socket read-only. A read-only socket mount still permits privileged Docker API operations and can control the host, so do not enable this mode on an untrusted or shared host. The gateway remains loopback-bound, and startup fails closed unless the configured Unix socket answers as a Docker daemon. See [Gateway Compatibility Container](../security/security-controls/gateway-authentication-controls#gateway-compatibility-container) for the container boundary and removal conditions. @@ -532,6 +535,11 @@ A gateway port that keeps listening after you uninstall, or that keeps serving a Onboarding under a non-default `NEMOCLAW_GATEWAY_PORT` registers the sandbox on gateway `nemoclaw-` and stores its registry and state under `~/.nemoclaw/gateways//`. Gateway-lifecycle commands such as `$$nemoclaw onboard`, `$$nemoclaw uninstall`, and the deprecated `$$nemoclaw stop` act on the state root that their own `NEMOCLAW_GATEWAY_PORT` selects, while sandbox-scoped commands that read, exec into, or inspect a sandbox, such as `$$nemoclaw exec` and `$$nemoclaw status`, resolve the named sandbox across every per-port registry root and address its recorded gateway binding. A gateway-lifecycle command without that variable uses port `8080` unless the installer recorded an automatic alternate-port selection; in that case, the CLI restores the recorded port. Clearing an explicitly supplied variable does not move an existing sandbox back to the default port. +`status` prints `Recorded gateway` from the sandbox registry: the gateway name, a recorded port when no name exists, or `not recorded`. +`recover` and `connect --probe-only` print `Target gateway` for the selected target. +Use these labels to identify the gateway involved in a failed operation. +Neither label proves gateway reachability; `Target gateway` alone does not establish registered sandbox ownership. + List the gateways the host still has: ```bash @@ -591,24 +599,27 @@ If the error names an invalid gateway binding, restore the affected row's known- ## Onboarding -### Enable Native Local Inference on a Reused Gateway +### Resume Native Local Inference on a Reused Gateway -If onboarding reports that native local inference requires `providers_v2_enabled=true`, ask the gateway administrator to review attached providers and run the command printed in the error. -For a gateway named `nemoclaw`, that command is: +OpenShell `0.1.7` composes attached-provider policies automatically when no active global policy overrides them. +The `providers_v2_enabled` setting no longer exists. +If onboarding reports a global policy override, ask the gateway administrator to review it before resuming. +For a gateway named `nemoclaw`, inspect its policy history with: ```bash -openshell settings set -g nemoclaw --global --key providers_v2_enabled --value true --yes +openshell policy list -g nemoclaw --global --page-size 1 --output json ``` -NemoClaw does not change this setting on an existing gateway. -After the administrator enables it, resume the failed onboarding session: +NemoClaw requires verified empty history or a superseded current global revision. +It does not remove an override or change gateway settings. +After the administrator resolves the reported policy condition, resume the failed onboarding session: ```bash $$nemoclaw onboard --resume ``` Resume reruns native local inference setup when NemoClaw has no recorded native local provider for this sandbox, even if another sandbox already has the same shared gateway route. -If the setting is still missing, onboarding stops before creating the sandbox and retains the session for another attempt. +An active override or an unverified policy response stops onboarding before sandbox creation and retains the session for another attempt. ### Ollama installer download fails @@ -905,7 +916,12 @@ Rerunning onboarding with a different provider recreates the sandbox because the ### Sandbox containers cannot reach the gateway -On native Linux Docker-driver hosts, `$$nemoclaw onboard` verifies the route that sandbox containers use to reach the OpenShell gateway. If a host firewall blocks that path, onboarding exits with output like: +OpenShell `0.1.7` Docker supervisors use host networking by default, and their workloads have no direct network interface. +Onboarding skips the legacy bridge-to-gateway probe for this runtime, even when an old `openshell-docker` network remains. +This check does not remove the retained network or change firewall rules. + +For runtimes that use bridge-based gateway access, a host firewall can block the route. +Apply the following guidance only when onboarding reports this bridge diagnostic: ```text ✗ Sandbox containers cannot reach the gateway at host.openshell.internal:8080. @@ -1737,8 +1753,10 @@ credential projection, but must not add the retired NemoClaw gateway controller. ### Hermes config reports a mutation already in progress -Hermes host config writes and lifecycle seals use the same root-only mutation lock. -A host-side config write binds to the SHA-256 digest of the matching read, installs fresh config inodes, refreshes both config hashes, and restores mutable access before releasing that lock. +This diagnostic applies to legacy root-owned configuration and lifecycle transactions. +Current host-side Hermes `config set` uses a guarded transaction as the sandbox user through the recorded OpenShell gateway. +It binds the write to the matching read, refreshes the integrity hashes, and verifies a gateway reload; it does not use the legacy root-only seal. +For transaction failure and rollback checks, refer to [Hermes Runtime Changes](../manage-sandboxes/configure-sandboxes/understand-runtime-changes#hermes-runtime-changes). If a concurrent command reports `Hermes config mutation is already in progress`, do not remove `/run/nemoclaw/hermes-config-mutation.lock` manually. Wait for the active config, recovery, or restart command to finish, then retry. @@ -2025,7 +2043,7 @@ When you run a command through `$$nemoclaw exec -- ...` and it exits non- ```text curl: (56) CONNECT tunnel failed, response 403 $$nemoclaw: recent network policy denial detected for example.com:443 inside sandbox 'oc-fresh'. - The sandbox's egress policy blocked this request; the tool above only saw the proxy's 403. + Match the denial's destination and time to the failed request before changing policy. See the denied flow: $$nemoclaw oc-fresh logs --tail 50 Review applied presets: $$nemoclaw oc-fresh policy list Allow the host: $$nemoclaw oc-fresh policy add @@ -2035,6 +2053,9 @@ $$nemoclaw: recent network policy denial detected for example.com:443 inside san If the log probe fails, no breadcrumb is added. Unsafe endpoint data is omitted from the breadcrumb, and an invalid sandbox name is shown as ``. The breadcrumb reports a recent denial. Match its destination and time to the failed request before treating policy as the cause. +On OpenShell `0.1.7`, a fresh `NET:REFUSE` audit event with `policy_dns_ineligible` also produces the hint, without a destination. +A DNS resolution error alone does not confirm a policy denial. + An IPv6 target is named in its RFC 3986 bracketed form: ```text @@ -2054,6 +2075,8 @@ $$nemoclaw logs --tail 50 These logs combine gateway and OpenShell entries. Match the denial's destination and time to the failed request before adding a preset. A denial for another request does not confirm this request was blocked. If the last 50 lines have no match, inspect more log lines before deciding why the request failed. Reading sandbox logs, or checking for a policy denial after a failed `$$nemoclaw exec` when policy hints are enabled, attempts to enable OpenShell OCSF JSON audit logging for that sandbox. NemoClaw prints a notice before attempting to update `ocsf_json_enabled`. +OpenShell `0.1.7` emits Open Cybersecurity Schema Framework (OCSF) `1.8.0` JSON records. +Its schema-version selectors are deprecated and ignored; the `ocsf_json_enabled` switch and shorthand audit output remain available. To keep the current OpenShell audit setting, set `NEMOCLAW_NO_AUTO_AUDIT_LOGS=1` before running either command. Policy denial hints may be unavailable if audit logging is disabled. You can enable it explicitly with `openshell settings set --key ocsf_json_enabled --value true`. @@ -2201,11 +2224,29 @@ Verify that the inference provider endpoint is reachable from the host. Check th $$nemoclaw status ``` -On a normal status read with verified gateway and sandbox state, the main `Inference` line checks the provider path from inside the sandbox and sends an inference request over the reachable path the agent uses. If gateway or sandbox state is not verified, status reports `Inference: not verified (gateway/sandbox state not verified)` and sends no inference request. For shared-route providers, including owned Local NVIDIA NIM, `status` probes `https://inference.local/v1/models` before it sends the request through `https://inference.local`. For NVIDIA Endpoints, fixed hosted selections, new hosted compatible selections outside the Personal tier, and qualified new local selections with a recorded native attachment, it verifies that attachment and sends the request to the selected native endpoint; it does not probe `inference.local`. When an inference request returns a transient gateway status, `status` makes up to three request attempts before it reports a failure. Shared-route retries repeat the route probe and request, while native providers retry only the native request. It prints the next attempt and two-second delay to stderr before each retry. Each inference request attempt has a 90-second request timeout. If a verified native NVIDIA attachment returns HTTP `429`, `500`, `502`, `503`, or `504`, or a confirmed timeout after bounded retries, `status` reports `degraded` and exits `0`. The model response remains unverified. Run `$$nemoclaw connect --probe-only` to require a successful inference request before accepting readiness. If the main line shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop. An `unauthorized` line means the route answered but rejected the request, so refresh the provider credential rather than the route. - -If status reports `Inference: not probed (another lifecycle operation is in progress)`, the unlocked read skipped provider-path and inference probes; JSON output sets `inferenceProbeSuppressed: true`. Wait for the lifecycle operation to finish and run status again before diagnosing the route or credential. - -For Hermes native NVIDIA routes, status also checks the managed credential reference in the Hermes configuration. If this check reports an invalid or stale credential, run `$$nemoclaw rebuild --yes` to refresh it. Then rerun `$$nemoclaw status` and confirm inference is healthy. If status reports that the installed image lacks the credential verifier, rebuild to install it, then rerun status. For other unavailable-verification results, follow the printed sandbox health and access recovery guidance. +On a normal status read with verified gateway and sandbox state, the main `Inference` line checks the provider path from inside the sandbox. +It sends an inference request over the reachable path the agent uses. +If gateway or sandbox state is not verified, status reports `Inference: not verified (gateway/sandbox state not verified)` and sends no inference request. +For shared-route providers, `status` probes `https://inference.local/v1/models` before it sends the request through `https://inference.local`. +For NVIDIA Endpoints, fixed hosted selections, qualifying custom hosted endpoints, and new Deep Agents Code and Hermes Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections, plus new OpenClaw Ollama, standalone vLLM, and llama.cpp selections with a recorded native attachment, it verifies that attachment and sends the request to the selected native endpoint; it does not probe `inference.local`. +When an inference request returns a transient gateway status, `status` makes up to three request attempts before it reports a failure. +Shared-route retries repeat the route probe and request, while native providers retry only the native request. +It prints the next attempt and two-second delay to stderr before each retry. +Each inference request attempt has a 90-second request timeout. +If a verified native NVIDIA attachment returns HTTP `429`, `500`, `502`, `503`, or `504`, or a confirmed timeout after bounded retries, `status` reports `degraded` and exits `0`. +The model response remains unverified. +Run `$$nemoclaw connect --probe-only` to require a successful inference request before accepting readiness. +If the main line shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop. +An `unauthorized` line means the route answered but rejected the request, so refresh the provider credential rather than the route. + +If status reports `Inference: not probed (another lifecycle operation is in progress)`, the unlocked read skipped provider-path and inference probes; JSON output sets `inferenceProbeSuppressed: true`. +Wait for the lifecycle operation to finish and run status again before diagnosing the route or credential. + +For Hermes native NVIDIA routes, status also checks the managed credential reference in the Hermes configuration. +If this check reports an invalid or stale credential, run `$$nemoclaw rebuild --yes` to refresh it. +Then rerun `$$nemoclaw status` and confirm inference is healthy. +If status reports that the installed image lacks the credential verifier, rebuild to install it, then rerun status. +For other unavailable-verification results, follow the printed sandbox health and access recovery guidance. Hermes live qualification for the new local lifecycle remains pending. @@ -2331,7 +2372,10 @@ $$nemoclaw onboard --fresh --name --recreate-sandbox -OpenClaw onboarding sends a validation request over the selected inference path from inside the sandbox, even when you select no messaging channel. New hosted custom endpoints outside the Personal tier use the attached native provider; a fresh Personal-tier hosted selection and custom endpoints without an attachment use `inference.local`. +OpenClaw onboarding validates a custom OpenAI-compatible endpoint from inside the sandbox, even when you select no messaging channel. +For a qualifying custom hosted selection, validation uses its sandbox-attached native provider. +Custom loopback and `host.openshell.internal` selections do not qualify for that hosted path. +Retained shared-route selections use `inference.local` on OpenShell `0.0.x`; their migration to `0.1.7` is incomplete and blocks upgrade qualification. If that sandbox validation request fails, fix the compatible-endpoint base URL, credentials, model, or network route before continuing onboarding. Messaging setup is not the root cause. @@ -2644,25 +2688,49 @@ NemoClaw rejects an explicit custom endpoint when it resolves a public HTTPS hos Use an HTTPS IP-literal endpoint whose certificate is valid for that address. If your deployment permits non-TLS provider traffic, you can instead use a public HTTP endpoint that NemoClaw can rewrite to a DNS-pinned address. Do not bypass the check with a private or internal address or by editing the persisted sandbox config directly. For the full endpoint rules, refer to [Meet Custom Endpoint Security Requirements](../inference/custom-endpoints/custom-endpoint-security). -### Agent cannot reach external hosts through a proxy + -NemoClaw uses a default proxy address of `10.200.0.1:3128` (the OpenShell-injected gateway). If your environment uses a different proxy, set `NEMOCLAW_PROXY_HOST` and `NEMOCLAW_PROXY_PORT` before onboarding: +### Agent Cannot Reach External Hosts -```bash -export NEMOCLAW_PROXY_HOST=proxy.example.com -export NEMOCLAW_PROXY_PORT=8080 -$$nemoclaw onboard -``` + + +For managed Deep Agents Code on OpenShell `0.1.7`, check the sandbox network policy and recent denial logs when an external request fails. +The launchers clear inherited proxy variables, and OpenShell mediates the request. +Do not set `NEMOCLAW_PROXY_HOST` or `NEMOCLAW_PROXY_PORT` to repair this route. +Rebuild an older Deep Agents sandbox to install the updated launchers. + + + + +For managed OpenClaw on OpenShell `0.1.7`, check the sandbox network policy and recent denial logs when an external request fails. +OpenClaw starts without application proxy settings, and OpenShell mediates outbound requests. +Normal sandbox startup clears inherited proxy variables and removes obsolete global and Telegram proxy settings from restored configuration. +Do not set `NEMOCLAW_PROXY_HOST` or `NEMOCLAW_PROXY_PORT` to repair this route. +If the gateway already runs OpenShell `0.1.7`, rebuild the OpenClaw sandbox to remove previous proxy settings. +Rebuilding a registered sandbox on an OpenShell `0.0.x` gateway does not upgrade that gateway; follow [Update Sandboxes](../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before planning replacement. -These are build-time settings baked into the sandbox image. Changing them after onboarding requires re-running `$$nemoclaw onboard` to rebuild the image. + + + +For managed Hermes on OpenShell `0.1.7`, check the sandbox network policy and recent denial logs when an external request fails. +Hermes startup and managed connect shells clear inherited proxy variables, and OpenShell mediates outbound requests. +Do not set `NEMOCLAW_PROXY_HOST` or `NEMOCLAW_PROXY_PORT` to repair this route. +If the gateway already runs OpenShell `0.1.7`, rebuild the Hermes sandbox to install the updated startup and shell environment. +Rebuilding a registered sandbox on an OpenShell `0.0.x` gateway does not upgrade that gateway; follow [Update Sandboxes](../manage-sandboxes/operate-sandboxes/update-sandboxes#use-the-automatic-upgrade-path) before planning replacement. + +Corporate certificate authority (CA) trust remains separate from application proxy settings. +If an enterprise proxy re-signs TLS, follow [Configure Corporate CA Trust](../security/configure-corporate-ca-trust) before onboarding or rebuilding. +On a successful merge, Hermes appends the validated corporate CA without replacing the OpenShell trust bundle. -When `HTTP_PROXY` or `HTTPS_PROXY` is set on the host, NemoClaw adds `localhost`, `127.0.0.1`, `::1`, `0.0.0.0`, the container-host aliases `host.docker.internal` and `host.containers.internal`, and the managed inference hostname `inference.local` to `NO_PROXY` for host-side subprocesses and for the env forwarded into `openshell sandbox create`. This keeps local Ollama health checks, model pulls, and managed inference traffic from being chained through a corporate or desktop proxy at the sandbox-create boundary, while preserving the proxy for external hosts. For the local provider validation probe, NemoClaw removes `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` from the probe process and sets `NO_PROXY=*` instead. A host proxy therefore cannot answer for the local endpoint, including the `host.docker.internal` alias used for Windows-host Ollama. Inside the running sandbox, processes continue to use the OpenShell L7 proxy for `inference.local` so OpenShell's internal routing, DNS, and audit boundaries stay intact. +When `HTTP_PROXY` or `HTTPS_PROXY` is set on the host, NemoClaw adds `localhost`, `127.0.0.1`, `::1`, `0.0.0.0`, the container-host aliases `host.docker.internal` and `host.containers.internal`, and the managed inference hostname `inference.local` to `NO_PROXY` for host-side subprocesses and for the env forwarded into `openshell sandbox create`. This keeps local Ollama health checks, model pulls, and managed inference traffic from being chained through a corporate or desktop proxy at the sandbox-create boundary, while preserving the proxy for external hosts. For the local provider validation probe, NemoClaw removes `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` from the probe process and sets `NO_PROXY=*` instead. A host proxy therefore cannot answer for the local endpoint, including the `host.docker.internal` alias used for Windows-host Ollama. + + ### Agent cannot reach a host-side HTTP service When a sandbox needs to call an HTTP service running on the host, use the normal OpenShell network policy path. Expose the service on a host IP address that the OpenShell gateway can reach, create a custom NemoClaw policy preset for that IP and port, and apply it with `$$nemoclaw policy add --from-file`. The sandbox request then flows through the OpenShell proxy while NemoClaw preserves the existing live policy entries. -Do not rely on `host.docker.internal` or `host.openshell.internal` as a general-purpose host-service path. Those names may appear in the sandbox's `/etc/hosts`, but in OpenShell's sandbox network they are not guaranteed to point at a reachable host gateway. Bypassing the proxy with `--noproxy '*'` also bypasses network policy enforcement and audit. +Do not rely on `host.docker.internal` or `host.openshell.internal` as a general-purpose host-service path. Those names may appear in the sandbox's `/etc/hosts`, but in OpenShell's sandbox network they are not guaranteed to point at a reachable host gateway. First, make sure the host-side service listens on a non-loopback address. For example, a health endpoint on port `50001` should be reachable from the host IP, not only from `127.0.0.1`: @@ -2836,7 +2904,10 @@ Only an accepted Windows-host Ollama route uses direct Docker Desktop access to ### `host.docker.internal` does not reliably reach the host from the sandbox -Configuring an inference provider with a base URL like `http://host.docker.internal:11434/v1` does not reliably reach a host Ollama service from inside the OpenShell sandbox. OpenShell runs sandboxes inside a k3s network, where `host.docker.internal` is not a reliable host-service route. Depending on the platform, it may fail DNS resolution or resolve to an internal gateway/bridge address where the host's port `11434` is not forwarded. The sandbox then sees a DNS failure or `connection refused`: +Configuring an inference provider with a base URL like `http://host.docker.internal:11434/v1` does not reliably reach a host Ollama service from inside the OpenShell sandbox. +The sandbox runtime network determines how `host.docker.internal` reaches host services. +Depending on the platform, it may fail DNS resolution or resolve to an internal gateway/bridge address where the host's port `11434` is not forwarded. +The sandbox then sees a DNS failure or `connection refused`: ```bash getent hosts host.docker.internal @@ -2851,6 +2922,20 @@ A native Linux Docker bridge can return an address such as this example: With Docker Desktop WSL integration, the aliases can instead resolve to an IPv6 unique-local address in `fc00::/7`, such as `fdc4:f303:9324::254`. New native-local host-alias profiles admit private IPv4 and IPv6 unique-local addresses while retaining the selected hostname and port restrictions. Public, link-local, and loopback IPv6 addresses remain excluded. +Existing IPv4-only attachments retain their permissions during resume and rebuild. +Do not edit or delete an attached profile to add IPv6 access. + + +To replace an IPv4-only attachment, reselect the recorded local provider and model: + +```bash +$$nemoclaw inference set --provider --model +``` + +The command checks ownership, creates the replacement attachment, and verifies the route before reporting success. +If verification fails, follow the printed recovery guidance before retrying. + + A valid DNS result does not prove that the host service is reachable. In the IPv4 example, this request still fails if the host service is not forwarded to port `11434`: @@ -2906,7 +2991,7 @@ journalctl -k --since "24 hours ago" --no-pager | grep -Ei 'NVRM|OOM|out of memo ``` For a NemoClaw-managed Spark profile, derive the host port from the managed container's fixed `8000/tcp` mapping. -New containers publish that host port on `127.0.0.1` and the private `openshell-docker` bridge gateway, independently of bearer authentication. +New containers publish that host port only on `127.0.0.1`, independently of bearer authentication. Existing containers keep their published bindings until they are recreated. Managed route recovery rejects owned bearerless containers with unsafe bindings. To recreate the container, follow [Handle a Running vLLM Server](../inference/local-inference/set-up-vllm#handle-a-running-vllm-server) with the original model and port settings. @@ -3489,7 +3574,7 @@ Confirm a real Hermes search after rebuilding. Do not replace an OpenShell placeholder with a raw Langfuse key in `~/.hermes/.env`. That workaround bypasses the sandbox credential boundary. -OpenShell 0.0.116 resolves static credentials only at endpoints declared by their provider profile. +Since OpenShell 0.1.2, static credentials resolve only at endpoints declared by their provider profile. Older provider-backed Langfuse setups could rely on destination-independent placeholder rewriting, so they can stop working after a NemoClaw upgrade or sandbox rebuild even though the Hermes placeholder validator still accepts the value. For Langfuse Cloud, register the two keys through the checked-in endpoint profile and rebuild the sandbox: diff --git a/docs/security/best-practices.mdx b/docs/security/best-practices.mdx index 3aeee580ec4..7cf0b526f95 100644 --- a/docs/security/best-practices.mdx +++ b/docs/security/best-practices.mdx @@ -114,7 +114,7 @@ NemoClaw allows a credential source only in the process that is intended to hold | Execution path | Rule | |---|---| -| Host-brokered provider | Keep provider credentials outside the sandbox. NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible endpoints outside the Personal tier, and qualified new local selections use sandbox-attached providers where supported. A fresh Personal-tier hosted compatible selection, owned Local NVIDIA NIM, compatible endpoints without an attachment, and retained shared routes use `inference.local`. The Personal selection creates no native provider attachment. Bedrock AWS credentials stay in the host-local adapter. The sandbox receives no raw host credential. | +| Host-brokered provider | Keep provider credentials outside the sandbox. Bedrock AWS credentials stay in its host-local adapter. NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, and qualifying local selections use sandbox-attached native providers. Remaining shared selections retain `inference.local` on OpenShell `0.0.x`. The sandbox receives no raw host credential. | | Sandbox-direct provider | Keep only credential sources that are deliberately provisioned and reachable inside the sandbox. | | Ambient metadata provider | Disable SDK discovery when the metadata endpoint cannot exist across the sandbox boundary. Do not make an internal or link-local endpoint reachable merely to satisfy an automatic probe. | | New agent or provider | Declare the enabled credential sources and add agent-specific tests proving that unavailable sources are not probed. | @@ -145,9 +145,9 @@ NemoClaw's host-local Amazon Bedrock adapter runs outside this sandbox credentia #### Hermes Managed Inference Credential Discovery NemoClaw configures Hermes with a custom provider. -NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible endpoints outside the Personal tier, and qualifying local selections use sandbox-attached OpenShell providers. +NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, and new qualifying local selections use sandbox-attached OpenShell providers. Hermes receives a scoped credential reference; OpenShell keeps the upstream credential outside the sandbox. -Owned Local NVIDIA NIM, nonqualifying custom endpoints, and retained legacy routes use `inference.local`. +Remaining custom shared selections and retained legacy routes use `inference.local` on OpenShell `0.0.x`. Hermes live qualification for the new local lifecycle remains pending. The shipped Hermes image does not install the optional native Bedrock dependency. AWS credential discovery for supported Bedrock inference therefore remains in NemoClaw's host-local adapter, outside the sandbox. @@ -159,9 +159,9 @@ If direct Hermes Bedrock support is added later, it must apply the same unavaila #### Deep Agents Managed Inference Credential Discovery NemoClaw configures the managed `dcode` runtime with a generated OpenAI-compatible route. -New NVIDIA Endpoints, supported fixed hosted providers, hosted compatible endpoints outside the Personal tier, Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections use sandbox-attached providers. -A fresh Personal-tier hosted compatible selection uses `inference.local` without creating a native attachment. -Owned Local NVIDIA NIM, other compatible endpoints without an attachment, and retained shared routes also use `inference.local`. +New NVIDIA Endpoints, supported fixed hosted providers, Ollama, standalone vLLM, llama.cpp, and qualifying local compatible selections use sandbox-attached native providers. +Qualifying custom hosted endpoints also use sandbox-attached providers. +Remaining shared selections retain `inference.local` on OpenShell `0.0.x`. The sandbox receives a scoped credential reference for native providers or a placeholder key for shared routes. OpenShell keeps the upstream provider credential outside the sandbox. The managed launchers reject credential-shaped environment values and upstream auth files before Deep Agents Code starts, so provider discovery stays on the host-side OpenShell boundary. @@ -189,7 +189,7 @@ Endpoint rules restrict allowed HTTP methods and URL paths. | Aspect | Detail | |---|---| -| Default | Some endpoints allow GET and POST on `/**` (for example, `clawhub.ai`). Others restrict methods and paths to specific API routes. The sandbox-attached NVIDIA inference provider allows POST only to `/v1/chat/completions` and GET only to `/v1/models`; it does not permit embedding requests. Read-only endpoints such as `docs.openclaw.ai`, the OpenClaw `npm_registry` baseline entry, and the `pypi` preset allow GET only (PyPI also allows HEAD). The broader `npm` preset is an intentional exception: npm/Yarn registry traffic uses L4 pass-through for Node 22 undici CONNECT compatibility. While that preset is active, NemoClaw temporarily aligns the overlapping baseline npm endpoint with the same reviewed L4 metadata because OpenShell 0.0.116 requires a single connection mode. | +| Default | Some endpoints allow GET and POST on `/**` (for example, `clawhub.ai`). Others restrict methods and paths to specific API routes. The sandbox-attached NVIDIA inference provider allows POST only to `/v1/chat/completions` and GET only to `/v1/models`; it does not permit embedding requests. Read-only endpoints such as `docs.openclaw.ai`, the OpenClaw `npm_registry` baseline entry, and the `pypi` preset allow GET only (PyPI also allows HEAD). The broader `npm` preset is an intentional exception: npm/Yarn registry traffic uses L4 pass-through for Node 22 undici CONNECT compatibility. While that preset is active, NemoClaw temporarily aligns the overlapping baseline npm endpoint with the same reviewed L4 metadata. OpenShell 0.1.7 rejects overlapping selectors with conflicting connection metadata, including TLS inspection mode. | | What you can change | Add methods (PUT, DELETE, PATCH) or restrict paths to specific prefixes. | | Risk if relaxed | Allowing all methods on an API endpoint gives the agent write and delete access. For example, allowing DELETE on `api.github.com` lets the agent delete repositories. | | Recommendation | Use GET-only rules for endpoints that the agent only reads. Add write methods only for endpoints where the agent must create or modify resources. Restrict paths to specific API routes when possible. | @@ -344,7 +344,7 @@ The agent never receives the provider API key. | Aspect | Detail | |---|---| -| Default | NVIDIA Endpoints, supported fixed hosted providers, new hosted compatible endpoints outside the Personal tier, and qualified new local selections use sandbox-attached providers where supported. A fresh Personal-tier hosted compatible selection, owned Local NVIDIA NIM, compatible endpoints without an attachment, and retained shared routes use `inference.local`; the Personal selection creates no native provider attachment. Provider credentials stay outside the sandbox. Bedrock AWS credentials stay in the host-local adapter. | +| Default | NVIDIA Endpoints, supported fixed hosted providers, qualifying custom hosted endpoints, and qualifying local selections use sandbox-attached native providers. Remaining shared selections retain `inference.local` on OpenShell `0.0.x`. OpenShell owns the direct provider credential and upstream endpoint. Bedrock AWS credentials stay in its host-local adapter. | | What you can change | You select a supported provider. NemoClaw selects and enforces that provider's managed path. | | Risk if bypassed | Adding an unowned provider endpoint to network policy bypasses NemoClaw's managed provider boundary and could let an agent use a stolen or hardcoded key. The NVIDIA Endpoints path is not a bypass: its checked-in OpenShell provider profile restricts the endpoint, methods, paths, and binaries, then rewrites a placeholder credential at the egress boundary. | | Recommendation | Do not add inference provider hosts to network policy for NemoClaw model traffic. Use the provider path created by onboarding. The `claude-code` preset is a separate opt-in exception for running the Claude Code CLI with its own credentials, not a way to configure NemoClaw inference. | diff --git a/docs/security/configure-corporate-ca-trust.mdx b/docs/security/configure-corporate-ca-trust.mdx index 670bd765d41..0f61f92e7b6 100644 --- a/docs/security/configure-corporate-ca-trust.mdx +++ b/docs/security/configure-corporate-ca-trust.mdx @@ -3,8 +3,8 @@ # SPDX-License-Identifier: Apache-2.0 title: "Configure Corporate Certificate Authority Trust" sidebar-title: "Configure Corporate CA Trust" -description: "Import a bounded corporate proxy CA chain into NemoClaw sandbox runtime trust and supported build-time flows." -description-agent: "Configures corporate proxy CA trust for runtime TLS and supported build-time operations. Use when a corporate MITM proxy re-signs external TLS or certificate verification fails behind an enterprise proxy." +description: "Configure corporate CA trust for sandbox TLS clients, supervisor destinations, and supported build-time flows." +description-agent: "Configures corporate CA trust for sandbox TLS clients, supervisor destinations, and supported build-time operations. Use when an enterprise proxy re-signs TLS or a managed endpoint uses a private CA." keywords: ["nemoclaw corporate ca", "corporate proxy tls", "certificate verification"] content: type: "how_to" @@ -18,6 +18,13 @@ NemoClaw imports a bounded corporate CA chain without replacing the OpenShell tr Point NemoClaw at a PEM bundle that contains your corporate root and any required intermediates. + +With OpenShell `0.1.7`, NemoClaw also configures these roots for destination TLS verification by the gateway's network supervisors. +This trust applies across that gateway, rather than only to one MCP endpoint or sandbox. +Import only CA roots approved for every sandbox that uses the gateway. +Network policy, destination pins, and TLS hostname checks still apply. + + ```bash export NEMOCLAW_CORPORATE_CA_BUNDLE=/path/to/corporate-ca.pem $$nemoclaw onboard @@ -32,7 +39,11 @@ For an existing sandbox, use the [explicit recreation command `NEMOCLAW_CORPORAT NemoClaw validates the selected bundle and bakes it into the sandbox image as `NEMOCLAW_CORPORATE_CA_B64`. The managed Dockerfile decodes it to the root-owned, read-only file `/usr/local/share/nemoclaw/corporate-ca.pem`. It also installs each validated certificate as a separate operating-system trust anchor and refreshes the OS trust bundle. -This lets the in-sandbox OpenShell proxy validate TLS when it opens the upstream connection to an HTTPS inference, MCP, or custom-policy endpoint signed by that CA. +Sandbox TLS clients can use these anchors. +OpenShell verifies destination TLS in a separate network supervisor. +Installing a CA in the workload image alone does not establish supervisor trust. +For OpenShell `0.1.7`, NemoClaw supplies the validated bundle through `openshell.supervisor.network.additional_ca_bundle` in the managed gateway configuration. +Private-CA MCP runtime qualification remains pending. @@ -40,7 +51,7 @@ When onboarding supplies a corporate CA, the Dockerfile sets `NODE_EXTRA_CA_CERT This includes registry-backed npm remediation and runtime dependency installation that cross a TLS-inspecting proxy. The build does not retain either variable in the image configuration. -NemoClaw keeps `NEMOCLAW_MANAGED_IMAGE_RUNTIME_USER=sandbox` for OpenShell 0.0.116. +NemoClaw keeps `NEMOCLAW_MANAGED_IMAGE_RUNTIME_USER=sandbox` when it imports a corporate CA into the managed OpenClaw image. The OpenClaw managed startup runtime creates `/tmp/nemoclaw-managed-startup-ca-bundle.pem` as `root:root` with mode `0444` before it releases the sandbox-user startup command. The `sandbox` user can read the merged bundle but cannot modify or replace it. Sandboxes still using the staged legacy direct-start path create `/tmp/nemoclaw-ca-bundle.pem` with mode `0444` as the same `sandbox` identity that runs the entrypoint and agent, so that compatibility path is read-only by mode but is not a privilege boundary. @@ -71,6 +82,29 @@ This supports cold builds on hosts where the base image is not cached. For sandbox child processes, NemoClaw also appends the corporate CA to the OpenShell client trust bundle instead of replacing it. It points `SSL_CERT_FILE`, `CURL_CA_BUNDLE`, `REQUESTS_CA_BUNDLE`, `GIT_SSL_CAINFO`, and `NODE_EXTRA_CA_CERTS` at the merged bundle so curl, Python, Git, and Node.js trust both roots. +## Understand Supervisor Destination Trust + +NemoClaw stores the validated public certificates under `network-trust/additional-ca-.pem` in the managed gateway state directory. +The gateway owner controls the directory with mode `0700`; each certificate snapshot has mode `0444`. +The snapshot contains certificates only, never a private key or provider credential. +NemoClaw checks its ownership, permissions, contents, and digest before reuse. +A missing, changed, or unsafe saved snapshot stops gateway configuration instead of silently dropping the requested trust. + +The destination bundle augments the supervisor's trust roots. +It does not change the mutual TLS credentials or trust used to authenticate gateway clients. + +When no CA source resolves during later gateway preparation, NemoClaw preserves the saved snapshot. +Selecting another validated bundle creates another generation and updates the gateway configuration. +NemoClaw retains previous generations because a running supervisor can still use them. +They remain in the gateway state until that owned state is removed. +Do not remove them manually while the gateway or its sandboxes are running. + +Changing gateway configuration does not prove that existing supervisors or workload images use the new CA. +For an existing sandbox, follow the [explicit recreation procedure](../reference/commands#--resume-and--fresh) and its state-preservation limits. +After recreation, verify the required HTTPS operation from the affected sandbox. +For managed MCP, require a verified credential-resolution result from `$$nemoclaw mcp status `. +An inconclusive result does not establish TLS connectivity or credential replacement. + ## Understand Automatic Source Selection NemoClaw checks corporate CA sources in this order: @@ -101,7 +135,9 @@ If no selection message appears, no source passed validation. Managed NemoClaw Dockerfiles implement the corporate CA build contract automatically. A custom Dockerfile must declare `ARG NEMOCLAW_CORPORATE_CA_B64`, validate and decode it into `/usr/local/share/nemoclaw/corporate-ca.pem`, install each certificate as an individual OS trust anchor, and refresh the OS trust bundle. -Decoding only the application bundle is insufficient for OpenShell-inspected HTTPS because the proxy verifies the upstream TLS connection itself. +Decoding only the application bundle does not establish trust for OpenShell-inspected HTTPS because the supervisor verifies the upstream TLS connection. +NemoClaw configures supervisor destination trust independently of the custom Dockerfile. +Skipping image CA import does not skip gateway destination trust. If its build needs network access behind the corporate proxy, establish build-time trust before the dependency operations that require TLS. @@ -111,14 +147,14 @@ Dockerfile instruction keywords are case-insensitive, but the argument name must Before the final entrypoint, the stage must set `USER ${NEMOCLAW_MANAGED_IMAGE_RUNTIME_USER}`. The final entrypoint must use the trusted exec form `ENTRYPOINT ["/usr/local/bin/nemoclaw-start"]`. NemoClaw preserves `sandbox` when it bakes the corporate CA; managed startup creates the root-owned merged bundle before it releases the entrypoint. -Automatic fallback and host-store sources are a no-op unless the Dockerfile declares `NEMOCLAW_CORPORATE_CA_B64` and meets this startup contract. +NemoClaw skips image CA import from automatic fallback and host-store sources unless the Dockerfile declares `NEMOCLAW_CORPORATE_CA_B64` and meets this startup contract. The explicit `NEMOCLAW_CORPORATE_CA_BUNDLE` path fails onboarding if either condition is not met. -Automatic fallback and host-store sources are a no-op when a custom Dockerfile omits `NEMOCLAW_CORPORATE_CA_B64`. +NemoClaw skips image CA import from automatic fallback and host-store sources when a custom Dockerfile omits `NEMOCLAW_CORPORATE_CA_B64`. The explicit `NEMOCLAW_CORPORATE_CA_BUNDLE` path fails onboarding when that argument is absent so the requested trust change cannot silently disappear. @@ -129,7 +165,10 @@ Every imported source must be a regular, readable, non-symlink PEM file that is The file must be non-empty, stay within the corporate-chain size and certificate-count limits, and contain only certificates that parse as X.509 CA certificates with `basicConstraints CA:TRUE`. NemoClaw rejects or skips a source that would widen trust to a full operating-system bundle instead of truncating it. -Set `NEMOCLAW_CORPORATE_CA_IMPORT=0` to disable corporate CA import entirely. +Set `NEMOCLAW_CORPORATE_CA_IMPORT=0` to disable corporate CA import. +During the next managed gateway preparation, this explicit setting also removes the additional destination bundle from the gateway configuration. +It retains saved certificate generations and does not remove CA material from existing images or running supervisors. +Recreate affected sandboxes before relying on removal, then verify that the disallowed private-CA connection fails TLS validation. ## Related Topics diff --git a/docs/security/credential-rotation.mdx b/docs/security/credential-rotation.mdx index 96495b1a103..b26d4339445 100644 --- a/docs/security/credential-rotation.mdx +++ b/docs/security/credential-rotation.mdx @@ -279,8 +279,8 @@ If a credential was exposed, act in this order: 2. For inference, keep the provider registered if you have a replacement or can issue one promptly. Issue a needed replacement in step 4, then use [Rotate an Inference API key](#rotate-an-inference-api-key) in step 5. If you cannot yet issue an inference replacement, follow [Remove and Re-register a Provider Credential](#remove-and-re-register-a-provider-credential). - For native inference in that removal path, preserve required state and destroy every attached sandbox before resetting the provider. - For web search without a replacement, remove the provider shown by `$$nemoclaw credentials list`. + For native NVIDIA and supported fixed hosted inference in that removal path, preserve required state and destroy every attached sandbox before reset. + For web search without a replacement, follow the same provider removal prerequisites. 3. For messaging without a replacement, use `channels remove` to retire the integration. Follow [Remove a Channel](../manage-sandboxes/messaging-channels/manage-messaging-channels#remove-a-channel) to complete its cleanup. Review recent logs as in step 7. diff --git a/docs/security/credential-storage.mdx b/docs/security/credential-storage.mdx index 782e5cc99f6..870665b6169 100644 --- a/docs/security/credential-storage.mdx +++ b/docs/security/credential-storage.mdx @@ -167,9 +167,9 @@ openshell sandbox provider detach This command keeps the provider and stored credential in OpenShell and does not change its attachments to other sandboxes. Rerun the original rebuild command. -For ordinary gateway providers, `$$nemoclaw credentials reset ` detaches every sandbox using the provider and deletes its stored credential. -Native NVIDIA and fixed hosted inference providers refuse reset while a sandbox records an attachment. -Refer to [Rotate or Remove a Stored Credential](#rotate-or-remove-a-stored-credential) before resetting one of these native providers. +Credential reset deletes a gateway-wide provider and can detach other sandboxes. +For inference providers, follow the prerequisites under [Rotate or Remove a Stored Credential](#rotate-or-remove-a-stored-credential). +For other gateway providers, `$$nemoclaw credentials reset ` detaches every sandbox using the provider and deletes its stored credential. When the host environment is empty, day-two operations such as `$$nemoclaw rebuild` and remote-provider updates can reuse the credential already registered with the OpenShell gateway. Export the credential only when you want to create, replace, or rotate the stored provider value. @@ -249,12 +249,14 @@ To replace a stored inference credential, rerun onboarding with the new value in NVIDIA_INFERENCE_API_KEY=nvapi-new-value $$nemoclaw onboard ``` -Native NVIDIA and fixed hosted inference providers refuse reset while a sandbox records an attachment. -To rotate their credentials in place, supply the selected provider's replacement credential and rerun `$$nemoclaw onboard --name `. -Before removing one of these providers, preserve required sandbox state and destroy every sandbox that records its attachment. -Then retry reset. +Before removing a credential, check whether a sandbox records the provider as its inference selection. +Reset refuses removal of a separately registered inference provider while that selection remains, without changing the provider or its attachments. +Switch each affected sandbox to another provider, or preserve required state and destroy those sandboxes first. +Native NVIDIA and supported fixed hosted providers also refuse reset while a sandbox records their attachment. +To rotate those credentials in place, supply the selected provider's replacement credential and rerun `$$nemoclaw onboard --name `. +Before removing one of those providers, preserve required state and destroy every sandbox that records its attachment. -To remove an unreferenced native provider or an ordinary gateway provider entirely: +After resolving these dependencies, remove the credential by provider name: ```bash $$nemoclaw credentials reset diff --git a/docs/security/filesystem-controls.mdx b/docs/security/filesystem-controls.mdx index c37235707cb..3020083c649 100644 --- a/docs/security/filesystem-controls.mdx +++ b/docs/security/filesystem-controls.mdx @@ -27,7 +27,7 @@ The container mounts system directories read-only to prevent the agent from modi -The Hermes policy also grants read-only access to Direct Rendering Manager (DRM) device metadata under `/sys/class/drm`. +The Hermes policy permits listing Direct Rendering Manager (DRM) device names in `/sys/class/drm`. This grant does not allow writes or expose the rest of `/sys`. diff --git a/docs/security/gateway-authentication-controls.mdx b/docs/security/gateway-authentication-controls.mdx index a131c4ed882..0f6197610bc 100644 --- a/docs/security/gateway-authentication-controls.mdx +++ b/docs/security/gateway-authentication-controls.mdx @@ -23,9 +23,9 @@ On Docker-driver deployments, NemoClaw gives host CLI calls and sandbox callback | Aspect | Detail | |---|---| -| Default | NemoClaw enables local TLS, mTLS user authentication, and sandbox JWT authentication. Host-side OpenShell CLI calls use local mTLS. Sandbox callbacks use the guest mTLS bundle plus a sandbox-scoped JWT. The generated config sets `allow_unauthenticated_users = false`, and gateway launch removes an inherited `OPENSHELL_DISABLE_GATEWAY_AUTH=true`. | -| Supervisor TLS name | OpenShell 0.0.116 removes a user-supplied `OPENSHELL_GATEWAY_TLS_SERVER_NAME` from Docker, Podman, and VM supervisor environments after it merges sandbox settings. The supervisor therefore verifies the gateway name chosen by the trusted driver rather than one chosen inside the sandbox specification. | -| Token lifetime | Local sandbox JWTs use OpenShell's `ttl_secs = 0` contract for a non-expiring token on a local single-user gateway. Sandbox identity checks and the local mTLS boundary still apply to each callback. | +| Default | NemoClaw enables local TLS, mTLS user authentication, and sandbox JWT authentication. Host-side OpenShell CLI calls use local mTLS. Sandbox callbacks use server-verified TLS plus a sandbox-scoped JWT. The generated config sets `allow_unauthenticated_users = false`, and gateway launch removes an inherited `OPENSHELL_DISABLE_GATEWAY_AUTH=true`. | +| Supervisor TLS name | OpenShell 0.1.7 excludes sandbox-supplied `OPENSHELL_GATEWAY_TLS_SERVER_NAME` values from Docker, Podman, and VM runtime environments. A sandbox specification cannot choose the gateway TLS name used by the trusted driver. | +| Token lifetime | Local sandbox JWTs use OpenShell's `ttl_secs = 0` contract for a non-expiring token on a local single-user gateway. Sandbox identity checks and server-verified TLS still apply to each callback. | | What you can change | These authentication controls are not user-facing settings. Use NemoClaw to configure and start the Docker-driver gateway. | | Risk if relaxed | Disabling gateway authentication or widening the gateway listener can expose privileged gateway methods to another local or network client. | | Recommendation | Keep the OpenShell gateway on `127.0.0.1`. Use the dashboard forward when a supported agent dashboard needs remote access. | @@ -40,7 +40,7 @@ This path requires the explicit opt-in `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PAT | Default | NemoClaw does not auto-enable the compatibility container on ABI mismatch. If `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=1` is set, the container keeps the main gateway listener on `127.0.0.1`, uses host networking so OpenShell computes the same Docker bridge callback addresses as a host-side gateway, mounts the Docker socket read-only, drops Linux capabilities, sets `no-new-privileges`, and publishes no extra Docker ports. | | What you can change | Opt in with `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=1`, keep the path disabled with `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=0`, or run on a host/OpenShell build combination where the gateway binary launches directly. | | Risk if relaxed | The Docker socket remains a privileged host API even when bind-mounted read-only. Treat this mode as equivalent to trusting the host user that can drive Docker, and do not enable it on untrusted shared hosts. | -| Recommendation | Prefer a host with glibc 2.39 or newer, which OpenShell 0.0.116 supports directly, and use the compatibility container only as an explicit local bridge on an older trusted host. | +| Recommendation | Prefer a host whose glibc satisfies the installed `openshell-gateway` binary. NemoClaw checks the binary's required glibc versions against the host version. Use the compatibility container only as an explicit local bridge on an older trusted host. | OpenShell owns the native Linux glibc support floor. NemoClaw owns the explicit opt-in, host-networking configuration, read-only socket mount, and gateway authentication controls for this fallback. @@ -161,9 +161,9 @@ For remote API access, use the [SSH forwarding procedure](/user-guide/hermes/get Protect any remotely exposed dashboard with your own access controls. NemoClaw keeps provider credentials in OpenShell. -NVIDIA Endpoints and supported fixed hosted providers use native endpoints through sandbox-attached OpenShell providers. +NVIDIA Endpoints, supported fixed hosted providers, and qualifying custom hosted endpoints use native endpoints through sandbox-attached OpenShell providers. New Ollama, standalone vLLM, llama.cpp, and qualifying host-local compatible selections also use sandbox-attached providers and scoped credential references. -Owned Local NVIDIA NIM, nonqualifying custom endpoints, and retained legacy shared routes continue to use `inference.local`. +Remaining custom shared selections and retained legacy shared routes continue to use `inference.local` on OpenShell `0.0.x`. Live qualification of the Hermes native local lifecycle is pending. Generated Hermes runtime files use OpenShell resolver placeholders for managed-tool and messaging credentials. diff --git a/docs/security/process-controls.mdx b/docs/security/process-controls.mdx index 403e66fad54..8ff0d6183ea 100644 --- a/docs/security/process-controls.mdx +++ b/docs/security/process-controls.mdx @@ -52,15 +52,16 @@ Refer to [Review Sandbox Hardening](../../manage-sandboxes/configure-sandboxes/r Gateway and agent UID isolation depends on the container process topology. The stock OpenClaw image defaults to the `sandbox` user for OpenShell compatibility. -An OpenShell-managed container has OpenShell as PID 1 and launches `nemoclaw-start` as a non-root process, so the supervisor, gateway, and agent all use the `sandbox` UID. -The managed-image publication workflow explicitly sets `NEMOCLAW_MANAGED_IMAGE_RUNTIME_USER=sandbox` for its reviewed release images because OpenShell 0.0.116 rejects a root OCI image user. +OpenShell 0.1.7's Docker driver separates the supervisor from the workload container. +The workload container launches `nemoclaw-start` as a non-root process, so the OpenClaw gateway and agent share the `sandbox` UID. +The managed-image publication workflow explicitly sets `NEMOCLAW_MANAGED_IMAGE_RUNTIME_USER=sandbox` for its reviewed release images. A direct container runtime can override the image user to `root`. That root-entrypoint topology runs the gateway as the separate `gateway` user and agent commands as the `sandbox` user. | Aspect | Detail | |---|---| | Default | A stock Dockerfile build declares `USER sandbox`. The OpenShell-managed topology runs the gateway and agent under the same sandbox UID because `no-new-privileges` prevents the non-root entrypoint from changing users. | -| What you can change | OpenClaw managed and custom images must keep `USER sandbox` as the default for OpenShell 0.0.116. A direct container runtime outside OpenShell can still override the image user to `root`. | +| What you can change | Keep `USER sandbox` for NemoClaw's managed OpenClaw image contract with OpenShell 0.1.7. A direct container runtime outside OpenShell can still override the image user to `root`. | | Risk if relaxed | A same-UID agent can signal peer processes and can attempt to imitate the expected gateway process shape. Native restart and host health checks prove service availability, not provenance against a malicious same-user process. | | Recommendation | Keep the OpenShell-managed image user as `sandbox`. Use a direct root-entrypoint deployment only outside OpenShell when separate gateway and agent UIDs are required. Treat native lifecycle and readiness checks as availability controls, not proof that the target process is trustworthy. | @@ -69,12 +70,12 @@ That root-entrypoint topology runs the gateway as the separate `gateway` user an ## Gateway Process Isolation -The stock Hermes image and the managed-image publication workflow set the OCI default user to `sandbox` for OpenShell 0.0.116. -The supervisor, agent gateway, and agent commands share that UID. +The stock Hermes image and the managed-image publication workflow set the OCI default user to `sandbox` for OpenShell 0.1.7. +The Hermes gateway and agent commands share that UID in the workload container; OpenShell's Docker supervisor runs in a separate container. | Aspect | Detail | |---|---| -| Default | The OCI default user is `sandbox`, as required by OpenShell 0.0.116. | +| Default | NemoClaw's managed Hermes image uses the OCI default user `sandbox`. | | What you can change | A direct container runtime outside OpenShell can override the image user. A root override can provide separate gateway and agent UIDs. | | Risk if relaxed | A same-UID agent can signal the agent gateway and attempt to imitate its process shape. Running agent commands as `root` also expands filesystem access and container escape impact. | | Recommendation | Keep the sandbox entrypoint under OpenShell. Use root only for a direct-container topology that requires separate gateway and agent UIDs. | @@ -85,7 +86,7 @@ The supervisor, agent gateway, and agent commands share that UID. ## Gateway Process Isolation The stock Deep Agents image defaults to the `sandbox` user. -The managed-image publication workflow also sets the OCI default user to `sandbox` for OpenShell 0.0.116. +The managed-image publication workflow also sets the OCI default user to `sandbox` for OpenShell 0.1.7. Deep Agents does not use a separate agent gateway UID in this topology. | Aspect | Detail | @@ -212,7 +213,7 @@ The managed headless path automatically approves non-shell tool requests such as The sandbox runs agent processes as a dedicated `sandbox` user and group. The stock OpenClaw image starts the entrypoint as `sandbox` for OpenShell compatibility. -A build from the managed-image publication workflow keeps the entrypoint, gateway, and agent commands under the sandbox UID for OpenShell 0.0.116. +A build from the managed-image publication workflow keeps the entrypoint, gateway, and agent commands under the sandbox UID for OpenShell 0.1.7. A direct runtime can override the image user to `root`, which lets the entrypoint separate the `gateway` and `sandbox` UIDs before it runs agent commands. | Aspect | Detail | @@ -227,7 +228,7 @@ A direct runtime can override the image user to `root`, which lets the entrypoin ## Non-Root User -The Hermes entrypoint, agent gateway, dashboard, and agent commands run as `sandbox` under OpenShell 0.0.116. +The Hermes entrypoint, agent gateway, dashboard, and agent commands run as `sandbox` under OpenShell 0.1.7. | Aspect | Detail | |---|---| diff --git a/docs/security/tcb-boundary.mdx b/docs/security/tcb-boundary.mdx index 52decb7def2..b92f6679ea8 100644 --- a/docs/security/tcb-boundary.mdx +++ b/docs/security/tcb-boundary.mdx @@ -96,6 +96,14 @@ They do not make the resulting agent config or state immutable after the transac The direct supervisor records the gateway process ID and kernel start identity and verifies the parent relationship before signalling or reaping the process. The Docker health fallback reads the same record and verifies process identity before and after reading its command line. +For OpenShell bootstrap, the Hermes guard permits supplementary GPU groups only when validated Direct Rendering Manager (DRM) devices account for them. +The Hermes policy grants read-only access to `/sys/class/drm` for device-name discovery; this grant does not add GPU device access. +The guard pins `/dev` and `/dev/dri`, rejects symlinks and unsafe directory ownership or write permissions, and bounds device inspection. +It reads device metadata without reading those directories or opening device nodes. +It checks device type, number, owner, group, and permissions before accepting a group beyond the sandbox's primary group. +The rootless launcher must still clear supplementary groups. +Process identity, lifetime, capability, and `NoNewPrivs` checks remain required. + The Hermes cron-restore controller pins the initial gateway identity while it validates restored cron state. It observes the replacement identity around health verification and releases the restore gate only for that live replacement. This controller protects the cron-restore transaction and does not own ordinary Hermes gateway lifecycle. diff --git a/managed-inference/provider-profiles/nemoclaw-anthropic-inference-v1.yaml b/managed-inference/provider-profiles/nemoclaw-anthropic-inference-v1.yaml index 6a9a99bd42a..e869cff101f 100644 --- a/managed-inference/provider-profiles/nemoclaw-anthropic-inference-v1.yaml +++ b/managed-inference/provider-profiles/nemoclaw-anthropic-inference-v1.yaml @@ -28,6 +28,8 @@ binaries: - /usr/bin/node - /opt/hermes/.venv/bin/python - /opt/hermes/.venv/bin/python3 + # OpenShell checks the kernel-resolved Hermes interpreter path. + - /usr/bin/python3.13 - /opt/venv/bin/python3 - /usr/local/bin/curl - /usr/bin/curl diff --git a/managed-inference/provider-profiles/nemoclaw-gemini-inference-v1.yaml b/managed-inference/provider-profiles/nemoclaw-gemini-inference-v1.yaml index 8a3146b860a..6b9c70e59f1 100644 --- a/managed-inference/provider-profiles/nemoclaw-gemini-inference-v1.yaml +++ b/managed-inference/provider-profiles/nemoclaw-gemini-inference-v1.yaml @@ -27,6 +27,8 @@ binaries: - /usr/bin/node - /opt/hermes/.venv/bin/python - /opt/hermes/.venv/bin/python3 + # OpenShell checks the kernel-resolved Hermes interpreter path. + - /usr/bin/python3.13 - /opt/venv/bin/python3 - /usr/local/bin/curl - /usr/bin/curl diff --git a/managed-inference/provider-profiles/nemoclaw-hermes-inference-v1.yaml b/managed-inference/provider-profiles/nemoclaw-hermes-inference-v1.yaml index 69dc82b6a39..d31ddd36836 100644 --- a/managed-inference/provider-profiles/nemoclaw-hermes-inference-v1.yaml +++ b/managed-inference/provider-profiles/nemoclaw-hermes-inference-v1.yaml @@ -27,6 +27,8 @@ binaries: - /usr/bin/node - /opt/hermes/.venv/bin/python - /opt/hermes/.venv/bin/python3 + # OpenShell checks the kernel-resolved Hermes interpreter path. + - /usr/bin/python3.13 - /opt/venv/bin/python3 - /usr/local/bin/curl - /usr/bin/curl diff --git a/managed-inference/provider-profiles/nemoclaw-nvidia-inference-v1.yaml b/managed-inference/provider-profiles/nemoclaw-nvidia-inference-v1.yaml index 079f53978be..3732457cde4 100644 --- a/managed-inference/provider-profiles/nemoclaw-nvidia-inference-v1.yaml +++ b/managed-inference/provider-profiles/nemoclaw-nvidia-inference-v1.yaml @@ -27,6 +27,8 @@ binaries: - /usr/bin/node - /opt/hermes/.venv/bin/python - /opt/hermes/.venv/bin/python3 + # OpenShell checks the kernel-resolved Hermes interpreter path. + - /usr/bin/python3.13 - /opt/venv/bin/python3 - /usr/local/bin/curl - /usr/bin/curl diff --git a/managed-inference/provider-profiles/nemoclaw-openai-inference-v1.yaml b/managed-inference/provider-profiles/nemoclaw-openai-inference-v1.yaml index d4badead86d..e8ba8bdac53 100644 --- a/managed-inference/provider-profiles/nemoclaw-openai-inference-v1.yaml +++ b/managed-inference/provider-profiles/nemoclaw-openai-inference-v1.yaml @@ -28,6 +28,8 @@ binaries: - /usr/bin/node - /opt/hermes/.venv/bin/python - /opt/hermes/.venv/bin/python3 + # OpenShell checks the kernel-resolved Hermes interpreter path. + - /usr/bin/python3.13 - /opt/venv/bin/python3 - /usr/local/bin/curl - /usr/bin/curl diff --git a/managed-inference/provider-profiles/nemoclaw-openrouter-inference-v1.yaml b/managed-inference/provider-profiles/nemoclaw-openrouter-inference-v1.yaml index 906dd38bc51..69298eb76aa 100644 --- a/managed-inference/provider-profiles/nemoclaw-openrouter-inference-v1.yaml +++ b/managed-inference/provider-profiles/nemoclaw-openrouter-inference-v1.yaml @@ -27,6 +27,8 @@ binaries: - /usr/bin/node - /opt/hermes/.venv/bin/python - /opt/hermes/.venv/bin/python3 + # OpenShell checks the kernel-resolved Hermes interpreter path. + - /usr/bin/python3.13 - /opt/venv/bin/python3 - /usr/local/bin/curl - /usr/bin/curl diff --git a/nemoclaw-blueprint/blueprint.yaml b/nemoclaw-blueprint/blueprint.yaml index cd9a90d5988..2c6c66feea7 100644 --- a/nemoclaw-blueprint/blueprint.yaml +++ b/nemoclaw-blueprint/blueprint.yaml @@ -3,8 +3,8 @@ version: "0.1.0" # Requires OpenShell MCP/JSON-RPC L7 policy support from NVIDIA/OpenShell#1865. -min_openshell_version: "0.0.116" -max_openshell_version: "0.0.116" +min_openshell_version: "0.1.7" +max_openshell_version: "0.1.7" min_openclaw_version: "2026.9.2" # Mirrors the components.sandbox.image manifest digest below. Lets a # downstream consumer (or release tooling) verify the blueprint declares diff --git a/nemoclaw-blueprint/provider-profiles/nemoclaw-messaging-v1.yaml b/nemoclaw-blueprint/provider-profiles/nemoclaw-messaging-v1.yaml new file mode 100644 index 00000000000..4128ed32345 --- /dev/null +++ b/nemoclaw-blueprint/provider-profiles/nemoclaw-messaging-v1.yaml @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +id: nemoclaw-messaging-v1 +display_name: NemoClaw Messaging +description: Endpointless credentials for manifest-owned messaging channels +category: agent +# Each provider holds its selected channel credential; the applier requires that key. +credentials: + - name: TELEGRAM_BOT_TOKEN + env_vars: [TELEGRAM_BOT_TOKEN] + required: false + auth_style: "" + header_name: "" + query_param: "" + - name: DISCORD_BOT_TOKEN + env_vars: [DISCORD_BOT_TOKEN] + required: false + auth_style: "" + header_name: "" + query_param: "" + - name: WECHAT_BOT_TOKEN + env_vars: [WECHAT_BOT_TOKEN] + required: false + auth_style: "" + header_name: "" + query_param: "" + - name: SLACK_BOT_TOKEN + env_vars: [SLACK_BOT_TOKEN] + required: false + auth_style: "" + header_name: "" + query_param: "" + - name: SLACK_APP_TOKEN + env_vars: [SLACK_APP_TOKEN] + required: false + auth_style: "" + header_name: "" + query_param: "" + - name: MSTEAMS_APP_PASSWORD + env_vars: [MSTEAMS_APP_PASSWORD] + required: false + auth_style: "" + header_name: "" + query_param: "" +endpoints: [] +binaries: [] +inference_capable: false diff --git a/nemoclaw-blueprint/scripts/http-proxy-fix.js b/nemoclaw-blueprint/scripts/http-proxy-fix.js deleted file mode 100644 index 78e663e0924..00000000000 --- a/nemoclaw-blueprint/scripts/http-proxy-fix.js +++ /dev/null @@ -1,180 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 -// -// http-proxy-fix.js — http.request() wrapper resolving the double-proxy -// conflict between NODE_USE_ENV_PROXY=1 (Node.js 22+) and HTTP libraries -// that independently read HTTPS_PROXY (axios, follow-redirects, -// proxy-from-env). See NemoClaw#2109. -// -// Problem: -// Node.js 22 with NODE_USE_ENV_PROXY=1 (baked into the OpenShell base -// image) intercepts https.request() calls and handles proxying via a -// CONNECT tunnel. HTTP libraries also read HTTPS_PROXY and configure -// HTTP FORWARD mode, so the request is processed twice and the L7 proxy -// rejects it with "FORWARD rejected: HTTPS requires CONNECT". -// -// Fix: -// Wrap http.request() — the lowest common denominator every HTTP client -// bottoms out at. Detect FORWARD-mode requests (hostname = proxy IP, -// path = full https:// URL) and rewrite them as https.request() against -// the real target host, letting NODE_USE_ENV_PROXY handle the CONNECT -// tunnel correctly. -// -// Earlier PR #2110 tried a Module._load hook intercepting require('axios'). -// That could not catch follow-redirects + proxy-from-env bundled as ESM in -// OpenClaw's dist/ — there are no require() calls to intercept. The -// http.request wrapper sits below all libraries and catches every path. -// -// This file is the canonical source for review and tests. The Dockerfile -// copies it into /usr/local/lib/nemoclaw/preloads/, then at sandbox boot -// nemoclaw-start.sh writes an identical copy to /tmp/nemoclaw-http-proxy-fix.js -// and loads it via NODE_OPTIONS=--require. - -(function () { - 'use strict'; - if (process.env.NODE_USE_ENV_PROXY !== '1') return; - - var http = require('http'); - var origRequest = http.request; - - var proxyUrl = - process.env.HTTPS_PROXY || - process.env.https_proxy || - process.env.HTTP_PROXY || - process.env.http_proxy || - ''; - var proxyHost = ''; - try { - proxyHost = new URL(proxyUrl).hostname; - } catch (_e) { - /* no usable proxy configured */ - } - if (!proxyHost) return; - - // Strip headers that were meaningful for the proxy hop only. Once we - // re-issue against the target via https.request, the original Host - // points at the proxy and the hop-by-hop headers (RFC 7230 §6.1) leak - // upstream — they describe the connection between the caller and the - // proxy, not the rewritten connection to the target. - // - // RFC 7230 §6.1 hop-by-hop set (request direction): - // Connection, Keep-Alive, Proxy-Authorization, TE, Trailer, - // Transfer-Encoding, Upgrade. - // Also stripped: Host (points at the proxy); Proxy-Connection (de - // facto deprecated header still emitted by some clients); and - // Proxy-Authenticate (response-only per RFC 7235 §4.3, included - // belt-and-suspenders for clients that echo response headers into - // retry-request options). Plus: per RFC 7230 §6.1, any token named in - // the Connection header is itself hop-by-hop and must be stripped. - var STATIC_HOP_BY_HOP = [ - 'host', - 'connection', - 'keep-alive', - 'proxy-authenticate', - 'proxy-authorization', - 'proxy-connection', - 'te', - 'trailer', - 'transfer-encoding', - 'upgrade', - ]; - - function sanitizeHeaders(headers) { - if (!headers || typeof headers !== 'object') return undefined; - // Collect tokens named in the Connection header — those become - // hop-by-hop transitively per RFC 7230 §6.1. - var dynamic = new Set(); - for (var k in headers) { - if ( - !Object.prototype.hasOwnProperty.call(headers, k) || - String(k).toLowerCase() !== 'connection' - ) { - continue; - } - var raw = headers[k]; - var listed = Array.isArray(raw) ? raw.join(',') : raw; - if (typeof listed === 'string') { - listed.split(',').forEach(function (token) { - var t = token.trim().toLowerCase(); - if (t) dynamic.add(t); - }); - } - } - var staticSet = new Set(STATIC_HOP_BY_HOP); - var out = {}; - for (var key in headers) { - if (!Object.prototype.hasOwnProperty.call(headers, key)) continue; - var lower = String(key).toLowerCase(); - if (staticSet.has(lower) || dynamic.has(lower)) continue; - out[key] = headers[key]; - } - return out; - } - - http.request = function (options, callback) { - if (typeof options === 'string' || !options) { - return origRequest.apply(http, arguments); - } - if ( - options.hostname === proxyHost && - options.path && - options.path.startsWith('https://') - ) { - var target; - try { - target = new URL(options.path); - } catch (_e) { - return origRequest.apply(http, arguments); - } - var https = require('https'); - // Clone caller's options and overwrite proxy-specific routing - // fields. Strip fields that were set up for the proxy hop and - // would misbehave on the rewritten https.request to the target: - // - agent: a forward-proxy http.Agent cannot speak TLS. Leaving - // it attached caused upstreams like deepinfra to surface as - // "LLM request failed: network connection error" while other - // upstreams that don't end up on this code path still worked. - // On Node 22 https.request throws a synchronous TypeError; on - // Node 18/20 it falls through and the TLS handshake fails. - // - auth: basic-auth meant for the proxy hop. Leaving it on - // would Basic-auth the target server with proxy credentials. - // - servername / checkServerIdentity: TLS SNI + cert validation - // pre-computed for the proxy hop. Wrong cert chain and wrong - // SNI must not survive into the rewrite — drop them so Node - // re-derives from the new `hostname`. - // - socketPath: Unix-socket proxies exist (e.g. cntlm-style - // local proxies). Routing TLS bytes into the proxy's Unix - // socket would defeat the entire rewrite. - // - localAddress / lookup / family / hints: source-binding and - // DNS hints picked for reachability to the proxy. The - // rewritten target may not be reachable from the same NIC or - // DNS family. - // - Host / hop-by-hop headers (RFC 7230 §6.1): stripped via - // sanitizeHeaders so Node regenerates Host from `host`/`port` - // to point at the real target. - // Signal (AbortController) and TLS material (ca/cert/key/ - // rejectUnauthorized), timeout, body, and target-intent headers - // (Authorization, Content-Type, …) are preserved. - var rewritten = Object.assign({}, options, { - method: options.method || 'GET', - hostname: target.hostname, - host: target.hostname, - port: target.port || 443, - path: target.pathname + target.search, - protocol: 'https:', - headers: sanitizeHeaders(options.headers), - }); - delete rewritten.agent; - delete rewritten.auth; - delete rewritten.servername; - delete rewritten.checkServerIdentity; - delete rewritten.socketPath; - delete rewritten.localAddress; - delete rewritten.lookup; - delete rewritten.family; - delete rewritten.hints; - return https.request(rewritten, callback); - } - return origRequest.apply(http, arguments); - }; -})(); diff --git a/nemoclaw/src/blueprint/runner-error-paths.test.ts b/nemoclaw/src/blueprint/runner-error-paths.test.ts index 38942dff474..5afeb5b3398 100644 --- a/nemoclaw/src/blueprint/runner-error-paths.test.ts +++ b/nemoclaw/src/blueprint/runner-error-paths.test.ts @@ -1,3 +1,4 @@ +import { withNativeInferenceFixture } from "./runner-test-fixtures.js"; // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 @@ -52,8 +53,14 @@ vi.mock("./ssrf.js", async (importOriginal) => ({ validateEndpointUrl: vi.fn(async (url: string) => resolvedEndpointFor(url)), })); -const { actionApply, actionReconcile, actionStatus, loadBlueprint, main } = - await import("./runner.js"); +const { + actionApply: applyWithoutFixture, + actionReconcile, + actionStatus, + loadBlueprint, + main, +} = await import("./runner.js"); +const actionApply = withNativeInferenceFixture(applyWithoutFixture); const stdoutCapture = createStdoutCapture(); function captureStdout(): void { @@ -207,6 +214,51 @@ describe("runner error paths", () => { await expect(actionReconcile("nc-drifted")).rejects.toThrow(/gateway binding changed/); }); + it("routes native recovery through the host and rechecks the saved gateway", async () => { + const rid = "nc-resume"; + const runDir = `${FAKE_HOME}/.nemoclaw/state/runs/${rid}`; + const gateway = { name: "test-gateway", host: "127.0.0.1", port: 8080 }; + addDir(runDir); + addFile( + `${runDir}/plan.json`, + JSON.stringify({ + run_id: rid, + sandbox_name: "sb", + gateway, + native_inference: { configPublished: false }, + }), + ); + captureStdout(); + const resume = vi.fn(async (input) => { + await input.revalidate(); + }); + await main(["reconcile", "--run-id", rid], { nativeInferenceResumer: resume }); + expect(resume).toHaveBeenCalledWith({ + runId: rid, + gateway, + sandboxName: "sb", + revalidate: expect.any(Function), + }); + expect(stdoutCapture.text()).toContain("native inference configuration is complete"); + expect( + mockExeca.mock.calls.every(([, args]) => args?.join(" ") === "gateway info -g test-gateway"), + ).toBe(true); + }); + + it("refuses native recovery without the host resumer", async () => { + const runDir = `${FAKE_HOME}/.nemoclaw/state/runs/nc-resume`; + addDir(runDir); + addFile( + `${runDir}/plan.json`, + JSON.stringify({ + sandbox_name: "sb", + gateway: { name: "test-gateway", host: "127.0.0.1", port: 8080 }, + native_inference: {}, + }), + ); + await expect(actionReconcile("nc-resume")).rejects.toThrow("host native inference resumer"); + }); + it("throws when reconcile has no --run-id", async () => { await expect(main(["reconcile"])).rejects.toThrow(/--run-id is required for reconcile/); }); diff --git a/nemoclaw/src/blueprint/runner-identity.test.ts b/nemoclaw/src/blueprint/runner-identity.test.ts index 3cfd5c018ba..3dcef3d71b7 100644 --- a/nemoclaw/src/blueprint/runner-identity.test.ts +++ b/nemoclaw/src/blueprint/runner-identity.test.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { withNativeInferenceFixture } from "./runner-test-fixtures.js"; + import type fs from "node:fs"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -19,7 +21,7 @@ import { MATCHING_INFERENCE_PROVIDER_LISTING, MATCHING_INFERENCE_ROUTE_LISTING, MATCHING_RUNTIME_PROVIDER_LISTING, - providersV2EnabledResult, + absentGlobalPolicyResult, resultWithBlueprintPolicy, successResult, TEST_SANDBOX_POLICY, @@ -70,8 +72,14 @@ vi.mock("./private-networks.js", async (importOriginal) => ({ isPrivateHostname: () => false, })); -const { actionApply, actionPlan, actionRollback, actionStatus, loadBlueprint } = - await import("./runner.js"); +const { + actionApply: applyWithoutFixture, + actionPlan, + actionRollback, + actionStatus, + loadBlueprint, +} = await import("./runner.js"); +const actionApply = withNativeInferenceFixture(applyWithoutFixture); const matchingProvider = MATCHING_RUNTIME_PROVIDER_LISTING; const matchingInferenceProvider = MATCHING_INFERENCE_PROVIDER_LISTING; @@ -79,7 +87,7 @@ const matchingInferenceRoute = MATCHING_INFERENCE_ROUTE_LISTING; const runtimeIdentityProfilePath = "/blueprint/provider-profiles/okta-runtime-v1.yaml"; const success = successResult(); -const providersV2Enabled = providersV2EnabledResult(); +const absentGlobalPolicy = absentGlobalPolicyResult(); const identityPolicyAdditions = { identity_api: { name: "identity_api", @@ -110,7 +118,7 @@ function responseQueue( { exitCode: 0, stdout: "Name: test-sandbox\nPhase: Ready", stderr: "" }, ], ["provider get test-provider", { exitCode: 0, stdout: matchingInferenceProvider, stderr: "" }], - ["settings get --global --json", providersV2Enabled], + ["policy list --global --page-size 1 --output json", absentGlobalPolicy], ]); mockExeca.mockImplementation(async (_command: string, args: string[]) => { const command = args.join(" "); @@ -168,9 +176,10 @@ function installPolicyIdentityResponses(policyWriteFailure?: string): void { runtimeProviderListing: matchingProvider, policyWriteFailure, }); - mockExeca.mockImplementation( - async (_command: string, args: string[]) => - runtimeIdentityProfileExport(args) ?? identityPolicyResult(args), + mockExeca.mockImplementation(async (_command: string, args: string[]) => + args.join(" ") === "sandbox get test-sandbox" + ? failureResult("sandbox not found") + : (runtimeIdentityProfileExport(args) ?? identityPolicyResult(args)), ); } @@ -185,7 +194,9 @@ describe("blueprint identity wrapper", () => { resultWithBlueprintPolicy( args, runtimeIdentityProfileExport(args) ?? - (args.join(" ") === "settings get --global --json" ? providersV2Enabled : success), + (args.join(" ") === "policy list --global --page-size 1 --output json" + ? absentGlobalPolicy + : success), ), ); process.env.NEMOCLAW_BLUEPRINT_PATH = "/blueprint"; @@ -350,9 +361,9 @@ describe("blueprint identity wrapper", () => { expect.objectContaining({ reject: false, env: expect.any(Object) }), ); const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); - expect( - commands.indexOf("inference set --provider test-provider --model test-model"), - ).toBeLessThan(commands.indexOf("sandbox provider attach test-sandbox acme-okta-runtime")); + expect(commands.indexOf("forward start 18789 test-sandbox")).toBeLessThan( + commands.indexOf("sandbox provider attach test-sandbox acme-okta-runtime"), + ); expect( commands.indexOf( "provider refresh configure acme-okta-runtime --credential-key OKTA_ACCESS_TOKEN --strategy oauth2-refresh-token --material client_id=client-id --secret-material-env refresh_token=OKTA_REFRESH_TOKEN --secret-material-env client_secret=OKTA_CLIENT_SECRET", @@ -408,9 +419,7 @@ describe("blueprint identity wrapper", () => { index > policySet && command === "policy get -g test-gateway --full --output json test-sandbox", ); - const inferenceProviderCreate = commands.findIndex((command) => - command.startsWith("provider create --name test-provider "), - ); + const nativeReady = commands.indexOf("forward start 18789 test-sandbox"); const runtimeProviderCreate = commands.findIndex((command) => command.startsWith("provider create --name acme-okta-runtime "), ); @@ -420,7 +429,7 @@ describe("blueprint identity wrapper", () => { expect(policySet).toBeGreaterThanOrEqual(0); expect(policyVerification).toBeGreaterThan(policySet); - expect(inferenceProviderCreate).toBeGreaterThan(policyVerification); + expect(nativeReady).toBeGreaterThan(policyVerification); expect(runtimeProviderCreate).toBeGreaterThan(policyVerification); expect(runtimeAttachment).toBeGreaterThan(policyVerification); }); @@ -476,29 +485,15 @@ describe("blueprint identity wrapper", () => { const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); expect(commands).not.toContain( - "sandbox create -g test-gateway --from openclaw --name test-sandbox --policy /tmp/nemoclaw-test-policy.yaml --forward 18789", + `sandbox create -g test-gateway --from openclaw --name test-sandbox --policy /tmp/nemoclaw-test-policy.yaml --no-auto-providers --label ai.nvidia.nemoclaw.create-attempt=${"a".repeat(62)} --detach`, ); expect(commands.some((command) => command.startsWith("provider create "))).toBe(false); }); - it("reuses the ANSI-formatted OpenShell v0.0.99 inference route", async () => { + it("preserves verified native inference while attaching a runtime identity", async () => { process.env.OKTA_CLIENT_ID = "client-id"; process.env.OKTA_REFRESH_TOKEN = "refresh-secret"; process.env.OKTA_CLIENT_SECRET = "client-secret"; - const routeOutput = [ - "\u001b[1mInference:\u001b[0m", - "", - " Workspace: default", - " Provider: test-provider", - " Model: test-model", - " Version: 1", - " Timeout: 180s", - "", - "\u001b[1mSystem inference:\u001b[0m", - "", - " Not configured", - "", - ].join("\n"); responseQueue([ [ "sandbox get test-sandbox", @@ -508,7 +503,6 @@ describe("blueprint identity wrapper", () => { "provider get test-provider", [{ exitCode: 0, stdout: matchingInferenceProvider, stderr: "" }], ], - ["inference get", [{ exitCode: 0, stdout: routeOutput, stderr: "" }]], [ "provider get acme-okta-runtime", [ @@ -522,17 +516,113 @@ describe("blueprint identity wrapper", () => { ], ]); - await actionApply("default", blueprint({ identity: oktaIdentity() })); + const nativeInferenceObserver = vi.fn(async () => "verified-native-identity"); + await actionApply("default", blueprint({ identity: oktaIdentity() }), { + nativeInferenceObserver, + }); + expect(nativeInferenceObserver).toHaveBeenCalledWith({ + gatewayName: "test-gateway", + sandboxName: "test-sandbox", + provider: "test-provider", + model: "test-model", + api: "openai-completions", + endpointUrl: "https://api.example.com/v1", + }); const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); expect(commands).not.toContain( - "sandbox create -g test-gateway --from openclaw --name test-sandbox --policy /tmp/nemoclaw-test-policy.yaml --forward 18789", + `sandbox create -g test-gateway --from openclaw --name test-sandbox --policy /tmp/nemoclaw-test-policy.yaml --no-auto-providers --label ai.nvidia.nemoclaw.create-attempt=${"a".repeat(62)} --detach`, ); - expect(commands).not.toContain("inference set --provider test-provider --model test-model"); + expect( + commands.some((command) => + /^(?:inference |provider (?:create|update).*--name test-provider)/u.test(command), + ), + ).toBe(false); expect(commands).toContain("sandbox provider attach test-sandbox acme-okta-runtime"); }); - it("sets the route when OpenShell reports inference is not configured", async () => { + it("refuses concurrent sandbox creation without recorded native authority", async () => { + responseQueue([ + [ + `sandbox create -g test-gateway --from openclaw --name test-sandbox --policy /tmp/nemoclaw-test-policy.yaml --no-auto-providers --label ai.nvidia.nemoclaw.create-attempt=${"a".repeat(62)} --detach`, + [failureResult("sandbox already exists")], + ], + ]); + const nativeInferenceObserver = vi.fn(async (): Promise => { + throw new Error("native authority missing"); + }); + await expect( + actionApply("default", blueprint({ identity: oktaIdentity() }), { + nativeInferenceObserver, + }), + ).rejects.toThrow(/Failed to create sandbox.*already exists/); + expect(nativeInferenceObserver).not.toHaveBeenCalled(); + const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); + expect( + commands.some((command) => + /^(?:inference |provider create --name test-provider)/u.test(command), + ), + ).toBe(false); + }); + + it.each([ + ["missing host observer", undefined, /requires the host native inference observer/], + [ + "missing native authority", + async (): Promise => { + throw new Error("native authority missing"); + }, + /native authority missing/, + ], + ["invalid native observation", async (): Promise => "", /could not be verified/], + ] as const)( + "refuses reuse with %s before mutation", + async (_label, nativeInferenceObserver, expected) => { + responseQueue([ + [ + "sandbox get test-sandbox", + [{ exitCode: 0, stdout: "Name: test-sandbox\nPhase: Ready", stderr: "" }], + ], + ]); + await expect( + actionApply("default", blueprint({ identity: oktaIdentity() }), { + nativeInferenceObserver, + }), + ).rejects.toThrow(expected); + const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); + expect( + commands.some((command) => + /^(?:policy set|inference set|provider (?:create|refresh)|sandbox (?:create|provider attach))/u.test( + command, + ), + ), + ).toBe(false); + }, + ); + + it("stops identity mutation when the verified native selection changes", async () => { + responseQueue([ + [ + "sandbox get test-sandbox", + [{ exitCode: 0, stdout: "Name: test-sandbox\nPhase: Ready", stderr: "" }], + ], + ]); + const nativeInferenceObserver = vi + .fn() + .mockResolvedValueOnce("original") + .mockResolvedValue("replacement"); + await expect( + actionApply("default", blueprint({ identity: oktaIdentity() }), { nativeInferenceObserver }), + ).rejects.toThrow(/Native inference identity changed/); + const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); + expect( + commands.some((command) => + /^(?:inference set|provider (?:create|refresh)|sandbox provider attach)/u.test(command), + ), + ).toBe(false); + }); + + it("creates the sandbox native route without reading the retired global inference route", async () => { process.env.OKTA_CLIENT_ID = "client-id"; process.env.OKTA_REFRESH_TOKEN = "refresh-secret"; process.env.OKTA_CLIENT_SECRET = "client-secret"; @@ -567,10 +657,11 @@ describe("blueprint identity wrapper", () => { await actionApply("default", blueprint({ identity: oktaIdentity() })); const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); - expect(commands).toContain("inference set --provider test-provider --model test-model"); + expect(commands).toContain("forward start 18789 test-sandbox"); + expect(commands.some((command) => command.startsWith("inference "))).toBe(false); }); - it("sets an exact reused route when the requested timeout differs", async () => { + it("refuses a timeout change without the host configuration boundary", async () => { process.env.OKTA_CLIENT_ID = "client-id"; process.env.OKTA_REFRESH_TOKEN = "refresh-secret"; process.env.OKTA_CLIENT_SECRET = "client-secret"; @@ -600,52 +691,81 @@ describe("blueprint identity wrapper", () => { const inferenceProfile = input.components?.inference?.profiles?.default; expect(inferenceProfile).toBeDefined(); inferenceProfile!.timeout_secs = 300; - await actionApply("default", input); + await expect( + actionApply("default", input, { + nativeInferenceObserver: async () => "verified-native-identity", + }), + ).rejects.toThrow(/require the host configuration boundary/); const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); - expect(commands).toContain( - "inference set --provider test-provider --model test-model --timeout 300", - ); + expect( + commands.some((command) => + /^(?:inference set|provider create|provider refresh|sandbox provider attach)/u.test( + command, + ), + ), + ).toBe(false); }); - it("revalidates a mismatched inference provider created after preflight", async () => { + it("preserves credential presence and updates native timeout before attaching runtime identity", async () => { process.env.OKTA_CLIENT_ID = "client-id"; process.env.OKTA_REFRESH_TOKEN = "refresh-secret"; process.env.OKTA_CLIENT_SECRET = "client-secret"; + vi.stubEnv("BLUEPRINT_TEST_CREDENTIAL", "host-secret-not-rotated"); responseQueue([ + ["sandbox get test-sandbox", [{ exitCode: 0, stdout: "Name: test-sandbox", stderr: "" }]], [ - "provider get test-provider", + "provider get acme-okta-runtime", [ failureResult("provider not found"), - { + ...Array.from({ length: 4 }, () => ({ exitCode: 0, - stdout: matchingInferenceProvider.replace("Type: openai", "Type: anthropic"), + stdout: matchingProvider, stderr: "", - }, - ], - ], - [ - "provider get acme-okta-runtime", - [ - failureResult("provider not found"), - { exitCode: 0, stdout: matchingProvider, stderr: "" }, - { exitCode: 0, stdout: matchingProvider, stderr: "" }, - { exitCode: 0, stdout: matchingProvider, stderr: "" }, + })), ], ], - [ - "provider create --name test-provider --type openai --config OPENAI_BASE_URL=https://api.example.com/v1", - [failureResult("provider already exists")], - ], ]); - - await expect(actionApply("default", blueprint({ identity: oktaIdentity() }))).rejects.toThrow( - /Inference provider 'test-provider' does not match the requested non-secret binding/, + const input = blueprint({ identity: oktaIdentity() }); + const inference = input.components!.inference!.profiles!.default!; + inference.timeout_secs = 300; + inference.credential_env = "BLUEPRINT_TEST_CREDENTIAL"; + const observer = vi.fn(async () => "verified-native-identity"); + const preparer = vi.fn(async () => { + const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); + expect(commands.some((command) => command.startsWith("sandbox provider attach"))).toBe(false); + }); + await actionApply("default", input, { + nativeInferenceObserver: observer, + nativeInferencePreparer: preparer, + }); + expect(observer).toHaveBeenCalledWith(expect.objectContaining({ requiresCredential: true })); + expect(preparer).toHaveBeenCalledWith( + expect.objectContaining({ + timeoutSeconds: 300, + requiresCredential: true, + model: "test-model", + }), ); + expect( + JSON.stringify(preparer.mock.calls) + JSON.stringify(mockExeca.mock.calls), + ).not.toContain("host-secret-not-rotated"); + }); + it("propagates native provider identity rejection without creating runtime credentials", async () => { + responseQueue([]); + const nativeInferenceCreator = vi.fn(async () => { + throw new Error("Native inference provider identity changed"); + }); + await expect( + actionApply("default", blueprint({ identity: oktaIdentity() }), { nativeInferenceCreator }), + ).rejects.toThrow("Native inference provider identity changed"); const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); - expect(commands).not.toContain("inference set --provider test-provider --model test-model"); - expect(commands).not.toContain("provider delete test-provider"); + expect( + commands.some((command) => + /^(?:inference |provider (?:create|delete|refresh)|sandbox provider attach)/u.test(command), + ), + ).toBe(false); }); it("plans only the non-secret runtime identity binding", async () => { @@ -674,10 +794,7 @@ describe("blueprint identity wrapper", () => { { exitCode: 0, stdout: matchingProvider, stderr: "" }, ], ], - [ - "inference set --provider test-provider --model test-model", - [failureResult("route failed")], - ], + ["sandbox provider attach test-sandbox acme-okta-runtime", [failureResult("route failed")]], ]); await expect(actionApply("default", blueprint({ identity: oktaIdentity() }))).rejects.toThrow( @@ -783,12 +900,7 @@ describe("blueprint identity wrapper", () => { }); it("preserves a sandbox after a later failure without an identity component (#9833)", async () => { - responseQueue([ - [ - "inference set --provider test-provider --model test-model", - [failureResult("route failed")], - ], - ]); + responseQueue([["forward start 18789 test-sandbox", [failureResult("route failed")]]]); await expect(actionApply("default", blueprint())).rejects.toThrow( /route failed.*automatic cleanup was refused/u, @@ -828,10 +940,10 @@ describe("blueprint identity wrapper", () => { const applyCommands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); expect(applyCommands).toContain( - "sandbox create -g test-gateway --from openclaw --name test-sandbox --policy /tmp/nemoclaw-test-policy.yaml --forward 18789", + `sandbox create -g test-gateway --from openclaw --name test-sandbox --policy /tmp/nemoclaw-test-policy.yaml --no-auto-providers --label ai.nvidia.nemoclaw.create-attempt=${"a".repeat(62)} --detach`, ); - expect(applyCommands).toContain("provider get test-provider"); - expect(applyCommands).toContain("inference get"); + expect(applyCommands).not.toContain("provider get test-provider"); + expect(applyCommands).not.toContain("inference get"); expect(applyCommands).not.toContain( "inference set --provider test-provider --model test-model", ); @@ -842,7 +954,11 @@ describe("blueprint identity wrapper", () => { expect(planEntry?.content).toBeDefined(); const plan = JSON.parse(planEntry!.content!); expect(plan.sandbox_created_by_apply).toBe(true); - expect(plan.inference_provider_created_by_apply).toBe(false); + expect(plan.inference_provider_created_by_apply).toBe(true); + expect(plan.native_inference).toMatchObject({ + sandboxId: "sandbox-id-1", + providerAttachment: { providerId: "test-native-provider-id" }, + }); mockExeca.mockClear(); responseQueue([ @@ -906,47 +1022,14 @@ describe("blueprint identity wrapper", () => { await actionApply("default", blueprint({ identity: oktaIdentity() })); - expect(mockExeca).toHaveBeenCalledWith( - "openshell", - [ - "provider", - "create", - "--name", - "test-provider", - "--type", - "openai", - "--config", - "OPENAI_BASE_URL=https://api.example.com/v1", - ], - expect.objectContaining({ reject: false }), + const plan = JSON.parse( + [...store.entries()].find(([path]) => path.endsWith("/plan.json"))![1].content!, ); - const planEntry = [...store.entries()].find(([path]) => path.endsWith("/plan.json"))?.[1]; - expect(JSON.parse(planEntry!.content!)).toMatchObject({ - sandbox_created_by_apply: true, - inference_provider_created_by_apply: true, + expect(plan.native_inference).toMatchObject({ + providerAttachment: { providerName: "test-provider", providerId: "test-native-provider-id" }, + sandboxId: "sandbox-id-1", }); - - mockExeca.mockClear(); - responseQueue([ - [ - "provider get test-provider", - [{ exitCode: 0, stdout: matchingInferenceProvider, stderr: "" }], - ], - [ - "provider get acme-okta-runtime", - [ - { exitCode: 0, stdout: matchingProvider, stderr: "" }, - { exitCode: 0, stdout: matchingProvider, stderr: "" }, - ], - ], - ]); - await expect(actionRollback(JSON.parse(planEntry!.content!).run_id)).rejects.toThrow( - /mutable sandbox and provider names/u, - ); - const rollbackCommands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); - expect(rollbackCommands).not.toContain("provider delete test-provider"); - expect(rollbackCommands).not.toContain("sandbox stop -g test-gateway test-sandbox"); - expect(rollbackCommands).not.toContain("sandbox remove -g test-gateway test-sandbox"); + expect(plan.inference_provider_created_by_apply).toBe(true); }); it("refuses name-only rollback before attempting an owned sandbox removal (#9833)", async () => { @@ -1080,10 +1163,7 @@ describe("blueprint identity wrapper", () => { [failureResult("delete denied"), { exitCode: 0, stdout: "", stderr: "" }], ], ["inference get", [failureResult("inference route not found")]], - [ - "inference set --provider test-provider --model test-model", - [failureResult("route failed")], - ], + ["sandbox provider attach test-sandbox acme-okta-runtime", [failureResult("route failed")]], ]); await expect(actionApply("default", blueprint({ identity: oktaIdentity() }))).rejects.toThrow( diff --git a/nemoclaw/src/blueprint/runner-name-validation.test.ts b/nemoclaw/src/blueprint/runner-name-validation.test.ts index 257f3b5c88a..edc3e156e6b 100644 --- a/nemoclaw/src/blueprint/runner-name-validation.test.ts +++ b/nemoclaw/src/blueprint/runner-name-validation.test.ts @@ -17,6 +17,7 @@ import { resolvedEndpointFor, } from "./runner-mock-fixtures.js"; import { + createNativeInferenceFixture, minimalBlueprint, resultWithBlueprintPolicy, successResult, @@ -54,6 +55,7 @@ vi.mock("node:fs", async (importOriginal) => { }); const mockExeca = vi.fn(); +const mockNativeCreator = vi.fn(createNativeInferenceFixture); vi.mock("execa", () => ({ execa: (...args: unknown[]) => mockExeca(...args), })); @@ -123,6 +125,7 @@ describe("blueprint name validation (fail-closed integration)", () => { beforeEach(() => { store.clear(); + mockNativeCreator.mockReset().mockImplementation(createNativeInferenceFixture); addFile(TEST_SANDBOX_POLICY_PATH, TEST_SANDBOX_POLICY); vi.stubEnv("OPENSHELL_SANDBOX_POLICY", TEST_SANDBOX_POLICY_PATH); stdout.reset(); @@ -144,9 +147,12 @@ describe("blueprint name validation (fail-closed integration)", () => { const bp = minimalBlueprint(); corrupt(bp.components as BlueprintComponents); - await expect(actionApply("default", bp)).rejects.toThrow(expected); + await expect( + actionApply("default", bp, { nativeInferenceCreator: mockNativeCreator }), + ).rejects.toThrow(expected); // Validation runs before any command, so no provider or sandbox command // may have executed — not merely no sandbox create. + expect(mockNativeCreator).not.toHaveBeenCalled(); expect(mockExeca).not.toHaveBeenCalled(); expect(createCalls()).toEqual([]); }, @@ -159,12 +165,17 @@ describe("blueprint name validation (fail-closed integration)", () => { const components = bp.components as BlueprintComponents; components.inference.profiles.default.provider_name = providerName; - await expect(actionApply("default", bp)).resolves.toBeUndefined(); - expect(mockExeca).toHaveBeenCalledWith( - "openshell", - expect.arrayContaining(["provider", "create", "--name", providerName]), - expect.any(Object), + await expect( + actionApply("default", bp, { nativeInferenceCreator: mockNativeCreator }), + ).resolves.toBeUndefined(); + expect(mockNativeCreator).toHaveBeenCalledExactlyOnceWith( + expect.objectContaining({ + gatewayName: "test-gateway", + sandboxName: "test-sandbox", + provider: providerName, + }), ); + expect(createCalls()).toHaveLength(1); }, ); diff --git a/nemoclaw/src/blueprint/runner-openshell-072-policy.test.ts b/nemoclaw/src/blueprint/runner-openshell-072-policy.test.ts index 6f5c0f9df2e..9fbafed8e4b 100644 --- a/nemoclaw/src/blueprint/runner-openshell-072-policy.test.ts +++ b/nemoclaw/src/blueprint/runner-openshell-072-policy.test.ts @@ -1,3 +1,4 @@ +import { withNativeInferenceFixture } from "./runner-test-fixtures.js"; // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 @@ -60,7 +61,12 @@ vi.mock("./private-networks.js", async (importOriginal) => ({ new Set(["0.0.0.0", "10.0.0.1", "127.0.0.1", "169.254.169.254"]).has(host), })); -const { actionApply, actionReconcile, actionStatus } = await import("./runner.js"); +const { + actionApply: applyWithoutFixture, + actionReconcile, + actionStatus, +} = await import("./runner.js"); +const actionApply = withNativeInferenceFixture(applyWithoutFixture); const { validateEndpointUrl } = await import("./ssrf.js"); const mockedValidateEndpoint = vi.mocked(validateEndpointUrl); @@ -139,13 +145,24 @@ describe("blueprint policy convenience", () => { joined, ); switch (joined) { + case "sandbox get test-sandbox": + return { exitCode: 1, stdout: "", stderr: "sandbox not found" }; case "status": return gatewayStatusResult(); case "gateway info -g test-gateway": return gatewayInfoResult(); - case "policy list -g test-gateway --global --limit 1": + case "policy list -g test-gateway --global --page-size 1 --output json": return globalActive - ? { exitCode: 0, stdout: "revision 1", stderr: "" } + ? { + exitCode: 0, + stdout: JSON.stringify({ + revisions: [ + { scope: "global", version: 1, hash: "sha256:global-policy", status: "loaded" }, + ], + next_page_token: "", + }), + stderr: "", + } : absentGlobalPolicyHistoryResult(); case "policy get -g test-gateway --global --full --output json": return globalPolicyResult(livePolicy.network_policies as Record); @@ -317,13 +334,12 @@ describe("blueprint policy convenience", () => { index > policySet && command === "policy get -g test-gateway --full --output json test-sandbox", ); - const providerCreate = commands.findIndex((command) => command.startsWith("provider create ")); - const inferenceSet = commands.findIndex((command) => command.startsWith("inference set ")); + const nativeReady = commands.indexOf("forward start 18789 test-sandbox"); expect(policySet).toBeGreaterThanOrEqual(0); expect(policyVerification).toBeGreaterThan(policySet); - expect(providerCreate).toBeGreaterThan(policyVerification); - expect(inferenceSet).toBeGreaterThan(providerCreate); + expect(nativeReady).toBeGreaterThan(policyVerification); + expect(commands.some((command) => command.startsWith("inference "))).toBe(false); }); it("rebases additions when an unrelated host edit races the initial base-policy read", async () => { @@ -533,7 +549,7 @@ describe("blueprint policy convenience", () => { const implementation = mockExeca.getMockImplementation(); expect(implementation).toBeDefined(); mockExeca.mockImplementation(async (command: string, args: string[]) => - args.join(" ") === "policy list -g test-gateway --global --limit 1" + args.join(" ") === "policy list -g test-gateway --global --page-size 1 --output json" ? { exitCode: 1, stdout: "", @@ -545,7 +561,9 @@ describe("blueprint policy convenience", () => { : implementation!(command, args), ); - const failure = await actionApply("default", blueprint()).then( + const failure = await actionApply("default", blueprint(), { + nativeInferenceObserver: async () => "verified-native-authority", + }).then( () => null, (error: unknown) => error, ); @@ -571,7 +589,7 @@ describe("blueprint policy convenience", () => { expect(policyCalls).toHaveLength(1); expect(policyCalls[0]).toEqual([ "openshell", - ["policy", "list", "-g", "test-gateway", "--global", "--limit", "1"], + ["policy", "list", "-g", "test-gateway", "--global", "--page-size", "1", "--output", "json"], expect.objectContaining({ extendEnv: false, env: expect.objectContaining({ @@ -598,8 +616,8 @@ describe("blueprint policy convenience", () => { expect(implementation).toBeDefined(); mockExeca.mockImplementation(async (command: string, args: string[]) => { const joined = args.join(" "); - return joined.startsWith("sandbox create -g test-gateway ") - ? { exitCode: 1, stdout: "", stderr: "sandbox already exists" } + return joined === "sandbox get test-sandbox" + ? { exitCode: 0, stdout: "Name: test-sandbox", stderr: "" } : joined === "policy get -g test-gateway --full --output json test-sandbox" ? { exitCode: 1, @@ -612,7 +630,9 @@ describe("blueprint policy convenience", () => { : implementation!(command, args); }); - const failure = await actionApply("default", blueprint()).then( + const failure = await actionApply("default", blueprint(), { + nativeInferenceObserver: async () => "verified-native-authority", + }).then( () => null, (error: unknown) => error, ); @@ -632,9 +652,9 @@ describe("blueprint policy convenience", () => { expect(detail!.length).toBeLessThanOrEqual(240); const commands = mockExeca.mock.calls.map(([, args]) => (args ?? []).join(" ")); - expect(commands).toContain("policy list -g test-gateway --global --limit 1"); + expect(commands).toContain("policy list -g test-gateway --global --page-size 1 --output json"); expect(commands).toContain("policy get -g test-gateway --full --output json test-sandbox"); - expect(commands.filter((command) => command.startsWith("sandbox create "))).toHaveLength(1); + expect(commands.filter((command) => command.startsWith("sandbox create "))).toHaveLength(0); expect(mockExeca).toHaveBeenCalledWith( "openshell", ["policy", "get", "-g", "test-gateway", "--full", "--output", "json", "test-sandbox"], @@ -684,7 +704,7 @@ describe("blueprint policy convenience", () => { const implementation = mockExeca.getMockImplementation(); expect(implementation).toBeDefined(); mockExeca.mockImplementation(async (command: string, args: string[]) => - args.join(" ") === "policy list -g test-gateway --global --limit 1" + args.join(" ") === "policy list -g test-gateway --global --page-size 1 --output json" ? { exitCode: 0, stdout: "", stderr: "unexpected diagnostic" } : implementation!(command, args), ); diff --git a/nemoclaw/src/blueprint/runner-test-fixtures.ts b/nemoclaw/src/blueprint/runner-test-fixtures.ts index 2a75f4996ca..f044e765691 100644 --- a/nemoclaw/src/blueprint/runner-test-fixtures.ts +++ b/nemoclaw/src/blueprint/runner-test-fixtures.ts @@ -1,6 +1,48 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import type { actionApply, BlueprintNativeInferenceCreator } from "./runner.js"; + +/** The host transaction is tested in native-provider/blueprint.test.ts; these fixtures own plugin ordering. */ +export const createNativeInferenceFixture: BlueprintNativeInferenceCreator = async (input) => { + const receipt = { + schemaVersion: 1, + gatewayName: input.gatewayName, + sandboxName: input.sandboxName, + provider: input.provider, + model: input.model, + api: input.api, + endpointUrl: input.endpointUrl, + createAttemptNonce: "a".repeat(62), + credentialEnv: input.credentialValue ? "OPENAI_API_KEY" : null, + providerAttachment: { + schemaVersion: 1, + profileId: "test-native-profile", + providerName: input.provider, + providerId: "test-native-provider-id", + }, + }; + input.persist({ ...receipt, providerAttachment: undefined }); + await input.createSandbox(input.provider, receipt.createAttemptNonce); + const completed = { ...receipt, sandboxId: "sandbox-id-1" }; + input.persist({ ...completed, providerAttachment: undefined }); + await input.prepareSandboxPolicy(); + input.persist(completed); + const identity = JSON.stringify(completed); + return { identity, observe: async () => identity }; +}; + +export function withNativeInferenceFixture( + apply: typeof actionApply, + creator: BlueprintNativeInferenceCreator = createNativeInferenceFixture, +): typeof actionApply { + return (profile, blueprint, options) => + apply(profile, blueprint, { + nativeInferenceCreator: creator, + ...options, + }); +} + /** The smallest valid blueprint: one inference profile, one sandbox, empty policy. */ export function minimalBlueprint(overrides?: Record): Record { return { @@ -99,13 +141,9 @@ export function successResult(): { return { exitCode: 0, stdout: "", stderr: "" }; } -/** OpenShell 0.0.106 output when no global policy revision exists. */ +/** The empty JSON history page returned by OpenShell 0.1.3. */ export function absentGlobalPolicyHistoryResult(): CommandResult { - return { - exitCode: 0, - stdout: "", - stderr: "No global policy history found\n", - }; + return absentGlobalPolicyResult(); } export type CommandResult = { exitCode: number; stdout: string; stderr: string }; @@ -176,9 +214,8 @@ export function globalPolicyResult(networkPolicies: Record = {} stdout: JSON.stringify({ scope: "global", status: "loaded", - policy_source: "global", hash: "sha256:global-policy", - active_version: 1, + version: 1, policy: { version: 1, network_policies: networkPolicies }, }), stderr: "", @@ -191,11 +228,19 @@ export function resultWithBlueprintPolicy( fallback: CommandResult, gateway = "test-gateway", ): CommandResult { + if ( + args[0] === "sandbox" && + args[1] === "get" && + args[2] !== "-g" && + fallback.stdout === "" && + fallback.stderr === "" + ) + return failureResult("sandbox not found"); return args.join(" ") === "status" ? gatewayStatusResult(gateway) : args.join(" ") === `gateway info -g ${gateway}` ? gatewayInfoResult() - : args.join(" ") === `policy list -g ${gateway} --global --limit 1` + : args.join(" ") === `policy list -g ${gateway} --global --page-size 1 --output json` ? absentGlobalPolicyHistoryResult() : args[0] === "policy" && args[1] === "get" && @@ -283,7 +328,8 @@ export function createMutableIdentityPolicyResult(options: { const policyResult = createMutableSandboxPolicyResult(options.readMergedPolicy); return (args) => { const command = args.join(" "); - if (command === "settings get --global --json") return providersV2EnabledResult(); + if (command === "policy list --global --page-size 1 --output json") + return absentGlobalPolicyResult(); if (command === "provider get test-provider") return failureResult("provider not found"); if (command === "provider get acme-okta-runtime") { runtimeProviderReads += 1; @@ -338,8 +384,8 @@ export const MATCHING_INFERENCE_ROUTE_LISTING = [ "", ].join("\n"); -/** A `settings get` payload with gateway providers v2 enabled. */ -export function providersV2EnabledResult(): { +/** An empty page of global policy history from OpenShell 0.1.3. */ +export function absentGlobalPolicyResult(): { exitCode: number; stdout: string; stderr: string; @@ -347,9 +393,8 @@ export function providersV2EnabledResult(): { return { exitCode: 0, stdout: JSON.stringify({ - scope: "global", - settings_revision: 1, - settings: { providers_v2_enabled: "true" }, + revisions: [], + next_page_token: "", }), stderr: "", }; diff --git a/nemoclaw/src/blueprint/runner.test.ts b/nemoclaw/src/blueprint/runner.test.ts index 527287810cb..d75a82407c0 100644 --- a/nemoclaw/src/blueprint/runner.test.ts +++ b/nemoclaw/src/blueprint/runner.test.ts @@ -1,3 +1,7 @@ +import { + withNativeInferenceFixture, + createNativeInferenceFixture, +} from "./runner-test-fixtures.js"; // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 @@ -74,9 +78,18 @@ vi.mock("./private-networks.js", async (importOriginal) => ({ const { validateEndpointUrl } = await import("./ssrf.js"); const mockedValidateEndpoint = vi.mocked(validateEndpointUrl); -const { emitRunId, loadBlueprint, actionPlan, actionApply, actionStatus, actionRollback, main } = - await import("./runner.js"); -const { readFileSync } = await import("node:fs"); +const { + emitRunId, + loadBlueprint, + actionPlan, + actionApply: applyWithoutFixture, + actionStatus, + actionRollback, + main, +} = await import("./runner.js"); +const mockNativeCreator = vi.fn(createNativeInferenceFixture); +const actionApply = withNativeInferenceFixture(applyWithoutFixture, mockNativeCreator); +const { readFileSync, existsSync } = await import("node:fs"); const mockedReadFileSync = vi.mocked(readFileSync); // ── Helpers ───────────────────────────────────────────────────── @@ -111,6 +124,7 @@ function mockCurrentPolicy(stdout: string): void { describe("runner", () => { beforeEach(() => { store.clear(); + mockNativeCreator.mockReset().mockImplementation(createNativeInferenceFixture); addFile(TEST_SANDBOX_POLICY_PATH, TEST_SANDBOX_POLICY); vi.stubEnv("OPENSHELL_SANDBOX_POLICY", TEST_SANDBOX_POLICY_PATH); stdoutCapture.reset(); @@ -575,8 +589,10 @@ describe("runner", () => { "test-sandbox", "--policy", TEST_SANDBOX_POLICY_PATH, - "--forward", - "18789", + "--no-auto-providers", + "--label", + `ai.nvidia.nemoclaw.create-attempt=${"a".repeat(62)}`, + "--detach", ], expect.objectContaining({ reject: false }), ); @@ -635,16 +651,15 @@ describe("runner", () => { const hasPlanJson = (): boolean => [...store.keys()].some((k) => k.endsWith("plan.json")); - it("rejects provider creation failure with a compensated ownership plan (#6703)", async () => { + it("redacts provider creation failure while retaining the ownership plan (#6703)", async () => { const credential = "provider-secret-value"; process.env.MY_API_KEY = credential; - mockExeca.mockImplementation(async (_cmd: string, args: string[]) => - resultForCommandFailure( - args, - ["provider", "create"], - `provider setup failed\nOPENAI_API_KEY=${credential}\nAuthorization: Bearer opaque-bearer`, - ), - ); + mockNativeCreator.mockImplementationOnce(async (input) => { + input.persist({ schemaVersion: 1, sandboxId: "sandbox-id-1" }); + throw new Error( + `Failed to create inference provider 'my-provider': provider setup failed\nOPENAI_API_KEY=${credential}\nAuthorization: Bearer opaque-bearer`, + ); + }); try { const error = await actionApply("default", minimalBlueprint()).then( @@ -668,42 +683,28 @@ describe("runner", () => { } }); - it("reuses an already-existing provider instead of failing (#6703)", async () => { - mockExeca.mockImplementation(async (_cmd: string, args: string[]) => - resultForCommandFailure( - args, - ["provider", "create"], - "provider 'my-provider' already exists", - ), + it("reuses an already-recorded native sandbox without creating another provider (#6703)", async () => { + mockExeca.mockImplementation(async (_command: string, args: string[]) => + args.join(" ") === "sandbox get test-sandbox" + ? { exitCode: 0, stdout: "Name: test-sandbox", stderr: "" } + : resultWithBlueprintPolicy(args, { exitCode: 0, stdout: "", stderr: "" }), ); - - // Matches the sandbox-create contract: already-existing is a reuse, so the - // apply proceeds and completes. - await actionApply("default", minimalBlueprint()); + const nativeInferenceObserver = vi.fn(async () => "verified-native-binding"); + await actionApply("default", minimalBlueprint(), { nativeInferenceObserver }); + expect(nativeInferenceObserver).toHaveBeenCalled(); + expect(mockNativeCreator).not.toHaveBeenCalled(); expect(hasPlanJson()).toBe(true); expect(stdoutText()).toContain("Apply complete"); }); it("preserves an owned inference provider when name-only cleanup is unsafe (#9833)", async () => { - mockExeca.mockImplementation(async (_cmd: string, args: string[]) => { - if (args.join(" ") === "provider get my-provider") { - return { - exitCode: 0, - stdout: [ - "Name: my-provider", - "Type: openai", - "Credential keys: ", - "Config keys: OPENAI_BASE_URL", - "", - ].join("\n"), - stderr: "", - }; - } - return resultForCommandFailure(args, ["inference", "set"], "inference route rejected"); + mockNativeCreator.mockImplementationOnce(async (input) => { + await createNativeInferenceFixture(input); + throw new Error("Native inference route rejected"); }); await expect(actionApply("default", minimalBlueprint())).rejects.toThrow( - /Failed to set inference route .*inference route rejected.*automatic cleanup was refused/iu, + /Native inference route rejected.*automatic cleanup was refused/iu, ); expect(hasPlanJson()).toBe(true); @@ -815,25 +816,14 @@ describe("runner", () => { ); }); - it("passes credential via subprocess env, not global env", async () => { - process.env.MY_API_KEY = "secret-key-123"; - try { - await actionApply("default", minimalBlueprint()); - - // The provider create call should scope credentials to env - const providerCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("provider"), - ); - if (!providerCall) throw new Error("provider create call not found"); - expect(providerCall[2].env.OPENAI_API_KEY).toBe("secret-key-123"); - expect(providerCall[2].env.MY_API_KEY).toBeUndefined(); - // Args pass the env var NAME, not the value - expect(providerCall[1]).toContain("--credential"); - expect(providerCall[1]).toContain("OPENAI_API_KEY"); - expect(providerCall[1]).not.toContain("secret-key-123"); - } finally { - delete process.env.MY_API_KEY; - } + it("passes the selected credential only to the host creator", async () => { + vi.stubEnv("MY_API_KEY", "secret-key-123"); + await actionApply("default", minimalBlueprint()); + expect(mockNativeCreator).toHaveBeenCalledWith( + expect.objectContaining({ credentialValue: "secret-key-123" }), + ); + expect(JSON.stringify(mockExeca.mock.calls)).not.toContain("secret-key-123"); + expect(process.env.OPENAI_API_KEY).not.toBe("secret-key-123"); }); it("saves run state to disk", async () => { @@ -897,6 +887,7 @@ describe("runner", () => { [ "inference", "inference_provider_created_by_apply", + "native_inference", "gateway", "profile", "run_id", @@ -922,8 +913,7 @@ describe("runner", () => { await actionApply("default", minimalBlueprint()); const out = stdoutText(); expect(out).toContain("PROGRESS:20:Creating OpenClaw sandbox"); - expect(out).toContain("PROGRESS:50:Configuring inference provider"); - expect(out).toContain("PROGRESS:70:Setting inference route"); + expect(out).toContain("PROGRESS:70:Verifying native inference route"); expect(out).toContain("PROGRESS:85:Saving run state"); expect(out).toContain("PROGRESS:100:Apply complete"); }); @@ -938,13 +928,14 @@ describe("runner", () => { await actionApply("bare", sparseBlueprint); - // Provider create should use fallback defaults - const providerCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("provider"), + expect(mockNativeCreator).toHaveBeenCalledWith( + expect.objectContaining({ + provider: "default", + api: "openai-completions", + model: "", + endpointUrl: "", + }), ); - if (!providerCall) throw new Error("provider create call not found"); - expect(providerCall[1]).toContain("default"); // provider_name fallback - expect(providerCall[1]).toContain("openai"); // provider_type fallback // Sandbox create should use fallback defaults const sandboxCall = mockExeca.mock.calls.find( @@ -975,11 +966,9 @@ describe("runner", () => { await actionApply("nocred", noCredBlueprint); - const providerCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("provider"), + expect(mockNativeCreator).toHaveBeenCalledWith( + expect.objectContaining({ credentialValue: "" }), ); - if (!providerCall) throw new Error("provider create call not found"); - expect(providerCall[1]).not.toContain("--credential"); }); it("does not leak parent secrets into subprocess env", async () => { @@ -998,14 +987,16 @@ describe("runner", () => { try { await actionApply("default", minimalBlueprint()); - const providerCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("provider"), + const sandboxCall = mockExeca.mock.calls.find( + (c) => Array.isArray(c[1]) && c[1][0] === "sandbox" && c[1][1] === "create", ); - if (!providerCall) throw new Error("provider create call not found"); - const subEnv = providerCall[2].env; + const subEnv = sandboxCall![2].env; - // The explicitly injected credential must be present - expect(subEnv.OPENAI_API_KEY).toBe("secret-key-123"); + // The host creator receives the selected credential; other subprocesses receive none. + expect(mockNativeCreator).toHaveBeenCalledWith( + expect.objectContaining({ credentialValue: "secret-key-123" }), + ); + expect(subEnv).not.toHaveProperty("OPENAI_API_KEY"); // Secrets from the parent process must NOT be present expect(subEnv).not.toHaveProperty("GITHUB_TOKEN"); @@ -1053,11 +1044,9 @@ describe("runner", () => { await actionApply("withdefault", bp); - const providerCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("provider"), + expect(mockNativeCreator).toHaveBeenCalledWith( + expect.objectContaining({ credentialValue: "fallback-key" }), ); - if (!providerCall) throw new Error("provider create call not found"); - expect(providerCall[2].env.OPENAI_API_KEY).toBe("fallback-key"); }); it("validates and applies endpoint URL override", async () => { @@ -1084,7 +1073,7 @@ describe("runner", () => { expect(mockExeca).not.toHaveBeenCalled(); }); - it("passes --timeout when timeout_secs is set in profile", async () => { + it("passes the selected request timeout to the native creator", async () => { const bp = { components: { inference: { @@ -1110,22 +1099,15 @@ describe("runner", () => { delete process.env.OPENAI_API_KEY; } - const inferenceCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("inference") && c[1].includes("set"), + expect(mockNativeCreator).toHaveBeenCalledWith( + expect.objectContaining({ timeoutSeconds: 180 }), ); - if (!inferenceCall) throw new Error("inference set call not found"); - expect(inferenceCall[1]).toContain("--timeout"); - expect(inferenceCall[1]).toContain("180"); }); - it("omits --timeout when timeout_secs is not set in profile", async () => { + it("leaves the request timeout unspecified when the profile omits it", async () => { await actionApply("default", minimalBlueprint()); - const inferenceCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("inference") && c[1].includes("set"), - ); - if (!inferenceCall) throw new Error("inference set call not found"); - expect(inferenceCall[1]).not.toContain("--timeout"); + expect(mockNativeCreator.mock.calls[0]?.[0].timeoutSeconds).toBeUndefined(); }); it("passes endpoint as-is from blueprint (no rewriting)", async () => { @@ -1133,14 +1115,9 @@ describe("runner", () => { try { await actionApply("routed", routedBlueprint()); - const providerCall = mockExeca.mock.calls.find( - (c) => Array.isArray(c[1]) && c[1].includes("provider"), - ); - if (!providerCall) throw new Error("provider create call not found"); - const configArg = (providerCall[1] as string[]).find((a: string) => - a.startsWith("OPENAI_BASE_URL="), + expect(mockNativeCreator).toHaveBeenCalledWith( + expect.objectContaining({ endpointUrl: "http://localhost:4000/v1" }), ); - expect(configArg).toBe("OPENAI_BASE_URL=http://localhost:4000/v1"); } finally { delete process.env.NVIDIA_INFERENCE_API_KEY; } @@ -1387,6 +1364,24 @@ describe("runner", () => { }, ); + it("preserves an uncertain native create intent instead of marking rollback complete", async () => { + addDir(`${RUNS_DIR}/nc-run-1`); + addFile( + `${RUNS_DIR}/nc-run-1/plan.json`, + JSON.stringify({ + sandbox_name: "sb", + sandbox_created_by_apply: false, + inference_provider_created_by_apply: false, + native_inference: { schemaVersion: 1, createAttemptNonce: "a".repeat(62) }, + }), + ); + await expect(actionRollback("nc-run-1")).rejects.toThrow( + /preserved for identity-bound recovery/, + ); + expect(mockExeca).not.toHaveBeenCalled(); + expect(existsSync(`${RUNS_DIR}/nc-run-1/rolled_back`)).toBe(false); + }); + it("throws when rollback plan has no sandbox_name", async () => { const runDir = `${RUNS_DIR}/nc-run-1`; addDir(runDir); @@ -1453,7 +1448,9 @@ describe("runner", () => { }); it("parses apply with --profile and --endpoint-url", async () => { - await main(["apply", "--profile", "default", "--endpoint-url", "https://override.test/v1"]); + await main(["apply", "--profile", "default", "--endpoint-url", "https://override.test/v1"], { + nativeInferenceCreator: mockNativeCreator, + }); expect(mockedValidateEndpoint).toHaveBeenCalledWith("https://override.test/v1"); expect(stdoutText()).toContain("PROGRESS:100:Apply complete"); }); diff --git a/nemoclaw/src/blueprint/runner.ts b/nemoclaw/src/blueprint/runner.ts index d3683b9dd0a..444af29712c 100644 --- a/nemoclaw/src/blueprint/runner.ts +++ b/nemoclaw/src/blueprint/runner.ts @@ -58,7 +58,6 @@ import { isRuntimeIdentityConfig, isRuntimeIdentityReceipt, mintRuntimeIdentityCredential, - parseRuntimeIdentityProviderMetadata, prepareRuntimeIdentity, type RuntimeIdentityCommandDeps, type RuntimeIdentityCommandOptions, @@ -122,6 +121,7 @@ type Action = "plan" | "apply" | "status" | "reconcile" | "rollback"; type ExternalOpenShellTargetStatus = ExternalOpenShellGatewayStatus & Readonly<{ run_id: string }>; type RollbackPlanSource = { + native_inference?: unknown; sandbox_name?: unknown; sandbox_created_by_apply?: unknown; inference_provider_created_by_apply?: unknown; @@ -187,8 +187,6 @@ const BLUEPRINT_KEYS = new Set([ "openshell_target", "components", ]); -const MISSING_PROVIDER_INSPECTION_PATTERN = - /(?:\bprovider\b[^\r\n]*\b(?:not found|does not exist)\b|\b(?:not found|does not exist)\b[^\r\n]*\bprovider\b|\bunknown provider\b)/i; const MISSING_SANDBOX_INSPECTION_PATTERN = /(?:\bsandbox\b[^\r\n]*\b(?:not found|does not exist)\b|\b(?:not found|does not exist)\b[^\r\n]*\bsandbox\b|\bunknown sandbox\b)/i; const POLICY_INSPECTION_MAX_BYTES = 1024 * 1024; @@ -197,49 +195,6 @@ const POLICY_INSPECTION_DETAIL_MAX_CHARS = 240; const BLUEPRINT_POLICY_REBASE_ATTEMPTS = 3; const UNRESTRICTED_POLICY_HOSTS = new Set(["*", "0.0.0.0", "0.0.0.0/0", "::", "::/0"]); -interface InferenceRouteBinding { - provider: string; - model: string; - timeoutSeconds?: number; -} - -function parseInferenceRouteBinding(output: string): InferenceRouteBinding | null { - const lines = output.replace(/\u001b\[[0-9;]*m/g, "").split(/\r?\n/); - let inGatewayInference = false; - let provider = ""; - let model = ""; - let timeoutSeconds: number | undefined; - for (const line of lines) { - if (/^(?:Gateway )?Inference:\s*$/i.test(line)) { - inGatewayInference = true; - continue; - } - if (inGatewayInference && /^\S.*:$/.test(line)) break; - if (!inGatewayInference) continue; - const trimmed = line.trim(); - const providerMatch = /^Provider:\s*(.+)$/i.exec(trimmed); - const modelMatch = /^Model:\s*(.+)$/i.exec(trimmed); - const timeoutMatch = /^Timeout:\s*(\d+)s?$/i.exec(trimmed); - if (providerMatch) provider = providerMatch[1].trim(); - if (modelMatch) model = modelMatch[1].trim(); - if (timeoutMatch) timeoutSeconds = Number(timeoutMatch[1]); - } - return provider && model ? { provider, model, timeoutSeconds } : null; -} - -function isUnconfiguredInferenceRoute(output: string): boolean { - const lines = output.replace(/\u001b\[[0-9;]*m/g, "").split(/\r?\n/); - const headingIndex = lines.findIndex((line) => /^(?:Gateway )?Inference:\s*$/i.test(line)); - if (headingIndex < 0) return false; - const routeLines: string[] = []; - for (const line of lines.slice(headingIndex + 1)) { - if (/^\S.*:$/.test(line)) break; - const trimmed = line.trim(); - if (trimmed) routeLines.push(trimmed); - } - return routeLines.length === 1 && routeLines[0]?.toLowerCase() === "not configured"; -} - function isAction(value: string | undefined): value is Action { return ( value === "plan" || @@ -599,33 +554,6 @@ function readRollbackInferenceProviderBinding(value: RollbackPlanSource): { }; } -function assertReusableInferenceProvider( - output: string, - expected: { - name: string; - type: string; - requiresEndpointConfig: boolean; - requiresCredential: boolean; - }, -): void { - const metadata = parseRuntimeIdentityProviderMetadata(output); - const hasExpectedConfigShape = - !expected.requiresEndpointConfig || metadata?.configKeys.includes("OPENAI_BASE_URL") === true; - const hasExpectedCredentialShape = - !expected.requiresCredential || metadata?.credentialKeys.includes("OPENAI_API_KEY") === true; - if ( - !metadata || - metadata.name !== expected.name || - metadata.type !== expected.type || - !hasExpectedConfigShape || - !hasExpectedCredentialShape - ) { - throw new Error( - `Inference provider '${expected.name}' does not match the requested non-secret binding; refusing runtime identity apply`, - ); - } -} - // ── Utilities ─────────────────────────────────────────────────── export function emitRunId(): string { @@ -951,7 +879,18 @@ async function inspectBlueprintPolicy( } } const history = await runBlueprintInspectionCommand( - ["openshell", "policy", "list", "-g", gateway, "--global", "--limit", "1"], + [ + "openshell", + "policy", + "list", + "-g", + gateway, + "--global", + "--page-size", + "1", + "--output", + "json", + ], gateway, { kind: "policy", subject: "global" }, ); @@ -1364,6 +1303,7 @@ interface PersistedRunPlan { gateway: GatewayBinding; inference: SafeInferencePlan; identity?: RuntimeIdentityReceipt; + native_inference?: unknown; timestamp: string; } @@ -1458,6 +1398,7 @@ function buildPersistedRunPlan(args: { gateway: GatewayBinding; inferenceCfg: InferenceProfile; runtimeIdentityReceipt?: RuntimeIdentityReceipt; + nativeInferenceReceipt?: unknown; timestamp: string; }): PersistedRunPlan { const plan: PersistedRunPlan = { @@ -1473,6 +1414,7 @@ function buildPersistedRunPlan(args: { if (args.runtimeIdentityReceipt) { plan.identity = args.runtimeIdentityReceipt; } + if (args.nativeInferenceReceipt) plan.native_inference = args.nativeInferenceReceipt; return plan; } @@ -1700,6 +1642,29 @@ export async function actionExternalOpenShellTargetStatus( return observation; } +export type BlueprintNativeInferenceObserver = ( + requested: Readonly<{ + gatewayName: string; + sandboxName: string; + provider: string; + model: string; + api: string; + endpointUrl?: string; + requiresCredential?: boolean; + }>, +) => Promise; + +export type BlueprintNativeInferenceCreator = ( + input: Parameters[0] & + Readonly<{ + credentialValue: string; + timeoutSeconds?: number; + persist: (receipt: unknown) => void; + createSandbox: (providerName: string, createAttemptNonce: string) => Promise; + prepareSandboxPolicy: () => Promise; + }>, +) => Promise<{ identity: string; observe: () => Promise }>; + export async function actionApply( profile: string, blueprint: Blueprint, @@ -1708,6 +1673,12 @@ export async function actionApply( endpointUrl?: string; /** Code-only conformance-test seam. No CLI flag or environment input populates this policy. */ runtimeIdentityProfilePolicy?: RuntimeIdentityProfilePolicy; + nativeInferenceObserver?: BlueprintNativeInferenceObserver; + nativeInferenceCreator?: BlueprintNativeInferenceCreator; + nativeInferencePreparer?: ( + requested: Parameters[0] & + Readonly<{ timeoutSeconds: number }>, + ) => Promise; }, ): Promise { if (options?.planPath) { @@ -1752,6 +1723,38 @@ export async function actionApply( mkdirSync(stateDir, { recursive: true }); let runtimeIdentityReceipt: RuntimeIdentityReceipt | undefined; + let nativeInferenceIdentity: string | undefined; + let nativeInferenceReceipt: unknown; + let observeCreatedInference: (() => Promise) | undefined; + const observeNativeInference = async (): Promise => { + if (observeCreatedInference) return observeCreatedInference(); + if (!options?.nativeInferenceObserver) + throw new Error( + "Existing runtime identity apply requires the host native inference observer.", + ); + if (!["openai", "nvidia", "anthropic"].includes(providerType)) + throw new Error("Runtime identity apply cannot verify the requested native inference API."); + const identity = await options.nativeInferenceObserver({ + gatewayName: policyGateway.name, + sandboxName, + provider: providerName, + model, + api: providerType === "anthropic" ? "anthropic-messages" : "openai-completions", + ...(endpoint ? { endpointUrl: endpoint } : {}), + ...(credential ? { requiresCredential: true } : {}), + }); + if (typeof identity !== "string" || identity.length === 0) + throw new Error("Native inference authority could not be verified."); + return identity; + }; + const revalidateNativeInference = async (): Promise => { + if ( + nativeInferenceIdentity !== undefined && + nativeInferenceIdentity !== (await observeNativeInference()) + ) + throw new Error("Native inference identity changed during runtime identity apply."); + }; + let sandboxCreatedByApply = false; let inferenceProviderCreatedByApply = false; const persistRunPlan = (): void => { @@ -1766,6 +1769,7 @@ export async function actionApply( gateway: policyGateway, inferenceCfg, runtimeIdentityReceipt, + nativeInferenceReceipt, timestamp: new Date().toISOString(), }), ); @@ -1787,6 +1791,13 @@ export async function actionApply( throw new Error("The OpenShell gateway binding changed before sandbox creation."); } }; + const prepareSandboxPolicy = async (): Promise => { + if (Object.keys(policyAdditions).length > 0) { + progress(30, "Applying policy additions"); + await applyBlueprintPolicyAdditions(policyGateway, sandboxName, policyAdditions, stateDir); + } + assertBlueprintPolicyRequirements(await requireLivePolicy(), policyAdditions); + }; const identityDeps = runtimeIdentityDeps( (receipt) => { const previousReceipt = runtimeIdentityReceipt; @@ -1803,8 +1814,6 @@ export async function actionApply( ); try { - let reuseExistingInferenceProvider = false; - let reuseExistingInferenceRoute = false; let reuseExistingSandbox = false; progress(20, "Creating OpenClaw sandbox"); const createArgs = [ @@ -1821,12 +1830,9 @@ export async function actionApply( if (configuredSandboxPolicy) { createArgs.push("--policy", configuredSandboxPolicy.path); } - for (const port of forwardPorts) { - createArgs.push("--forward", String(port)); - } await requireCreatePolicyBoundary(); - if (runtimeIdentityConfig) { + { const sandboxResult = await runCmd(["openshell", "sandbox", "get", sandboxName], { gateway: policyGateway.name, reject: false, @@ -1834,6 +1840,7 @@ export async function actionApply( const sandboxOutput = `${sandboxResult.stderr}\n${sandboxResult.stdout}`; if (sandboxResult.exitCode === 0) { reuseExistingSandbox = true; + nativeInferenceIdentity = await observeNativeInference(); log(`Sandbox '${sandboxName}' already exists; using its current OpenShell policy.`); } else if (!MISSING_SANDBOX_INSPECTION_PATTERN.test(sandboxOutput)) { throw new Error( @@ -1842,184 +1849,117 @@ export async function actionApply( } } if (!reuseExistingSandbox) { - const createResult = await runCmd(createArgs, { - gateway: policyGateway.name, - omitSandboxPolicy: true, - reject: false, - }); - if (createResult.exitCode !== 0) { - if (createResult.stderr.includes("already exists")) { - log(`Sandbox '${sandboxName}' already exists; using its current OpenShell policy.`); - } else { - throw new Error(`Failed to create sandbox: ${boundedCommandError(createResult.stderr)}`); - } - } else { - sandboxCreatedByApply = true; - persistRunPlan(); - } - } - - persistRunPlan(); - - if (Object.keys(policyAdditions).length > 0) { - progress(30, "Applying policy additions"); - await applyBlueprintPolicyAdditions(policyGateway, sandboxName, policyAdditions, stateDir); - } - assertBlueprintPolicyRequirements(await requireLivePolicy(), policyAdditions); - - if (runtimeIdentityConfig) { - const providerResult = await runCmd(["openshell", "provider", "get", providerName], { - gateway: policyGateway.name, - reject: false, - }); - const providerOutput = `${providerResult.stderr}\n${providerResult.stdout}`; - if (providerResult.exitCode === 0) { - assertReusableInferenceProvider(providerResult.stdout, { - name: providerName, - type: providerType, - requiresEndpointConfig: endpoint !== "", - requiresCredential: credential !== "", - }); - reuseExistingInferenceProvider = true; - } else if (!MISSING_PROVIDER_INSPECTION_PATTERN.test(providerOutput)) { - throw new Error( - `Failed to inspect inference provider '${providerName}' before runtime identity apply: ${boundedCommandError(providerOutput)}`, - ); - } - if (reuseExistingInferenceProvider) { - const routeResult = await runCmd(["openshell", "inference", "get"], { - gateway: policyGateway.name, - reject: false, - }); - if (routeResult.exitCode !== 0) { + if (!options?.nativeInferenceCreator) + throw new Error("Fresh blueprint apply requires the host native inference creator."); + const created = await options + .nativeInferenceCreator({ + gatewayName: policyGateway.name, + sandboxName, + provider: providerName, + model, + api: providerType === "anthropic" ? "anthropic-messages" : "openai-completions", + endpointUrl: endpoint, + credentialValue: credential, + timeoutSeconds: inferenceCfg.timeout_secs, + persist: (receipt) => { + nativeInferenceReceipt = receipt; + inferenceProviderCreatedByApply = + isPlainObject(receipt) && isPlainObject(receipt.providerAttachment); + sandboxCreatedByApply = isPlainObject(receipt) && typeof receipt.sandboxId === "string"; + persistRunPlan(); + }, + prepareSandboxPolicy, + createSandbox: async (_nativeProviderName, createAttemptNonce) => { + await requireCreatePolicyBoundary(); + const result = await runCmd( + [ + ...createArgs, + "--no-auto-providers", + "--label", + `ai.nvidia.nemoclaw.create-attempt=${createAttemptNonce}`, + "--detach", + ], + { gateway: policyGateway.name, omitSandboxPolicy: true, reject: false }, + ); + if (result.exitCode !== 0) + throw new Error( + `Failed to create sandbox: ${boundedCommandError(result.stderr, [credential])}`, + ); + }, + }) + .catch((error: unknown) => { throw new Error( - `Failed to inspect the active inference route before runtime identity apply: ${boundedCommandError(`${routeResult.stderr}\n${routeResult.stdout}`)}`, + boundedCommandError(error instanceof Error ? error.message : String(error), [ + credential, + ]), ); - } - const activeRoute = parseInferenceRouteBinding(routeResult.stdout); - if (!activeRoute && !isUnconfiguredInferenceRoute(routeResult.stdout)) { + }); + nativeInferenceIdentity = created.identity; + observeCreatedInference = created.observe; + for (const port of forwardPorts) { + await revalidateNativeInference(); + const forwarded = await runCmd( + ["openshell", "forward", "start", String(port), sandboxName], + { + gateway: policyGateway.name, + reject: false, + }, + ); + if (forwarded.exitCode !== 0) throw new Error( - "Failed to parse the active inference route before runtime identity apply", + `Failed to forward sandbox port ${port}: ${boundedCommandError(forwarded.stderr)}`, ); - } - reuseExistingInferenceRoute = - activeRoute?.provider === providerName && - activeRoute.model === model && - (inferenceCfg.timeout_secs === undefined || - activeRoute.timeoutSeconds === inferenceCfg.timeout_secs); + await revalidateNativeInference(); } - progress(40, "Configuring runtime identity"); - runtimeIdentityReceipt = await prepareRuntimeIdentity(runtimeIdentityConfig, identityDeps); - persistRunPlan(); } - // Keep runtime credentials unattached until OpenShell accepts the - // sandbox's requested inference route. - progress(50, "Configuring inference provider"); - if (reuseExistingInferenceProvider) { - log(`Provider '${providerName}' already exists, reusing.`); - } else { - const providerArgs = [ - "openshell", - "provider", - "create", - "--name", - providerName, - "--type", - providerType, - ]; - // Pass the env-var NAME (not the value) to --credential; openshell reads the value from the env. - // Scope the credential to the subprocess to avoid leaking into later commands. - const credEnv: Record = {}; - if (credential) { - credEnv.OPENAI_API_KEY = credential; - providerArgs.push("--credential", "OPENAI_API_KEY"); - } - if (endpoint) { - providerArgs.push("--config", `OPENAI_BASE_URL=${endpoint}`); - } - const providerResult = await execa(providerArgs[0], providerArgs.slice(1), { - reject: false, - stdout: "pipe", - stderr: "pipe", - env: buildBlueprintOpenShellEnv(policyGateway.name, credEnv), - extendEnv: false, - }); - // A required mutation: a silently-ignored failure would persist plan.json and - // report a ready sandbox that cannot perform inference. Mirror the - // sandbox-create contract above — tolerate an already-existing provider as a - // reuse (keeps re-apply idempotent) and fail on any other non-zero result. - // The credential is passed via env (never argv); redact it from stderr before - // surfacing bounded diagnostic context. (#6703) - if (providerResult.exitCode !== 0) { - if (providerResult.stderr.includes("already exists")) { - if (runtimeIdentityConfig) { - const racedProvider = await runCmd(["openshell", "provider", "get", providerName], { - gateway: policyGateway.name, - reject: false, - }); - if (racedProvider.exitCode !== 0) { - throw new Error( - `Failed to inspect inference provider '${providerName}' after concurrent creation: ${boundedCommandError(`${racedProvider.stderr}\n${racedProvider.stdout}`)}`, - ); - } - assertReusableInferenceProvider(racedProvider.stdout, { - name: providerName, - type: providerType, - requiresEndpointConfig: endpoint !== "", - requiresCredential: credential !== "", - }); - } - log(`Provider '${providerName}' already exists, reusing.`); - } else { + persistRunPlan(); + + if (reuseExistingSandbox) { + await prepareSandboxPolicy(); + if (inferenceCfg.timeout_secs !== undefined) { + if (!options?.nativeInferencePreparer) throw new Error( - `Failed to create inference provider '${providerName}': ${boundedCommandError(providerResult.stderr, [credential])}`, + "Native inference timeout updates require the host configuration boundary.", ); - } - } else { - inferenceProviderCreatedByApply = true; - // Persist inference-provider ownership before a later route or policy mutation can fail. - try { - persistRunPlan(); - } catch (error) { - inferenceProviderCreatedByApply = false; - throw error; - } + await revalidateNativeInference(); + await options.nativeInferencePreparer({ + gatewayName: policyGateway.name, + sandboxName, + provider: providerName, + model, + api: providerType === "anthropic" ? "anthropic-messages" : "openai-completions", + ...(endpoint ? { endpointUrl: endpoint } : {}), + ...(credential ? { requiresCredential: true } : {}), + timeoutSeconds: inferenceCfg.timeout_secs, + }); + await revalidateNativeInference(); } } - progress(70, "Setting inference route"); - if (reuseExistingInferenceRoute) { - log(`Inference route '${providerName} / ${model}' is already active, reusing.`); - } else { - const inferenceArgs = [ - "openshell", - "inference", - "set", - "--provider", - providerName, - "--model", - model, - ]; - if (inferenceCfg.timeout_secs !== undefined) { - inferenceArgs.push("--timeout", String(inferenceCfg.timeout_secs)); - } - const inferenceResult = await runCmd(inferenceArgs, { - gateway: policyGateway.name, - reject: false, - }); - // Another required mutation: without a routed provider the sandbox cannot - // perform inference, so a non-zero result must abort the apply. (#6703) - if (inferenceResult.exitCode !== 0) { - throw new Error( - `Failed to set inference route (provider '${providerName}', model '${model}'): ${boundedCommandError(inferenceResult.stderr)}`, - ); - } + if (runtimeIdentityConfig) { + await revalidateNativeInference(); + progress(40, "Configuring runtime identity"); + log(`Runtime identity configure started at ${new Date().toISOString()}`); + runtimeIdentityReceipt = await prepareRuntimeIdentity(runtimeIdentityConfig, identityDeps); + log(`Runtime identity configure completed at ${new Date().toISOString()}`); + persistRunPlan(); } + progress(70, "Verifying native inference route"); + await revalidateNativeInference(); + log("Preserving the verified sandbox-attached inference route."); + if (runtimeIdentityReceipt) { + await revalidateNativeInference(); + log(`Runtime identity attach started at ${new Date().toISOString()}`); await attachRuntimeIdentity(runtimeIdentityReceipt, sandboxName, identityDeps); + log(`Runtime identity attach completed at ${new Date().toISOString()}`); + await revalidateNativeInference(); + log(`Runtime identity rotate started at ${new Date().toISOString()}`); await mintRuntimeIdentityCredential(runtimeIdentityReceipt, identityDeps); + log(`Runtime identity rotate completed at ${new Date().toISOString()}`); + await revalidateNativeInference(); } progress(85, "Saving run state"); @@ -2032,6 +1972,7 @@ export async function actionApply( const message = error instanceof Error ? error.message : String(error); if ( runtimeIdentityReceipt !== undefined || + nativeInferenceReceipt !== undefined || inferenceProviderCreatedByApply || sandboxCreatedByApply ) { @@ -2152,7 +2093,19 @@ export function actionStatus(rid?: string): void { log(JSON.stringify(safePlan, null, 2)); } -export async function actionReconcile(rid: string): Promise { +export type BlueprintNativeInferenceResumer = ( + input: Readonly<{ + runId: string; + gateway: Readonly; + sandboxName: string; + revalidate: () => Promise; + }>, +) => Promise; + +export async function actionReconcile( + rid: string, + resume?: BlueprintNativeInferenceResumer, +): Promise { emitRunId(); const runsDir = join(homedir(), ".nemoclaw", "state", "runs"); @@ -2161,12 +2114,14 @@ export async function actionReconcile(rid: string): Promise { let sandboxName: string; let gateway: GatewayBinding; + let nativeInferencePresent = false; try { const parsedPlan: unknown = JSON.parse(readFileSync(join(stateDir, "plan.json"), "utf-8")); if (!isPlainObject(parsedPlan)) { throw new Error("plan.json must contain a JSON object"); } const plan = parsedPlan as ReconciliationPlanSource; + nativeInferencePresent = plan.native_inference !== undefined; sandboxName = readRollbackSandboxName(plan); if (!isGatewayBinding(plan.gateway)) throw new Error("gateway binding is invalid"); gateway = plan.gateway; @@ -2174,9 +2129,20 @@ export async function actionReconcile(rid: string): Promise { const detail = error instanceof Error ? error.message : String(error); throw new Error(`Cannot read reconciliation plan for run ${rid}: ${detail}`); } - const endpoint = await inspectGatewayEndpoint(gateway.name); - if (endpoint.host !== gateway.host || endpoint.port !== gateway.port) { - throw new Error("Cannot reconcile the blueprint: the OpenShell gateway binding changed."); + const revalidate = async (): Promise => { + const endpoint = await inspectGatewayEndpoint(gateway.name); + if (endpoint.host !== gateway.host || endpoint.port !== gateway.port) { + throw new Error("Cannot reconcile the blueprint: the OpenShell gateway binding changed."); + } + }; + await revalidate(); + if (nativeInferencePresent) { + if (!resume) throw new Error("Blueprint recovery requires the host native inference resumer."); + await resume({ runId: rid, gateway, sandboxName, revalidate }); + log( + `Run ${rid}: native inference configuration is complete. Reapply the blueprint for later setup phases.`, + ); + return; } log( `Run ${rid} targets sandbox '${sandboxName}', whose policy is managed directly by OpenShell; NemoClaw has no stored policy intent to reconcile.`, @@ -2192,6 +2158,7 @@ export async function actionRollback(rid: string): Promise { const planFile = join(stateDir, "plan.json"); let sandboxName: string; + let nativeInferenceReceiptPresent = false; let sandboxCreatedByApply = false; let inferenceProviderCreatedByApply = false; let runtimeIdentityReceipt: RuntimeIdentityReceipt | undefined; @@ -2203,6 +2170,7 @@ export async function actionRollback(rid: string): Promise { ? parsedPlan : null; sandboxName = readRollbackSandboxName(rollbackPlan); + nativeInferenceReceiptPresent = rollbackPlan?.native_inference !== undefined; sandboxCreatedByApply = rollbackPlan?.sandbox_created_by_apply === true; inferenceProviderCreatedByApply = rollbackPlan?.inference_provider_created_by_apply === true; if (inferenceProviderCreatedByApply) { @@ -2220,6 +2188,7 @@ export async function actionRollback(rid: string): Promise { } if ( + nativeInferenceReceiptPresent || runtimeIdentityReceipt !== undefined || sandboxCreatedByApply || inferenceProviderCreatedByApply @@ -2246,6 +2215,13 @@ export async function main( snapshotCommand?: SnapshotCommandOptions; /** Code-only conformance-test seam. No CLI flag or environment input populates this policy. */ runtimeIdentityProfilePolicy?: RuntimeIdentityProfilePolicy; + nativeInferenceObserver?: BlueprintNativeInferenceObserver; + nativeInferenceCreator?: BlueprintNativeInferenceCreator; + nativeInferenceResumer?: BlueprintNativeInferenceResumer; + nativeInferencePreparer?: ( + requested: Parameters[0] & + Readonly<{ timeoutSeconds: number }>, + ) => Promise; } = {}, ): Promise { const rawAction = argv.at(0); @@ -2325,6 +2301,9 @@ export async function main( planPath, endpointUrl, runtimeIdentityProfilePolicy: options.runtimeIdentityProfilePolicy, + nativeInferenceObserver: options.nativeInferenceObserver, + nativeInferenceCreator: options.nativeInferenceCreator, + nativeInferencePreparer: options.nativeInferencePreparer, }); break; } @@ -2348,7 +2327,7 @@ export async function main( if (!runId) { throw new Error("--run-id is required for reconcile"); } - await actionReconcile(runId); + await actionReconcile(runId, options.nativeInferenceResumer); break; case "rollback": if (!runId) { diff --git a/nemoclaw/src/blueprint/runtime-identity.test.ts b/nemoclaw/src/blueprint/runtime-identity.test.ts index ea7d00e18f4..5d4b5649319 100644 --- a/nemoclaw/src/blueprint/runtime-identity.test.ts +++ b/nemoclaw/src/blueprint/runtime-identity.test.ts @@ -34,12 +34,11 @@ import { } from "./runtime-identity.js"; const success: RuntimeIdentityCommandResult = { exitCode: 0, stdout: "", stderr: "" }; -const providersV2Enabled: RuntimeIdentityCommandResult = { +const absentGlobalPolicy: RuntimeIdentityCommandResult = { exitCode: 0, stdout: JSON.stringify({ - scope: "global", - settings_revision: 1, - settings: { providers_v2_enabled: "true" }, + revisions: [], + next_page_token: "", }), stderr: "", }; @@ -126,6 +125,9 @@ const gatewayExportDocument = profileDocument " - name: client_id\n required: true", " - name: client_id\n required: true\n secret: false", ); +const gatewayDurationExportDocument = gatewayExportDocument + .replace(/refresh_before_seconds: (\d+)/u, 'refresh_before: "$1s"') + .replace(/max_lifetime_seconds: (\d+)/u, 'max_lifetime: "$1s"'); const entraProfileDocument = readFileSync( new URL("../../../nemoclaw-blueprint/provider-profiles/entra-runtime-v1.yaml", import.meta.url), "utf8", @@ -186,7 +188,7 @@ describe("runtime identity contract", () => { profilePath = realpathSync(profilePath); calls = []; responses = new Map(); - responses.set("settings get --global --json", [providersV2Enabled]); + responses.set("policy list --global --page-size 1 --output json", [absentGlobalPolicy]); validatedDestinations = []; persistedReceipts = []; importedProfilePaths = []; @@ -347,7 +349,7 @@ describe("runtime identity contract", () => { await expect(prepareRuntimeIdentity(config, deps)).resolves.toMatchObject(createdReceipt); expect(calls.map(({ args }) => commandKey(args))).toEqual([ - "settings get --global --json", + "policy list --global --page-size 1 --output json", "provider get acme-okta-runtime", "provider profile import --file", "provider profile export okta-runtime-v1 --output yaml", @@ -372,23 +374,28 @@ describe("runtime identity contract", () => { expect(existsSync(importedProfilePaths[0])).toBe(false); }); - it("fails before identity mutation when provider-derived policy is disabled", async () => { - responses.set("settings get --global --json", [ + it("fails before identity mutation when a global policy overrides provider composition", async () => { + responses.set("policy list --global --page-size 1 --output json", [ { exitCode: 0, stdout: JSON.stringify({ - scope: "global", - settings_revision: 0, - settings: { providers_v2_enabled: "" }, + revisions: [{ scope: "global", status: "loaded" }], + next_page_token: "", }), stderr: "", }, ]); + responses.set("policy get --global --output json", [ + { exitCode: 0, stdout: JSON.stringify({ scope: "global", status: "loaded" }), stderr: "" }, + ]); await expect(prepareRuntimeIdentity(config, deps)).rejects.toThrow( - /requires OpenShell global setting 'providers_v2_enabled=true'/, + /requires sandbox-owned policy/, ); - expect(calls.map(({ args }) => commandKey(args))).toEqual(["settings get --global --json"]); + expect(calls.map(({ args }) => commandKey(args))).toEqual([ + "policy list --global --page-size 1 --output json", + "policy get --global --output json", + ]); }); it("rejects a matching pre-existing provider before any mutable refresh operation", async () => { @@ -398,7 +405,7 @@ describe("runtime identity contract", () => { /cannot be safely reused.*prior refresh configuration cannot be restored/, ); expect(calls.map(({ args }) => commandKey(args))).toEqual([ - "settings get --global --json", + "policy list --global --page-size 1 --output json", "provider get acme-okta-runtime", ]); }); @@ -415,19 +422,34 @@ describe("runtime identity contract", () => { await expect(prepareRuntimeIdentity(config, deps)).resolves.toMatchObject(createdReceipt); }); - it("accepts OpenShell's catalog metadata and explicit default fields without a false post-mutation warning", async () => { - responses.set("provider get acme-okta-runtime", [missingProvider]); - responses.set("provider profile export okta-runtime-v1 --output yaml", [ - { exitCode: 0, stdout: gatewayExportDocument, stderr: "" }, - { exitCode: 0, stdout: gatewayExportDocument, stderr: "" }, - ]); + it.each([gatewayExportDocument, gatewayDurationExportDocument])( + "accepts equivalent exported metadata and durations without a post-mutation warning", + async (exported) => { + responses.set("provider get acme-okta-runtime", [missingProvider]); + responses.set("provider profile export okta-runtime-v1 --output yaml", [ + { exitCode: 0, stdout: exported, stderr: "" }, + { exitCode: 0, stdout: exported, stderr: "" }, + ]); - await expect(prepareRuntimeIdentity(config, deps)).resolves.toMatchObject(createdReceipt); - expect(warnings).toEqual([]); - expect(persistedReceipts).toHaveLength(1); - }); + await expect(prepareRuntimeIdentity(config, deps)).resolves.toMatchObject(createdReceipt); + expect(warnings).toEqual([]); + expect(persistedReceipts).toHaveLength(1); + }, + ); it.each([ + ...["0s", "-1s", "60.5s", "1e2s", "999999999999999999999s", "60"].map((duration) => + gatewayDurationExportDocument.replace( + /refresh_before: "[0-9]+s"/u, + `refresh_before: "${duration}"`, + ), + ), + gatewayDurationExportDocument.replace(/refresh_before: "[0-9]+s"/u, 'refresh_before: "61s"'), + gatewayDurationExportDocument.replace(/max_lifetime: "[0-9]+s"/u, 'max_lifetime: "61s"'), + gatewayDurationExportDocument.replace( + "refresh_before:", + "refresh_before_seconds: 60\n refresh_before:", + ), gatewayExportDocument.replace("source: user", "source: builtin"), gatewayExportDocument.replace("scope: workspace", "scope: platform"), gatewayExportDocument.replace("query_param: ''", "query_param: credential"), @@ -523,7 +545,7 @@ describe("runtime identity contract", () => { /profile 'okta-runtime-v1' has an incompatible binding/, ); expect(calls.map(({ args }) => commandKey(args))).toEqual([ - "settings get --global --json", + "policy list --global --page-size 1 --output json", "provider get acme-okta-runtime", "provider profile import --file", "provider profile export okta-runtime-v1 --output yaml", @@ -965,7 +987,7 @@ describe("runtime identity contract", () => { await expect(prepareRuntimeIdentity(config, deps)).rejects.toThrow(/invalid profile/); expect(calls.map(({ args }) => commandKey(args))).toEqual([ - "settings get --global --json", + "policy list --global --page-size 1 --output json", "provider get acme-okta-runtime", "provider profile import --file", ]); @@ -1033,7 +1055,7 @@ describe("runtime identity contract", () => { await expect(prepareRuntimeIdentity(config, deps)).rejects.toThrow(/cannot be safely reused/); expect(calls.map(({ args }) => commandKey(args))).toEqual([ - "settings get --global --json", + "policy list --global --page-size 1 --output json", "provider get acme-okta-runtime", ]); }); @@ -1150,14 +1172,17 @@ describe("runtime identity contract", () => { await expect(attachRuntimeIdentity(createdReceipt, "sandbox", deps)).resolves.toBe(true); expect(calls.map(({ args }) => commandKey(args))).toEqual([ - "settings get --global --json", + "policy list --global --page-size 1 --output json", "provider get acme-okta-runtime", "sandbox provider attach sandbox acme-okta-runtime", ]); }); it("warns after attachment if the verified profile changed without changing the attach result", async () => { - responses.set("settings get --global --json", [providersV2Enabled, providersV2Enabled]); + responses.set("policy list --global --page-size 1 --output json", [ + absentGlobalPolicy, + absentGlobalPolicy, + ]); responses.set("provider get acme-okta-runtime", [missingProvider, matchingProviderResult]); responses.set("provider profile export okta-runtime-v1 --output yaml", [ { exitCode: 0, stdout: profileDocument, stderr: "" }, @@ -1178,7 +1203,10 @@ describe("runtime identity contract", () => { }); it("warns after attachment if the profile export fails without changing the attach result", async () => { - responses.set("settings get --global --json", [providersV2Enabled, providersV2Enabled]); + responses.set("policy list --global --page-size 1 --output json", [ + absentGlobalPolicy, + absentGlobalPolicy, + ]); responses.set("provider get acme-okta-runtime", [missingProvider, matchingProviderResult]); responses.set("provider profile export okta-runtime-v1 --output yaml", [ { exitCode: 0, stdout: profileDocument, stderr: "" }, @@ -1204,7 +1232,7 @@ describe("runtime identity contract", () => { await expect(attachRuntimeIdentity(createdReceipt, "sandbox", deps)).resolves.toBe(true); expect(calls.map(({ args }) => commandKey(args))).toEqual([ - "settings get --global --json", + "policy list --global --page-size 1 --output json", "provider get acme-okta-runtime", "provider refresh status acme-okta-runtime --credential-key OKTA_ACCESS_TOKEN", "sandbox provider attach sandbox acme-okta-runtime", @@ -1222,23 +1250,28 @@ describe("runtime identity contract", () => { ); }); - it("fails before attachment when provider-derived policy was disabled after preparation", async () => { - responses.set("settings get --global --json", [ + it("fails before attachment when a global policy override appeared after preparation", async () => { + responses.set("policy list --global --page-size 1 --output json", [ { exitCode: 0, stdout: JSON.stringify({ - scope: "global", - settings_revision: 2, - settings: { providers_v2_enabled: "false" }, + revisions: [{ scope: "global", status: "loaded" }], + next_page_token: "", }), stderr: "", }, ]); + responses.set("policy get --global --output json", [ + { exitCode: 0, stdout: JSON.stringify({ scope: "global", status: "loaded" }), stderr: "" }, + ]); await expect(attachRuntimeIdentity(createdReceipt, "sandbox", deps)).rejects.toThrow( - /requires OpenShell global setting 'providers_v2_enabled=true'/, + /requires sandbox-owned policy/, ); - expect(calls.map(({ args }) => commandKey(args))).toEqual(["settings get --global --json"]); + expect(calls.map(({ args }) => commandKey(args))).toEqual([ + "policy list --global --page-size 1 --output json", + "policy get --global --output json", + ]); }); it("treats an existing attachment as reused", async () => { diff --git a/nemoclaw/src/blueprint/runtime-identity.ts b/nemoclaw/src/blueprint/runtime-identity.ts index 6b2c625977b..06f2ff6ba76 100644 --- a/nemoclaw/src/blueprint/runtime-identity.ts +++ b/nemoclaw/src/blueprint/runtime-identity.ts @@ -37,11 +37,10 @@ const RUNTIME_IDENTITY_REFRESH_STATUSES = new Set([ "refreshed", "rotation_requested", ]); -const MAX_SETTINGS_OUTPUT_BYTES = 16 * 1024; +const MAX_POLICY_OUTPUT_BYTES = 16 * 1024; const MAX_PROFILE_BYTES = 64 * 1024; const MAX_PROFILE_ENDPOINTS = 32; const MAX_DESTINATION_URL_LENGTH = 2048; -const PROVIDERS_V2_SETTING = "providers_v2_enabled"; const RUNTIME_IDENTITY_BINARIES = Object.freeze([ "/usr/local/bin/node", "/usr/bin/node", @@ -437,6 +436,22 @@ function parseRuntimeIdentityProfile( if (!isPlainObject(credential.refresh) || !Array.isArray(credential.refresh.material)) { continue; } + // OpenShell 0.1.3 exports protobuf durations instead of legacy seconds. + // Only whole-second values can match the reviewed imported profile. + for (const field of ["refresh_before", "max_lifetime"]) { + if (!(field in credential.refresh)) continue; + const duration = credential.refresh[field]; + const secondsField = `${field}_seconds`; + if ( + typeof duration !== "string" || + !/^[1-9][0-9]*s$/u.test(duration) || + secondsField in credential.refresh + ) { + throw new Error(`${label} has an unsupported refresh duration`); + } + credential.refresh[secondsField] = Number(duration.slice(0, -1)); + delete credential.refresh[field]; + } for (const material of credential.refresh.material) { if ( isPlainObject(material) && @@ -775,31 +790,41 @@ function requiredEnvironmentValue( } async function requireProviderDerivedPolicy(deps: RuntimeIdentityCommandDeps): Promise { - const result = await deps.run(["openshell", "settings", "get", "--global", "--json"]); - if (result.exitCode !== 0) { - throw new Error( - `Failed to inspect OpenShell provider-policy prerequisite: ${deps.formatError(commandOutput(result))}`, - ); - } - - let enabled = false; - if (Buffer.byteLength(result.stdout, "utf8") <= MAX_SETTINGS_OUTPUT_BYTES) { - try { - const document: unknown = JSON.parse(result.stdout); - enabled = - isPlainObject(document) && - isPlainObject(document.settings) && - document.settings[PROVIDERS_V2_SETTING] === "true"; - } catch { - enabled = false; + const read = async (args: string[]) => { + const result = await deps.run(["openshell", "policy", ...args]); + if (result.exitCode !== 0) { + throw new Error( + `Failed to inspect OpenShell provider-policy prerequisite: ${deps.formatError(commandOutput(result))}`, + ); } + if (Buffer.byteLength(result.stdout, "utf8") <= MAX_POLICY_OUTPUT_BYTES) { + try { + const document: unknown = JSON.parse(result.stdout); + if (isPlainObject(document)) return document; + } catch { + /* Reject incomplete or incompatible policy responses. */ + } + } + throw new Error("OpenShell returned invalid provider-policy prerequisites."); + }; + // OpenShell 0.1.3 composes providers automatically unless a global policy overrides them. + const history = await read(["list", "--global", "--page-size", "1", "--output", "json"]); + if ( + !Array.isArray(history.revisions) || + history.revisions.length > 1 || + typeof history.next_page_token !== "string" + ) + throw new Error("OpenShell returned invalid global policy history."); + if (history.revisions.length === 0) { + if (history.next_page_token !== "") + throw new Error("OpenShell returned incomplete global policy history."); + return; } - if (!enabled) { - throw new Error( - "Runtime identity requires OpenShell global setting 'providers_v2_enabled=true' " + - "so the attached provider's enforced network policy and credential injection are active", - ); - } + const policy = await read(["get", "--global", "--output", "json"]); + if (policy.scope === "global" && policy.status === "superseded") return; + throw new Error( + "Runtime identity requires sandbox-owned policy. Ask the gateway administrator to review its global policy override before retrying.", + ); } export function resolveRuntimeIdentityProfilePath( diff --git a/nemoclaw/src/shared/openshell-external-target-boundary.cts b/nemoclaw/src/shared/openshell-external-target-boundary.cts index b9d1cfc6211..27e0d473d1b 100644 --- a/nemoclaw/src/shared/openshell-external-target-boundary.cts +++ b/nemoclaw/src/shared/openshell-external-target-boundary.cts @@ -32,7 +32,7 @@ export interface OpenShellCompatibilityRange { readonly maxVersion: string; } -export const EXTERNAL_OPENSHELL_RELEASE = "0.0.116" as const; +export const EXTERNAL_OPENSHELL_RELEASE = "0.1.7" as const; export interface SanitizedExternalOpenShellTargetPlan { readonly endpoint: string; diff --git a/nemoclaw/src/shared/openshell-external-target-boundary.test.ts b/nemoclaw/src/shared/openshell-external-target-boundary.test.ts index 1f75eb6cdee..4f4eb0aa637 100644 --- a/nemoclaw/src/shared/openshell-external-target-boundary.test.ts +++ b/nemoclaw/src/shared/openshell-external-target-boundary.test.ts @@ -29,7 +29,7 @@ const CA_FILE = "/var/run/openshell-target/private-ca.pem"; const AUTHENTICATION_FILE = "/var/run/openshell-target/private-authentication"; const AUTHENTICATION_CONTENTS = "private-authentication-material"; const CA_PEM = rootCertificates[0]; -const COMPATIBILITY = { minVersion: "0.0.116", maxVersion: "0.0.116" }; +const COMPATIBILITY = { minVersion: "0.1.7", maxVersion: "0.1.7" }; const REGULAR_FILE_METADATA = { isFile: () => true, isSymbolicLink: () => false, @@ -74,7 +74,7 @@ function externalTarget() { return { endpoint: "https://openshell.example.test:8443", workspace: "default", - expected_release: "0.0.116", + expected_release: "0.1.7", lifecycle: "external", trust: { ca_file: CA_FILE }, authentication: { credential_file: AUTHENTICATION_FILE }, @@ -135,7 +135,7 @@ describe("external OpenShell target boundary", () => { expect(plan).toEqual({ endpoint: "https://openshell.example.test:8443", workspace: "default", - expected_release: "0.0.116", + expected_release: "0.1.7", lifecycle: "external", authentication_source: "file", ca_fingerprint: `sha256:${createHash("sha256") @@ -272,19 +272,19 @@ describe("external OpenShell target boundary", () => { }); it("rejects a range-compatible release that public health does not support (#9872)", () => { - const target = { ...externalTarget(), expected_release: "0.0.105" }; - const compatibility = { minVersion: "0.0.105", maxVersion: "0.0.116" }; + const target = { ...externalTarget(), expected_release: "0.1.2" }; + const compatibility = { minVersion: "0.1.2", maxVersion: "0.1.7" }; expect(() => buildSanitizedExternalOpenShellTargetPlan(target, compatibility)).toThrow( - "external OpenShell target expected_release must be 0.0.116", + "external OpenShell target expected_release must be 0.1.7", ); expect(fsMocks.openSync).not.toHaveBeenCalled(); }); it.each([ - ["non-semantic", { minVersion: "current", maxVersion: "0.0.116" }], - ["unsafe", { minVersion: "0.0.116", maxVersion: "9007199254740992.0.0" }], - ["reversed", { minVersion: "0.0.117", maxVersion: "0.0.116" }], + ["non-semantic", { minVersion: "current", maxVersion: "0.1.7" }], + ["unsafe", { minVersion: "0.1.7", maxVersion: "9007199254740992.0.0" }], + ["reversed", { minVersion: "0.1.8", maxVersion: "0.1.7" }], ])("rejects a %s compatibility range before reading files", (_name, compatibility) => { expect(() => buildSanitizedExternalOpenShellTargetPlan(externalTarget(), compatibility), diff --git a/nemoclaw/src/shared/openshell-policy-boundary.cts b/nemoclaw/src/shared/openshell-policy-boundary.cts index 52e615b0124..784b4f9f79d 100644 --- a/nemoclaw/src/shared/openshell-policy-boundary.cts +++ b/nemoclaw/src/shared/openshell-policy-boundary.cts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { isDeepStrictEqual } from "node:util"; +import { isIP } from "node:net"; import YAML from "yaml"; @@ -166,15 +167,31 @@ export type ActiveGlobalPolicyInspection = export type OpenShellGlobalPolicyHistoryState = "absent" | "present" | "invalid"; -const OPENSHELL_GLOBAL_POLICY_HISTORY_ABSENT = "No global policy history found"; - -/** Classify the exact OpenShell 0.0.106 global-policy history output contract. */ +/** Classify the first OpenShell 0.1.3 global-policy history page. */ export function classifyOpenShellGlobalPolicyHistory( stdout: string, stderr: string, ): OpenShellGlobalPolicyHistoryState { - if (stdout.trim().length > 0) return "present"; - return stderr.trim() === OPENSHELL_GLOBAL_POLICY_HISTORY_ABSENT ? "absent" : "invalid"; + if (stderr.trim().length > 0) return "invalid"; + let page: OpenShellPolicyMapping; + try { + page = parseJsonMapping(stdout, "Invalid global policy history"); + } catch { + return "invalid"; + } + if (!Array.isArray(page.revisions) || typeof page.next_page_token !== "string") return "invalid"; + if (page.revisions.length === 0) return page.next_page_token === "" ? "absent" : "invalid"; + const revision: unknown = page.revisions[0]; + return page.revisions.length === 1 && + isMapping(revision) && + revision.scope === "global" && + !Object.hasOwn(revision, "sandbox") && + positiveInteger(revision.version) && + typeof revision.hash === "string" && + POLICY_HASH_PATTERN.test(revision.hash) && + ["loaded", "superseded", "pending", "failed", "unspecified"].includes(String(revision.status)) + ? "present" + : "invalid"; } const MISSING_POLICY_DOCUMENT = @@ -254,11 +271,14 @@ export function parseActiveGlobalPolicyMetadata(raw: string): ActiveGlobalPolicy if ( metadata.scope !== "global" || (metadata.status !== "loaded" && metadata.status !== "superseded") || - metadata.policy_source !== "global" || Object.hasOwn(metadata, "sandbox") ) { throw new Error("OpenShell returned invalid global policy metadata"); } + const policyIdentity = parsePolicyIdentity( + { hash: metadata.hash, active_version: metadata.version }, + "OpenShell returned invalid global policy metadata", + ); if (metadata.status === "superseded") return { state: "absent" }; if (!isMapping(metadata.policy)) { throw new Error("OpenShell returned invalid global policy metadata"); @@ -268,10 +288,7 @@ export function parseActiveGlobalPolicyMetadata(raw: string): ActiveGlobalPolicy inspection: { policySource: "global", effectivePolicy: metadata.policy, - policyIdentity: parsePolicyIdentity( - metadata, - "OpenShell returned invalid global policy metadata", - ), + policyIdentity, }, }; } @@ -481,23 +498,7 @@ function policyHostMatches(pattern: string, host: string): boolean { const normalized = pattern.includes(":") ? normalizedHost(pattern) : pattern.toLowerCase().replace(/\.$/u, ""); - // Track consumed prefixes to avoid backtracking through repeated wildcards. - let positions = [0]; - for (const part of normalized.split(/(\*+)/u)) { - positions = [ - ...new Set( - positions.flatMap((start) => { - if (!part.startsWith("*")) { - return host.startsWith(part, start) ? [start + part.length] : []; - } - const dot = part.length === 1 ? host.indexOf(".", start) : -1; - const end = dot < 0 ? host.length : dot; - return Array.from({ length: end - start + 1 }, (_, offset) => start + offset); - }), - ), - ]; - } - return positions.includes(host.length); + return openShellHostMatches(normalized, host); } /** Match OpenShell's set comparison of configured address pins. */ @@ -553,3 +554,77 @@ export function assertOpenShellPolicyEndpointPins( } } } + +type LabelToken = "*" | ((character: string) => boolean); + +function labelTokens(label: string): LabelToken[] { + const tokens: LabelToken[] = []; + const chars = Array.from(label); + for (let index = 0; index < chars.length; index++) { + const character = chars[index]!; + if (character === "*") { + if (chars[index + 1] === "*") throw new Error("Invalid OpenShell host pattern"); + tokens.push("*"); + } else if (character === "?") tokens.push(() => true); + else if (character === "[") { + const negated = chars[index + 1] === "!"; + const start = index + (negated ? 2 : 1); + const end = chars.indexOf("]", start + 1); + if (end < 0) throw new Error("Invalid OpenShell host pattern"); + const ranges: [string, string][] = []; + for (let position = start; position < end; position++) { + const first = chars[position]!; + if (position + 2 < end && chars[position + 1] === "-") { + ranges.push([first, chars[position + 2]!]); + position += 2; + } else ranges.push([first, first]); + } + tokens.push( + (value) => ranges.some(([first, last]) => value >= first && value <= last) !== negated, + ); + index = end; + } else tokens.push((value) => value === character); + } + return tokens; +} + +function labelMatches(tokens: LabelToken[], label: string): boolean { + const characters = Array.from(label); + let positions = new Set([0]); + for (const token of tokens) { + const next = new Set(); + for (const position of positions) { + if (token === "*") { + for (let end = position; end <= characters.length; end++) next.add(end); + } else if (position < characters.length && token(characters[position]!)) + next.add(position + 1); + } + positions = next; + } + return positions.has(characters.length); +} + +/** Match OpenShell DNS-label globs without regular-expression backtracking. */ +export function openShellHostMatches(pattern: string, hostname: string): boolean { + const labels = pattern.toLowerCase().split("."); + const host = hostname.toLowerCase().split("."); + if (!pattern || /[\s{}]/u.test(pattern) || labels.some((label) => !label)) + throw new Error("Invalid OpenShell host pattern"); + if (host.some((label) => !label)) return false; + if (pattern.toLowerCase() === hostname.toLowerCase() && isIP(pattern.replace(/^\[|\]$/gu, ""))) + return true; + const compiled = labels.map((label) => (label === "**" ? "**" : labelTokens(label))); + let positions = new Set([0]); + for (const label of compiled) { + const next = new Set(); + for (const position of positions) { + if (label === "**") { + // OpenShell requires a recursive wildcard to consume at least one DNS label. + for (let end = position + 1; end <= host.length; end++) next.add(end); + } else if (position < host.length && labelMatches(label, host[position]!)) + next.add(position + 1); + } + positions = next; + } + return positions.has(host.length); +} diff --git a/nemoclaw/src/shared/openshell-policy-boundary.test.ts b/nemoclaw/src/shared/openshell-policy-boundary.test.ts index 4e56d06a407..dbb802fbe74 100644 --- a/nemoclaw/src/shared/openshell-policy-boundary.test.ts +++ b/nemoclaw/src/shared/openshell-policy-boundary.test.ts @@ -8,6 +8,7 @@ import { assertOpenShellPolicyEndpointPins, assertPolicyRequirementContainment, classifyOpenShellGlobalPolicyHistory, + openShellHostMatches, parseActiveGlobalPolicyMetadata, parseOpenShellPolicy, parseSandboxPolicyMetadata, @@ -43,9 +44,8 @@ describe("OpenShell policy boundary", () => { JSON.stringify({ scope: "global", status: "loaded", - policy_source: "global", hash: "sha256:global", - active_version: 2, + version: 2, policy: { version: 1, network_policies: {} }, }), ), @@ -144,9 +144,12 @@ describe("OpenShell policy boundary", () => { }); it("classifies the OpenShell global history absence contract", () => { - expect(classifyOpenShellGlobalPolicyHistory("", "No global policy history found\n")).toBe( - "absent", - ); + expect( + classifyOpenShellGlobalPolicyHistory( + JSON.stringify({ revisions: [], next_page_token: "" }), + "", + ), + ).toBe("absent"); }); it.each([ @@ -172,26 +175,103 @@ describe("OpenShell policy boundary", () => { }); it.each([ - ["active revision", "VERSION STATUS\n1 loaded\n", "", "present"], - ["fresh history", "", "No global policy history found\n", "absent"], - ["empty output", "", "", "invalid"], - ["unexpected diagnostic", "", "gateway warning", "invalid"], - ] as const)("classifies $0", (_name, stdout, stderr, expected) => { - expect(classifyOpenShellGlobalPolicyHistory(stdout, stderr)).toBe(expected); + ["loaded", "loaded"], + ["deleted", "superseded"], + ])("classifies a %s global revision as present history", (_label, status) => { + expect( + classifyOpenShellGlobalPolicyHistory( + JSON.stringify({ + revisions: [{ scope: "global", version: 2, hash: "sha256:global", status }], + next_page_token: "older-revision", + }), + "", + ), + ).toBe("present"); }); - it("treats a superseded global revision as absent without an identity", () => { + it.each([ + ["fresh history", { revisions: [], next_page_token: "" }, "", "absent"], + ["empty page with continuation", { revisions: [], next_page_token: "more" }, "", "invalid"], + ["missing continuation", { revisions: [] }, "", "invalid"], + ["wrong collection", { revisions: {}, next_page_token: "" }, "", "invalid"], + [ + "sandbox history", + { + revisions: [{ scope: "sandbox", version: 1, hash: "one", status: "loaded" }], + next_page_token: "", + }, + "", + "invalid", + ], + [ + "invalid revision", + { + revisions: [{ scope: "global", version: 0, hash: "one", status: "loaded" }], + next_page_token: "", + }, + "", + "invalid", + ], + [ + "unknown status", + { + revisions: [{ scope: "global", version: 1, hash: "one", status: "future" }], + next_page_token: "", + }, + "", + "invalid", + ], + [ + "diagnostic with empty page", + { revisions: [], next_page_token: "" }, + "gateway warning", + "invalid", + ], + ] as const)("classifies %s", (_name, page, stderr, expected) => { + expect(classifyOpenShellGlobalPolicyHistory(JSON.stringify(page), stderr)).toBe(expected); + }); + + it.each(["", "not json", "null", "[]", "VERSION STATUS\n1 loaded", "{}"])( + "rejects malformed global history %s", + (raw) => { + expect(classifyOpenShellGlobalPolicyHistory(raw, "")).toBe("invalid"); + }, + ); + + it("treats a superseded global revision as absent", () => { expect( parseActiveGlobalPolicyMetadata( JSON.stringify({ scope: "global", status: "superseded", - policy_source: "global", + version: 2, + hash: "sha256:global", + policy: null, }), ), ).toEqual({ state: "absent" }); }); + it.each([ + { status: "pending", version: 2 }, + { status: "failed", version: 2 }, + { status: "superseded", version: 0 }, + { status: "loaded", active_version: 2 }, + { status: "loaded", version: 2, sandbox: "alpha" }, + { status: "loaded", version: 2, hash: "invalid hash" }, + ])("refuses invalid global revision metadata %j", (fields) => { + expect(() => + parseActiveGlobalPolicyMetadata( + JSON.stringify({ + scope: "global", + hash: "sha256:global", + policy: { version: 1 }, + ...fields, + }), + ), + ).toThrow(/invalid global policy metadata/); + }); + it("requires requested sections while allowing unrelated live policy", () => { const inspection = { policySource: "global" as const, @@ -299,9 +379,79 @@ describe("OpenShell endpoint pin admission", () => { ).toThrow(/existing_policy.*IP pins.*No provider was changed/); }); + it.each([ + ["DNS", "api.example.com", "https://api.example.com/v1"], + ["IPv6", "[2001:db8::1]", "https://[2001:db8::1]/v1"], + ["all hosts", "", "https://api.example.com/v1"], + ])("admits equal address sets for %s endpoints", (_label, host, endpointUrl) => { + const base = YAML.stringify({ + version: 1, + network_policies: { + existing: { + endpoints: [{ host, port: 443, allowed_ips: ["2001:DB8::1", " 8.8.8.8 ", "8.8.8.8"] }], + }, + }, + }); + expect(() => + assertOpenShellPolicyEndpointPins(base, endpointUrl, ["8.8.8.8", "2001:db8::1"]), + ).not.toThrow(); + expect(() => assertOpenShellPolicyEndpointPins(base, endpointUrl, ["8.8.4.4"])).toThrow( + /conflicting IP pins.*No provider was changed/, + ); + }); + + it.each([ + ["policy", { existing: false }], + ["endpoint list", { existing: { endpoints: {} } }], + ["endpoint", { existing: { endpoints: ["api.example.com"] } }], + ])("refuses malformed %s before provider mutation", (_label, policies) => { + expect(() => + assertOpenShellPolicyEndpointPins( + YAML.stringify({ version: 1, network_policies: policies }), + "https://api.example.com/v1", + ["8.8.8.8"], + ), + ).toThrow(/malformed sandbox.*No provider was changed/); + }); + it("admits a host outside the canonical wildcard label boundary", () => { expect(() => assertOpenShellPolicyEndpointPins(document, "https://deep.api.example.com/v1", ["8.8.8.8"]), ).not.toThrow(); }); }); + +describe("OpenShell host patterns", () => { + it.each([ + ["api?.example", "api1.example", true], + ["api?.example", "api12.example", false], + ["api[0-9].example", "api1.example", true], + ["api[0-9].example", "apia.example", false], + ["api[!a-z].example", "api1.example", true], + ["api[!a-z].example", "apia.example", false], + ["8.8.8.8", "8.8.8.8", true], + ["[2001:db8::1]", "[2001:db8::1]", true], + ["*.example", "api..example", false], + ["*.example", "deep.api.example", false], + ["*.example", "example", false], + ["*.example", "api.example", true], + ["**.example", "deep.api.example", true], + ["**.example", "example", false], + ["api.**.example", "api.x.y.example", true], + ["api.**.example", "api.example", false], + ["API.*.example", "api.internal.example", true], + ["api[^a].example", "api1.example", false], + ["api[!a].example", "api1.example", true], + ["api[-a].example", "api-.example", true], + ["api[a-].example", "api-.example", true], + ["api[]].example", "api].example", true], + ["api[!]].example", "api1.example", true], + ] as const)("matches OpenShell pattern %s against %s: %s", (pattern, host, expected) => { + expect(openShellHostMatches(pattern, host)).toBe(expected); + }); + + it.each(["api[.example", "api**.example", "*.{prod,stage}.example", "api..example"])( + "rejects invalid host pattern %s", + (pattern) => expect(() => openShellHostMatches(pattern, "api.example")).toThrow("host pattern"), + ); +}); diff --git a/package-lock.json b/package-lock.json index 7457c4af93b..c8689250dbe 100644 --- a/package-lock.json +++ b/package-lock.json @@ -63,7 +63,7 @@ "@bufbuild/protobuf": "2.12.1", "@connectrpc/connect": "2.1.2", "@connectrpc/connect-node": "2.1.2", - "@nvidia/openshell-sdk": "0.0.116" + "@nvidia/openshell-sdk": "0.1.7" } }, "node_modules/@aws-crypto/crc32": { @@ -3953,9 +3953,9 @@ "license": "MIT" }, "node_modules/@nvidia/openshell-sdk": { - "version": "0.0.116", - "resolved": "https://npm.pkg.github.com/download/@nvidia/openshell-sdk/0.0.116/3380652847354127ed1bb8010490f8815040ada3", - "integrity": "sha512-MFnyX2vs47f/yVJRUXKmA7C6wZDH03ZGFFg+R5gZME+yMQDg9WF6G4Kdz/APM7nCUpaoGk2bwNEdUUmR60J54A==", + "version": "0.1.7", + "resolved": "https://npm.pkg.github.com/download/@nvidia/openshell-sdk/0.1.7/e34e85f18bf581cce2e99a74edc3424644cf4ed6", + "integrity": "sha512-38Zs30o/lvAxHFH+7eUINbXG+QrfFOFUY3rIxoPNXFbURSdWGoo+SNwzCHtP9ulFs9Q0odVsmRr50AImXDM5hA==", "inBundle": true, "license": "Apache-2.0", "optional": true, diff --git a/package.json b/package.json index 7ed90000790..6d96543f60d 100644 --- a/package.json +++ b/package.json @@ -126,7 +126,7 @@ "@bufbuild/protobuf": "2.12.1", "@connectrpc/connect": "2.1.2", "@connectrpc/connect-node": "2.1.2", - "@nvidia/openshell-sdk": "0.0.116" + "@nvidia/openshell-sdk": "0.1.7" }, "bundleDependencies": [ "p-retry", diff --git a/schemas/network-policy.schema.json b/schemas/network-policy.schema.json index a83562f2153..bf479f06b19 100644 --- a/schemas/network-policy.schema.json +++ b/schemas/network-policy.schema.json @@ -946,6 +946,20 @@ }, "allow_all_known_mcp_methods": { "type": "boolean" + }, + "versions": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "string", + "enum": [ + "2025-03-26", + "2025-06-18", + "2025-11-25", + "2026-07-28" + ] + } } } }, diff --git a/schemas/sandbox-policy.schema.json b/schemas/sandbox-policy.schema.json index 45a1ed2c1a5..6c4876d5863 100644 --- a/schemas/sandbox-policy.schema.json +++ b/schemas/sandbox-policy.schema.json @@ -40,7 +40,7 @@ "compatibility": { "type": "string", "enum": [ - "strict", + "hard_requirement", "best_effort" ] } diff --git a/scripts/brev-launchable-ci-cpu.sh b/scripts/brev-launchable-ci-cpu.sh index fe7d560c500..0e27f671a58 100755 --- a/scripts/brev-launchable-ci-cpu.sh +++ b/scripts/brev-launchable-ci-cpu.sh @@ -145,20 +145,20 @@ assert_openshell_version() { case "${NEMOCLAW_OPENSHELL_CHANNEL:-stable}" in stable | auto) ;; - dev) fail "NemoClaw requires exact stable OpenShell 0.0.116; the dev channel is not supported." ;; + dev) fail "NemoClaw requires exact stable OpenShell 0.1.7; the dev channel is not supported." ;; *) fail "NEMOCLAW_OPENSHELL_CHANNEL must be one of: stable, auto" ;; esac if [ -z "$OPENSHELL_VERSION" ]; then case "${NEMOCLAW_OPENSHELL_CHANNEL:-stable}" in - stable | auto) OPENSHELL_VERSION="v0.0.116" ;; + stable | auto) OPENSHELL_VERSION="v0.1.7" ;; esac fi assert_openshell_version "$OPENSHELL_VERSION" if [[ "$OPENSHELL_VERSION" != v* ]]; then OPENSHELL_VERSION="v${OPENSHELL_VERSION}" fi -if [[ "$OPENSHELL_VERSION" != "v0.0.116" ]]; then - fail "NemoClaw requires exact stable OpenShell 0.0.116; OPENSHELL_VERSION resolved to '${OPENSHELL_VERSION}'." +if [[ "$OPENSHELL_VERSION" != "v0.1.7" ]]; then + fail "NemoClaw requires exact stable OpenShell 0.1.7; OPENSHELL_VERSION resolved to '${OPENSHELL_VERSION}'." fi if [ "${1:-}" = "--print-openshell-version" ]; then printf '%s\n' "$OPENSHELL_VERSION" @@ -217,14 +217,14 @@ openshell_cli_asset_for_arch() { openshell_cli_pinned_sha256() { local release_tag="$1" asset="$2" case "${release_tag}:${asset}" in - v0.0.116:openshell-x86_64-unknown-linux-musl.tar.gz) - printf '%s\n' "4fb4476d80a1875a0b83547ec3aba999cf0a2e2d75f95f2f709b622e2103520e" + v0.1.7:openshell-x86_64-unknown-linux-musl.tar.gz) + printf '%s\n' "5c1adfcae797eac9b2e8b9280c66c1f75e4e84d9e31ce6938c5b994314ef6098" ;; - v0.0.116:openshell-aarch64-unknown-linux-musl.tar.gz) - printf '%s\n' "7a949c48d1e000cd280869eea1e203e24816b9cfefc575b68a8b72b939cb3f43" + v0.1.7:openshell-aarch64-unknown-linux-musl.tar.gz) + printf '%s\n' "696b523c263895b69eaa7c7e6afbb19b8af6c65cd79c7ffac7b685b6e3aa23da" ;; - v0.0.116:openshell-checksums-sha256.txt) - printf '%s\n' "f8b6ec65366f9d256737b884ba4d9f184b4dbbbb9540711ed9e4934d772eba7e" + v0.1.7:openshell-checksums-sha256.txt) + printf '%s\n' "a53cd40267d57969f2a1c917ad93a1b7c280dc13826211e95b276f9bd2c62f7e" ;; *) return 1 diff --git a/scripts/checks/docker-engine-27-receipt-transfer-e2e.ts b/scripts/checks/docker-engine-27-receipt-transfer-e2e.ts index b2579f77234..d8dee14bb7e 100644 --- a/scripts/checks/docker-engine-27-receipt-transfer-e2e.ts +++ b/scripts/checks/docker-engine-27-receipt-transfer-e2e.ts @@ -215,7 +215,10 @@ export function cleanupDockerEngine27ReceiptDaemon(daemonName: string): void { inspected.stdout.trim() === daemonName, "refusing to remove a Docker Engine 27 daemon with mismatched ownership", ); - requireSuccess(runDocker(["rm", "-f", daemonName], "cleanup"), "remove Docker Engine 27 daemon"); + requireSuccess( + runDocker(["rm", "-f", "--volumes", daemonName], "cleanup"), + "remove Docker Engine 27 daemon", + ); requireDockerResourceAbsent( runDocker(["container", "inspect", daemonName], "cleanup"), "Docker Engine 27 daemon", diff --git a/scripts/checks/layer-import-boundaries.mts b/scripts/checks/layer-import-boundaries.mts index b494dd378e4..9870a279cba 100644 --- a/scripts/checks/layer-import-boundaries.mts +++ b/scripts/checks/layer-import-boundaries.mts @@ -45,7 +45,7 @@ const PROVIDER_NEUTRAL_MANAGED_RUNTIME_MODULES = [ "src/lib/onboard/gateway-sandbox-reachability.ts", "src/lib/onboard/host-gateway-process.ts", "src/lib/onboard/host-service-reachability.ts", - "src/lib/onboard/managed-workload/hermes-state-volume.ts", + "src/lib/onboard/managed-workload/agent-state-volumes.ts", "src/lib/onboard/sandbox-create/orchestration.ts", "src/lib/adapters/sandbox/command-transport.ts", "src/lib/sandbox/config.ts", diff --git a/scripts/checks/managed-image-protected-runtime-contract.ts b/scripts/checks/managed-image-protected-runtime-contract.ts index 9703b2b6874..d7d756eb7e6 100644 --- a/scripts/checks/managed-image-protected-runtime-contract.ts +++ b/scripts/checks/managed-image-protected-runtime-contract.ts @@ -18,7 +18,7 @@ import type { ShippedManagedImageAgent } from "../../src/lib/onboard/managed-ima import type { ManagedStartupProfile } from "../../src/lib/onboard/managed-startup/profile.ts"; import { gatewayIdForStateDir } from "../../src/lib/onboard/gateway/process-environment.ts"; -const MANAGED_IMAGE_HARNESS_OPENSHELL_VERSIONS = ["0.0.116", "0.1.3", "0.1.5"] as const; +const MANAGED_IMAGE_HARNESS_OPENSHELL_VERSIONS = ["0.0.116", "0.1.3", "0.1.5", "0.1.7"] as const; export type ManagedImageHarnessOpenShellVersion = (typeof MANAGED_IMAGE_HARNESS_OPENSHELL_VERSIONS)[number]; @@ -63,7 +63,9 @@ export function isManagedImageHarnessWorkload( labels?.["openshell.ai/managed-by"] === "openshell" && labels["openshell.ai/sandbox-name"] === expected.sandbox && record?.Image === expected.imageContentId && - (expected.runtimeVersion === "0.1.3" || expected.runtimeVersion === "0.1.5" + (expected.runtimeVersion === "0.1.3" || + expected.runtimeVersion === "0.1.5" || + expected.runtimeVersion === "0.1.7" ? labels["openshell.ai/isolation-role"] === "sandbox" && labels["openshell.ai/sandbox-namespace"] === gatewayIdForStateDir(expected.stateDir) && record.HostConfig?.NetworkMode === "none" @@ -567,7 +569,7 @@ export function managedImageHarnessContainerListArgs( "label=openshell.ai/managed-by=openshell", "--filter", `label=openshell.ai/sandbox-name=${sandbox}`, - ...(runtimeVersion === "0.1.3" || runtimeVersion === "0.1.5" + ...(runtimeVersion === "0.1.3" || runtimeVersion === "0.1.5" || runtimeVersion === "0.1.7" ? ["--filter", "label=openshell.ai/isolation-role=sandbox"] : []), ]; @@ -576,5 +578,7 @@ export function managedImageHarnessContainerListArgs( export function managedImageSandboxListLimitFlag( runtimeVersion: ManagedImageHarnessOpenShellVersion, ): string { - return runtimeVersion === "0.1.3" || runtimeVersion === "0.1.5" ? "--page-size" : "--limit"; + return runtimeVersion === "0.1.3" || runtimeVersion === "0.1.5" || runtimeVersion === "0.1.7" + ? "--page-size" + : "--limit"; } diff --git a/scripts/checks/run-managed-image-direct-e2e.ts b/scripts/checks/run-managed-image-direct-e2e.ts index 778b775e050..39d8dbeebc4 100755 --- a/scripts/checks/run-managed-image-direct-e2e.ts +++ b/scripts/checks/run-managed-image-direct-e2e.ts @@ -132,7 +132,48 @@ function waitForNativeStartup(containerId: string): void { if (probe.status === 0) return; spawnSync("sleep", ["1"]); } - throw new Error("native managed-image startup did not reach its command boundary"); + const state = docker(["inspect", "--format", "{{json .State}}", containerId], true, 5_000); + const logs = docker(["logs", "--tail", "80", containerId], true, 5_000); + throw new Error( + `native managed-image startup did not reach its command boundary: ${JSON.stringify( + managedImageStartupFailureSummary(state, logs), + )}`, + ); +} + +/** Container output can contain runtime handles; emit only classified evidence. */ +export function managedImageStartupFailureSummary(state: CommandResult, logs: CommandResult) { + let parsed: Record = {}; + try { + const value: unknown = JSON.parse(state.stdout); + if ( + state.status === 0 && + value !== null && + typeof value === "object" && + !Array.isArray(value) + ) { + parsed = value as Record; + } + } catch { + // An unreadable state remains unknown; the startup failure still applies. + } + const statuses = ["created", "running", "paused", "restarting", "removing", "exited", "dead"]; + const output = `${logs.stdout}\n${logs.stderr}`; + return { + stateReadSucceeded: state.status === 0, + containerStatus: + typeof parsed.Status === "string" && statuses.includes(parsed.Status) ? parsed.Status : null, + running: typeof parsed.Running === "boolean" ? parsed.Running : null, + exitCode: Number.isSafeInteger(parsed.ExitCode) ? parsed.ExitCode : null, + oomKilled: typeof parsed.OOMKilled === "boolean" ? parsed.OOMKilled : null, + logsReadSucceeded: logs.status === 0, + missingProviderFunction: /fixedHostedNativeProvider is not a function/u.test(output), + providerRefreshFailed: /fixedHostedNativeProvider[\s\S]*returned non-zero exit status/u.test( + output, + ), + managedStartupFailed: /Managed startup (?:image application|coordination) failed/u.test(output), + permissionDenied: /(?:Permission denied|EACCES)/u.test(output), + }; } export function runManagedImageDirectE2e(input: ManagedImageDirectE2eInputs): void { diff --git a/scripts/checks/verify-openshell-policy-boundary-dependencies.mts b/scripts/checks/verify-openshell-policy-boundary-dependencies.mts index 48293bb026e..b4f99e4de3e 100644 --- a/scripts/checks/verify-openshell-policy-boundary-dependencies.mts +++ b/scripts/checks/verify-openshell-policy-boundary-dependencies.mts @@ -5,7 +5,7 @@ import fs from "node:fs"; import path from "node:path"; import { pathToFileURL } from "node:url"; -const ALLOWED_POLICY_BOUNDARY_MODULES = new Set(["node:util", "yaml"]); +const ALLOWED_POLICY_BOUNDARY_MODULES = new Set(["node:net", "node:util", "yaml"]); const STATIC_REQUIRE = /\brequire\s*\(\s*(["'])([^"'\\\r\n]+)\1\s*\)/g; const STATIC_IMPORT = /\bimport\s*\(\s*(["'])([^"'\\\r\n]+)\1\s*\)/g; const ANY_UNCLASSIFIED_REQUIRE = /\brequire\b/; diff --git a/scripts/find-source-shape-tests.mts b/scripts/find-source-shape-tests.mts index 7bdcda7814b..898eeadf149 100755 --- a/scripts/find-source-shape-tests.mts +++ b/scripts/find-source-shape-tests.mts @@ -79,6 +79,7 @@ type SourceFunction = { const REPO_ROOT = join(dirname(fileURLToPath(import.meta.url)), ".."); const TEST_NAME_PATTERN = /\.(test|spec)\.(js|ts|mjs|mts|cjs|cts)$/; const SKIP_DIRS = new Set([ + ".e2e", ".git", ".venv", "coverage", diff --git a/scripts/generate-openclaw-config.mts b/scripts/generate-openclaw-config.mts index ea9a8e90cdb..4b5a6645732 100755 --- a/scripts/generate-openclaw-config.mts +++ b/scripts/generate-openclaw-config.mts @@ -20,8 +20,7 @@ // NEMOCLAW_DISABLE_DEVICE_AUTH, // NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE, // NEMOCLAW_EXTRA_AGENTS_JSON_B64, -// NEMOCLAW_PROXY_HOST, NEMOCLAW_PROXY_PORT, -// NEMOCLAW_OPENCLAW_MANAGED_PROXY, NEMOCLAW_WEB_SEARCH_ENABLED, +// NEMOCLAW_WEB_SEARCH_ENABLED, // NEMOCLAW_WEB_SEARCH_PROVIDER, // NEMOCLAW_OPENCLAW_OTEL, NEMOCLAW_OPENCLAW_OTEL_ENDPOINT, // NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME, NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE, @@ -774,10 +773,6 @@ export function buildManagedInferenceSafeguardCompaction( export function buildConfig(env: Env = process.env): JsonObject { const providerless = hasProviderlessInferenceEnvironment(env); - const proxyHost = env.NEMOCLAW_PROXY_HOST || "10.200.0.1"; - const proxyPort = env.NEMOCLAW_PROXY_PORT || "3128"; - const proxyUrl = `http://${proxyHost}:${proxyPort}`; - const emitOpenClawManagedProxy = truthyEnvDefault(env, "NEMOCLAW_OPENCLAW_MANAGED_PROXY", true); const model = env.NEMOCLAW_MODEL as string; if (!providerless && !model) throw new Error("NEMOCLAW_MODEL is required"); const rawChatUiUrl = env.CHAT_UI_URL || ""; @@ -975,16 +970,22 @@ export function buildConfig(env: Env = process.env): JsonObject { }); } const nativeHosted = nativeHostedAgentConfig(upstreamProvider, inferenceBaseUrl); - const nativeLocal = ["ollama-local", "vllm-local", "llama-cpp-local", "nvidia-router"].includes( - upstreamProvider, - ) - ? nativeLocalCredentialReference(upstreamProvider, inferenceBaseUrl, "openclaw") - : null; + const nativeLocal = + ["ollama-local", "vllm-local", "llama-cpp-local", "compatible-endpoint"].includes( + upstreamProvider, + ) || env.NEMOCLAW_LOCAL_INFERENCE_TOKEN + ? nativeLocalCredentialReference( + upstreamProvider, + inferenceBaseUrl, + undefined, + env.NEMOCLAW_LOCAL_INFERENCE_TOKEN, + ) + : null; const providers = { [providerKey]: { baseUrl: inferenceBaseUrl, apiKey: - nativeHosted?.apiKey ?? nativeLocal ?? managedInferenceApiKey(inferenceBaseUrl, "unused"), + nativeLocal ?? nativeHosted?.apiKey ?? managedInferenceApiKey(inferenceBaseUrl, "unused"), ...(nativeHosted?.headers ? { headers: nativeHosted.headers } : {}), api: inferenceApi, timeoutSeconds: agentTimeout, @@ -1091,13 +1092,6 @@ export function buildConfig(env: Env = process.env): JsonObject { }, }; - if (emitOpenClawManagedProxy) { - config.proxy = { - enabled: true, - proxyUrl, - loopbackMode: "gateway-only", - }; - } if (openclawOtel) { config.diagnostics = { enabled: true, @@ -1107,7 +1101,7 @@ export function buildConfig(env: Env = process.env): JsonObject { const tools = config.tools; tools.web ??= {}; - tools.web.fetch = { enabled: true, useTrustedEnvProxy: true }; + tools.web.fetch = { enabled: true, useTrustedEnvProxy: false }; if (!webSearchProvider) { // OpenClaw treats OPENROUTER_API_KEY as Perplexity search auth when search is unspecified. tools.web.search = { enabled: false }; diff --git a/scripts/install-openshell.sh b/scripts/install-openshell.sh index e4568b7eb85..1e34f474afa 100755 --- a/scripts/install-openshell.sh +++ b/scripts/install-openshell.sh @@ -117,10 +117,10 @@ info "Detected $OS_LABEL ($ARCH_LABEL)" # round-trippable base policies: WebSocket text frames, provider-shaped # aliases, REST request bodies, MCP/JSON-RPC L7 enforcement, and # `policy get --base` for MCP/JSON-RPC-safe read-modify-write operations. -MIN_VERSION="0.0.116" +MIN_VERSION="0.1.7" # Maximum version validated for this NemoClaw release. Newer OpenShell builds # may change sandbox semantics; upgrade NemoClaw before upgrading past this. -MAX_VERSION="0.0.116" +MAX_VERSION="0.1.7" # Pin fresh installs to this version. The TS installer normally overrides this # via NEMOCLAW_OPENSHELL_PIN_VERSION after resolving the highest published # OpenShell release that satisfies the blueprint's max_openshell_version @@ -128,7 +128,7 @@ MAX_VERSION="0.0.116" PIN_VERSION="$MAX_VERSION" # Keep the base-trusted template selector aligned with the immutable release; # the dev channel is rejected below and cannot consume it. -DEV_MIN_VERSION="0.0.116" +DEV_MIN_VERSION="0.1.7" CHANNEL="${NEMOCLAW_OPENSHELL_CHANNEL:-auto}" case "$CHANNEL" in @@ -203,6 +203,233 @@ fi RELEASE_TAG="v${PIN_VERSION}" +# Prove absence through searchable parents; an inaccessible or dangling path +# cannot establish that gateway state is absent. +gateway_registration_path_is_absent() { + local current="$1" + [[ "$current" = /* ]] || return 1 + [ ! -e "$current" ] && [ ! -L "$current" ] || return 1 + while [ "$current" != "/" ]; do + current="${current%/*}" + current="${current:-/}" + if [ -e "$current" ] || [ -L "$current" ]; then + [ -d "$current" ] && [ -r "$current" ] && [ -x "$current" ] || return 1 + fi + done +} + +legacy_openshell_state_is_absent() { + local config="${XDG_CONFIG_HOME:-${HOME:+$HOME/.config}}" + local system="${OPENSHELL_SYSTEM_GATEWAY_DIR-/etc/openshell}" + ! command -v openshell >/dev/null 2>&1 || return 1 + [[ "$config" = /* && "$system" = /* && "${HOME:-}" = /* ]] || return 1 + gateway_registration_path_is_absent "$config/openshell/gateways" \ + && gateway_registration_path_is_absent "/etc/openshell/gateways" \ + && gateway_registration_path_is_absent "$system/gateways" \ + && gateway_registration_path_is_absent "$HOME/.nemoclaw/sandboxes.json" \ + && gateway_registration_path_is_absent "$HOME/.nemoclaw/gateways" +} + +# Every installation entrypoint must verify legacy state before downloading or +# replacing components. The Homebrew service-control operation above remains +# available so the installed legacy gateway can still be stopped and retired. +verify_legacy_openshell_state() { + if legacy_openshell_state_is_absent; then + return + fi + command -v node >/dev/null 2>&1 || fail "Node.js is required to verify legacy OpenShell state." + # Keep this guard in the installer: protected execution projects this file alone. + # shellcheck disable=SC2016 # JavaScript template literals are evaluated by Node.js. + node --input-type=module -e ' +import { spawnSync } from "node:child_process"; +import { closeSync, constants, fstatSync, openSync, readSync, readdirSync, lstatSync } from "node:fs"; +import { isAbsolute, join } from "node:path"; + +function deny(reason) { + throw new Error( + `Cannot replace OpenShell: ${reason}. Keep the existing gateway and sandboxes intact until an approved migration path is available. Existing components and gateway data were not changed.`, + ); +} + +function query(binary, args) { + const result = spawnSync(binary, args, { + encoding: "utf8", + timeout: 15_000, + killSignal: "SIGKILL", + maxBuffer: 1024 * 1024, + }); + if (result.error || result.status !== 0 || result.stderr.trim()) { + deny("the installed CLI could not verify legacy gateway state"); + } + return result.stdout.trim(); +} + +function missing(error) { + return error instanceof Error && "code" in error && error.code === "ENOENT"; +} + +function readRetainedJson(descriptor, label) { + const limit = 16 * 1024 * 1024; + const stat = fstatSync(descriptor); + if (!stat.isFile()) deny(`${label} is not a regular file`); + if (stat.size > limit) deny(`${label} exceeds the ${limit} byte limit`); + // Keep allocation bounded even if the opened file grows after fstat. + const bytes = Buffer.alloc(limit + 1); + let length = 0; + while (length < bytes.length) { + const count = readSync(descriptor, bytes, length, bytes.length - length, null); + if (count === 0) break; + length += count; + } + if (length > limit) deny(`${label} exceeds the ${limit} byte limit`); + return JSON.parse(bytes.toString("utf8", 0, length)); +} + +function gatewayNames() { + const config = process.env.XDG_CONFIG_HOME || join(process.env.HOME || "", ".config"); + const system = process.env.OPENSHELL_SYSTEM_GATEWAY_DIR ?? "/etc/openshell"; + if (!isAbsolute(config) || !isAbsolute(system)) + deny("gateway configuration paths are not absolute"); + const names = new Map(); + // An alternate root does not prove that default system registrations were retired. + for (const root of new Set([join(config, "openshell", "gateways"), "/etc/openshell/gateways", join(system, "gateways")])) { + let entries; + try { + entries = readdirSync(root, { withFileTypes: true }); + } catch (error) { + if (missing(error)) continue; + deny("legacy gateway registrations could not be read"); + } + for (const entry of entries) { + if (entry.isSymbolicLink()) deny("a legacy gateway registration is a symbolic link"); + if (!entry.isDirectory()) deny("a legacy gateway registration is not a directory"); + const metadataPath = join(root, entry.name, "metadata.json"); + // An absent record cannot prove retirement. Check and read the opened + // file, without following a substituted symlink or blocking on a FIFO. + const descriptor = openSync( + metadataPath, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK, + ); + let metadata; + try { + metadata = readRetainedJson(descriptor, "legacy gateway metadata"); + } finally { + closeSync(descriptor); + } + if ( + !/^[A-Za-z0-9][A-Za-z0-9_-]*$/.test(entry.name) || + metadata?.name !== entry.name || + typeof metadata.gateway_endpoint !== "string" || + !metadata.gateway_endpoint.trim() + ) { + deny("legacy gateway metadata is invalid"); + } + if (names.has(entry.name) && names.get(entry.name) !== metadata.gateway_endpoint) { + deny("user and system legacy gateway registrations disagree"); + } + names.set(entry.name, metadata.gateway_endpoint); + } + } + return [...names.keys()]; +} + +function verifyRetainedNemoClawRegistry() { + if (!isAbsolute(process.env.HOME || "")) deny("the NemoClaw state path is not absolute"); + const root = join(process.env.HOME, ".nemoclaw"); + const roots = [root]; + // Component replacement affects every gateway, including non-default ports. + for (const directory of [root, join(root, "gateways")]) { + try { + if (!lstatSync(directory).isDirectory()) deny("the NemoClaw state path is not a directory"); + if (directory !== root) { + for (const entry of readdirSync(directory, { withFileTypes: true })) { + if (!/^\d{1,5}$/.test(entry.name) || Number(entry.name) < 1 || + Number(entry.name) > 65535 || Number(entry.name) === 8080) continue; + if (!entry.isDirectory()) deny("a NemoClaw gateway state path is not a directory"); + roots.push(join(directory, entry.name)); + } + } + } catch (error) { + if (!missing(error)) throw error; + } + } + for (const directory of roots) { + let descriptor; + try { + descriptor = openSync(join(directory, "sandboxes.json"), + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + } catch (error) { + if (missing(error)) continue; + throw error; + } + try { + const registry = readRetainedJson(descriptor, "the NemoClaw sandbox registry"); + if (!registry || typeof registry !== "object" || Array.isArray(registry) || + !registry.sandboxes || typeof registry.sandboxes !== "object" || Array.isArray(registry.sandboxes)) { + deny("the NemoClaw sandbox registry is invalid"); + } + const names = Object.keys(registry.sandboxes); + if (names.some((name) => !name.trim() || name.length > 512 || + !registry.sandboxes[name] || typeof registry.sandboxes[name] !== "object" || + Array.isArray(registry.sandboxes[name]) || registry.sandboxes[name].name !== name)) { + deny("the NemoClaw sandbox registry has invalid sandbox identities"); + } + if (names.length) { + deny(`retained NemoClaw sandbox records at ${JSON.stringify(join(directory, "sandboxes.json"))}: ${names.map((name) => JSON.stringify(name)).join(", ")} do not prove that legacy state was retired`); + } + } finally { + closeSync(descriptor); + } + } +} + +function verify() { + verifyRetainedNemoClawRegistry(); + // Match the component installer PATH selection, not a different CLI override. + const binary = query("bash", ["-c", "command -v openshell || true"]); + if (binary) { + const version = /^openshell (\d+)\.(\d+)\.(\d+)(?:[.+-][A-Za-z0-9.+-]+)?$/.exec( + query(binary, ["--version"]), + ); + if (!version) deny("the installed CLI version is unknown"); + } + // Native gateway list silently omits corrupt metadata. Inspect registrations + // before querying each gateway, rather than trusting an empty gateway list. + const names = gatewayNames(); + if (names.length && (!binary || process.env.OPENSHELL_GATEWAY_ENDPOINT?.trim())) { + deny("legacy gateway inventory requires its installed CLI and recorded endpoint"); + } + for (const name of names) { + const inventory = JSON.parse(query(binary, ["sandbox", "list", "-g", name, "-o", "json"])); + const sandboxes = Array.isArray(inventory) + ? inventory + : inventory !== null && + Array.isArray(inventory.sandboxes) && + inventory.next_page_token === "" + ? inventory.sandboxes + : null; + if (!sandboxes || sandboxes.some((sandbox) => + typeof sandbox?.name !== "string" || !sandbox.name.trim() || sandbox.name.length > 512 + )) deny(`legacy gateway ${JSON.stringify(name)} has an invalid or incomplete sandbox inventory`); + if (sandboxes.length) { + deny(`legacy gateway ${JSON.stringify(name)} has sandboxes: ${sandboxes.map((sandbox) => JSON.stringify(sandbox.name)).join(", ")}`); + } + } +} + +try { + verify(); +} catch (error) { + console.error( + error instanceof Error && error.message.startsWith("Cannot replace OpenShell:") + ? error.message + : "Cannot replace OpenShell: legacy gateway or NemoClaw registry state is unreadable. Preserve the installed CLI and recorded state before retrying.", + ); + process.exitCode = 1; +} +' +} + # invalidState: a consumed OpenShell release asset differs from the digest # published for the selected immutable release, or a mutable registry tag moves. # sourceBoundary: NVIDIA/OpenShell owns the release workflow, GitHub release @@ -217,41 +444,41 @@ RELEASE_TAG="v${PIN_VERSION}" openshell_pinned_sha256() { local release_tag="$1" asset="$2" case "${release_tag}:${asset}" in - v0.0.116:openshell-x86_64-unknown-linux-musl.tar.gz) - printf '%s\n' "4fb4476d80a1875a0b83547ec3aba999cf0a2e2d75f95f2f709b622e2103520e" + v0.1.7:openshell-x86_64-unknown-linux-musl.tar.gz) + printf '%s\n' "5c1adfcae797eac9b2e8b9280c66c1f75e4e84d9e31ce6938c5b994314ef6098" ;; - v0.0.116:openshell-aarch64-unknown-linux-musl.tar.gz) - printf '%s\n' "7a949c48d1e000cd280869eea1e203e24816b9cfefc575b68a8b72b939cb3f43" + v0.1.7:openshell-aarch64-unknown-linux-musl.tar.gz) + printf '%s\n' "696b523c263895b69eaa7c7e6afbb19b8af6c65cd79c7ffac7b685b6e3aa23da" ;; - v0.0.116:openshell-aarch64-apple-darwin.tar.gz) - printf '%s\n' "e582f2374053bebac8e6aaeb4a369931b7d4bb97bd55055e2c02e85502627e22" + v0.1.7:openshell-aarch64-apple-darwin.tar.gz) + printf '%s\n' "c5899a2fe6b40c371e7d660bd8983cefb766d04658bd9193b12a80525b12c742" ;; - v0.0.116:openshell-gateway-x86_64-unknown-linux-gnu.tar.gz) - printf '%s\n' "59c6da724eae7a00c28826f9191efbdf4fbaa5c768afdc8dea6a80a949ebcc89" + v0.1.7:openshell-gateway-x86_64-unknown-linux-gnu.tar.gz) + printf '%s\n' "e1714dcfd85c949e2f53260bcfbf04b203f6f5cbd3a61e61c11c38cbc91cb102" ;; - v0.0.116:openshell-gateway-aarch64-unknown-linux-gnu.tar.gz) - printf '%s\n' "292c379193a339220234ffea585350901468bb8f4076e2076bc074e8ed18974b" + v0.1.7:openshell-gateway-aarch64-unknown-linux-gnu.tar.gz) + printf '%s\n' "eb0ba91efebbcc61e4cfae5fb18934c5fe308ece1a09ba16c999d00c452109dc" ;; - v0.0.116:openshell-gateway-aarch64-apple-darwin.tar.gz) - printf '%s\n' "f192d3d737c125264e13ef73458541df2ca6a9eb2fa599736a7f2587d5d2ce8d" + v0.1.7:openshell-gateway-aarch64-apple-darwin.tar.gz) + printf '%s\n' "fe47a385c16c10cde8905bf1bf627e4a9fbf4a81bcf44b388dec4d22618fbb48" ;; - v0.0.116:openshell-sandbox-x86_64-unknown-linux-musl.tar.gz) - printf '%s\n' "0bb160f73e5007338b94e3c868f66f50c71cd65c27c932ed9a4fa67c49e6d423" + v0.1.7:openshell-sandbox-x86_64-unknown-linux-musl.tar.gz) + printf '%s\n' "c1b3166d1c232948293ff84b0fac2d0419a24e4b694d354ba6aeb100efa2f039" ;; - v0.0.116:openshell-sandbox-aarch64-unknown-linux-musl.tar.gz) - printf '%s\n' "959d9a88270e0336f04342560df750591da603424d0a9bfb481ee29670342557" + v0.1.7:openshell-sandbox-aarch64-unknown-linux-musl.tar.gz) + printf '%s\n' "ed60eab7612f372a0f669186ac2637fe3223fe90a1c20ad51d747e1a87ca5c6d" ;; - v0.0.116:openshell-checksums-sha256.txt) - printf '%s\n' "f8b6ec65366f9d256737b884ba4d9f184b4dbbbb9540711ed9e4934d772eba7e" + v0.1.7:openshell-checksums-sha256.txt) + printf '%s\n' "a53cd40267d57969f2a1c917ad93a1b7c280dc13826211e95b276f9bd2c62f7e" ;; - v0.0.116:openshell-gateway-checksums-sha256.txt) - printf '%s\n' "572d80ded99fab0c2cf75f8108c62ab3e8455356b3c3b38de1be98806a2440e9" + v0.1.7:openshell-gateway-checksums-sha256.txt) + printf '%s\n' "20fb91f0d63581d911ea98b41d0e07175eb859950bd5f032159c168fcdf7dc1a" ;; - v0.0.116:openshell-sandbox-checksums-sha256.txt) - printf '%s\n' "0cb63b3b4436214224872c1ba245bda0d92d904822aa4f28015081269f398f93" + v0.1.7:openshell-sandbox-checksums-sha256.txt) + printf '%s\n' "f5070d63a49da60e801813ccadadc08e88cfecb5e781a7eef149d933ce144a9b" ;; - v0.0.116:openshell.rb) - printf '%s\n' "cf00a9441589702ffe006720fd6a9dffc0f0745b337036aad26dc53eb94c1558" + v0.1.7:openshell.rb) + printf '%s\n' "daa268c73d627cc72fb5c36b041e0e8625b1125cc83706f76772d52b5da081d7" ;; *) return 1 @@ -394,6 +621,26 @@ pinned_sandbox_build_version() { 7052a87d2b46ef52ecc0f7c64b9bac008dd3010c467881b0648045334eb0ed1d) printf '%s\n' "0.0.116" ;; + # OpenShell v0.1.2 standalone sandbox binaries. + 5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1 | \ + 9b527c257e7917d11cee34075369cdfb69a57764198da6e72cc0847cb9b427aa) + printf '%s\n' "0.1.2" + ;; + # OpenShell v0.1.3 standalone sandbox binaries. + 38fe063f19c72680a1bc4b0c343a3ec5b7a5cb087a627faab5b2aefd9d50d094 | \ + 23147b42d31bd0b13e1a9ed7b3834b8e6075836cba0872fffa9e7d481f6dd281) + printf '%s\n' "0.1.3" + ;; + # OpenShell v0.1.7 standalone sandbox binaries. + b4ae1d60b238c84e290bcd08a962997e5c36ed04237fa59b09b5799b11363feb | \ + d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21) + printf '%s\n' "0.1.7" + ;; + # OpenShell v0.1.5 standalone sandbox binaries. + a048b1fd2acd78f44f50a4cf0e02cdefec691656f3941e92c5ec0e09a69990e2 | \ + 65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc) + printf '%s\n' "0.1.5" + ;; *) return 1 ;; @@ -602,9 +849,9 @@ openshell_has_required_messaging_features() { return 1 fi - # MCP policy enforcement and credential replacement execute in - # openshell-sandbox. When that host artifact is present, require the native - # MCP policy marker from that exact binary. + # OpenShell 0.1.2, 0.1.3, 0.1.5, and 0.1.7 enforce MCP policy in their supervisor images. + # Require a pinned sandbox digest matching the CLI, not version text alone. + # Runtime policy verification still precedes credential/provider changes. if [ -z "$sandbox_bin" ] || [ ! -f "$sandbox_bin" ]; then # VM drivers embed a compressed supervisor, so scanning the host driver is # not authoritative. Docker/VM packaging can also keep the supervisor out @@ -616,6 +863,15 @@ openshell_has_required_messaging_features() { fi sandbox_strings="$(strings "$sandbox_bin" 2>/dev/null || true)" if [[ "$sandbox_strings" != *"$OPENSHELL_SANDBOX_MCP_FEATURE"* ]]; then + local sandbox_digest sandbox_version openshell_version + sandbox_digest="$(file_sha256 "$sandbox_bin")" || return 1 + sandbox_version="$(pinned_sandbox_build_version "$sandbox_digest")" || return 1 + openshell_version="$(component_build_version "$openshell_bin" cli)" || return 1 + case "$sandbox_version" in + 0.1.2 | 0.1.3 | 0.1.5 | 0.1.7) + [ "$sandbox_version" = "$openshell_version" ] && return 0 + ;; + esac OPENSHELL_FEATURE_CHECK_ERROR="OpenShell sandbox runtime is missing MCP/JSON-RPC L7 policy support." return 1 fi @@ -872,6 +1128,8 @@ if command -v openshell >/dev/null 2>&1; then fi fi +verify_legacy_openshell_state + info "Installing OpenShell from release '$RELEASE_TAG'..." if [ "$OS" = "Darwin" ]; then @@ -1046,7 +1304,7 @@ user_local_executable_directory_safe() { [[ -d "$current" ]] || return 1 if metadata="$(stat -c '%u %a' -- "$current" 2>/dev/null)"; then : - elif metadata="$(stat -f '%u %Lp' "$current" 2>/dev/null)"; then + elif metadata="$(stat -f '%u %Mp%03Lp' "$current" 2>/dev/null)"; then : else return 1 diff --git a/scripts/install.sh b/scripts/install.sh index cf51506e8ad..090bd9053b9 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -1158,14 +1158,14 @@ usage() { printf " HUGGING_FACE_HUB_TOKEN Compatibility alias for HF_TOKEN\n" printf " NEMOCLAW_SINGLE_SESSION=1 Abort if active sandbox sessions exist\n" printf " NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE=1\n" - printf " Allow automatic pre-0.0.37 OpenShell gateway upgrade\n" + printf " Allow automatic upgrade only for eligible gateways\n" + printf " Registered sandboxes prevent replacement of an out-of-range OpenShell version\n" printf " NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1\n" - printf " Continue only after the current CLI completes strict backup and forward retirement:\n" - printf " NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS=1 nemoclaw backup-all --retire-legacy-forwards\n" - printf " Then retire the selected gateway process:\n" - printf " openshell gateway destroy -g nemoclaw || openshell gateway destroy\n" - printf " For NEMOCLAW_GATEWAY_PORT=, destroy nemoclaw- with -g and omit the unnamed fallback\n" - printf " Resolve any failure before setting this variable\n" + printf " Resume an eligible upgrade after strict backup, forward retirement, and gateway retirement\n" + printf " With registered sandboxes, the installed OpenShell version must be verifiable and within the supported range\n" + printf " This flag does not migrate registered sandboxes from an out-of-range OpenShell version\n" + printf " Retained NemoClaw sandbox records still block component replacement, including on OpenShell 0.1.x\n" + printf " Preserve the legacy gateway, registrations, sandbox records, and data\n" printf " NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE\n" printf " Exact JSON array of pre-fingerprint managed sandbox names\n" printf " NEMOCLAW_RECREATE_SANDBOX=1 Recreate an existing sandbox\n" @@ -2826,7 +2826,7 @@ user_local_executable_directory_safe() { [[ -d "$current" ]] || return 1 if metadata="$(stat -c '%u %a' -- "$current" 2>/dev/null)"; then : - elif metadata="$(stat -f '%u %Lp' "$current" 2>/dev/null)"; then + elif metadata="$(stat -f '%u %Mp%03Lp' "$current" 2>/dev/null)"; then : else return 1 @@ -3457,7 +3457,7 @@ finish_nemoclaw_install() { if [ "$retry_gateway_port" -ne 8080 ]; then retry_gateway_port_env="NEMOCLAW_GATEWAY_PORT=${retry_gateway_port} " fi - error "Could not install the OpenShell version pinned by the prepared source after retiring the gateway. The installer preserved the sandbox backups and did not start recovery. Rerun the installer with ${retry_gateway_port_env}NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1 to reuse the prepared upgrade state and retry the OpenShell install." + error "Could not install the OpenShell version pinned by the prepared source after retiring the gateway. The installer preserved the sandbox backups and did not start recovery. If component replacement reports retained NemoClaw sandbox records, stop and preserve those records and all gateway data. Otherwise, verify the installed version and resolve the reported failure before retrying with ${retry_gateway_port_env}NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1. This flag does not bypass version or registry checks." fi # The retired gateway registration is intentionally gone. Start the newly # installed, identity-checked NemoClaw service before upgrade-sandboxes @@ -3469,7 +3469,7 @@ finish_nemoclaw_install() { info "Starting the current OpenShell gateway before sandbox recovery…" restart_selected_openshell_gateway_user_service \ "systemd:${NEMOCLAW_GATEWAY_SERVICE_NAME}.service" \ - || error "The current OpenShell gateway service could not start after the legacy gateway was retired. Sandbox backups were preserved; fix the user service and rerun with NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1." + || error "The current OpenShell gateway service could not start after the legacy gateway was retired. Sandbox backups were preserved; fix the user service and rerun with NEMOCLAW_OPENSHELL_UPGRADE_PREPARED=1. If component replacement reports retained NemoClaw sandbox records, stop and preserve those records and all gateway data." fi _OPENSHELL_INSTALL_REQUIRED_BEFORE_RECOVERY=false else @@ -3775,6 +3775,10 @@ if (process.argv[3] === "count") { process.stdout.write(String(sandboxes.length)); process.exit(0); } +if (process.argv[3] === "names") { + process.stdout.write(JSON.stringify(sandboxes.map(([name]) => name).sort())); + process.exit(0); +} if (process.argv[3] === "incompatible-names") { const compatiblePattern = /^[a-z]([a-z0-9-]*[a-z0-9])?$/; const incompatible = sandboxes @@ -4100,7 +4104,9 @@ installed_openshell_version() { command_exists openshell || return 1 local version_output version_output="$(openshell --version 2>/dev/null)" || return 1 - printf "%s\n" "$version_output" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1 + [[ "$version_output" =~ ^openshell\ ([0-9]+\.[0-9]+\.[0-9]+)([.+-][A-Za-z0-9.+-]+)?$ ]] \ + || return 1 + printf '%s\n' "${BASH_REMATCH[1]}" } # Fail closed when OpenShell is present but cannot report its version. An absent @@ -4311,6 +4317,11 @@ print_openshell_upgrade_manual_commands() { gateway_retire_command="openshell gateway destroy -g ${gateway_name} || openshell gateway destroy" fi cat </dev/null || return 0 require_openshell_compatible_sandbox_names "$reg_file" + local supported_range="" min_openshell_version="" max_openshell_version="" + supported_range="$(resolve_current_openshell_version_range)" \ + || error "Could not resolve the current OpenShell version range. Existing gateway and sandboxes were left unchanged." + read -r min_openshell_version max_openshell_version <<<"$supported_range" + # Keep the 0.1.x retained-state refusal before prepared-state handling, + # backup, gateway retirement, and the install-openshell.sh component handoff. + # The component installer repeats the guard for direct invocations. + if version_gte "$max_openshell_version" "0.1.0"; then + if ! command_exists openshell; then prefer_user_local_openshell; fi + command_exists openshell \ + || error "Could not verify the installed OpenShell version before the 0.1.x cutover. Existing gateway and sandboxes were left unchanged." + local old_cutover_version="" sandbox_names="" scope="selected" + old_cutover_version="$(installed_openshell_version || true)" + [[ "$old_cutover_version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] \ + || error "Could not verify the installed OpenShell version before the 0.1.x cutover. Existing gateway and sandboxes were left unchanged." + if ! version_gte "$old_cutover_version" "$min_openshell_version" \ + || ! version_gte "$max_openshell_version" "$old_cutover_version"; then + if [ "$(resolve_nemoclaw_gateway_port)" -eq 8080 ] \ + || [ "$reg_file" = "${HOME}/.nemoclaw/sandboxes.json" ]; then scope="legacy"; fi + sandbox_names="$(inspect_sandbox_registry_for_upgrade "$reg_file" names "$scope")" \ + || error "Could not inspect legacy sandbox names before the 0.1.x cutover. Existing gateway and sandboxes were left unchanged." + error "OpenShell 0.1.x requires sandbox recreation. Refusing to retire gateway ${gateway_name} with registered sandboxes ${sandbox_names} on OpenShell ${old_cutover_version}; existing gateway and sandboxes were left unchanged." + fi + fi if truthy_env "${NEMOCLAW_OPENSHELL_UPGRADE_PREPARED:-}"; then if [[ "${NEMOCLAW_SINGLE_SESSION:-}" == "1" ]]; then error "Aborting — NEMOCLAW_SINGLE_SESSION is set. Destroy existing sessions with '${_CLI_BIN} destroy' before reinstalling." @@ -4911,11 +4946,6 @@ preinstall_backup_and_retire_legacy_gateway() { fi confirm_legacy_managed_image_recovery "$reg_file" - local supported_range="" min_openshell_version="" max_openshell_version="" - supported_range="$(resolve_current_openshell_version_range || true)" - if [[ -n "$supported_range" ]]; then - read -r min_openshell_version max_openshell_version <<<"$supported_range" - fi local retire_legacy_forwards=false if [[ -n "$old_openshell_version" && -n "$supported_range" ]] \ && { ! version_gte "$old_openshell_version" "$min_openshell_version" \ diff --git a/scripts/lib/patch-openclaw-native-network.mts b/scripts/lib/patch-openclaw-native-network.mts new file mode 100755 index 00000000000..5dad208cde5 --- /dev/null +++ b/scripts/lib/patch-openclaw-native-network.mts @@ -0,0 +1,115 @@ +#!/usr/bin/env node +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const SCRIPT_PATH = fileURLToPath(import.meta.url); +const VERSION = "2026.9.2"; + +// OpenShell 0.1.3 returns addresses from these pools after evaluating DNS policy. +// Its supervisor checks each mapping again before connecting to the real target. +// Accept these answers only at the resolved-address boundary. URL literals, +// hostname policy, redirects, and all other address checks retain upstream rules. +export const SYNTHETIC_DNS_HELPER = [ + 'import { isIP as nemoclawIsIP } from "node:net";', + "function nemoclawIsSyntheticDnsAnswer(address) {", + '\tif (process.env.OPENSHELL_SANDBOX !== "1") return false;', + "\tif (nemoclawIsIP(address) === 4) {", + '\t\tconst [first, second] = address.split(".").map(Number);', + '\t\treturn first === 198 && (second === 18 || second === 19) && address !== "198.18.0.1";', + "\t}", + '\treturn nemoclawIsIP(address) === 6 && !address.includes("%") && /^fd23:6f70:656e:/i.test(address);', + "}", + "", +].join("\n"); + +const DNS_CHECK = "if (isBlockedHostnameOrIp(entry.address, policy))"; +const NATIVE_DNS_CHECK = + "if (!nemoclawIsSyntheticDnsAnswer(entry.address) && isBlockedHostnameOrIp(entry.address, policy))"; + +const WEB_FETCH = + "return fetchWithSsrFGuard(useEnvProxy ? withTrustedEnvProxyGuardedFetchMode(resolved) : withStrictGuardedFetchMode(resolved));"; +const NATIVE_WEB_FETCH = [ + // This existing host-gateway exception now applies without an HTTP proxy. + // The caller's blocklist and hostname allowlist still apply before DNS. + 'const hostGatewayPolicy = process.env.OPENSHELL_SANDBOX === "1" && new URL(resolved.url).hostname === "host.openshell.internal" ? { ...resolved.policy, allowedHostnames: [...resolved.policy?.allowedHostnames ?? [], "host.openshell.internal"] } : resolved.policy;', + "\tconst nativeResolved = { ...resolved, policy: hostGatewayPolicy };", + "\treturn fetchWithSsrFGuard(useEnvProxy ? withTrustedEnvProxyGuardedFetchMode(nativeResolved) : withStrictGuardedFetchMode(nativeResolved));", +].join("\n"); + +export function patchNativeNetworkText(source: string, kind: "dns" | "web-fetch"): string { + const original = kind === "dns" ? DNS_CHECK : WEB_FETCH; + if (source.split(original).length !== 2) { + throw new Error(`OpenClaw ${kind} boundary changed; expected one reviewed target`); + } + const replacement = kind === "dns" ? NATIVE_DNS_CHECK : NATIVE_WEB_FETCH; + return (kind === "dns" ? SYNTHETIC_DNS_HELPER : "") + source.replace(original, replacement); +} + +const TARGETS = [ + { + file: "ssrf-0QyXWOVG.js", + kind: "dns", + before: "cb3ab9c9e3f1cbfe22a2f74b1668e0d18f03ebf1544f47954bee7ef4060cd032", + after: "e1e1809e5ab2a298099aa842f1fa9b35e6d4f1af5ecde7700a79b8556103f757", + }, + { + file: "web-guarded-fetch-DfTFSIPT.js", + kind: "web-fetch", + before: "9c12e6147c90804b817aee328662a4837769a1928167b336a74815485d2dc633", + after: "01e6b848c1aef3bedf0e11f3e6b3ea3f46d1a9e993b9f9e505140626aa790914", + }, +] as const; + +function sha256(source: string): string { + return createHash("sha256").update(source).digest("hex"); +} + +export function patchOpenClawNativeNetwork(distDirectory: string): void { + const metadata = JSON.parse(fs.readFileSync(path.join(distDirectory, "../package.json"), "utf8")); + if (metadata.name !== "openclaw" || metadata.version !== VERSION) { + throw new Error(`Native network patch requires reviewed openclaw@${VERSION}`); + } + const descriptors: number[] = []; + try { + // Keep validation and writes on the same files if a path is replaced. + // Validate both modules before modifying either one. + const changes = TARGETS.map((target) => { + const file = path.join(distDirectory, target.file); + const fd = fs.openSync(file, fs.constants.O_RDWR | fs.constants.O_NOFOLLOW); + descriptors.push(fd); + if (!fs.fstatSync(fd).isFile()) + throw new Error(`Expected a regular OpenClaw module: ${file}`); + const source = fs.readFileSync(fd, "utf8"); + const digest = sha256(source); + if (digest === target.after) return { fd, text: source }; + if (digest !== target.before) throw new Error(`Unreviewed OpenClaw network module: ${file}`); + const text = patchNativeNetworkText(source, target.kind); + if (sha256(text) !== target.after) + throw new Error(`Unexpected native network patch output: ${file}`); + return { fd, text }; + }); + for (const { fd, text } of changes) { + const bytes = Buffer.from(text); + let offset = 0; + while (offset < bytes.length) { + const written = fs.writeSync(fd, bytes, offset, bytes.length - offset, offset); + if (written === 0) throw new Error("OpenClaw module write made no progress"); + offset += written; + } + fs.ftruncateSync(fd, bytes.length); + } + } finally { + for (const fd of descriptors) fs.closeSync(fd); + } +} + +if (process.argv[1] && path.resolve(process.argv[1]) === SCRIPT_PATH) { + if (process.argv.length !== 3) + throw new Error("Usage: patch-openclaw-native-network.mts "); + patchOpenClawNativeNetwork(path.resolve(process.argv[2]!)); +} diff --git a/scripts/nemoclaw-start.sh b/scripts/nemoclaw-start.sh index 50167edafdd..2dad2b4a8ed 100755 --- a/scripts/nemoclaw-start.sh +++ b/scripts/nemoclaw-start.sh @@ -888,11 +888,11 @@ parsed = subprocess.run( "/usr/local/bin/node", "-e", 'const JSON5=require("/usr/local/lib/node_modules/openclaw/node_modules/json5");' - 'const {hostedNativeProvider,NVIDIA_HOSTED_NATIVE_ENDPOINT,NVIDIA_HOSTED_CREDENTIAL_ENV}=' + 'const {fixedHostedNativeProvider,NVIDIA_HOSTED_NATIVE_ENDPOINT,NVIDIA_HOSTED_CREDENTIAL_ENV}=' 'require("/usr/local/lib/nemoclaw/managed-startup-image-runtime.cjs");' 'process.stdout.write(JSON.stringify({' 'config:JSON5.parse(require("node:fs").readFileSync(0,"utf8")),' - 'openrouter:hostedNativeProvider("openrouter-api"),' + 'openrouter:fixedHostedNativeProvider("openrouter-api"),' 'nvidia:{endpoint:NVIDIA_HOSTED_NATIVE_ENDPOINT,credentialEnv:NVIDIA_HOSTED_CREDENTIAL_ENV}}));', ], input=source, @@ -1522,7 +1522,7 @@ validate_nemoclaw_tmp_permissions() { [ -n "$_target" ] && _dynamic_targets+=("$_target") done < <(messaging_runtime_preload_targets) - validate_tmp_permissions "$_SANDBOX_SAFETY_NET" "$_PROXY_FIX_SCRIPT" "$_NEMOTRON_FIX_SCRIPT" "${_dynamic_targets[@]+"${_dynamic_targets[@]}"}" + validate_tmp_permissions "$_SANDBOX_SAFETY_NET" "$_NEMOTRON_FIX_SCRIPT" "${_dynamic_targets[@]+"${_dynamic_targets[@]}"}" } verify_messaging_runtime_secret_scans() { @@ -1667,6 +1667,18 @@ try { hooks = hooks[key] ?? (hooks[key] = {}); } hooks.allowConversationAccess = true; + // Restored configs must use the same native egress path as fresh images. + delete cfg.proxy; + if (cfg.tools?.web?.fetch && typeof cfg.tools.web.fetch === "object") { + cfg.tools.web.fetch.useTrustedEnvProxy = false; + } + const telegram = cfg.channels?.telegram; + if (telegram && typeof telegram === "object") { + delete telegram.proxy; + for (const account of Object.values(telegram.accounts ?? {})) { + if (account && typeof account === "object") delete account.proxy; + } + } const existingToken = cfg.gateway?.auth?.token; if (process.argv[4] !== "if-missing" || typeof existingToken !== "string" || !existingToken || existingToken === "[STRIPPED_BY_MIGRATION]") { @@ -2935,28 +2947,10 @@ prepare_auto_pair_log() { fi } -# ── Proxy environment ──────────────────────────────────────────── -# OpenShell injects HTTP_PROXY/HTTPS_PROXY/NO_PROXY into the sandbox, but its -# NO_PROXY is limited to 127.0.0.1,localhost,::1 — missing the gateway IP. -# The gateway IP itself must bypass the proxy to avoid proxy loops. -# -# Do NOT add inference.local here. OpenShell intentionally routes that hostname -# through the proxy path; bypassing the proxy forces a direct DNS lookup inside -# the sandbox, which breaks inference.local resolution. -# -# NEMOCLAW_PROXY_HOST / NEMOCLAW_PROXY_PORT can be overridden at sandbox -# creation time if the gateway IP or port changes in a future OpenShell release. -# Ref: https://github.com/NVIDIA/NemoClaw/issues/626 -PROXY_HOST="${NEMOCLAW_PROXY_HOST:-10.200.0.1}" -PROXY_PORT="${NEMOCLAW_PROXY_PORT:-3128}" -_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}" -_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}" -export HTTP_PROXY="$_PROXY_URL" -export HTTPS_PROXY="$_PROXY_URL" -export NO_PROXY="$_NO_PROXY_VAL" -export http_proxy="$_PROXY_URL" -export https_proxy="$_PROXY_URL" -export no_proxy="$_NO_PROXY_VAL" +# OpenShell 0.1.3 mediates native egress. Clear obsolete application proxy settings. +unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy +unset ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY +unset NODE_USE_ENV_PROXY NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT # Corporate proxy CA merge (NemoClaw#6210). # OpenShell injects SSL_CERT_FILE for its own L7 proxy CA at runtime. When a @@ -2992,22 +2986,6 @@ if [ -n "${SSL_CERT_FILE:-}" ] && [ -f "${SSL_CERT_FILE}" ]; then export GIT_SSL_CAINFO="$SSL_CERT_FILE" fi -# HTTP library + NODE_USE_ENV_PROXY double-proxy fix (NemoClaw#2109). -# Node.js 22 sets NODE_USE_ENV_PROXY=1 in the OpenShell base image, which -# intercepts https.request() calls and handles proxying via CONNECT tunnel. -# HTTP libraries (axios, follow-redirects, proxy-from-env) also read -# HTTPS_PROXY and configure HTTP FORWARD mode, double-processing the -# request — the L7 proxy rejects with "FORWARD rejected: HTTPS requires -# CONNECT". -# -# The preload wraps http.request() — the lowest common denominator every -# HTTP client bottoms out at — and rewrites FORWARD-mode requests back to -# https.request() so NODE_USE_ENV_PROXY can handle the CONNECT tunnel. -# -# Earlier PR #2110 intercepted require('axios') via a Module._load hook; -# that could not catch follow-redirects + proxy-from-env bundled as ESM -# in OpenClaw's dist/ (no require() calls to intercept). -# # Node runtime preload modules are copied into /usr/local/lib/nemoclaw/preloads/ # at image build time, then copied to /tmp before NODE_OPTIONS=--require so # the sandbox user can read them under Landlock-constrained runtimes. @@ -3030,9 +3008,6 @@ fi _SANDBOX_SAFETY_NET="/tmp/nemoclaw-sandbox-safety-net.js" _SANDBOX_SAFETY_NET_SOURCE="/usr/local/lib/nemoclaw/preloads/sandbox-safety-net.js" -_PROXY_FIX_SCRIPT="/tmp/nemoclaw-http-proxy-fix.js" -_PROXY_FIX_SOURCE="/usr/local/lib/nemoclaw/preloads/http-proxy-fix.js" - # NVIDIA endpoint model-specific inference parameter injection # (NemoClaw#1193, NemoClaw#2051). # Nemotron models may return empty content (tool call instead of text) or @@ -3067,31 +3042,14 @@ install_core_runtime_preloads() { emit_sandbox_sourced_file "$_SANDBOX_SAFETY_NET" <"$_SANDBOX_SAFETY_NET_SOURCE" || return 1 append_node_require_once "$_SANDBOX_SAFETY_NET" - if [ "${NODE_USE_ENV_PROXY:-}" = "1" ]; then - emit_sandbox_sourced_file "$_PROXY_FIX_SCRIPT" <"$_PROXY_FIX_SOURCE" || return 1 - append_node_require_once "$_PROXY_FIX_SCRIPT" - fi - emit_sandbox_sourced_file "$_NEMOTRON_FIX_SCRIPT" <"$_NEMOTRON_FIX_SOURCE" || return 1 append_node_require_once "$_NEMOTRON_FIX_SCRIPT" } install_core_runtime_preloads || exit 1 -# OpenShell re-injects narrow NO_PROXY/no_proxy=127.0.0.1,localhost,::1 every -# time a user connects via `openshell sandbox connect`. Dynamic connect-session -# config lives in /tmp/nemoclaw-proxy-env.sh and is sourced by system-wide shell -# hooks from the base image, keeping per-user rc files free of proxy entries. -# -# SECURITY: The proxy-env file is written via emit_sandbox_sourced_file() -# which ensures root:root 444 in root mode (sandbox cannot modify) and -# best-effort 444 in non-root mode. The /tmp sticky bit prevents the -# sandbox user from deleting or replacing the root-owned file. -# Ref: https://github.com/NVIDIA/NemoClaw/issues/2181 -# -# Both uppercase and lowercase variants are required: Node.js undici prefers -# lowercase (no_proxy) over uppercase (NO_PROXY) when both are set. -# curl/wget use uppercase. gRPC C-core uses lowercase. +# Base-image shell hooks source this file on connect. Preserve its established +# path and root-owned mode while clearing obsolete proxy settings. _RUNTIME_SHELL_ENV_FILE="/tmp/nemoclaw-proxy-env.sh" write_runtime_shell_env() { @@ -3123,13 +3081,10 @@ GATEWAYTOKENRECONCILEEND } { cat <&2 @@ -3639,8 +3573,8 @@ _nemoclaw_maybe_policy_denial_hint() { [ -t 2 ] || return 0 # Top-level connect shell only — don't repeat in every subshell/pane. [ "${SHLVL:-1}" -le 1 ] || return 0 - # Nothing is proxied (no egress restriction) ⇒ nothing to explain. - [ -n "${HTTPS_PROXY:-${https_proxy:-}}" ] || return 0 + # Show the hint only inside an OpenShell sandbox. + [ -n "${OPENSHELL_SANDBOX:-}" ] || return 0 _NEMOCLAW_POLICY_HINT_SHOWN=1 _nemoclaw_policy_denial_hint_text } @@ -3649,12 +3583,6 @@ _nemoclaw_maybe_policy_denial_hint GUARDENVEOF # Global sandbox safety net for connect sessions — must be first. echo "export NODE_OPTIONS=\"\${NODE_OPTIONS:+\$NODE_OPTIONS }--require $_SANDBOX_SAFETY_NET\"" - # HTTP library double-proxy fix: also expose NODE_OPTIONS in connect - # sessions so interactive shells and user commands started via - # `openshell sandbox connect` benefit from the preload. (NemoClaw#2109) - if [ "${NODE_USE_ENV_PROXY:-}" = "1" ]; then - echo "export NODE_OPTIONS=\"\${NODE_OPTIONS:+\$NODE_OPTIONS }--require $_PROXY_FIX_SCRIPT\"" - fi # Git TLS CA bundle for connect sessions (NemoClaw#2270) if [ -n "${GIT_SSL_CAINFO:-}" ]; then printf 'export GIT_SSL_CAINFO=%q\n' "$GIT_SSL_CAINFO" diff --git a/scripts/security/build-native-security-packages.sh b/scripts/security/build-native-security-packages.sh index ac08a13face..38c31124769 100755 --- a/scripts/security/build-native-security-packages.sh +++ b/scripts/security/build-native-security-packages.sh @@ -28,6 +28,7 @@ trap cleanup EXIT download() { local url="$1" local output="$2" + printf 'Downloading %s\n' "${url}" >&2 curl --proto '=https' --tlsv1.2 -fsSL \ --retry 5 --retry-all-errors --retry-delay 2 \ --connect-timeout 15 --max-time 120 \ @@ -243,6 +244,7 @@ build_libssh2_package() { build_zlib_package() { local architecture="$1" local original_sha256 + local original_snapshot_file local debian_version="1:1.3.dfsg+really1.3.1-1+b1" local package_version="${debian_version}+nemoclaw1" local source_dir="${build_root}/zlib-source" @@ -251,8 +253,14 @@ build_zlib_package() { local multiarch case "${architecture}" in - amd64) original_sha256="015be740d6236ad114582dea500c1d907f29e16d6db00566ca32fb68d71ac90d" ;; - arm64) original_sha256="209aa5cf671e97b9eb0410844fa6df4cae2e75b0c72e7802ab6c8ece13e6ddef" ;; + amd64) + original_sha256="015be740d6236ad114582dea500c1d907f29e16d6db00566ca32fb68d71ac90d" + original_snapshot_file="6521c4fb859cda27142c655d2aa30c9d31e40ff7" + ;; + arm64) + original_sha256="209aa5cf671e97b9eb0410844fa6df4cae2e75b0c72e7802ab6c8ece13e6ddef" + original_snapshot_file="37dbdb01f106b4b2246e8b4e2a09988081baae4d" + ;; *) return 64 ;; esac download \ @@ -281,7 +289,7 @@ build_zlib_package() { ./write-error-test ) download \ - "https://snapshot.debian.org/archive/debian/20260906T023042Z/pool/main/z/zlib/zlib1g_1.3.dfsg%2Breally1.3.1-1%2Bb1_${architecture}.deb" \ + "https://snapshot.debian.org/file/${original_snapshot_file}" \ "${original_deb}" verify_sha256 "${original_sha256}" "${original_deb}" test "$(dpkg-deb -f "${original_deb}" Package)" = zlib1g @@ -309,6 +317,7 @@ build_zlib_package() { build_python_fix_package() { local architecture="$1" local original_sha256 + local original_snapshot_file local original_deb="${build_root}/python-stdlib-original.deb" local original_root="${build_root}/python-stdlib-original" local package_root="${build_root}/python-htmlparser-fix" @@ -317,9 +326,11 @@ build_python_fix_package() { case "${architecture}" in amd64) original_sha256="db161322a3481d2c0c3b9a3b9a03c3ab0e2b1718f54b88755fb4a3f939165b84" + original_snapshot_file="c8d9cc73aab111fac7dc1b26c6e2ffda7af08f04" ;; arm64) original_sha256="d1178d24e4d143cc6c577d9dc0f26dd982efccc2e600d25ce86361f168a22be0" + original_snapshot_file="35cc8669f33410e90648c1dee9ed1395920b9b9e" ;; *) printf 'Unsupported architecture: %s\n' "${architecture}" >&2 @@ -328,7 +339,7 @@ build_python_fix_package() { esac download \ - "https://snapshot.debian.org/archive/debian/20260906T023042Z/pool/main/p/python3.13/libpython3.13-stdlib_${PYTHON_DEBIAN_VERSION}_${architecture}.deb" \ + "https://snapshot.debian.org/file/${original_snapshot_file}" \ "${original_deb}" verify_sha256 "${original_sha256}" "${original_deb}" dpkg-deb -x "${original_deb}" "${original_root}" diff --git a/scripts/update-hermes-agent.sh b/scripts/update-hermes-agent.sh index 4a7df05c19b..1a56dd2a614 100755 --- a/scripts/update-hermes-agent.sh +++ b/scripts/update-hermes-agent.sh @@ -199,7 +199,7 @@ installed_copy_schema_error() { "validate-hermes-env-secret-boundary.py" \ "sha256sum /sandbox/.hermes/config.yaml /sandbox/.hermes/.env" \ "hermes-mcp-config-transaction.py" \ - "openshell-child-visible-credentials.v0.0.116.json" \ + "openshell-child-visible-credentials.v0.1.7.json" \ "HERMES_HOME=/sandbox/.hermes /usr/local/bin/hermes doctor --fix" \ "node /opt/nemoclaw-hermes-config/generate-config.ts"; do grep -Fq "$item" "$dockerfile" || missing+=("marker ${item}") diff --git a/scripts/vendor/openshell-sdk/LICENSE.v0.1.5 b/scripts/vendor/openshell-sdk/LICENSE.v0.1.5 new file mode 100644 index 00000000000..f13d24c3fa4 --- /dev/null +++ b/scripts/vendor/openshell-sdk/LICENSE.v0.1.5 @@ -0,0 +1,203 @@ +Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/scripts/vendor/openshell-sdk/LICENSE.v0.1.7 b/scripts/vendor/openshell-sdk/LICENSE.v0.1.7 new file mode 100644 index 00000000000..f13d24c3fa4 --- /dev/null +++ b/scripts/vendor/openshell-sdk/LICENSE.v0.1.7 @@ -0,0 +1,203 @@ +Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/scripts/vendor/openshell-sdk/README.md b/scripts/vendor/openshell-sdk/README.md index b56e2925801..0ba3d50f0f3 100644 --- a/scripts/vendor/openshell-sdk/README.md +++ b/scripts/vendor/openshell-sdk/README.md @@ -3,10 +3,10 @@ # OpenShell SDK archive -This directory carries the reviewed `@nvidia/openshell-sdk@0.0.116` package under its upstream Apache-2.0 license. +This directory carries the reviewed `@nvidia/openshell-sdk@0.1.7` package under its upstream Apache-2.0 license. The archive matches the GitHub Packages URL and SHA-512 integrity in the root `package-lock.json`. -Its source is NVIDIA/OpenShell tag `v0.0.116`, commit `d1155aa70042d3e2ee49dbfa15346b108b7c1d92`. -The included `LICENSE` is copied from that source. +Its source is NVIDIA/OpenShell tag `v0.1.7`, commit `5d25211b25a8fc887e8d5c606d57d6ab22f1d76c`. +The included `LICENSE.v0.1.7` is copied from that source. The installer and `npm run dev:setup` verify this archive, seed npm's cache offline, and run the normal dependency installation. Users do not need GitHub credentials or an SDK build toolchain. @@ -32,9 +32,12 @@ Do not substitute a locally repacked archive: its compressed bytes can differ fr Run the SDK installation package contract after each update. Remove this archive and cache preparation when the pinned SDK is available through the public npm registry. +The `0.0.116` and `0.1.5` archives and their license files remain for historical packaging evidence. + ## Trusted E2E Setup -The production SDK remains `0.0.116`; `ci/reviewed-npm-audit.json` also admits `0.1.5` as the reviewed replacement for the OpenShell upgrade. +The production SDK is `0.1.7`. +`ci/reviewed-npm-audit.json` admits the `0.0.116` baseline and the reviewed `0.1.7` replacement for the OpenShell upgrade. When compiling the candidate CLI and plugin, the E2E planner requires successful SDK packaging, downloads that run's archive bundle, and uses explicit artifact mode. Explicit candidate dispatches that select only `llama-cpp-dgx-spark-qualification` or `native-runtime-qualification-producer`, with no target selector, skip SDK packaging and candidate compilation. Those dispatches use their trusted producer plans. diff --git a/scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.1.5.tgz b/scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.1.5.tgz new file mode 100644 index 00000000000..db73226bace Binary files /dev/null and b/scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.1.5.tgz differ diff --git a/scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.1.7.tgz b/scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.1.7.tgz new file mode 100644 index 00000000000..de86edcbd61 Binary files /dev/null and b/scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.1.7.tgz differ diff --git a/src/commands/inference/get.ts b/src/commands/inference/get.ts index 33b0d5c5ba5..f9bb09ccf3a 100644 --- a/src/commands/inference/get.ts +++ b/src/commands/inference/get.ts @@ -10,7 +10,7 @@ export default class InferenceGetCommand extends NemoClawCommand { static enableJsonFlag = true; static summary = "Show the selected NemoClaw inference path"; static description = - "Read the selected sandbox's native NVIDIA provider path or the live shared OpenShell route."; + "Read the selected sandbox's recorded native provider attachment or the live shared OpenShell route."; static usage = ["inference get [--json]"]; static examples = ["<%= config.bin %> inference get", "<%= config.bin %> inference get --json"]; static flags = {}; diff --git a/src/commands/sandbox/inference/get.ts b/src/commands/sandbox/inference/get.ts index 4b8f9124bab..4c653070241 100644 --- a/src/commands/sandbox/inference/get.ts +++ b/src/commands/sandbox/inference/get.ts @@ -6,14 +6,14 @@ import { NemoClawCommand } from "../../../lib/cli/nemoclaw-oclif-command"; import { sandboxNameArg } from "../../../lib/sandbox/command-support"; // Sandbox-first mirror of the global inference:get command; both delegate to -// the shared runInferenceGet action that selects native NVIDIA or shared-route output. +// the shared runInferenceGet action that selects native attachment or shared-route output. export default class SandboxInferenceGetCommand extends NemoClawCommand { static id = "sandbox:inference:get"; static strict = true; static enableJsonFlag = true; static summary = "Show the selected NemoClaw inference path"; static description = - "Read the sandbox's native NVIDIA provider path or its gateway's live shared route. NEMOCLAW_GATEWAY_PORT selects the sandbox registry and fallback gateway; a registered sandbox's binding selects its recorded gateway."; + "Read the sandbox's recorded native provider attachment or its gateway's live shared route. NEMOCLAW_GATEWAY_PORT selects the sandbox registry and fallback gateway; a registered sandbox's binding selects its recorded gateway."; static usage = [" inference get [--json]"]; static examples = [ "<%= config.bin %> my-assistant inference get", diff --git a/src/commands/sandbox/oclif-command-adapters.test.ts b/src/commands/sandbox/oclif-command-adapters.test.ts index 80af80df774..a2b424b53b8 100644 --- a/src/commands/sandbox/oclif-command-adapters.test.ts +++ b/src/commands/sandbox/oclif-command-adapters.test.ts @@ -439,14 +439,11 @@ describe("sandbox oclif command adapters", () => { ); expect(RecoverCliCommand.description).not.toContain("repair stopped gateways"); expect(RecoverCliCommand.description).toContain( - "the Hermes supervisor can wait for authenticated gated recovery", + "A stopped native gateway is reported without relaunching it", ); - expect(RecoverCliCommand.description).toContain("After a clean exit or exit status 79"); + expect(RecoverCliCommand.description).not.toContain("supervised gateway relaunch"); expect(RecoverCliCommand.description).toContain( - "In that state, `recover` requests a supervised gateway relaunch before verification", - ); - expect(RecoverCliCommand.description).toContain( - "For other stopped gateways or a forced runtime reload, use `gateway restart`", + "Use `gateway restart` when you need a forced runtime reload", ); expect(RecoverCliCommand.description).toContain( "validate and release a NemoClaw cron restore gate", diff --git a/src/commands/sandbox/recover.ts b/src/commands/sandbox/recover.ts index e5b30da90bb..412a0c38520 100644 --- a/src/commands/sandbox/recover.ts +++ b/src/commands/sandbox/recover.ts @@ -11,7 +11,7 @@ export default class RecoverCliCommand extends NemoClawCommand { static strict = true; static summary = "Verify sandbox gateway health and repair host forwards"; static description = - "Probe sandbox gateway health and repair host forwards without opening an SSH session. For Hermes, also validate and release a NemoClaw cron restore gate left by an interrupted rebuild. A healthy gateway is not restarted. After a clean exit or exit status 79, the Hermes supervisor can wait for authenticated gated recovery. In that state, `recover` requests a supervised gateway relaunch before verification. For other stopped gateways or a forced runtime reload, use `gateway restart`."; + "Probe sandbox gateway health and repair host forwards without opening an SSH session. For Hermes, also validate and release a NemoClaw cron restore gate left by an interrupted rebuild. A healthy gateway is not restarted. A stopped native gateway is reported without relaunching it. Use `gateway restart` when you need a forced runtime reload."; static usage = [""]; static examples = ["<%= config.bin %> sandbox recover alpha"]; static args = { diff --git a/src/lib/acp/command.test.ts b/src/lib/acp/command.test.ts index 2a555cdd502..c5be8d8a005 100644 --- a/src/lib/acp/command.test.ts +++ b/src/lib/acp/command.test.ts @@ -6,7 +6,10 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import type { HermesAcpSshTransport } from "../adapters/openshell/hermes-acp-ssh"; import type { OpenShellSandboxObserver } from "../adapters/openshell/sandbox-observer"; -import { HERMES_LIFECYCLE_DEFINITION } from "../domain/lifecycle/hermes-definition"; +import { + HERMES_LIFECYCLE_DEFINITION, + HERMES_CLI_OPENSHELL_VERSION, +} from "../domain/lifecycle/hermes-definition"; import type { HostGatewayRegistryEntry } from "../state/gateway-registry"; import { type HermesAcpCommandDeps, @@ -42,7 +45,7 @@ function registryEntry( lifecycleGeneration: "generation-1", lifecycleLiveIdentityFingerprint: FINGERPRINT, nemoclawVersion: VERSION, - openshellVersion: HERMES_LIFECYCLE_DEFINITION.openshellVersion, + openshellVersion: HERMES_CLI_OPENSHELL_VERSION, ...overrides, }, }; @@ -74,6 +77,8 @@ function commandHarness( overrides: { entries?: HostGatewayRegistryEntry[]; fingerprint?: string; + inspectIdentity?: NonNullable; + listRegistry?: NonNullable; observer?: OpenShellSandboxObserver; recovered?: boolean; transport?: HermesAcpSshTransport; @@ -120,8 +125,10 @@ function commandHarness( { currentVersion: () => VERSION, home: () => "/home/operator", - inspectIdentity: vi.fn(() => overrides.fingerprint ?? FINGERPRINT), - listRegistry: vi.fn(() => overrides.entries ?? [registryEntry("alpha")]), + inspectIdentity: + overrides.inspectIdentity ?? vi.fn(() => overrides.fingerprint ?? FINGERPRINT), + listRegistry: + overrides.listRegistry ?? vi.fn(() => overrides.entries ?? [registryEntry("alpha")]), observer: overrides.observer ?? readyObserver(), recoverGateway: recoverGateway as never, transport, @@ -134,6 +141,7 @@ function commandHarness( describe("Hermes ACP command", () => { afterEach(() => { + vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs(); }); @@ -193,6 +201,11 @@ describe("Hermes ACP command", () => { [registryEntry("alpha", 8080, { openshellVersion: "0.0.106" })], "incompatible", ], + [ + "the frozen headless API version", + [registryEntry("alpha", 8080, { openshellVersion: "0.1.5" })], + "incompatible", + ], [ "custom image", [registryEntry("alpha", 8080, { fromDockerfile: "/tmp/Dockerfile" })], @@ -387,6 +400,94 @@ describe("Hermes ACP command", () => { expect(fixture.transport.run).not.toHaveBeenCalled(); }); + it("waits for the same provisioning sandbox after gateway recovery within the session deadline", async () => { + vi.useFakeTimers(); + const observer = readyObserver(); + observer.listSandboxes.mockResolvedValueOnce({ + ok: true, + value: { sandboxes: [{ name: "alpha", phase: "Provisioning", readiness: "not_ready" }] }, + }); + const inspectIdentity = vi.fn(() => FINGERPRINT); + const fixture = commandHarness({ observer, inspectIdentity }); + const completion = fixture.run(["--sandbox", "alpha", "--timeout", "1"]); + await vi.advanceTimersByTimeAsync(250); + expect(await completion).toBe(0); + expect(observer.listSandboxes).toHaveBeenCalledTimes(2); + expect(observer.listSandboxes.mock.calls).toEqual([ + [{ target: { kind: "named", gatewayName: "nemoclaw" }, timeoutMs: 1000 }], + [{ target: { kind: "named", gatewayName: "nemoclaw" }, timeoutMs: 750 }], + ]); + expect(inspectIdentity).toHaveBeenCalledTimes(2); + expect(fixture.transport.run).toHaveBeenCalledWith(expect.objectContaining({ timeoutMs: 750 })); + }); + + it("refuses a replacement identity during provisioning recovery", async () => { + vi.useFakeTimers(); + const observer = readyObserver(); + observer.listSandboxes.mockResolvedValueOnce({ + ok: true, + value: { sandboxes: [{ name: "alpha", phase: "Provisioning", readiness: "not_ready" }] }, + }); + const inspectIdentity = vi + .fn() + .mockReturnValueOnce(FINGERPRINT) + .mockReturnValue("b".repeat(64)); + const fixture = commandHarness({ observer, inspectIdentity }); + const completion = fixture.run(["--sandbox", "alpha"]); + await vi.advanceTimersByTimeAsync(250); + expect(await completion).toBe(1); + expect(fixture.diagnostics.text()).toContain("identity does not match"); + expect(fixture.transport.run).not.toHaveBeenCalled(); + }); + + it("does not wait through provisioning without an actual gateway recovery", async () => { + const observer = readyObserver(); + observer.listSandboxes.mockResolvedValue({ + ok: true, + value: { sandboxes: [{ name: "alpha", phase: "Provisioning", readiness: "not_ready" }] }, + }); + const fixture = commandHarness({ observer }); + fixture.recoverGateway.mockResolvedValueOnce({ + recovered: true, + attempted: false, + before: { state: "healthy_named" }, + after: { state: "healthy_named" }, + }); + expect(await fixture.run(["--sandbox", "alpha"])).toBe(1); + expect(observer.listSandboxes).toHaveBeenCalledTimes(1); + expect(fixture.transport.run).not.toHaveBeenCalled(); + }); + + it("ends provisioning observation at the requested session deadline", async () => { + vi.useFakeTimers(); + const observer = readyObserver(); + observer.listSandboxes.mockResolvedValue({ + ok: true, + value: { sandboxes: [{ name: "alpha", phase: "Provisioning", readiness: "not_ready" }] }, + }); + const fixture = commandHarness({ observer }); + const completion = fixture.run(["--sandbox", "alpha", "--timeout", "1"]); + await vi.advanceTimersByTimeAsync(1000); + expect(await completion).toBe(1); + expect(fixture.transport.run).not.toHaveBeenCalled(); + expect(fixture.diagnostics.text()).toContain("stopped or not ready"); + expect(observer.listSandboxes.mock.calls.at(-1)).toEqual([ + { target: { kind: "named", gatewayName: "nemoclaw" }, timeoutMs: 375 }, + ]); + }); + + it("revalidates the recorded generation after readiness before starting transport", async () => { + const listRegistry = vi + .fn() + .mockReturnValueOnce([registryEntry("alpha")]) + .mockReturnValueOnce([registryEntry("alpha")]) + .mockReturnValue([registryEntry("alpha", 8080, { lifecycleGeneration: "replacement" })]); + const fixture = commandHarness({ listRegistry }); + expect(await fixture.run(["--sandbox", "alpha"])).toBe(1); + expect(fixture.transport.run).not.toHaveBeenCalled(); + expect(fixture.diagnostics.text()).toContain("changed or its deadline expired"); + }); + it("rejects a gateway that recovery cannot make ready", async () => { const fixture = commandHarness({ recovered: false }); diff --git a/src/lib/acp/command.ts b/src/lib/acp/command.ts index 055805ba9ae..4ddc040c490 100644 --- a/src/lib/acp/command.ts +++ b/src/lib/acp/command.ts @@ -6,7 +6,10 @@ import path from "node:path"; import type { Readable, Writable } from "node:stream"; import { withConnectSandboxLifecycleLock } from "../actions/sandbox/lifecycle/lock"; -import { createCliHermesAcpSshTransport } from "../adapters/openshell/hermes-acp-ssh-cli"; +import { + ACP_SETUP_TIMEOUT_MS, + createCliHermesAcpSshTransport, +} from "../adapters/openshell/hermes-acp-ssh-cli"; import type { HermesAcpSshTransport } from "../adapters/openshell/hermes-acp-ssh"; import { inspectOpenShellSandboxIdentityFingerprint } from "../adapters/openshell/sandbox-identity-cli"; import { @@ -17,7 +20,11 @@ import type { OpenShellSandboxObserver } from "../adapters/openshell/sandbox-obs import { captureSanitizedResolvedOpenshell } from "../adapters/openshell/sanitized-capture"; import { OPENSHELL_DEFAULT_WORKSPACE } from "../adapters/openshell/sandbox-ssh-host"; import { getVersion } from "../core/version"; -import { HERMES_LIFECYCLE_DEFINITION } from "../domain/lifecycle/hermes-definition"; +import { waitUntilAsync } from "../core/wait"; +import { + HERMES_LIFECYCLE_DEFINITION, + HERMES_CLI_OPENSHELL_VERSION, +} from "../domain/lifecycle/hermes-definition"; import { recoverNamedGatewayRuntime } from "../gateway-runtime-action"; import { assertNoOpenShellGatewayEndpointOverride } from "../openshell-gateway-endpoint-guard"; import { resolveGatewayName, resolveGatewayPortFromName } from "../onboard/gateway-binding"; @@ -157,7 +164,7 @@ function compatibleEntry(entry: HostGatewayRegistryEntry["entry"], version: stri return ( entry.agent === HERMES_LIFECYCLE_DEFINITION.agent && entry.agentVersion === HERMES_LIFECYCLE_DEFINITION.agentVersion && - entry.openshellVersion === HERMES_LIFECYCLE_DEFINITION.openshellVersion && + entry.openshellVersion === HERMES_CLI_OPENSHELL_VERSION && entry.nemoclawVersion === version && (entry.fromDockerfile === undefined || entry.fromDockerfile === null) && typeof entry.lifecycleGeneration === "string" && @@ -269,22 +276,52 @@ async function validateLiveTarget( target: HermesAcpTarget, observer: OpenShellSandboxObserver, inspectIdentity: typeof inspectOpenShellSandboxIdentityFingerprint, + recoveryDeadlineMs: number, + allowProvisioning: boolean, ): Promise { - const inventory = await observer.listSandboxes({ - target: { kind: "named", gatewayName: target.gatewayName }, - }); - if (!inventory.ok) return "OpenShell could not inspect the selected sandbox."; - const matches = inventory.value.sandboxes.filter(({ name }) => name === target.sandboxName); - if (matches.length === 0) return "The requested sandbox is missing."; - if (matches.length > 1) return "OpenShell returned an ambiguous sandbox identity."; - if (matches[0]!.readiness !== "ready") return "The requested sandbox is stopped or not ready."; - const fingerprint = inspectIdentity({ - gatewayName: target.gatewayName, - sandboxName: target.sandboxName, - }); - return fingerprint === target.entry.lifecycleLiveIdentityFingerprint - ? null - : "The live sandbox identity does not match its NemoClaw registration."; + let failure: string | null = "The requested sandbox is stopped or not ready."; + await waitUntilAsync( + async () => { + const remainingMs = Math.max(1, recoveryDeadlineMs - Date.now()); + const inventory = await observer.listSandboxes({ + target: { kind: "named", gatewayName: target.gatewayName }, + timeoutMs: remainingMs, + }); + if (!inventory.ok) { + failure = "OpenShell could not inspect the selected sandbox."; + return true; + } + const matches = inventory.value.sandboxes.filter(({ name }) => name === target.sandboxName); + if (matches.length !== 1) { + failure = + matches.length === 0 + ? "The requested sandbox is missing." + : "OpenShell returned an ambiguous sandbox identity."; + return true; + } + const observed = matches[0]!; + const canWait = + allowProvisioning && + observed.phase === "Provisioning" && + observed.readiness === "not_ready"; + if (observed.readiness !== "ready" && !canWait) return true; + const fingerprint = inspectIdentity({ + gatewayName: target.gatewayName, + sandboxName: target.sandboxName, + timeoutMs: Math.max(1, recoveryDeadlineMs - Date.now()), + }); + if (fingerprint !== target.entry.lifecycleLiveIdentityFingerprint) { + failure = "The live sandbox identity does not match its NemoClaw registration."; + return true; + } + if (Date.now() >= recoveryDeadlineMs) return true; + failure = + observed.readiness === "ready" ? null : "The requested sandbox is stopped or not ready."; + return !canWait; + }, + { deadlineMs: recoveryDeadlineMs, initialIntervalMs: 250, maxIntervalMs: 1_000 }, + ); + return failure; } /** Run the packaged host-side ACP adapter without writing ACP-adjacent progress to stdout. */ @@ -309,6 +346,8 @@ export async function runHermesAcpCommand( return 0; } + const sessionDeadlineMs = + options.timeoutMs === undefined ? Infinity : Date.now() + options.timeoutMs; const home = (deps.home ?? os.homedir)(); let entries: HostGatewayRegistryEntry[]; try { @@ -396,11 +435,34 @@ export async function runHermesAcpCommand( target, deps.observer ?? defaultObserver(), deps.inspectIdentity ?? inspectOpenShellSandboxIdentityFingerprint, + Math.min(sessionDeadlineMs, Date.now() + ACP_SETUP_TIMEOUT_MS), + recovery.attempted, ); } catch { liveError = "OpenShell could not validate the selected sandbox identity."; } if (liveError) return { error: liveError } as const; + const finalTarget = resolveHermesAcpTarget( + (deps.listRegistry ?? listHostGatewayRegistryEntries)(home), + { + gatewayName: target.gatewayName, + sandboxName: target.sandboxName, + nemoclawVersion: currentVersion, + }, + ); + if ( + !finalTarget.ok || + finalTarget.target.stateRoot !== target.stateRoot || + finalTarget.target.entry.lifecycleGeneration !== target.entry.lifecycleGeneration || + finalTarget.target.entry.lifecycleLiveIdentityFingerprint !== + target.entry.lifecycleLiveIdentityFingerprint || + Date.now() >= sessionDeadlineMs + ) { + return { + error: + "The requested sandbox changed or its deadline expired before the ACP adapter could start.", + } as const; + } let markSessionStarted!: () => void; const sessionStarted = new Promise((resolve) => { @@ -411,7 +473,7 @@ export async function runHermesAcpCommand( sandboxName: target.sandboxName, streams: io, onSessionStarted: markSessionStarted, - ...(options.timeoutMs ? { timeoutMs: options.timeoutMs } : {}), + ...(options.timeoutMs ? { timeoutMs: Math.max(1, sessionDeadlineMs - Date.now()) } : {}), }); await Promise.race([ sessionStarted, diff --git a/src/lib/actions/credentials-add.ts b/src/lib/actions/credentials-add.ts index 96abb5f267d..ea8920570ac 100644 --- a/src/lib/actions/credentials-add.ts +++ b/src/lib/actions/credentials-add.ts @@ -470,6 +470,8 @@ export async function runCredentialsAddAction( gatewayName: target.gatewayName, adapter: providerAdapter, endpointUrl: existing?.endpointUrl, + retainedHermesEndpoint: + hosted.logicalProvider === "hermes-provider" && Boolean(existing?.endpointUrl), credentialValue: fromExisting ? null : credentialValue, reuseExistingCredential: fromExisting, readAuthority, @@ -525,9 +527,9 @@ export async function runCredentialsAddAction( }); } return ok([ - ` Registered provider '${provider}' with the OpenShell gateway.`, + ` Registered provider '${NVIDIA_HOSTED_LOGICAL_PROVIDER}' with the OpenShell gateway.`, ` Verify with '${CLI_NAME} credentials list'.`, - ` Select it with '${CLI_NAME} inference set --provider ${provider} --model '.`, + ` Select it with '${CLI_NAME} inference set --provider ${NVIDIA_HOSTED_LOGICAL_PROVIDER} --model '.`, ]); } catch (error) { const detail = diff --git a/src/lib/actions/credentials-native-hosted.test.ts b/src/lib/actions/credentials-native-hosted.test.ts index b914dcbe6ab..c8c50c9b5ce 100644 --- a/src/lib/actions/credentials-native-hosted.test.ts +++ b/src/lib/actions/credentials-native-hosted.test.ts @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { HOSTED_NATIVE_PROVIDERS } from "../inference/native-provider/hosted"; +import { HOSTED_NATIVE_PROVIDERS, hostedNativeProvider } from "../inference/native-provider/hosted"; import { providerAdapter } from "../../../test/helpers/credentials-provider-adapter.ts"; import { setGlobalCliActionRuntimeHooksForTest } from "./global"; import { runCredentialsAddAction } from "./credentials-add"; @@ -150,6 +150,61 @@ describe("native hosted credential registration", () => { expect(JSON.stringify(result)).not.toContain("host-only-value"); }); + it.each([ + { endpointUrl: "http://8.8.8.8/team/v1", allowedIps: ["8.8.8.8"] }, + { endpointUrl: "http://host.openshell.internal:42103/team/v1", allowedIps: undefined }, + ])( + "rotates the owned Hermes credential at $endpointUrl without changing its identity", + async ({ endpointUrl, allowedIps }) => { + vi.stubEnv("NOUS_API_KEY", "host-only-value"); + const definition = hostedNativeProvider("hermes-provider", endpointUrl)!; + const receipt = { + schemaVersion: 1 as const, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "owned-id", + endpointUrl, + ...(allowedIps ? { allowedIps } : {}), + }; + const adapter = providerAdapter({ + getProvider: vi.fn(async () => ({ + ok: true as const, + value: { + name: definition.providerName, + type: definition.profileId, + credentialKeys: ["OPENAI_API_KEY"], + configKeys: [], + revision: { id: "owned-id", resourceVersion: 1 }, + }, + })), + }); + const save = vi.fn(); + const result = await runCredentialsAddAction( + { + provider: "hermes-provider", + type: "openai", + credentials: ["NOUS_API_KEY"], + configPairs: [], + fromExisting: false, + }, + { + providerAdapter: adapter, + getNativeHostedProviderAuthority: () => receipt, + setNativeHostedProviderAuthority: save, + }, + ); + expect(result.exitCode).toBe(0); + expect(adapter.updateProvider).toHaveBeenCalledWith( + expect.objectContaining({ + providerName: definition.providerName, + credentials: [{ name: "OPENAI_API_KEY", value: "host-only-value" }], + }), + ); + expect(adapter.createProvider).not.toHaveBeenCalled(); + expect(JSON.stringify([result, save.mock.calls])).not.toContain("host-only-value"); + }, + ); + it("reports a native registration failure without exposing the supplied credential", async () => { vi.stubEnv("OPENAI_API_KEY", "host-only-value"); const adapter = providerAdapter({ diff --git a/src/lib/actions/credentials-native-nvidia.test.ts b/src/lib/actions/credentials-native-nvidia.test.ts new file mode 100644 index 00000000000..06a4871db6f --- /dev/null +++ b/src/lib/actions/credentials-native-nvidia.test.ts @@ -0,0 +1,219 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { providerAdapter } from "../../../test/helpers/credentials-provider-adapter.ts"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import type { OpenShellProviderAdapter } from "../adapters/openshell/provider-adapter"; +import { setGlobalCliActionRuntimeHooksForTest } from "./global"; +import { runCredentialsAddAction } from "./credentials-add"; +import { runCredentialsListAction } from "./credentials/list"; + +vi.mock("../onboard/gateway-teardown-authority", () => ({ + resolveGatewayCredentialMutationAuthority: vi.fn(() => ({})), +})); + +vi.mock("../state/mcp-lifecycle-lock/credential-ownership", () => ({ + withMcpCredentialOwnershipLock: (operation: () => Promise | T) => operation(), +})); + +vi.mock("../gateway-start-guidance", () => ({ + gatewayStartGuidance: () => "Start the gateway again with `nemoclaw onboard`.", +})); + +function nativeNvidiaProviderAdapter(): OpenShellProviderAdapter { + let providerPresent = false; + return providerAdapter({ + getProvider: vi.fn(async (request) => + providerPresent + ? { + ok: true as const, + value: { + name: request.providerName, + type: "nemoclaw-nvidia-inference-v1", + credentialKeys: ["NVIDIA_INFERENCE_API_KEY"], + configKeys: [], + revision: { + id: "11111111-2222-4333-8444-555555555555", + resourceVersion: 1, + }, + }, + } + : { + ok: false as const, + error: { + kind: "command" as const, + reason: "not_found" as const, + message: "provider not found", + }, + }, + ), + createProvider: vi.fn(async () => { + providerPresent = true; + return { ok: true as const }; + }), + deleteProvider: vi.fn(async () => { + providerPresent = false; + return { ok: true as const }; + }), + }); +} + +describe("native NVIDIA credential registration and listing", () => { + beforeEach(() => { + setGlobalCliActionRuntimeHooksForTest({ + recoverNamedGatewayRuntime: async () => ({ recovered: true }), + recordExtraProvider: () => true, + forgetExtraProvider: () => true, + }); + }); + + afterEach(() => { + setGlobalCliActionRuntimeHooksForTest({}); + vi.unstubAllEnvs(); + }); + + it("maps the NVIDIA credential alias to the internal native provider", async () => { + vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "host-only-nvidia-value"); + const recordExtraProvider = vi.fn(() => true); + setGlobalCliActionRuntimeHooksForTest({ + recoverNamedGatewayRuntime: async () => ({ recovered: true }), + recordExtraProvider, + forgetExtraProvider: () => true, + }); + const providerId = "11111111-2222-4333-8444-555555555555"; + const adapter = nativeNvidiaProviderAdapter(); + const setNativeNvidiaProviderAuthority = vi.fn(); + + const result = await runCredentialsAddAction( + { + provider: "nvidia-prod", + type: "nvidia", + credentials: ["NVIDIA_INFERENCE_API_KEY"], + configPairs: [], + fromExisting: false, + }, + { + providerAdapter: adapter, + getNativeNvidiaProviderAuthority: () => undefined, + setNativeNvidiaProviderAuthority, + }, + ); + + expect(result.exitCode).toBe(0); + expect(adapter.importProviderProfile).toHaveBeenCalledWith({ + target: { kind: "named", gatewayName: "nemoclaw" }, + profilePath: expect.stringMatching(/provider-profiles\/nemoclaw-nvidia-inference-v1\.yaml$/u), + }); + expect(adapter.createProvider).toHaveBeenCalledWith({ + target: { kind: "named", gatewayName: "nemoclaw" }, + name: "nemoclaw-nvidia-prod-v1", + type: "nemoclaw-nvidia-inference-v1", + credentials: [{ name: "NVIDIA_INFERENCE_API_KEY", value: "host-only-nvidia-value" }], + config: [], + fromExisting: false, + }); + expect(setNativeNvidiaProviderAuthority).toHaveBeenCalledWith("nemoclaw", { + schemaVersion: 1, + profileId: "nemoclaw-nvidia-inference-v1", + providerName: "nemoclaw-nvidia-prod-v1", + providerId, + }); + expect(recordExtraProvider).not.toHaveBeenCalled(); + expect(JSON.stringify(result)).not.toContain("host-only-nvidia-value"); + expect(result.successLines).toContain( + " Select it with 'nemoclaw inference set --provider nvidia-prod --model '.", + ); + expect(result.successLines.join("\n")).not.toContain(" rebuild"); + }); + + it("removes the native NVIDIA provider when authority persistence fails", async () => { + vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "host-only-nvidia-value"); + const adapter = nativeNvidiaProviderAdapter(); + + const result = await runCredentialsAddAction( + { + provider: "nvidia-prod", + type: "nvidia", + credentials: ["NVIDIA_INFERENCE_API_KEY"], + configPairs: [], + fromExisting: false, + }, + { + providerAdapter: adapter, + getNativeNvidiaProviderAuthority: () => undefined, + setNativeNvidiaProviderAuthority: () => { + throw new Error("state directory is read-only"); + }, + }, + ); + + expect(result.exitCode).toBe(1); + expect(result.failureLines.join("\n")).toMatch( + /newly created provider was removed.*state directory is read-only/su, + ); + expect(adapter.deleteProvider).toHaveBeenCalledExactlyOnceWith({ + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: "nemoclaw-nvidia-prod-v1", + }); + expect(JSON.stringify(result)).not.toContain("host-only-nvidia-value"); + }); + + it("rejects a substituted native NVIDIA provider before replacing its credential", async () => { + vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "host-only-nvidia-value"); + const adapter = providerAdapter({ + getProvider: vi.fn(async (request) => ({ + ok: true as const, + value: { + name: request.providerName, + type: "nemoclaw-nvidia-inference-v1", + credentialKeys: ["NVIDIA_INFERENCE_API_KEY"], + configKeys: [], + revision: { id: "replacement-provider-id", resourceVersion: 2 }, + }, + })), + }); + const setNativeNvidiaProviderAuthority = vi.fn(); + + const result = await runCredentialsAddAction( + { + provider: "nvidia-prod", + type: "nvidia", + credentials: ["NVIDIA_INFERENCE_API_KEY"], + configPairs: [], + fromExisting: false, + }, + { + providerAdapter: adapter, + getNativeNvidiaProviderAuthority: () => ({ + schemaVersion: 1, + profileId: "nemoclaw-nvidia-inference-v1", + providerName: "nemoclaw-nvidia-prod-v1", + providerId: "11111111-2222-4333-8444-555555555555", + }), + setNativeNvidiaProviderAuthority, + }, + ); + + expect(result.exitCode).toBe(1); + expect(result.failureLines.join("\n")).toContain("changed identity"); + expect(adapter.updateProvider).not.toHaveBeenCalled(); + expect(setNativeNvidiaProviderAuthority).not.toHaveBeenCalled(); + }); + + it("lists the native NVIDIA provider only by its logical credential name", async () => { + const adapter = providerAdapter({ + listProviders: vi.fn(async () => ({ + ok: true as const, + value: { names: ["nemoclaw-nvidia-prod-v1", "custom-provider"] }, + })), + }); + + const result = await runCredentialsListAction("nemoclaw", { providerAdapter: adapter }); + + expect(result.exitCode).toBe(0); + expect(result.outputLines).toContain(" nvidia-prod"); + expect(result.outputLines).toContain(" custom-provider"); + expect(result.outputLines.join("\n")).not.toContain("nemoclaw-nvidia-prod-v1"); + }); +}); diff --git a/src/lib/actions/credentials-native-reset.test.ts b/src/lib/actions/credentials-native-reset.test.ts index 37c9a814ba3..f237461aa98 100644 --- a/src/lib/actions/credentials-native-reset.test.ts +++ b/src/lib/actions/credentials-native-reset.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../test/helpers/registered-inference"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { OpenShellProviderAdapter } from "../adapters/openshell/provider-adapter"; @@ -27,6 +28,28 @@ describe("native NVIDIA credential reset ownership", () => { afterEach(() => setGlobalCliActionRuntimeHooksForTest({})); + it("refuses reset while an external provider remains a recorded selection", async () => { + const f = registeredInferenceFixture(true); + const binding = { ...f.binding, gatewayName: "nemoclaw" }; + const result = await runCredentialsResetAction( + { provider: binding.providerName, confirmed: true }, + { + providerAdapter: f.adapter, + listRegisteredInferenceProviderAttachmentSandboxNames: (gateway, provider) => + gateway === binding.gatewayName && provider === binding.providerName + ? [binding.sandboxName] + : [], + withGatewayRouteMutationLock: async (_gateway, operation) => operation(), + }, + ); + expect(result.exitCode).toBe(1); + expect(result.failureLines.join(" ")).toContain( + "Switch those sandboxes to another provider or destroy them", + ); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + }); + it.each([ ["present", { ok: true as const }], [ diff --git a/src/lib/actions/credentials-provider-adapter.test.ts b/src/lib/actions/credentials-provider-adapter.test.ts index f1959f27233..91485344bf8 100644 --- a/src/lib/actions/credentials-provider-adapter.test.ts +++ b/src/lib/actions/credentials-provider-adapter.test.ts @@ -24,44 +24,6 @@ vi.mock("../gateway-start-guidance", () => ({ gatewayStartGuidance: () => "Start the gateway again with `nemoclaw onboard`.", })); -function nativeNvidiaProviderAdapter(): OpenShellProviderAdapter { - let providerPresent = false; - return providerAdapter({ - getProvider: vi.fn(async (request) => - providerPresent - ? { - ok: true as const, - value: { - name: request.providerName, - type: "nemoclaw-nvidia-inference-v1", - credentialKeys: ["NVIDIA_INFERENCE_API_KEY"], - configKeys: [], - revision: { - id: "11111111-2222-4333-8444-555555555555", - resourceVersion: 1, - }, - }, - } - : { - ok: false as const, - error: { - kind: "command" as const, - reason: "not_found" as const, - message: "provider not found", - }, - }, - ), - createProvider: vi.fn(async () => { - providerPresent = true; - return { ok: true as const }; - }), - deleteProvider: vi.fn(async () => { - providerPresent = false; - return { ok: true as const }; - }), - }); -} - describe("credential actions use typed OpenShell provider results", () => { beforeEach(() => { setGlobalCliActionRuntimeHooksForTest({ @@ -103,7 +65,10 @@ describe("credential actions use typed OpenShell provider results", () => { }); expect(adapter.importProviderProfile).not.toHaveBeenCalled(); expect(JSON.stringify(result)).not.toContain("credential-value"); - expect(result.successLines.join("\n")).toContain("Rebuild each sandbox"); + expect(result.successLines).toContain( + " Rebuild each sandbox that should use 'custom-provider' (`nemoclaw rebuild`).", + ); + expect(result.successLines.join("\n")).not.toContain("inference set"); }); it.each([false, true])( @@ -231,146 +196,6 @@ describe("credential actions use typed OpenShell provider results", () => { expect(run.mock.calls.some(([args]) => args.includes("import"))).toBe(false); }); - it("maps the NVIDIA credential alias to the internal native provider", async () => { - vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "host-only-nvidia-value"); - const recordExtraProvider = vi.fn(() => true); - setGlobalCliActionRuntimeHooksForTest({ - recoverNamedGatewayRuntime: async () => ({ recovered: true }), - recordExtraProvider, - forgetExtraProvider: () => true, - }); - const providerId = "11111111-2222-4333-8444-555555555555"; - const adapter = nativeNvidiaProviderAdapter(); - const setNativeNvidiaProviderAuthority = vi.fn(); - - const result = await runCredentialsAddAction( - { - provider: "nvidia-prod", - type: "nvidia", - credentials: ["NVIDIA_INFERENCE_API_KEY"], - configPairs: [], - fromExisting: false, - }, - { - providerAdapter: adapter, - getNativeNvidiaProviderAuthority: () => undefined, - setNativeNvidiaProviderAuthority, - }, - ); - - expect(result.exitCode).toBe(0); - expect(adapter.importProviderProfile).toHaveBeenCalledWith({ - target: { kind: "named", gatewayName: "nemoclaw" }, - profilePath: expect.stringMatching(/provider-profiles\/nemoclaw-nvidia-inference-v1\.yaml$/u), - }); - expect(adapter.createProvider).toHaveBeenCalledWith({ - target: { kind: "named", gatewayName: "nemoclaw" }, - name: "nemoclaw-nvidia-prod-v1", - type: "nemoclaw-nvidia-inference-v1", - credentials: [{ name: "NVIDIA_INFERENCE_API_KEY", value: "host-only-nvidia-value" }], - config: [], - fromExisting: false, - }); - expect(setNativeNvidiaProviderAuthority).toHaveBeenCalledWith("nemoclaw", { - schemaVersion: 1, - profileId: "nemoclaw-nvidia-inference-v1", - providerName: "nemoclaw-nvidia-prod-v1", - providerId, - }); - expect(recordExtraProvider).not.toHaveBeenCalled(); - expect(JSON.stringify(result)).not.toContain("host-only-nvidia-value"); - }); - - it("removes the native NVIDIA provider when authority persistence fails", async () => { - vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "host-only-nvidia-value"); - const adapter = nativeNvidiaProviderAdapter(); - - const result = await runCredentialsAddAction( - { - provider: "nvidia-prod", - type: "nvidia", - credentials: ["NVIDIA_INFERENCE_API_KEY"], - configPairs: [], - fromExisting: false, - }, - { - providerAdapter: adapter, - getNativeNvidiaProviderAuthority: () => undefined, - setNativeNvidiaProviderAuthority: () => { - throw new Error("state directory is read-only"); - }, - }, - ); - - expect(result.exitCode).toBe(1); - expect(result.failureLines.join("\n")).toMatch( - /newly created provider was removed.*state directory is read-only/su, - ); - expect(adapter.deleteProvider).toHaveBeenCalledExactlyOnceWith({ - target: { kind: "named", gatewayName: "nemoclaw" }, - providerName: "nemoclaw-nvidia-prod-v1", - }); - expect(JSON.stringify(result)).not.toContain("host-only-nvidia-value"); - }); - - it("rejects a substituted native NVIDIA provider before replacing its credential", async () => { - vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "host-only-nvidia-value"); - const adapter = providerAdapter({ - getProvider: vi.fn(async (request) => ({ - ok: true as const, - value: { - name: request.providerName, - type: "nemoclaw-nvidia-inference-v1", - credentialKeys: ["NVIDIA_INFERENCE_API_KEY"], - configKeys: [], - revision: { id: "replacement-provider-id", resourceVersion: 2 }, - }, - })), - }); - const setNativeNvidiaProviderAuthority = vi.fn(); - - const result = await runCredentialsAddAction( - { - provider: "nvidia-prod", - type: "nvidia", - credentials: ["NVIDIA_INFERENCE_API_KEY"], - configPairs: [], - fromExisting: false, - }, - { - providerAdapter: adapter, - getNativeNvidiaProviderAuthority: () => ({ - schemaVersion: 1, - profileId: "nemoclaw-nvidia-inference-v1", - providerName: "nemoclaw-nvidia-prod-v1", - providerId: "11111111-2222-4333-8444-555555555555", - }), - setNativeNvidiaProviderAuthority, - }, - ); - - expect(result.exitCode).toBe(1); - expect(result.failureLines.join("\n")).toContain("changed identity"); - expect(adapter.updateProvider).not.toHaveBeenCalled(); - expect(setNativeNvidiaProviderAuthority).not.toHaveBeenCalled(); - }); - - it("lists the native NVIDIA provider only by its logical credential name", async () => { - const adapter = providerAdapter({ - listProviders: vi.fn(async () => ({ - ok: true as const, - value: { names: ["nemoclaw-nvidia-prod-v1", "custom-provider"] }, - })), - }); - - const result = await runCredentialsListAction("nemoclaw", { providerAdapter: adapter }); - - expect(result.exitCode).toBe(0); - expect(result.outputLines).toContain(" nvidia-prod"); - expect(result.outputLines).toContain(" custom-provider"); - expect(result.outputLines.join("\n")).not.toContain("nemoclaw-nvidia-prod-v1"); - }); - it("registers both Langfuse keys through the checked-in endpoint profile (#10840)", async () => { vi.stubEnv("LANGFUSE_PUBLIC_KEY", "pk-lf-host-only"); vi.stubEnv("LANGFUSE_SECRET_KEY", "sk-lf-host-only"); diff --git a/src/lib/actions/credentials/reset.ts b/src/lib/actions/credentials/reset.ts index b02ae96fb58..a3043f56c12 100644 --- a/src/lib/actions/credentials/reset.ts +++ b/src/lib/actions/credentials/reset.ts @@ -29,6 +29,7 @@ import { import { prompt as askPrompt, KNOWN_CREDENTIAL_ENV_KEYS } from "../../credentials/store"; import { withGatewayRouteMutationLock } from "../../inference/gateway-route-mutation-lock"; import { + listRegisteredInferenceProviderAttachmentSandboxNames, clearNativeNvidiaProviderAuthority, listNativeNvidiaProviderAttachmentSandboxNames, } from "../../state/registry/native-provider-authority"; @@ -51,6 +52,7 @@ export type CredentialsResetResult = { export type CredentialsResetDeps = Readonly< NativeHostedResetDeps & { providerAdapter?: OpenShellProviderAdapter; + listRegisteredInferenceProviderAttachmentSandboxNames?: typeof listRegisteredInferenceProviderAttachmentSandboxNames; clearNativeNvidiaProviderAuthority?: typeof clearNativeNvidiaProviderAuthority; listNativeNvidiaProviderAttachmentSandboxNames?: typeof listNativeNvidiaProviderAttachmentSandboxNames; withGatewayRouteMutationLock?: typeof withGatewayRouteMutationLock; @@ -211,13 +213,28 @@ export async function runCredentialsResetAction( if (!target) return fail(recoveryFailureLines); const providerAdapter = deps.providerAdapter ?? createCliOpenShellProviderAdapter(); - if (hostedNativeProvider(key) || isNativeHostedProviderName(key)) { - return (deps.withGatewayRouteMutationLock ?? withGatewayRouteMutationLock)( - target.gatewayName, - async () => (await resetNativeHostedProvider(key, target, providerAdapter, deps))!, - ); - } const resetProvider = async (): Promise => { + let registeredSandboxes: readonly string[]; + try { + registeredSandboxes = ( + deps.listRegisteredInferenceProviderAttachmentSandboxNames ?? + listRegisteredInferenceProviderAttachmentSandboxNames + )(target.gatewayName, providerName); + } catch { + return fail([ + "Could not inspect registered inference selections. No provider or attachment was changed.", + ]); + } + if (registeredSandboxes.length) { + return fail([ + `Provider '${key}' is recorded by sandbox(es): ${registeredSandboxes.join(", ")}.`, + "Switch those sandboxes to another provider or destroy them before resetting this provider.", + "No external provider or existing attachment was changed.", + ]); + } + if (hostedNativeProvider(key) || isNativeHostedProviderName(key)) { + return (await resetNativeHostedProvider(key, target, providerAdapter, deps))!; + } if (nativeNvidiaProvider) { const blockers = nativeNvidiaResetBlockers(deps, target.gatewayName); if (!blockers.ok) { @@ -269,7 +286,6 @@ export async function runCredentialsResetAction( return ok(outcome.lines); }; - if (!nativeNvidiaProvider) return resetProvider(); return (deps.withGatewayRouteMutationLock ?? withGatewayRouteMutationLock)( target.gatewayName, resetProvider, diff --git a/src/lib/actions/inference-get.test.ts b/src/lib/actions/inference-get.test.ts index 67af216b351..7ad120af1f0 100644 --- a/src/lib/actions/inference-get.test.ts +++ b/src/lib/actions/inference-get.test.ts @@ -37,7 +37,7 @@ function configuredRoute(provider: string, model: string): OpenShellInferenceRou function nativeLocalSandbox( overrides: Record = {}, - agent: "langchain-deepagents-code" | "hermes" | "openclaw" = "langchain-deepagents-code", + agent: "langchain-deepagents-code" | "hermes" | "openclaw" | "pi" = "langchain-deepagents-code", ) { const binding: NativeLocalBinding = { provider: "ollama-local", @@ -263,6 +263,30 @@ describe("runInferenceGet", () => { }, ); + it.each(["openclaw", "hermes", "langchain-deepagents-code", "pi"] as const)( + "reads the %s Local NIM attachment without querying the shared route", + async (agent) => { + const deps = createDeps(configuredRoute("stale-provider", "stale-model")); + const row = nativeLocalSandbox({}, agent); + const binding = { + ...row.nativeLocalProviderAttachment, + provider: "vllm-local" as const, + authMode: "sentinel" as const, + endpointUrl: "http://host.openshell.internal:8000/v1", + }; + deps.getSandbox = () => ({ + ...row, + provider: binding.provider, + nimContainer: "owned-nim", + nativeLocalProviderAttachment: { ...binding, ...nativeLocalIdentity(binding) }, + }); + await expect( + runInferenceGet({ json: true, sandboxName: "alpha" }, deps), + ).resolves.toMatchObject({ provider: "vllm-local", endpointUrl: binding.endpointUrl }); + expect(deps.observeInferenceRoute).not.toHaveBeenCalled(); + }, + ); + it("prints the selected native-local provider, model, and endpoint (#12975)", async () => { const deps = createDeps(configuredRoute("stale-shared-provider", "stale-shared-model")); deps.getSandbox = () => nativeLocalSandbox(); @@ -1287,3 +1311,51 @@ describe("native hosted inference get", () => { }, ); }); + +it.each(["text", "json", "quiet"])( + "reads the registered sandbox selection without a shared route in %s mode", + async (mode) => { + const binding = { + schemaVersion: 1 as const, + gatewayName: "nemoclaw", + sandboxName: "alpha", + providerName: "operator-provider", + providerId: "operator-id", + profileId: "operator-profile", + profileDigest: "a".repeat(64), + endpointUrl: "https://api.example.test/v1", + inferenceApi: "openai-completions" as const, + credentialEnv: "OPERATOR_API_KEY", + attachedBySelection: false, + }; + const sandbox = { + name: "alpha", + agent: "openclaw", + gatewayName: binding.gatewayName, + provider: binding.providerName, + model: "model-a", + endpointUrl: binding.endpointUrl, + preferredInferenceApi: binding.inferenceApi, + credentialEnv: binding.credentialEnv, + registeredInferenceProviderAttachment: binding, + }; + const deps = createDeps(configuredRoute("unrelated", "other-model")); + deps.getSandbox = () => sandbox; + const result = await runInferenceGet( + { sandboxName: "alpha", json: mode === "json", quiet: mode === "quiet" }, + deps, + ); + expect(result).toEqual({ + provider: binding.providerName, + model: "model-a", + endpointUrl: binding.endpointUrl, + }); + expect(deps.observeInferenceRoute).not.toHaveBeenCalled(); + expect(deps.log).toHaveBeenCalledTimes(mode === "quiet" ? 0 : mode === "json" ? 1 : 3); + sandbox.gatewayName = "other"; + await expect(runInferenceGet({ sandboxName: "alpha" }, deps)).rejects.toThrow( + "does not match its sandbox route", + ); + expect(deps.observeInferenceRoute).not.toHaveBeenCalled(); + }, +); diff --git a/src/lib/actions/inference-get.ts b/src/lib/actions/inference-get.ts index 6c484956618..fa5994e6b29 100644 --- a/src/lib/actions/inference-get.ts +++ b/src/lib/actions/inference-get.ts @@ -10,6 +10,7 @@ import type { import { OPENSHELL_PROBE_TIMEOUT_MS } from "../adapters/openshell/timeouts"; import { unsafeEndpointUrlViolation } from "../core/endpoint-url-safety"; import { + recordedRegisteredInferenceProviderAttachment, getLlamaCppRouteDetails, sanitizeRouteValueForDisplay, type LlamaCppRouteDetails, @@ -316,6 +317,9 @@ export async function runInferenceGet( const selectedSandbox = selectedSandboxName ? (deps.getSandbox ?? getKnownSandboxTarget)(selectedSandboxName) : null; + const registeredAttachment = selectedSandbox + ? recordedRegisteredInferenceProviderAttachment(selectedSandbox) + : undefined; const nativeAttachment = selectedSandbox && selectedSandbox.nativeLocalProviderAttachment === undefined && @@ -343,10 +347,13 @@ export async function runInferenceGet( } return payload; } - if (selectedSandbox?.nativeLocalProviderAttachment !== undefined) { - const attachment = normalizeNativeLocalProviderAttachment( - selectedSandbox.nativeLocalProviderAttachment, - ); + if ( + selectedSandbox && + (registeredAttachment || selectedSandbox.nativeLocalProviderAttachment !== undefined) + ) { + const attachment = + registeredAttachment ?? + normalizeNativeLocalProviderAttachment(selectedSandbox.nativeLocalProviderAttachment); let recordedGatewayName: string | undefined; try { recordedGatewayName = getPersistedSandboxTargetGatewayName(selectedSandbox); @@ -362,21 +369,26 @@ export async function runInferenceGet( attachment.sandboxName !== selectedSandbox.name || attachment.gatewayName !== recordedGatewayName || attachment.gatewayName !== selectedSandbox.gatewayName || - attachment.provider !== selectedSandbox.provider || - attachment.credentialEnv !== NATIVE_LOCAL_CREDENTIAL_ENV || + (!registeredAttachment && + (!("provider" in attachment) || + attachment.provider !== selectedSandbox.provider || + attachment.credentialEnv !== NATIVE_LOCAL_CREDENTIAL_ENV)) || typeof selectedSandbox.model !== "string" || !selectedSandbox.model.trim() || - !(selectedSandbox.agent === "langchain-deepagents-code" - ? attachment.agent === undefined - : selectedSandbox.agent === "hermes" - ? attachment.agent === "hermes" - : selectedSandbox.agent === "openclaw" && - attachment.agent === "openclaw" && - isLocalInferenceProvider(selectedSandbox.provider) && - !selectedSandbox.nimContainer) + (!registeredAttachment && + !(selectedSandbox.agent === "langchain-deepagents-code" + ? !("agent" in attachment) || attachment.agent === undefined + : selectedSandbox.agent === "hermes" || selectedSandbox.agent === "pi" + ? "agent" in attachment && attachment.agent === selectedSandbox.agent + : selectedSandbox.agent === "openclaw" && + "agent" in attachment && + attachment.agent === "openclaw" && + isLocalInferenceProvider(selectedSandbox.provider))) ) { throw new InferenceGetError( - "Invalid native local provider attachment for the recorded inference selection", + registeredAttachment + ? "Invalid registered provider attachment for the recorded inference selection" + : "Invalid native local provider attachment for the recorded inference selection", ); } const endpoint = @@ -385,7 +397,7 @@ export async function runInferenceGet( ? { endpointUrl: attachment.endpointUrl } : endpointOmission("withheld"); const payload: InferenceGetResult = { - provider: selectedSandbox.provider, + provider: selectedSandbox.provider ?? null, model: selectedSandbox.model, ...endpoint, }; diff --git a/src/lib/actions/inference-set-compatible-provider.test.ts b/src/lib/actions/inference-set-compatible-provider.test.ts index 7e2bca169ff..94d18b850ae 100644 --- a/src/lib/actions/inference-set-compatible-provider.test.ts +++ b/src/lib/actions/inference-set-compatible-provider.test.ts @@ -9,10 +9,10 @@ import { baseSession, createCompatibleProviderCapture, createDeps, + HERMES_TARGET, + OPENCLAW_TARGET, } from "../../../test/helpers/inference-set"; -type ProbeSandboxRoute = NonNullable[0]["probeSandboxRoute"]>; - const OPENAI_PROFILE_OUTPUT = JSON.stringify({ id: "openai", credentials: [], @@ -27,106 +27,6 @@ const OPENAI_PROFILE_RESULT = { stderr: "", }; -async function runRejectedCompatibleSwitchScenario(options: { - targetFamily: "openai" | "anthropic"; - probeSandboxRoute: ProbeSandboxRoute; - expectedError: RegExp; -}) { - const target = - options.targetFamily === "anthropic" - ? { - provider: "compatible-anthropic-endpoint", - model: "mock-anthropic-model", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - inferenceApi: "anthropic-messages" as const, - captureType: "anthropic" as const, - configKey: "ANTHROPIC_BASE_URL" as const, - } - : { - provider: "compatible-endpoint", - model: "mock-model", - credentialEnv: "COMPATIBLE_API_KEY", - inferenceApi: "openai-completions" as const, - captureType: "openai" as const, - configKey: "OPENAI_BASE_URL" as const, - }; - const captureOpenshell = createCompatibleProviderCapture({ - name: target.provider, - type: target.captureType, - credentialEnv: target.credentialEnv, - configKey: target.configKey, - initiallyPresent: false, - }); - const probeSandboxRoute = vi.fn(options.probeSandboxRoute); - const deps = createDeps({ - config: { - agents: { defaults: { model: { primary: "inference/old-model" } } }, - models: { providers: { inference: { api: "openai-completions", models: [] } } }, - }, - entry: { - name: "alpha", - agent: "openclaw", - provider: "nvidia-prod", - model: "old-model", - }, - session: baseSession({ provider: "nvidia-prod", model: "old-model" }), - captureOpenshell, - probeSandboxRoute, - }); - - await expect( - runInferenceSet( - { - provider: target.provider, - model: target.model, - endpointUrl: "http://host.openshell.internal:18767/", - credentialEnv: target.credentialEnv, - inferenceApi: target.inferenceApi, - }, - deps, - ), - ).rejects.toThrow(options.expectedError); - - expect( - captureOpenshell.mock.calls - .filter(([args]) => args[0] === "inference" && args[1] === "set") - .map(([args]) => args), - ).toEqual([ - [ - "inference", - "set", - "-g", - "nemoclaw", - "--no-verify", - "--provider", - target.provider, - "--model", - target.model, - ], - [ - "inference", - "set", - "-g", - "nemoclaw", - "--no-verify", - "--provider", - "nvidia-prod", - "--model", - "old-model", - ], - ]); - expect( - captureOpenshell.mock.calls - .filter(([args]) => args[0] === "provider" && args[1] === "delete") - .map(([args]) => args), - ).toEqual([["provider", "delete", "-g", "nemoclaw", target.provider]]); - expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); - expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); - expect(deps.getSession()).toMatchObject({ provider: "nvidia-prod", model: "old-model" }); - - return { deps, probeSandboxRoute }; -} - describe("runInferenceSet compatible providers", () => { afterEach(() => vi.unstubAllEnvs()); @@ -851,272 +751,68 @@ describe("runInferenceSet compatible providers", () => { expect(deps.calls.restartSandboxGateway).toHaveBeenCalledWith("alpha", "nemoclaw"); }); - it("accepts explicit compatible Anthropic endpoint metadata for provider-family switches", async () => { - const config: ConfigObject = { - agents: { defaults: { model: { primary: "inference/nvidia/model-a" } } }, - models: { providers: { inference: { api: "openai-completions", models: [] } } }, - }; - const captureOpenshell = createCompatibleProviderCapture({ - name: "compatible-anthropic-endpoint", - type: "anthropic", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - configKey: "ANTHROPIC_BASE_URL", - initiallyPresent: false, - }); - const deps = createDeps({ - config, - entry: { - name: "alpha", - agent: "openclaw", - provider: "nvidia-prod", - model: "nvidia/model-a", - }, - session: baseSession({ - provider: "nvidia-prod", - model: "nvidia/model-a", - endpointUrl: "https://integrate.api.nvidia.com/v1", - credentialEnv: "NVIDIA_INFERENCE_API_KEY", - }), - captureOpenshell, - }); - - await runInferenceSet( - { - provider: "compatible-anthropic-endpoint", - model: "mock-anthropic-model", - endpointUrl: "http://host.openshell.internal:18767/", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - inferenceApi: "anthropic-messages", - }, - deps, - ); - - expect( - deps.calls.updateSandbox.mock.calls - .filter(([, fields]) => fields.provider !== undefined) - .at(-1), - ).toEqual([ - "alpha", - expect.objectContaining({ - provider: "compatible-anthropic-endpoint", - model: "mock-anthropic-model", - endpointUrl: "http://host.openshell.internal:18767", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - preferredInferenceApi: "anthropic-messages", - nimContainer: null, - }), - ]); - expect(deps.calls.rewriteConfigUrlsWithDnsPinning).not.toHaveBeenCalled(); - expect(captureOpenshell).toHaveBeenCalledWith( - [ - "inference", - "set", - "-g", - "nemoclaw", - "--no-verify", - "--provider", - "compatible-anthropic-endpoint", - "--model", - "mock-anthropic-model", - ], - expect.objectContaining({ ignoreError: true }), - ); - expect(deps.calls.probeSandboxRoute).toHaveBeenCalledWith({ - agentName: "openclaw", - gatewayName: "nemoclaw", - sandboxName: "alpha", - provider: "compatible-anthropic-endpoint", - model: "mock-anthropic-model", - preferredInferenceApi: "anthropic-messages", - }); - expect(deps.calls.probeSandboxRoute.mock.invocationCallOrder[0]).toBeLessThan( - deps.calls.updateSandbox.mock.invocationCallOrder[0], - ); - expect(deps.calls.sleep).toHaveBeenCalledWith(6_000); - }); - - it("waits for a changed API family to replace the previous sandbox route (#9467)", async () => { - const captureOpenshell = createCompatibleProviderCapture({ - name: "compatible-anthropic-endpoint", - type: "anthropic", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - configKey: "ANTHROPIC_BASE_URL", - initiallyPresent: false, - }); - const probeSandboxRoute = vi - .fn() - .mockReturnValueOnce({ - ok: false, - detail: "sandbox inference invocation probe returned HTTP 400", - httpStatus: 400, - }) - .mockReturnValueOnce({ ok: true }); - const deps = createDeps({ - config: { - agents: { defaults: { model: { primary: "inference/old-model" } } }, + it.each([OPENCLAW_TARGET, HERMES_TARGET])( + "persists admitted Anthropic API metadata for $agentName provider switches", + async (target) => { + const inferenceApi = + target.agentName === "hermes" ? "openai-completions" : "anthropic-messages"; + const config: ConfigObject = { + agents: { defaults: { model: { primary: "inference/nvidia/model-a" } } }, models: { providers: { inference: { api: "openai-completions", models: [] } } }, - }, - entry: { - name: "alpha", - agent: "openclaw", - provider: "compatible-endpoint", - model: "old-model", - }, - session: baseSession({ - provider: "compatible-endpoint", - model: "old-model", - preferredInferenceApi: "openai-completions", - }), - captureOpenshell, - probeSandboxRoute, - }); - - await runInferenceSet( - { - provider: "compatible-anthropic-endpoint", - model: "mock-anthropic-model", - endpointUrl: "http://host.openshell.internal:18767/", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - inferenceApi: "anthropic-messages", - }, - deps, - ); - - expect(probeSandboxRoute).toHaveBeenCalledTimes(2); - expect(deps.calls.sleep.mock.calls).toEqual([[6_000], [2_000]]); - expect(deps.calls.log).toHaveBeenCalledWith( - " Waiting 2s for OpenShell route convergence after HTTP 400 (probe 1/3)...", - ); - expect(deps.calls.updateSandbox).toHaveBeenCalled(); - }); - - it("restores the prior route after changed-family convergence retries are exhausted (#9467)", async () => { - const { deps, probeSandboxRoute } = await runRejectedCompatibleSwitchScenario({ - targetFamily: "anthropic", - probeSandboxRoute: vi - .fn() - .mockReturnValueOnce({ - ok: false, - detail: "sandbox inference invocation probe returned HTTP 400", - httpStatus: 400, - }) - .mockReturnValueOnce({ - ok: false, - detail: "sandbox inference invocation probe returned HTTP 404", - httpStatus: 404, - }) - .mockReturnValueOnce({ - ok: false, - detail: "sandbox inference invocation probe returned HTTP 400", - httpStatus: 400, - }), - expectedError: - /Sandbox-side verification rejected.*previous OpenShell inference selection was restored/s, - }); - - expect(probeSandboxRoute).toHaveBeenCalledTimes(3); - expect(deps.calls.sleep.mock.calls).toEqual([[6_000], [2_000], [4_000]]); - expect(deps.calls.log.mock.calls).toEqual( - expect.arrayContaining([ - [" Waiting 2s for OpenShell route convergence after HTTP 400 (probe 1/3)..."], - [" Waiting 4s for OpenShell route convergence after HTTP 404 (probe 2/3)..."], - ]), - ); - }); - - it.each([ - ["authentication", 401], - ["server", 500], - ])("does not retry a changed-family %s failure (#9467)", async (_failureClass, httpStatus) => { - const { deps, probeSandboxRoute } = await runRejectedCompatibleSwitchScenario({ - targetFamily: "anthropic", - probeSandboxRoute: async () => ({ - ok: false as const, - detail: `sandbox inference invocation probe returned HTTP ${httpStatus}`, - httpStatus, - }), - expectedError: - /Sandbox-side verification rejected.*previous OpenShell inference selection was restored/s, - }); - - expect(probeSandboxRoute).toHaveBeenCalledOnce(); - expect(deps.calls.sleep.mock.calls).toEqual([[6_000]]); - expect(deps.calls.log).not.toHaveBeenCalledWith(expect.stringContaining("route convergence")); - }); - - it("does not retry a target rejection when the API family did not change", async () => { - const { deps, probeSandboxRoute } = await runRejectedCompatibleSwitchScenario({ - targetFamily: "openai", - probeSandboxRoute: async () => ({ - ok: false as const, - detail: "sandbox inference invocation probe returned HTTP 400", - httpStatus: 400, - }), - expectedError: /Sandbox-side verification rejected/, - }); - - expect(probeSandboxRoute).toHaveBeenCalledOnce(); - expect(deps.calls.sleep.mock.calls).toEqual([[6_000]]); - }); - - it.each([ - [ - "returns a rejection", - async () => ({ - ok: false, - detail: "sandbox inference invocation probe exited with status 7", - httpStatus: null, - }), - /Sandbox-side verification rejected.*previous OpenShell inference selection was restored/s, - ], - [ - "throws", - async () => { - throw new Error("sandbox dial failed"); - }, - /sandbox inference invocation probe was unavailable: sandbox dial failed.*previous OpenShell inference selection was restored/s, - ], - ])( - "restores the prior route when sandbox-only provider verification %s", - async (_failureMode, probeSandboxRoute, expectedError) => { + }; const captureOpenshell = createCompatibleProviderCapture({ name: "compatible-anthropic-endpoint", - type: "anthropic", + type: target.agentName === "hermes" ? "openai" : "anthropic", credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - configKey: "ANTHROPIC_BASE_URL", + configKey: target.agentName === "hermes" ? "OPENAI_BASE_URL" : "ANTHROPIC_BASE_URL", initiallyPresent: false, }); const deps = createDeps({ - config: { agents: { defaults: { model: { primary: "inference/old-model" } } } }, + config, + target, entry: { name: "alpha", - agent: "openclaw", + agent: target.agentName, provider: "nvidia-prod", - model: "old-model", + model: "nvidia/model-a", }, - session: baseSession({ provider: "nvidia-prod", model: "old-model" }), + session: baseSession({ + provider: "nvidia-prod", + model: "nvidia/model-a", + endpointUrl: "https://integrate.api.nvidia.com/v1", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }), captureOpenshell, - probeSandboxRoute, }); - await expect( - runInferenceSet( - { - provider: "compatible-anthropic-endpoint", - model: "mock-anthropic-model", - endpointUrl: "http://host.openshell.internal:18767/", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - inferenceApi: "anthropic-messages", - }, - deps, - ), - ).rejects.toThrow(expectedError); + await runInferenceSet( + { + provider: "compatible-anthropic-endpoint", + model: "mock-anthropic-model", + endpointUrl: "https://anthropic.example/v1", + credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", + inferenceApi, + }, + deps, + ); expect( - captureOpenshell.mock.calls - .filter(([args]) => args[0] === "inference" && args[1] === "set") - .map(([args]) => args), + deps.calls.updateSandbox.mock.calls + .filter(([, fields]) => fields.provider !== undefined) + .at(-1), ).toEqual([ + "alpha", + expect.objectContaining({ + provider: "compatible-anthropic-endpoint", + model: "mock-anthropic-model", + endpointUrl: "http://host.openshell.internal:11438/route/test-route", + credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", + preferredInferenceApi: inferenceApi, + nimContainer: null, + }), + ]); + expect(deps.calls.rewriteConfigUrlsWithDnsPinning).not.toHaveBeenCalled(); + expect(captureOpenshell).toHaveBeenCalledWith( [ "inference", "set", @@ -1128,130 +824,35 @@ describe("runInferenceSet compatible providers", () => { "--model", "mock-anthropic-model", ], - [ - "inference", - "set", - "-g", - "nemoclaw", - "--no-verify", - "--provider", - "nvidia-prod", - "--model", - "old-model", - ], - ]); - expect( - captureOpenshell.mock.calls.some( - ([args]) => args[0] === "provider" && args[1] === "delete", - ), - ).toBe(true); - expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); - expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + expect.objectContaining({ ignoreError: true }), + ); }, ); - it("preserves redacted probe diagnostics when restoring the prior route fails", async () => { - const providerCapture = createCompatibleProviderCapture({ - name: "compatible-anthropic-endpoint", - type: "anthropic", - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - configKey: "ANTHROPIC_BASE_URL", - initiallyPresent: false, - }); - const inferenceSetResults = [ - null, - { - status: 19, - output: "restore rejected", - stdout: "", - stderr: "restore rejected", - }, - ]; - let inferenceSetCalls = 0; - const captureOpenshell = vi.fn((args: string[]) => { - switch (`${args[0]}:${args[1]}`) { - case "inference:set": - return inferenceSetResults[inferenceSetCalls++] ?? providerCapture(args); - default: - return providerCapture(args); - } - }); + it("rejects an unverified host bridge before provider or route mutation", async () => { + const captureOpenshell = vi.fn(); const deps = createDeps({ config: { agents: { defaults: { model: { primary: "inference/old-model" } } } }, - entry: { - name: "alpha", - agent: "openclaw", - provider: "nvidia-prod", - model: "old-model", - }, + entry: { name: "alpha", agent: "openclaw", provider: "nvidia-prod", model: "old-model" }, session: baseSession({ provider: "nvidia-prod", model: "old-model" }), captureOpenshell, - probeSandboxRoute: () => { - throw new Error("sandbox dial failed; NVIDIA_API_KEY=nvapi-secret-value"); - }, }); - let failure: unknown; - try { - await runInferenceSet( + await expect( + runInferenceSet( { provider: "compatible-anthropic-endpoint", model: "mock-anthropic-model", - endpointUrl: "http://host.openshell.internal:18767/", + endpointUrl: "http://host.openshell.internal:49152", credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", inferenceApi: "anthropic-messages", }, deps, - ); - } catch (error) { - failure = error; - } - - expect(failure).toBeInstanceOf(Error); - const failureMessage = (failure as Error).message; - expect(failureMessage).toContain( - "sandbox inference invocation probe was unavailable: sandbox dial failed", - ); - expect(failureMessage).toContain("NVIDIA_API_KEY="); - expect(failureMessage).not.toContain("nvapi-secret-value"); - expect(failureMessage).toMatch( - /Failed to restore the previous OpenShell inference selection.*status 19.*Re-run onboarding/s, - ); - expect( - deps.calls.captureOpenshell.mock.calls - .filter(([args]) => args[0] === "inference" && args[1] === "set") - .map(([args]) => args), - ).toEqual([ - [ - "inference", - "set", - "-g", - "nemoclaw", - "--no-verify", - "--provider", - "compatible-anthropic-endpoint", - "--model", - "mock-anthropic-model", - ], - [ - "inference", - "set", - "-g", - "nemoclaw", - "--no-verify", - "--provider", - "nvidia-prod", - "--model", - "old-model", - ], - ]); - expect( - deps.calls.captureOpenshell.mock.calls.some( - ([args]) => args[0] === "provider" && args[1] === "delete", ), - ).toBe(false); + ).rejects.toThrow("cannot use an unverified OpenShell host bridge endpoint"); + + expect(captureOpenshell).not.toHaveBeenCalled(); expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); - expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); }); it.each( @@ -1326,7 +927,7 @@ describe("runInferenceSet compatible providers", () => { }, ); - it("does not roll back a created direct provider after an ambiguous route result", async () => { + it("retains a legacy shared provider after an ambiguous route result", async () => { const captureOpenshell = createCompatibleProviderCapture({ name: "compatible-endpoint", type: "openai", @@ -1349,10 +950,10 @@ describe("runInferenceSet compatible providers", () => { entry: { name: "alpha", agent: "openclaw", - provider: "nvidia-prod", + provider: "compatible-endpoint", model: "old-model", }, - session: baseSession({ provider: "nvidia-prod", model: "old-model" }), + session: baseSession({ provider: "compatible-endpoint", model: "old-model" }), captureOpenshell, inferenceRouteMutator: { setInferenceRoute }, }); @@ -1363,7 +964,7 @@ describe("runInferenceSet compatible providers", () => { provider: "compatible-endpoint", model: "mock-model", noVerify: true, - endpointUrl: "http://host.openshell.internal:18767/v1", + endpointUrl: "http://8.8.8.8:18767/v1", credentialEnv: "COMPATIBLE_API_KEY", inferenceApi: "openai-completions", }, @@ -1383,7 +984,7 @@ describe("runInferenceSet compatible providers", () => { expect(deps.calls.setOpenClawConfigValues).not.toHaveBeenCalled(); }); - it("fails before provider or route mutation when a provider-backed route has no rollback authority", async () => { + it("rejects a retained shared route without rollback authority before mutation", async () => { const captureOpenshell = createCompatibleProviderCapture({ name: "compatible-endpoint", type: "openai", @@ -1394,7 +995,7 @@ describe("runInferenceSet compatible providers", () => { const setInferenceRoute = vi.fn(); const deps = createDeps({ config: {}, - entry: { name: "alpha", agent: "openclaw" }, + entry: { name: "alpha", agent: "openclaw", provider: "compatible-endpoint" }, captureOpenshell, inferenceRouteObserver: { observeInferenceRoute: vi.fn(async () => ({ @@ -1410,7 +1011,7 @@ describe("runInferenceSet compatible providers", () => { { provider: "compatible-endpoint", model: "mock-model", - endpointUrl: "http://host.openshell.internal:18767/v1", + endpointUrl: "http://8.8.8.8:18767/v1", credentialEnv: "COMPATIBLE_API_KEY", inferenceApi: "openai-completions", }, @@ -1426,7 +1027,7 @@ describe("runInferenceSet compatible providers", () => { ).toBe(false); }); - it("retries exactly once only for definite provider-not-found after direct binding", async () => { + it("retries a retained shared binding only after definite provider-not-found", async () => { const directCapture = createCompatibleProviderCapture({ name: "compatible-endpoint", type: "openai", @@ -1449,7 +1050,7 @@ describe("runInferenceSet compatible providers", () => { .mockResolvedValueOnce({ ok: true as const }); const directDeps = createDeps({ config: {}, - entry: { name: "alpha", agent: "openclaw", provider: "nvidia-prod", model: "old" }, + entry: { name: "alpha", agent: "openclaw", provider: "compatible-endpoint", model: "old" }, captureOpenshell: directCapture, inferenceRouteMutator: { setInferenceRoute: directSet }, }); @@ -1460,7 +1061,7 @@ describe("runInferenceSet compatible providers", () => { provider: "compatible-endpoint", model: "mock-model", noVerify: true, - endpointUrl: "http://host.openshell.internal:18767/v1", + endpointUrl: "http://8.8.8.8:18767/v1", credentialEnv: "COMPATIBLE_API_KEY", inferenceApi: "openai-completions", }, @@ -1481,11 +1082,11 @@ describe("runInferenceSet compatible providers", () => { })); const noBindingDeps = createDeps({ config: {}, - entry: { name: "alpha", agent: "openclaw", provider: "nvidia-prod", model: "old" }, + entry: { name: "alpha", agent: "openclaw", provider: "vllm-local", model: "old" }, inferenceRouteMutator: { setInferenceRoute: noBindingSet }, }); await expect( - runInferenceSet({ provider: "nvidia-nim", model: "gpt-test", noVerify: true }, noBindingDeps), + runInferenceSet({ provider: "vllm-local", model: "gpt-test", noVerify: true }, noBindingDeps), ).rejects.toThrow("provider not found"); expect(noBindingSet).toHaveBeenCalledOnce(); }); diff --git a/src/lib/actions/inference-set-context-window.test.ts b/src/lib/actions/inference-set-context-window.test.ts index edb49238f44..f414c4a647b 100644 --- a/src/lib/actions/inference-set-context-window.test.ts +++ b/src/lib/actions/inference-set-context-window.test.ts @@ -207,7 +207,7 @@ describe("runInferenceSet context window", () => { const options = { provider: "compatible-endpoint", model: "llama3.2:3b", - endpointUrl: "http://host.openshell.internal:11434/v1", + endpointUrl: "http://192.168.50.22:11434/v1", credentialEnv: "COMPATIBLE_API_KEY", inferenceApi: "openai-completions", noVerify: true, diff --git a/src/lib/actions/inference-set-degraded-state.test.ts b/src/lib/actions/inference-set-degraded-state.test.ts index 0357b839a69..08b4ab76014 100644 --- a/src/lib/actions/inference-set-degraded-state.test.ts +++ b/src/lib/actions/inference-set-degraded-state.test.ts @@ -174,9 +174,7 @@ describe("runInferenceSet degraded state handling", () => { const options = { provider: "compatible-endpoint", model: "openai/gpt-5.4-mini", - endpointUrl: initiallyPresent - ? "https://new.example/v1" - : "http://host.openshell.internal:11434/v1", + endpointUrl: initiallyPresent ? "https://new.example/v1" : "http://192.168.50.22:11434/v1", credentialEnv: "COMPATIBLE_API_KEY", inferenceApi: "openai-completions", noVerify: true, diff --git a/src/lib/actions/inference-set-endpoint-security.test.ts b/src/lib/actions/inference-set-endpoint-security.test.ts index 889db596006..f20c909d802 100644 --- a/src/lib/actions/inference-set-endpoint-security.test.ts +++ b/src/lib/actions/inference-set-endpoint-security.test.ts @@ -8,15 +8,15 @@ import { normalizeCustomEndpointUrl } from "./inference-set"; describe("custom inference endpoint DNS pinning", () => { it.each([1024, 65535])( - "allows the exact OpenShell bridge exemption at port %i without DNS rewriting", + "rejects an unverified OpenShell bridge at port %i without DNS rewriting", async (port) => { const rewriteUrl = vi.fn(async () => { - throw new Error("bridge exemption unexpectedly reached DNS validation"); + throw new Error("unverified bridge unexpectedly reached DNS validation"); }); await expect( normalizeCustomEndpointUrl(`http://host.openshell.internal:${port}/v1/`, rewriteUrl), - ).resolves.toBe(`http://host.openshell.internal:${port}/v1`); + ).rejects.toThrow("cannot use an unverified OpenShell host bridge endpoint"); expect(rewriteUrl).not.toHaveBeenCalled(); }, ); diff --git a/src/lib/actions/inference-set-failure-handling.test.ts b/src/lib/actions/inference-set-failure-handling.test.ts index 846c95e5a91..1a84f0dd199 100644 --- a/src/lib/actions/inference-set-failure-handling.test.ts +++ b/src/lib/actions/inference-set-failure-handling.test.ts @@ -3,7 +3,33 @@ import { describe, expect, it, vi } from "vitest"; import { InferenceSetError, runInferenceSet } from "./inference-set"; -import { createDeps } from "../../../test/helpers/inference-set"; +import { createDeps as createFixtureDeps } from "../../../test/helpers/inference-set"; + +// Existing compatible-endpoint sandboxes retain the shared-route transaction. +// Hermes and Model Router now require native provider admission. +function createDeps(options: Parameters[0]) { + const retained = { + name: "alpha", + agent: "openclaw", + provider: "compatible-endpoint", + endpointUrl: "https://retained-compatible.example/v1", + preferredInferenceApi: "openai-completions", + credentialEnv: "COMPATIBLE_API_KEY", + }; + return createFixtureDeps({ + ...options, + entry: options.entry === undefined ? retained : options.entry, + ...(options.entries + ? { + entries: options.entries.map((entry) => ({ + ...retained, + ...entry, + provider: "compatible-endpoint", + })), + } + : {}), + }); +} describe("runInferenceSet failure handling", () => { it("fails before OpenShell or config mutation for an unknown durable runtime provider", async () => { @@ -97,7 +123,7 @@ describe("runInferenceSet failure handling", () => { const deps = createDeps({ config: {}, openshellStatus: 17 }); await expect( - runInferenceSet({ provider: "nvidia-nim", model: "openai/model-a" }, deps), + runInferenceSet({ provider: "compatible-endpoint", model: "openai/model-a" }, deps), ).rejects.toThrow(/OpenShell inference route update with verification failed/); expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); @@ -108,7 +134,7 @@ describe("runInferenceSet failure handling", () => { const deps = createDeps({ config: {}, openshellStatus: 17 }); await expect( - runInferenceSet({ provider: "nvidia-nim", model: "openai/model-a" }, deps), + runInferenceSet({ provider: "compatible-endpoint", model: "openai/model-a" }, deps), ).rejects.toThrow( /route state is unknown.*Inspect gateway 'nemoclaw', then rerun the same `nemoclaw inference set` command\./su, ); @@ -123,7 +149,9 @@ describe("runInferenceSet failure handling", () => { const createProvider = vi.fn(); const base = createDeps({ config: {}, - entries: [{ name: "alpha", agent: "openclaw", provider: "nvidia-nim", model: "old-model" }], + entries: [ + { name: "alpha", agent: "openclaw", provider: "compatible-endpoint", model: "old-model" }, + ], inferenceRouteMutator: { setInferenceRoute }, inferenceRouteObserver: { observeInferenceRoute: vi.fn(async () => ({ @@ -142,7 +170,7 @@ describe("runInferenceSet failure handling", () => { }; await expect( - runInferenceSet({ provider: "nvidia-nim", model: "new-model" }, deps), + runInferenceSet({ provider: "compatible-endpoint", model: "new-model" }, deps), ).rejects.toThrow(/Cannot reconcile.*gateway 'nemoclaw'.*gateway unavailable/su); expect(createProvider).not.toHaveBeenCalled(); @@ -162,7 +190,7 @@ describe("runInferenceSet failure handling", () => { ok: true as const, value: { state: "configured" as const, - route: { provider: "nvidia-nim", model: "old-model" }, + route: { provider: "compatible-endpoint", model: "old-model" }, }, }; }); @@ -185,13 +213,15 @@ describe("runInferenceSet failure handling", () => { }); const deps = createDeps({ config: {}, - entries: [{ name: "alpha", agent: "openclaw", provider: "nvidia-nim", model: "old-model" }], + entries: [ + { name: "alpha", agent: "openclaw", provider: "compatible-endpoint", model: "old-model" }, + ], inferenceRouteObserver: { observeInferenceRoute }, inferenceRouteMutator: { setInferenceRoute }, }); await expect( - runInferenceSet({ provider: "nvidia-nim", model: "new-model" }, deps), + runInferenceSet({ provider: "compatible-endpoint", model: "new-model" }, deps), ).rejects.toThrow( /gateway schema mismatch.*Inspect gateway 'nemoclaw', then rerun the same `nemoclaw inference set` command\./su, ); @@ -199,8 +229,8 @@ describe("runInferenceSet failure handling", () => { expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); await expect( - runInferenceSet({ provider: "nvidia-nim", model: "new-model" }, deps), - ).resolves.toMatchObject({ provider: "nvidia-nim", model: "new-model" }); + runInferenceSet({ provider: "compatible-endpoint", model: "new-model" }, deps), + ).resolves.toMatchObject({ provider: "compatible-endpoint", model: "new-model" }); expect(events).toEqual([ "observe:nemoclaw", @@ -225,7 +255,7 @@ describe("runInferenceSet failure handling", () => { status: null, output: "", stdout: "", - stderr: `error: provider 'nvidia-nim' not found at https://${username}:${password}@gateway.example.test/v1?token=${querySecret} ${"x".repeat(3_000)}`, + stderr: `error: provider 'compatible-endpoint' not found at https://${username}:${password}@gateway.example.test/v1?token=${querySecret} ${"x".repeat(3_000)}`, error: Object.assign(new Error("spawnSync openshell ENOBUFS"), { code: "ENOBUFS" }), signal: "SIGTERM", }) @@ -237,7 +267,7 @@ describe("runInferenceSet failure handling", () => { }); const err = await runInferenceSet( - { provider: "nvidia-nim", model: "openai/gpt-5.4-mini" }, + { provider: "compatible-endpoint", model: "openai/gpt-5.4-mini" }, deps, ).catch((error: Error) => error); @@ -282,17 +312,17 @@ describe("runInferenceSet failure handling", () => { status: 1, output: "", stdout: "", - stderr: "error: provider 'nvidia-nim' not found in gateway", + stderr: "error: provider 'compatible-endpoint' not found in gateway", }); const err = await runInferenceSet( - { provider: "nvidia-nim", model: "openai/gpt-5.4-mini" }, + { provider: "compatible-endpoint", model: "openai/gpt-5.4-mini" }, deps, ).catch((e: Error) => e); expect(err).toBeInstanceOf(Error); const message = (err as Error).message; - expect(message).toMatch(/provider 'nvidia-nim' not found/); + expect(message).toMatch(/provider 'compatible-endpoint' not found/); expect(message).toMatch(/Registered providers: nvidia-prod/); expect(message).not.toMatch(/stale-local|telegram-bridge/); expect(message).toMatch(/Tip: register a new provider with `nemoclaw onboard`/); @@ -317,7 +347,7 @@ describe("runInferenceSet failure handling", () => { }); const err = await runInferenceSet( - { provider: "nvidia-nim", model: "openai/model-a" }, + { provider: "compatible-endpoint", model: "openai/model-a" }, deps, ).catch((e: Error) => e); @@ -342,8 +372,8 @@ describe("runInferenceSet failure handling", () => { deps.calls.captureOpenshell .mockReturnValueOnce({ status: 1, - output: "error: provider 'nvidia-nim' not found in gateway", - stdout: "error: provider 'nvidia-nim' not found in gateway", + output: "error: provider 'compatible-endpoint' not found in gateway", + stdout: "error: provider 'compatible-endpoint' not found in gateway", stderr: "", }) .mockReturnValueOnce({ @@ -354,7 +384,7 @@ describe("runInferenceSet failure handling", () => { }); const err = await runInferenceSet( - { provider: "nvidia-nim", model: "openai/gpt-5.4-mini" }, + { provider: "compatible-endpoint", model: "openai/gpt-5.4-mini" }, deps, ).catch((e: Error) => e); @@ -372,14 +402,14 @@ describe("runInferenceSet failure handling", () => { status: 1, output: "", stdout: "", - stderr: "error: provider 'nvidia-nim' not found in gateway", + stderr: "error: provider 'compatible-endpoint' not found in gateway", }) .mockImplementationOnce(() => { throw new Error(`gateway provider query failed token=${querySecret}`); }); const err = await runInferenceSet( - { provider: "nvidia-nim", model: "openai/gpt-5.4-mini" }, + { provider: "compatible-endpoint", model: "openai/gpt-5.4-mini" }, deps, ).catch((e: Error) => e); @@ -402,7 +432,7 @@ describe("runInferenceSet failure handling", () => { status: 1, output: "", stdout: "", - stderr: "error: provider 'nvidia-nim' not found in gateway", + stderr: "error: provider 'compatible-endpoint' not found in gateway", }) .mockReturnValueOnce({ status: null, @@ -414,7 +444,7 @@ describe("runInferenceSet failure handling", () => { }); const err = await runInferenceSet( - { provider: "nvidia-nim", model: "openai/gpt-5.4-mini" }, + { provider: "compatible-endpoint", model: "openai/gpt-5.4-mini" }, deps, ).catch((error: Error) => error); diff --git a/src/lib/actions/inference-set-gateway-restart.ts b/src/lib/actions/inference-set-gateway-restart.ts index 9ad0209d89a..05795b2916d 100644 --- a/src/lib/actions/inference-set-gateway-restart.ts +++ b/src/lib/actions/inference-set-gateway-restart.ts @@ -13,6 +13,9 @@ import { import type { GatewayRestartResult } from "./sandbox/gateway-restart"; import { observeOpenClawPairingSettlement, + OpenClawPairingQualificationError, + OPENCLAW_PAIRING_OBSERVATION_FAILURE_CATEGORIES, + type OpenClawPairingObservationFailureCategory, type OpenClawPairingSettlementObservation, } from "./sandbox/launch-readiness/openclaw-pairing-qualification"; @@ -36,6 +39,7 @@ export type InferenceSetOpenClawPairingResult = | { readonly ok: false; readonly failureLayer: InferenceSetOpenClawPairingFailureLayer; + readonly failureCategory?: OpenClawPairingObservationFailureCategory; }; export type InferenceSetOpenClawPairingDeps = { @@ -49,6 +53,13 @@ export type InferenceSetOpenClawPairingDeps = { ) => PortableOpenClawPairingApprovalReceipt; }; +function observationFailureCategory(error: unknown): OpenClawPairingObservationFailureCategory { + return error instanceof OpenClawPairingQualificationError && + OPENCLAW_PAIRING_OBSERVATION_FAILURE_CATEGORIES.includes(error.failureCategory) + ? error.failureCategory + : "unclassified"; +} + const defaultOpenClawPairingDeps: InferenceSetOpenClawPairingDeps = { observePairing: (target) => observeOpenClawPairingSettlement( @@ -82,8 +93,12 @@ export function settleInferenceSetOpenClawPairing( let initial: OpenClawPairingSettlementObservation; try { initial = deps.observePairing(target); - } catch { - return { ok: false, failureLayer: "initial-state-unavailable" }; + } catch (error) { + return { + ok: false, + failureLayer: "initial-state-unavailable", + failureCategory: observationFailureCategory(error), + }; } if (initial.state === "settled") return { ok: true }; @@ -98,8 +113,12 @@ export function settleInferenceSetOpenClawPairing( let final: OpenClawPairingSettlementObservation; try { final = deps.observePairing(target); - } catch { - return { ok: false, failureLayer: "final-state-unavailable" }; + } catch (error) { + return { + ok: false, + failureLayer: "final-state-unavailable", + failureCategory: observationFailureCategory(error), + }; } if (final.state === "settled") return { ok: true }; return { @@ -322,16 +341,19 @@ export async function completeInferencePostCommit { const deps = createDeps({ config: {}, entries: [ - entry("alpha", { provider: "nvidia-nim", model: "router/model-a" }), - entry("stopped-peer", { provider: "nvidia-nim", model: "router/model-a" }), + entry("alpha", { + provider: "compatible-endpoint", + model: "router/model-a", + endpointUrl: "http://192.168.50.22:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + preferredInferenceApi: "openai-completions", + }), + entry("stopped-peer", { + provider: "compatible-endpoint", + model: "router/model-a", + endpointUrl: "http://192.168.50.22:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + preferredInferenceApi: "openai-completions", + }), ], defaultSandbox: "alpha", }); await expect( runInferenceSet( - { provider: "nvidia-nim", model: "router/model-b", sandboxName: "alpha" }, + { provider: "compatible-endpoint", model: "router/model-b", sandboxName: "alpha" }, deps, ), ).resolves.toMatchObject({ sandboxName: "alpha", model: "router/model-b" }); @@ -121,7 +133,7 @@ describe("runtime shared gateway route containment", () => { "-g", "nemoclaw", "--provider", - "nvidia-nim", + "compatible-endpoint", "--model", "router/model-b", ], @@ -135,7 +147,9 @@ describe("runtime shared gateway route containment", () => { message.includes("Setting OpenShell inference route"), ); expect(warningIndex).toBeGreaterThanOrEqual(0); - expect(messages[warningIndex]).toContain("'stopped-peer' (nvidia-nim / router/model-a)"); + expect(messages[warningIndex]).toContain( + "'stopped-peer' (compatible-endpoint / router/model-a)", + ); expect(warningIndex).toBeLessThan(mutationIndex); }); @@ -147,7 +161,10 @@ describe("runtime shared gateway route containment", () => { entry("alpha", { gatewayName: "nemoclaw-9090", gatewayPort: 9090, - provider: "nvidia-nim", + provider: "compatible-endpoint", + endpointUrl: "http://192.168.50.22:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + preferredInferenceApi: "openai-completions", model: "router/model-a", }), entry("default-gateway-peer"), @@ -158,7 +175,7 @@ describe("runtime shared gateway route containment", () => { await expect( runInferenceSet( - { provider: "nvidia-nim", model: "router/model-b", sandboxName: "alpha" }, + { provider: "compatible-endpoint", model: "router/model-b", sandboxName: "alpha" }, deps, ), ).resolves.toMatchObject({ sandboxName: "alpha", model: "router/model-b" }); @@ -170,7 +187,7 @@ describe("runtime shared gateway route containment", () => { "-g", "nemoclaw-9090", "--provider", - "nvidia-nim", + "compatible-endpoint", "--model", "router/model-b", ], diff --git a/src/lib/actions/inference-set-hermes-local.test.ts b/src/lib/actions/inference-set-hermes-local.test.ts index 1838a71dda6..31b4fcfdff0 100644 --- a/src/lib/actions/inference-set-hermes-local.test.ts +++ b/src/lib/actions/inference-set-hermes-local.test.ts @@ -7,8 +7,13 @@ import { HERMES_TARGET, nativeLocalTestReceipt, } from "../../../test/helpers/inference-set"; +import { switchingFixture } from "../../../test/helpers/inference-set-openclaw-native-local"; +import { withNativePolicyFixture } from "../../../test/support/native-local-setup-harness"; +import { prepareNativeLocalSwitch } from "./inference/native-local"; +import { verifyNativeLocalProviderAttachment } from "../inference/native-local/profile"; import { applyHermesManagedRoute } from "../hermes-managed-route"; -import { nativeLocalIdentity } from "../inference/native-local/contract"; +import type { SandboxEntry } from "../state/registry"; +import { nativeLocalIdentity, type NativeLocalBinding } from "../inference/native-local/contract"; import type { ConfigObject } from "../security/credential-filter"; import { runInferenceSet } from "./inference-set"; import { @@ -23,15 +28,18 @@ const selection = { sandboxName: "alpha", noVerify: true, }; -function fixture() { - const binding = { ...nativeLocalTestReceipt(), agent: "hermes" as const }; +function fixture( + bindingOverride: Partial = {}, + entryOverride: Partial = {}, +) { + const binding = { ...nativeLocalTestReceipt(), agent: "hermes" as const, ...bindingOverride }; const receipt = { ...binding, ...nativeLocalIdentity(binding) }; const config: ConfigObject = { mcp_servers: { example: { command: "example-mcp" } } }; applyHermesManagedRoute(config, { upstreamProvider: receipt.provider, baseUrl: receipt.endpointUrl, model: "old-local-model", - inferenceApi: "openai-completions", + inferenceApi: receipt.api ?? "openai-completions", nativeCredentialHandle: issued("v6"), }); const deps = createDeps({ @@ -43,6 +51,7 @@ function fixture() { model: "old-local-model", endpointUrl: receipt.endpointUrl, nativeLocalProviderAttachment: receipt, + ...entryOverride, }, defaultSandbox: "alpha", target: HERMES_TARGET, @@ -269,3 +278,258 @@ describe("Hermes native local selection", () => { expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); }); }); + +it.each(["openai-completions", "openai-responses"] as const)( + "switches Hermes to reserved-alias %s with its selected credential", + async (api) => { + const { deps } = switchingFixture( + { + agent: "hermes", + ...(api === "openai-responses" ? { api } : {}), + provider: "compatible-endpoint", + authMode: "authenticated", + endpointUrl: "http://host.openshell.internal:42103/v1", + }, + [{ agent: "hermes" }], + ); + deps.getSandbox("alpha")!.agent = "hermes"; + deps.resolveAgentConfig = () => HERMES_TARGET; + const config: ConfigObject = {}; + applyHermesManagedRoute(config, { + upstreamProvider: "ollama-local", + baseUrl: "http://host.openshell.internal:11435/v1", + model: "old-model", + inferenceApi: "openai-completions", + nativeCredentialHandle: issued("v6"), + }); + deps.calls.readSandboxConfig.mockReturnValue(config); + deps.prepareNativeLocalSwitch = prepareNativeLocalSwitch; + deps.verifyNativeLocalProviderAttachment = verifyNativeLocalProviderAttachment; + deps.calls.resolveCredentialValue.mockReturnValue("fixture-only-host-key"); + const execute = vi.fn>(async () => ({ + status: 0, + stdout: + '200\n{"choices":[{"message":{"content":"OK"}}],"output":[{"type":"message","content":[{"type":"output_text","text":"OK"}]}]}', + stderr: "", + })); + deps.calls.probeSandboxRoute.mockImplementation((input) => + probeSandboxInferenceInvocation(input, { execute }), + ); + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + await withNativePolicyFixture(() => + runInferenceSet( + { + provider: "compatible-endpoint", + model: "served-alias", + sandboxName: "alpha", + endpointUrl: "http://host.openshell.internal:42103/v1", + credentialEnv: "COMPATIBLE_API_KEY", + inferenceApi: api, + }, + deps, + ), + ); + expect(create).toHaveBeenCalledWith( + expect.objectContaining({ + credentials: [{ name: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", value: "fixture-only-host-key" }], + }), + ); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + endpointUrl: "http://host.openshell.internal:42103/v1", + nativeLocalProviderAttachment: expect.objectContaining({ + agent: "hermes", + provider: "compatible-endpoint", + }), + }), + ); + expect(execute.mock.calls[0][1]).toContain( + api === "openai-responses" ? "/v1/responses" : "/v1/chat/completions", + ); + expect(config.model.api_mode ?? null).toBe( + api === "openai-responses" ? "codex_responses" : null, + ); + expect(config.model).toMatchObject({ + base_url: "http://host.openshell.internal:42103/v1", + api_key: issued("v1"), + }); + expect( + deps.calls.captureOpenshell.mock.calls.some( + ([args]) => Array.isArray(args) && args[0] === "inference", + ), + ).toBe(false); + }, +); + +it.each([ + "http://127.0.0.1:11434/v1", + "http://127.0.0.1:8000/other", + "http://host.openshell.internal:42103/v1", +])( + "rejects a Hermes no-auth backend replacement %s before preparing a provider", + async (endpointUrl) => { + const { deps, receipt } = fixture(); + const binding = { + ...receipt, + provider: "compatible-endpoint" as const, + authMode: "authenticated" as const, + }; + Object.assign(deps.getSandbox("alpha")!, { + provider: "compatible-endpoint", + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN", + nativeLocalProviderAttachment: { ...binding, ...nativeLocalIdentity(binding) }, + }); + const prepare = vi.spyOn(deps, "prepareNativeLocalSwitch"); + await expect( + runInferenceSet( + { + provider: "compatible-endpoint", + model: "served-alias", + sandboxName: "alpha", + endpointUrl, + inferenceApi: "openai-completions", + }, + deps, + ), + ).rejects.toThrow(/no-auth proxy is bound to the recorded backend/); + expect(prepare).not.toHaveBeenCalled(); + expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + }, +); + +it.each([undefined, "http://127.0.0.1:8000/v1", "http://127.0.0.1:8000/v1/"])( + "reuses authenticated recorded loopback %s through its verified native attachment", + async (endpointUrl) => { + const { deps, receipt, config } = fixture( + { + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8000/v1", + authMode: "authenticated", + api: "openai-responses", + }, + { + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + preferredInferenceApi: "openai-responses", + }, + ); + deps.prepareNativeLocalSwitch = prepareNativeLocalSwitch; + deps.verifyNativeLocalProviderAttachment = verifyNativeLocalProviderAttachment; + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + await withNativePolicyFixture(() => + runInferenceSet( + { provider: "compatible-endpoint", model: "new-model", sandboxName: "alpha", endpointUrl }, + deps, + ), + ); + expect(create).not.toHaveBeenCalled(); + expect(deps.calls.resolveCredentialValue).not.toHaveBeenCalled(); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + nativeLocalProviderAttachment: receipt, + preferredInferenceApi: "openai-responses", + }), + ); + expect(config.model).toMatchObject({ + base_url: "http://host.openshell.internal:8000/v1", + api_key: issued("v1"), + api_mode: "codex_responses", + }); + }, +); + +it.each([ + { port: 8000, credentialEnv: "COMPATIBLE_API_KEY" }, + { port: 11435, credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN" }, +])( + "changes a recorded Hermes backend to Responses through native port $port", + async ({ port, credentialEnv }) => { + const endpointUrl = `http://host.openshell.internal:${port}/v1`; + const { deps, next } = switchingFixture( + { + agent: "hermes", + provider: "compatible-endpoint", + endpointUrl, + authMode: "authenticated", + api: "openai-responses", + }, + [ + { + agent: "hermes", + provider: "compatible-endpoint", + endpointUrl, + authMode: "authenticated", + }, + credentialEnv, + "http://127.0.0.1:8000/v1", + ], + ); + deps.getSandbox("alpha")!.agent = "hermes"; + deps.resolveAgentConfig = () => HERMES_TARGET; + const config: ConfigObject = {}; + applyHermesManagedRoute(config, { + upstreamProvider: "compatible-endpoint", + baseUrl: endpointUrl, + model: "old-model", + inferenceApi: "openai-completions", + nativeCredentialHandle: issued("v6"), + }); + deps.calls.readSandboxConfig.mockReturnValue(config); + deps.prepareNativeLocalSwitch = prepareNativeLocalSwitch; + deps.verifyNativeLocalProviderAttachment = verifyNativeLocalProviderAttachment; + deps.calls.resolveCredentialValue.mockReturnValue("fixture-only-selected-token"); + await withNativePolicyFixture(() => + runInferenceSet( + { + provider: "compatible-endpoint", + model: "new-model", + sandboxName: "alpha", + endpointUrl: "http://127.0.0.1:8000/v1", + inferenceApi: "openai-responses", + }, + deps, + ), + ); + expect(deps.calls.resolveCredentialValue).toHaveBeenCalledWith(credentialEnv); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + nativeLocalProviderAttachment: next, + preferredInferenceApi: "openai-responses", + credentialEnv, + }), + ); + expect(config.model).toMatchObject({ + base_url: endpointUrl, + api_key: issued("v1"), + api_mode: "codex_responses", + }); + }, +); + +it("rejects a new Hermes raw loopback route before creating a provider", async () => { + const { deps } = fixture(); + deps.prepareNativeLocalSwitch = prepareNativeLocalSwitch; + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + await withNativePolicyFixture(() => + expect( + runInferenceSet( + { + provider: "compatible-endpoint", + model: "new-model", + sandboxName: "alpha", + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + inferenceApi: "openai-responses", + }, + deps, + ), + ).rejects.toThrow(/Invalid native host-local inference boundary/), + ); + expect(create).not.toHaveBeenCalled(); + expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); +}); diff --git a/src/lib/actions/inference-set-hermes-run.test.ts b/src/lib/actions/inference-set-hermes-run.test.ts index e56bc120f9f..0e654fbc9c4 100644 --- a/src/lib/actions/inference-set-hermes-run.test.ts +++ b/src/lib/actions/inference-set-hermes-run.test.ts @@ -309,7 +309,10 @@ describe("runInferenceSet Hermes routing", () => { terminal: { backend: "local" }, }); expect(deps.calls.writeSandboxConfig).toHaveBeenCalledTimes(1); - expect(deps.calls.writeSandboxConfig).toHaveBeenCalledWith("hermes", HERMES_TARGET, config); + expect(deps.calls.writeSandboxConfig).toHaveBeenCalledWith("hermes", HERMES_TARGET, config, { + gatewayName: "nemoclaw", + deferReload: true, + }); expect(deps.calls.writeSandboxConfig.mock.calls[0][1].configPath).toBe( "/sandbox/.hermes/config.yaml", ); @@ -442,7 +445,7 @@ describe("runInferenceSet Hermes routing", () => { ).toBe(false); }); - it("fails after commit when the config hash refresh fails (#7083)", async () => { + it("fails after route commit when the native config transaction cannot refresh hashes (#7083)", async () => { const config: ConfigObject = { model: { default: "moonshotai/kimi-k2.6", provider: "custom" }, }; @@ -458,7 +461,7 @@ describe("runInferenceSet Hermes routing", () => { target: HERMES_TARGET, session: baseSession({ agent: "hermes", sandboxName: "hermes" }), }); - deps.calls.recomputeSandboxConfigHash.mockImplementation(() => { + deps.calls.writeSandboxConfig.mockImplementation(() => { throw new Error("hash refresh failed"); }); @@ -487,7 +490,9 @@ describe("runInferenceSet Hermes routing", () => { }), ); const logs = deps.calls.log.mock.calls.map((call) => String(call[0])); - expect(logs.some((line) => line.includes("failed to refresh its integrity hash"))).toBe(true); + expect( + logs.some((line) => line.includes("in-sandbox config failed: hash refresh failed")), + ).toBe(true); expect(logs.some((line) => line.includes("Run 'nemoclaw hermes rebuild'"))).toBe(false); expect(logs.some((line) => line.includes("Inference route synced"))).toBe(false); }); @@ -734,7 +739,12 @@ describe("runInferenceSet Hermes routing", () => { await runInferenceSet({ provider: "hermes-provider", model: "z-ai/glm-5.1" }, deps); - expect(deps.calls.writeSandboxConfig).toHaveBeenCalledWith("hermes-one", HERMES_TARGET, config); + expect(deps.calls.writeSandboxConfig).toHaveBeenCalledWith( + "hermes-one", + HERMES_TARGET, + config, + { gatewayName: "nemoclaw", deferReload: true }, + ); expect(deps.calls.updateSandbox).toHaveBeenCalledWith( "hermes-one", expect.objectContaining({ diff --git a/src/lib/actions/inference-set-live-rollback.test.ts b/src/lib/actions/inference-set-live-rollback.test.ts index a09ca2732cc..d6d5d309995 100644 --- a/src/lib/actions/inference-set-live-rollback.test.ts +++ b/src/lib/actions/inference-set-live-rollback.test.ts @@ -6,7 +6,7 @@ import { runInferenceSet } from "./inference-set"; import { createCompatibleProviderCapture, createDeps } from "../../../test/helpers/inference-set"; describe("runInferenceSet live rollback authority", () => { - it("does not reapply or report restoration when the observed route already matches the rejected route", async () => { + it("allows an admitted endpoint retry after an ambiguous route result", async () => { const captureOpenshell = createCompatibleProviderCapture({ name: "compatible-endpoint", type: "openai", @@ -55,7 +55,7 @@ describe("runInferenceSet live rollback authority", () => { agent: "openclaw", provider: "compatible-endpoint", model: "old-model", - endpointUrl: "http://host.openshell.internal:18767/v1", + endpointUrl: "http://8.8.8.8:18767/v1", credentialEnv: "COMPATIBLE_API_KEY", preferredInferenceApi: "openai-completions", }, @@ -68,18 +68,19 @@ describe("runInferenceSet live rollback authority", () => { provider: "compatible-endpoint", model: "mock-model", noVerify: true, - endpointUrl: "http://host.openshell.internal:18767/v1", + endpointUrl: "http://8.8.8.8:18767/v1", credentialEnv: "COMPATIBLE_API_KEY", inferenceApi: "openai-completions" as const, }; await expect(runInferenceSet(request, deps)).rejects.toThrow("first route result unknown"); - await expect(runInferenceSet(request, deps)).rejects.toThrow( - /no distinct prior inference selection to restore/u, - ); + await expect(runInferenceSet(request, deps)).resolves.toMatchObject({ + provider: "compatible-endpoint", + model: "mock-model", + }); expect(observeInferenceRoute).toHaveBeenCalledTimes(2); expect(setInferenceRoute).toHaveBeenCalledTimes(2); - expect(probeSandboxRoute).toHaveBeenCalledTimes(3); + expect(probeSandboxRoute).not.toHaveBeenCalled(); }); }); diff --git a/src/lib/actions/inference-set-native-hosted.test.ts b/src/lib/actions/inference-set-native-hosted.test.ts index 6199c7f50e8..458fa32f79d 100644 --- a/src/lib/actions/inference-set-native-hosted.test.ts +++ b/src/lib/actions/inference-set-native-hosted.test.ts @@ -2,7 +2,11 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import * as endpointSecurity from "../security/trusted-private-endpoint"; +import fs from "node:fs"; +import YAML from "yaml"; import type { OpenShellProviderAdapter } from "../adapters/openshell/provider-adapter"; +import { nativeLocalIdentity } from "../inference/native-local/contract"; import { customNativeProvider } from "../inference/native-provider/custom"; import { HOSTED_NATIVE_PROVIDERS, hostedNativeProvider } from "../inference/native-provider/hosted"; import type { NativeProviderAttachment } from "../inference/native-provider/contract"; @@ -11,7 +15,12 @@ import { normalizeNativeNvidiaProviderAttachment } from "../inference/native-nvi import type { SandboxEntry } from "../state/registry"; import { runInferenceSet } from "./inference-set"; import { prepareRebuildResumeConfig } from "./sandbox/rebuild-resume-config"; -import { createDeps, HERMES_TARGET, OPENCLAW_TARGET } from "../../../test/helpers/inference-set"; +import { + createDeps, + HERMES_TARGET, + OPENCLAW_TARGET, + nativeLocalTestReceipt, +} from "../../../test/helpers/inference-set"; function fixture(extraDefinitions: ReturnType[] = []) { const definitions = [ @@ -95,7 +104,7 @@ function fixture(extraDefinitions: ReturnType[] = [ deps.setNativeHostedProviderAuthority = (gateway, provider, receipt) => { authorities.set(`${gateway}/${receipt.custom ? receipt.providerName : provider}`, receipt); }; - return { deps, calls, attached, authorities, entry }; + return { deps, calls, attached, authorities, entry, existing }; } it("rejects Pi runtime switching before changing its native attachment (#12589)", async () => { @@ -330,32 +339,80 @@ it("keeps a Hermes sandbox on its recorded endpoint after another sandbox change ); }); -it.each([OPENCLAW_TARGET, HERMES_TARGET])( - "preserves a user-supplied Hermes endpoint in $agentName configuration", - async (target) => { - const agent = target.agentName; - const { deps, calls, entry } = fixture(); - entry.provider = "hermes-provider"; - entry.agent = agent; - deps.resolveAgentConfig = () => target; - entry.endpointUrl = "https://custom.example/v1"; - entry.credentialEnv = "OPENAI_API_KEY"; - entry.preferredInferenceApi = "openai-completions"; - await runInferenceSet({ provider: "hermes-provider", model: "new-model" }, deps); - expect(calls.attachProvider).not.toHaveBeenCalled(); - expect(calls.createProvider).not.toHaveBeenCalled(); - expect(entry.nativeHostedProviderAttachment).toBeUndefined(); - const configWrites = - agent === "hermes" - ? deps.calls.writeSandboxConfig.mock.calls - : deps.calls.setOpenClawConfigValues.mock.calls; - expect(JSON.stringify(configWrites)).toContain("https://inference.local/v1"); - expect(JSON.stringify(configWrites)).not.toContain("https://inference-api.nousresearch.com"); - expect(JSON.stringify(configWrites)).not.toContain("openshell:resolve:env:OPENAI_API_KEY"); - expect(deps.inferenceRouteObserver.observeInferenceRoute).toHaveBeenCalled(); - expect(deps.calls.captureOpenshell).toHaveBeenCalledWith( - expect.arrayContaining(["inference", "set", "--provider", "hermes-provider"]), - expect.anything(), +describe.each([OPENCLAW_TARGET, HERMES_TARGET])( + "recorded Hermes endpoint for $agentName", + (target) => { + it.each([ + "http://custom.example/team/v1", + "https://8.8.8.8/team/v1", + "http://host.openshell.internal:42103/team/v1", + ])( + "migrates %s to an attached provider and preserves it across model changes and rebuild", + async (endpointUrl) => { + const validateEndpoint = endpointSecurity.assertEndpointResolvesPublic; + vi.spyOn(endpointSecurity, "assertEndpointResolvesPublic").mockImplementation((url) => + validateEndpoint(url, async () => [{ address: "8.8.8.8", family: 4 }]), + ); + const definition = hostedNativeProvider("hermes-provider", endpointUrl)!; + const { deps, calls, entry, attached } = fixture([definition]); + entry.provider = "hermes-provider"; + entry.agent = target.agentName; + entry.endpointUrl = endpointUrl; + entry.credentialEnv = "OPENAI_API_KEY"; + entry.preferredInferenceApi = "openai-completions"; + deps.resolveAgentConfig = () => target; + const imported: unknown[] = []; + calls.importProviderProfile.mockImplementation(({ profilePath }) => { + imported.push(YAML.parse(fs.readFileSync(profilePath, "utf8"))); + return { ok: true }; + }); + await runInferenceSet({ provider: "hermes-provider", model: "new-model" }, deps); + expect(entry.nativeHostedProviderAttachment).toMatchObject({ + endpointUrl, + providerName: definition.providerName, + }); + expect([...attached.get("alpha")!]).toEqual([definition.providerName]); + expect(imported[0]).toMatchObject({ + endpoints: [ + { + host: new URL(endpointUrl).hostname, + rules: [ + { allow: { method: "GET", path: "/team/v1/models" } }, + { allow: { method: "POST", path: "/team/v1/chat/completions" } }, + ], + }, + ], + }); + expect(calls.createProvider).toHaveBeenCalledWith( + expect.objectContaining({ + credentials: [{ name: "OPENAI_API_KEY", value: "canary-host-secret" }], + }), + ); + const configWrites = + target.agentName === "hermes" + ? deps.calls.writeSandboxConfig.mock.calls + : deps.calls.setOpenClawConfigValues.mock.calls; + expect(JSON.stringify(configWrites)).toContain(endpointUrl); + expect(JSON.stringify(configWrites)).toContain( + target.agentName === "hermes" + ? "${OPENAI_API_KEY}" + : "openshell:resolve:env:OPENAI_API_KEY", + ); + expect(JSON.stringify(configWrites)).not.toContain("canary-host-secret"); + expect(deps.inferenceRouteObserver.observeInferenceRoute).not.toHaveBeenCalled(); + deps.resolveCredentialValue = vi.fn(() => ""); + await runInferenceSet({ provider: "hermes-provider", model: "second-model" }, deps); + expect(deps.resolveCredentialValue).not.toHaveBeenCalled(); + expect(calls.updateProvider).not.toHaveBeenCalled(); + expect( + prepareRebuildResumeConfig("alpha", entry, target.agentName, vi.fn(), (message) => { + throw new Error(message); + }), + ).toMatchObject({ + endpointUrl, + nativeHostedProviderAttachment: entry.nativeHostedProviderAttachment, + }); + }, ); }, ); @@ -533,6 +590,56 @@ describe("Hermes native gateway credential refresh", () => { }); describe("custom native inference selection", () => { + it("rejects an unverified Anthropic bridge before replacing native local inference", async () => { + const binding = { ...nativeLocalTestReceipt(), agent: "hermes" as const }; + const local = { + ...binding, + ...nativeLocalIdentity(binding), + providerId: `id-${nativeLocalIdentity(binding).providerName}`, + }; + const { deps, entry, calls, attached, existing } = fixture([ + { + ...local, + logicalProvider: local.provider, + endpoint: local.endpointUrl, + api: "openai-completions", + label: "local", + }, + ]); + Object.assign(entry, { + agent: "hermes", + provider: local.provider, + endpointUrl: local.endpointUrl, + nativeLocalProviderAttachment: local, + }); + deps.resolveAgentConfig = () => HERMES_TARGET; + deps.providerAdapter.inspectProviderProfile = async () => ({ + ok: true, + value: { credentialKeys: [local.credentialEnv] }, + }); + attached.get("alpha")!.add(local.providerName); + existing.add(local.providerName); + const before = structuredClone(entry); + await expect( + runInferenceSet( + { + provider: "compatible-anthropic-endpoint", + model: "mock-anthropic-model", + endpointUrl: "http://host.openshell.internal:49152", + inferenceApi: "openai-completions", + }, + deps, + ), + ).rejects.toThrow("cannot use an unverified OpenShell host bridge endpoint"); + expect(entry).toEqual(before); + expect([...attached.get("alpha")!]).toEqual([local.providerName]); + expect([...attached.get("beta")!]).toEqual(["unrelated-provider"]); + expect(deps.inferenceRouteObserver.observeInferenceRoute).not.toHaveBeenCalled(); + expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + expect(deps.calls.restartSandboxGateway).not.toHaveBeenCalled(); + expect(calls.importProviderProfile).not.toHaveBeenCalled(); + expect(calls.createProvider).not.toHaveBeenCalled(); + }); const custom = customNativeProvider({ provider: "compatible-endpoint", api: "openai-completions", @@ -584,6 +691,38 @@ describe("custom native inference selection", () => { expect([...attached.get("alpha")!]).toEqual([custom.providerName]); expect(deps.inferenceRouteObserver.observeInferenceRoute).not.toHaveBeenCalled(); }); + it("waits for a replacement endpoint credential with unchanged provider and model (#12603)", async () => { + const { deps, entry } = fixture([custom, replacement]); + await runInferenceSet({ provider: "openai-api", model: "first-model" }, deps); + await runInferenceSet( + { + provider: "compatible-endpoint", + model: "custom-model", + endpointUrl: custom.endpoint, + inferenceApi: custom.api, + }, + deps, + ); + let elapsed = 0; + deps.sleep = async (milliseconds) => { + elapsed += milliseconds; + }; + deps.calls.probeSandboxRoute.mockImplementation(async () => + elapsed >= 10_000 + ? { ok: true } + : { ok: false, detail: "native credential handle pending", httpStatus: null }, + ); + await runInferenceSet( + { + provider: "compatible-endpoint", + model: "custom-model", + endpointUrl: replacement.endpoint, + inferenceApi: replacement.api, + }, + deps, + ); + expect(entry.nativeHostedProviderAttachment?.custom).toEqual(replacement.custom); + }); it("restores the prior attachment if a changed custom endpoint fails sandbox verification", async () => { const { deps, calls, entry, attached, authorities } = fixture([custom, replacement]); await runInferenceSet({ provider: "openai-api", model: "first-model" }, deps); @@ -631,6 +770,64 @@ describe("custom native inference selection", () => { }); }); +it("restores the prior attachment when a retained Hermes endpoint fails sandbox verification", async () => { + const endpointUrl = "http://host.openshell.internal:42103/team/v1"; + const definition = hostedNativeProvider("hermes-provider", endpointUrl)!; + const { deps, calls, entry, attached } = fixture([definition]); + entry.agent = "hermes"; + deps.resolveAgentConfig = () => HERMES_TARGET; + await runInferenceSet({ provider: "openai-api", model: "previous-model" }, deps); + const previous = structuredClone(entry); + deps.loadSession = () => + ({ + sandboxName: "alpha", + provider: "hermes-provider", + model: "retained-model", + endpointUrl, + credentialEnv: "NOUS_API_KEY", + preferredInferenceApi: "openai-completions", + }) as never; + deps.calls.probeSandboxRoute.mockResolvedValue({ + ok: false, + detail: "retained endpoint refused", + httpStatus: 401, + }); + await expect( + runInferenceSet({ provider: "hermes-provider", model: "retained-model" }, deps), + ).rejects.toThrow("retained endpoint refused"); + expect(entry).toEqual(previous); + expect([...attached.get("alpha")!]).toEqual([ + previous.nativeHostedProviderAttachment!.providerName, + ]); + expect(calls.detachProvider).toHaveBeenCalledWith( + expect.objectContaining({ providerName: definition.providerName }), + ); + expect([...attached.get("beta")!]).toEqual(["unrelated-provider"]); + expect(deps.inferenceRouteObserver.observeInferenceRoute).not.toHaveBeenCalled(); +}); + +it.each([ + { endpointUrl: "http://host.openshell.internal:42104/team/v1" }, + { preferredInferenceApi: "openai-responses" }, +])("rejects changed recorded Hermes authority %j before provider writes", async (change) => { + const endpointUrl = "http://host.openshell.internal:42103/team/v1"; + const definition = hostedNativeProvider("hermes-provider", endpointUrl)!; + const { deps, calls, entry } = fixture([definition]); + entry.provider = "hermes-provider"; + entry.endpointUrl = endpointUrl; + entry.credentialEnv = "OPENAI_API_KEY"; + entry.preferredInferenceApi = "openai-completions"; + await runInferenceSet({ provider: "hermes-provider", model: "first-model" }, deps); + Object.assign(entry, change); + calls.createProvider.mockClear(); + calls.attachProvider.mockClear(); + await expect( + runInferenceSet({ provider: "hermes-provider", model: "second-model" }, deps), + ).rejects.toThrow("Recorded Hermes selection does not match its endpoint or API authority"); + expect(calls.createProvider).not.toHaveBeenCalled(); + expect(calls.attachProvider).not.toHaveBeenCalled(); +}); + describe("custom NVIDIA to native NVIDIA transition", () => { const custom = customNativeProvider({ provider: "compatible-endpoint", diff --git a/src/lib/actions/inference-set-nim-local.test.ts b/src/lib/actions/inference-set-nim-local.test.ts new file mode 100644 index 00000000000..8cdca8e73b5 --- /dev/null +++ b/src/lib/actions/inference-set-nim-local.test.ts @@ -0,0 +1,327 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { + createDeps, + baseSession, + nativeLocalTestReceipt, + HERMES_TARGET, + OPENCLAW_TARGET, +} from "../../../test/helpers/inference-set"; +import * as local from "../inference/local"; +import * as nim from "../inference/nim"; +import { fixedNativeProvider } from "../inference/native-provider/fixed"; +import type { NativeLocalProviderAttachment } from "../inference/native-local/contract"; +import type { SandboxEntry } from "../state/registry"; +import type { ConfigObject } from "../security/credential-filter"; +import { applyHermesManagedRoute } from "../hermes-managed-route"; +import { runInferenceSet } from "./inference-set"; +import { resolveLocalSwitchRuntime } from "./inference-set/local-runtime"; +import { prepareNativeLocalSwitch } from "./inference/native-local"; + +vi.mock("../adapters/openshell/provider-policy", () => ({ + requireNativeProviderPolicy: vi.fn(async () => {}), +})); + +const MODEL = "local/nim-model"; +const hosted = fixedNativeProvider("nvidia-prod")!; +const selected = { provider: "vllm-local", model: MODEL, sandboxName: "alpha" }; + +function fixture( + agent: "openclaw" | "hermes", + managedState: "conflicting" | "unreadable" = "conflicting", +) { + const receipt = nativeLocalTestReceipt("vllm-local", nim.getNimInferenceEndpoints().baseUrl, { + agent, + authMode: "sentinel", + }); + const entry: SandboxEntry = { + name: "alpha", + agent, + gatewayName: "nemoclaw", + provider: "vllm-local", + model: MODEL, + endpointUrl: receipt.endpointUrl, + preferredInferenceApi: "openai-completions", + credentialEnv: null, + nimContainer: "nemoclaw-nim-alpha", + nativeLocalProviderAttachment: receipt, + }; + const config: ConfigObject = + agent === "openclaw" + ? { + agents: { defaults: { model: { primary: `inference/${MODEL}` } } }, + models: { + providers: { + inference: { + baseUrl: receipt.endpointUrl, + apiKey: "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + api: "openai-completions", + models: [{ id: MODEL }], + }, + }, + }, + } + : {}; + const configure = { + openclaw: () => {}, + hermes: () => + applyHermesManagedRoute(config, { + upstreamProvider: "vllm-local", + baseUrl: receipt.endpointUrl, + model: MODEL, + inferenceApi: "openai-completions", + nativeCredentialHandle: "openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }), + }; + configure[agent](); + const session = baseSession({ ...entry, agent, credentialEnv: null }); + const deps = createDeps({ + config, + entry, + session: structuredClone(session), + target: agent === "openclaw" ? OPENCLAW_TARGET : HERMES_TARGET, + updateSandbox: (_name, updates) => { + Object.assign(entry, updates); + return true; + }, + }); + const managed = vi.spyOn(local, "getManagedVllmProviderBinding"); + const arrange = { + conflicting: () => + managed.mockReturnValue({ + baseUrl: "http://host.openshell.internal:18000/v1", + apiKey: "unrelated-managed-secret", + }), + unreadable: () => + managed.mockImplementation(() => { + throw new Error("unreadable unrelated managed state"); + }), + }; + arrange[managedState](); + const servedModel = vi.spyOn(nim, "getServedModelId").mockReturnValue(MODEL); + vi.spyOn(nim, "getNimContextWindow").mockReturnValue(65536); + vi.stubEnv("NGC_API_KEY", "host-ngc-secret"); + deps.calls.validateLocalProvider.mockImplementation(() => { + throw new Error("generic vLLM validation must not inspect NIM"); + }); + deps.calls.ensureLocalProviderReachable.mockImplementation(() => { + throw new Error("generic vLLM health must not inspect NIM"); + }); + const authorities = new Map([ + [receipt.providerName, receipt], + ]); + deps.getNativeLocalProviderAuthority = (name) => authorities.get(name); + deps.setNativeLocalProviderAuthority = (value) => { + authorities.set(value.providerName, value); + }; + deps.clearNativeLocalProviderAuthority = (value) => { + authorities.delete(value.providerName); + }; + deps.resolveLocalSwitchRuntime = resolveLocalSwitchRuntime; + deps.prepareNativeLocalSwitch = prepareNativeLocalSwitch; + const inspect = deps.providerAdapter.inspectProviderProfile; + deps.providerAdapter.inspectProviderProfile = async (request) => + request.profileType === hosted.profileId + ? { ok: true, value: { credentialKeys: [hosted.credentialEnv] } } + : inspect(request); + vi.spyOn(deps.providerAdapter, "importProviderProfile").mockResolvedValue({ ok: true }); + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + const update = vi.spyOn(deps.providerAdapter, "updateProvider"); + return { deps, entry, receipt, session, config, managed, servedModel, create, update }; +} + +describe.each(["openclaw", "hermes"] as const)("%s NIM inference set", (agent) => { + it.each(["nim", "nim-local", "nim_local"])( + "selects the recorded native NIM route through alias %s", + async (provider) => { + const f = fixture(agent, "unreadable"); + await expect(runInferenceSet({ ...selected, provider }, f.deps)).resolves.toMatchObject({ + provider: "vllm-local", + }); + expect(f.entry.nativeLocalProviderAttachment).toEqual(f.receipt); + expect(f.deps.calls.probeSandboxRoute).toHaveBeenCalledWith( + expect.objectContaining({ nativeEndpointUrl: f.receipt.endpointUrl }), + ); + expect(f.managed).not.toHaveBeenCalled(); + expect( + f.deps.calls.captureOpenshell.mock.calls.filter(([args]) => args[0] === "inference"), + ).toEqual([]); + }, + ); + + it.each(["nvidia-nim", "nvidia_nim"])( + "selects native NVIDIA Endpoints through legacy hosted alias %s", + async (provider) => { + const f = fixture(agent); + await expect( + runInferenceSet({ ...selected, provider, model: "hosted-model" }, f.deps), + ).resolves.toMatchObject({ provider: "nvidia-prod" }); + expect(f.entry).toMatchObject({ + provider: "nvidia-prod", + nativeNvidiaProviderAttachment: { providerName: hosted.providerName }, + }); + expect(f.deps.calls.probeSandboxRoute).toHaveBeenCalledWith( + expect.objectContaining({ nativeProvider: true, provider: "nvidia-prod" }), + ); + expect( + f.deps.calls.captureOpenshell.mock.calls.filter(([args]) => args[0] === "inference"), + ).toEqual([]); + }, + ); + + it.each(["nim", "nim-local", "nim_local"])( + "refuses local alias %s without its NIM record before shared route access or mutation", + async (provider) => { + const f = fixture(agent); + f.entry.nimContainer = null; + f.deps.loadSession = () => null; + await expect(runInferenceSet({ ...selected, provider }, f.deps)).rejects.toThrow( + "Local NVIDIA NIM requires its recorded container and endpoint", + ); + expect(f.create).not.toHaveBeenCalled(); + expect(f.update).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(f.deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + expect( + f.deps.calls.captureOpenshell.mock.calls.filter(([args]) => args[0] === "inference"), + ).toEqual([]); + expect(f.managed).not.toHaveBeenCalled(); + }, + ); + + it.each(["conflicting", "unreadable"] as const)( + "reuses NIM with %s managed vLLM state", + async (managedState) => { + const f = fixture(agent, managedState); + await expect(runInferenceSet(selected, f.deps)).resolves.toMatchObject({ + provider: "vllm-local", + }); + expect(f.entry).toMatchObject({ + nimContainer: "nemoclaw-nim-alpha", + endpointUrl: f.receipt.endpointUrl, + nativeLocalProviderAttachment: f.receipt, + }); + expect(f.servedModel).toHaveBeenCalledOnce(); + expect(f.managed).not.toHaveBeenCalled(); + expect(f.deps.calls.resolveContextWindowForModel).not.toHaveBeenCalledWith( + "vllm-local", + MODEL, + ); + expect(f.deps.calls.log).toHaveBeenCalledWith( + ` Context window for '${MODEL}': 65536 tokens`, + ); + expect(f.deps.calls.validateLocalProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.ensureLocalProviderReachable).not.toHaveBeenCalled(); + expect(f.create).not.toHaveBeenCalled(); + expect(f.deps.calls.probeSandboxRoute).toHaveBeenCalledWith( + expect.objectContaining({ nativeEndpointUrl: f.receipt.endpointUrl }), + ); + }, + ); + + it.each([ + { managedState: "conflicting", provider: "vllm-local" }, + { managedState: "unreadable", provider: "vllm-local" }, + { managedState: "conflicting", provider: "nim" }, + { managedState: "unreadable", provider: "nim-local" }, + { managedState: "unreadable", provider: "nim_local" }, + ] as const)( + "restores NIM through $provider from its matching session with $managedState vLLM state", + async ({ managedState, provider }) => { + const f = fixture(agent, managedState); + await runInferenceSet( + { provider: "nvidia-prod", model: "hosted-model", sandboxName: "alpha" }, + f.deps, + ); + expect(f.entry.nativeLocalProviderAttachment).toBeUndefined(); + expect(f.entry.nimContainer).toBeNull(); + // Restore the matching onboarding record, the existing authority for a provider switch back. + f.deps.loadSession = () => f.session; + await expect(runInferenceSet({ ...selected, provider }, f.deps)).resolves.toMatchObject({ + provider: "vllm-local", + }); + expect(f.entry).toMatchObject({ + nimContainer: "nemoclaw-nim-alpha", + endpointUrl: f.receipt.endpointUrl, + nativeLocalProviderAttachment: expect.objectContaining({ + authMode: "sentinel", + endpointUrl: f.receipt.endpointUrl, + }), + }); + const nativeWrites = [...f.create.mock.calls, ...f.update.mock.calls] + .map(([request]) => request) + .filter( + (request) => + ("name" in request ? request.name : request.providerName) === f.receipt.providerName, + ); + expect(nativeWrites).toContainEqual( + expect.objectContaining({ + credentials: [{ name: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", value: "dummy" }], + }), + ); + expect(f.managed).not.toHaveBeenCalled(); + expect(f.deps.calls.resolveContextWindowForModel).not.toHaveBeenCalledWith( + "vllm-local", + MODEL, + ); + expect(f.deps.calls.log).toHaveBeenCalledWith( + ` Context window for '${MODEL}': 65536 tokens`, + ); + expect(f.deps.calls.validateLocalProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.ensureLocalProviderReachable).not.toHaveBeenCalled(); + expect( + f.deps.calls.captureOpenshell.mock.calls.some( + ([args]) => args[0] === "inference" && args[1] === "set", + ), + ).toBe(false); + }, + ); + + it("refuses a local NIM alias when the retained session belongs to a different model", async () => { + const f = fixture(agent); + await runInferenceSet({ ...selected, provider: "nvidia-prod", model: "hosted-model" }, f.deps); + f.deps.loadSession = () => ({ ...f.session, model: "other-nim-model" }); + vi.clearAllMocks(); + await expect(runInferenceSet({ ...selected, provider: "nim-local" }, f.deps)).rejects.toThrow( + "Local NVIDIA NIM requires its recorded container and endpoint", + ); + expect(f.create).not.toHaveBeenCalled(); + expect(f.update).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(f.deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + expect(f.deps.calls.probeSandboxRoute).not.toHaveBeenCalled(); + expect( + f.deps.calls.captureOpenshell.mock.calls.filter(([args]) => args[0] === "inference"), + ).toEqual([]); + }); + + it("rejects a different recorded NIM endpoint before probing or changing state", async () => { + const f = fixture(agent); + f.entry.endpointUrl = "http://host.openshell.internal:18000/v1"; + await expect(runInferenceSet(selected, f.deps)).rejects.toThrow( + "recorded NIM route does not match", + ); + expect(f.servedModel).not.toHaveBeenCalled(); + expect(f.managed).not.toHaveBeenCalled(); + expect(f.create).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it.each([null, "different-model"])( + "rejects NIM serving %s before changing provider or config", + async (served) => { + const f = fixture(agent); + f.servedModel.mockReturnValue(served); + await expect(runInferenceSet(selected, f.deps)).rejects.toThrow( + "NIM does not serve the selected model", + ); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(f.deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + expect(f.deps.calls.setOpenClawConfigValues).not.toHaveBeenCalled(); + expect(f.create).not.toHaveBeenCalled(); + expect(f.managed).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/src/lib/actions/inference-set-openclaw-native-local.test.ts b/src/lib/actions/inference-set-openclaw-native-local.test.ts index 74b4b30158a..9b35ea2568a 100644 --- a/src/lib/actions/inference-set-openclaw-native-local.test.ts +++ b/src/lib/actions/inference-set-openclaw-native-local.test.ts @@ -7,43 +7,54 @@ import { fixture, switchingFixture, } from "../../../test/helpers/inference-set-openclaw-native-local"; +import { createDeps, nativeLocalTestReceipt } from "../../../test/helpers/inference-set"; import { resolveOpenClawLocalSwitch } from "./inference-set/native-local"; -import { runInferenceSet } from "./inference-set"; +import { patchOpenClawInferenceConfig, runInferenceSet } from "./inference-set"; import { nativeLocalIdentity } from "../inference/native-local/contract"; -import { createDeps } from "../../../test/helpers/inference-set"; +import * as routerBlueprint from "../core/model-router-port"; +import * as routerProcess from "../onboard/model-router-process"; +import * as onboardSession from "../state/onboard-session"; +import type { ConfigObject } from "../security/credential-filter"; +import { hashCredential } from "../security/credential-hash"; vi.mock("../adapters/openshell/provider-policy", () => ({ requireNativeProviderPolicy: vi.fn(async () => {}), })); -describe("OpenClaw native local live selection", () => { - it.each([ - { provider: "vllm-local", expected: {} }, - { provider: "ollama-local", expected: { prepare: true } }, - ])( - "keeps owned NIM shared only when selecting $provider still uses NIM", - async ({ provider, expected }) => { - const deps = fixture(); - await expect( - resolveOpenClawLocalSwitch({ - agentName: "openclaw", - provider, - sandboxName: "alpha", - gatewayName: "nemoclaw", - deps, - entry: { - name: "alpha", - agent: "openclaw", - provider: "vllm-local", - nimContainer: "owned-nim", - }, - }), - ).resolves.toEqual(expected); - }, +it.each([ + ["http://host.openshell.internal:11434/v1", "NEMOCLAW_LOCAL_INFERENCE_TOKEN"], + ["http://host.docker.internal:11434/v1", "NEMOCLAW_LOCAL_INFERENCE_TOKEN"], + ["https://fixture.example/v1", "COMPATIBLE_API_KEY"], + ["http://10.1.1.225:8123/v1", "COMPATIBLE_API_KEY"], +])("restores compatible inference at %s with its selected credential family", (endpoint, key) => { + const config: ConfigObject = { + models: { providers: { inference: { apiKey: "openshell:resolve:env:v1_OTHER_API_KEY" } } }, + }; + patchOpenClawInferenceConfig( + config, + "compatible-endpoint", + "changed-model", + "openai-completions", + undefined, + "compatible-endpoint", + { effort: null, explicit: false }, + false, + endpoint, + true, ); + expect(config).toMatchObject({ + agents: { defaults: { model: { primary: "inference/changed-model" } } }, + models: { + providers: { + inference: { baseUrl: endpoint, apiKey: `openshell:resolve:env:${key}` }, + }, + }, + }); +}); +describe("OpenClaw native local live selection", () => { it.each(["local", "hosted", "legacy router"])( - "rejects a router switch from %s without a native router receipt before mutation", + "rejects a router switch from %s without verified ownership before mutation", async (previous) => { const deps = previous === "local" @@ -57,6 +68,20 @@ describe("OpenClaw native local live selection", () => { model: "old-model", }, }); + vi.spyOn(routerProcess, "doesModelRouterProcessOwnPort").mockReturnValue(false); + vi.spyOn(onboardSession, "loadSession").mockReturnValue({ + routerPort: 4000, + routerPid: 123, + routerCredentialHash: hashCredential("test-router-key"), + } as ReturnType); + deps.getNativeLocalProviderAuthority ??= () => undefined; + deps.setNativeLocalProviderAuthority ??= vi.fn(); + deps.resolveLocalSwitchRuntime = () => ({ + endpointUrl: "http://host.openshell.internal:4000/v1", + credentialValue: "test-router-key", + authMode: "authenticated", + ownedProxy: false, + }); const create = vi.spyOn(deps.providerAdapter, "createProvider"); const update = vi.spyOn(deps.providerAdapter, "updateProvider"); const attach = vi.spyOn(deps.providerAdapter, "attachProvider"); @@ -75,8 +100,12 @@ describe("OpenClaw native local live selection", () => { deps, ), ).rejects.toMatchObject({ - exitCode: 2, - message: expect.stringMatching(/Model Router.*native.*onboard/s), + ...(previous === "legacy router" ? { exitCode: 2 } : {}), + message: expect.stringContaining( + previous === "legacy router" + ? "Recreate this legacy Model Router sandbox" + : "ownership could not be verified", + ), }); expect(create).not.toHaveBeenCalled(); @@ -94,13 +123,194 @@ describe("OpenClaw native local live selection", () => { }, ); - it("detaches and retires local authority before publishing a shared selection", async () => { + it.each([ + { provider: "vllm-local", expected: {} }, + { provider: "ollama-local", expected: { prepare: true } }, + ])( + "preserves legacy NIM until recreation when selecting $provider", + async ({ provider, expected }) => { + const deps = fixture(); + deps.resolveCredentialValue = () => "test-hosted-secret"; + await expect( + resolveOpenClawLocalSwitch({ + agentName: "openclaw", + provider, + sandboxName: "alpha", + gatewayName: "nemoclaw", + deps, + entry: { + name: "alpha", + agent: "openclaw", + provider: "vllm-local", + nimContainer: "owned-nim", + }, + }), + ).resolves.toEqual(expected); + }, + ); + + it("changes a configured router model with only its native instance registered", async () => { + vi.spyOn(routerBlueprint, "loadBlueprintProfile").mockReturnValue({ + provider_name: "team-router", + model: "nvidia-routed", + endpoint: "http://localhost:4000/v1", + credential_env: "NVIDIA_INFERENCE_API_KEY", + router: { enabled: true, port: 4000 }, + }); + const credential = "test-router-key"; + vi.spyOn(routerProcess, "doesModelRouterProcessOwnPort").mockReturnValue(true); + vi.spyOn(onboardSession, "loadSession").mockReturnValue({ + routerPort: 4000, + routerPid: 123, + routerCredentialHash: hashCredential(credential), + } as ReturnType); + const binding = { + agent: "openclaw" as const, + provider: "team-router", + endpointUrl: "http://host.openshell.internal:4000/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated" as const, + gatewayName: "nemoclaw", + sandboxName: "alpha", + }; + const receipt = { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1 as const, + providerId: "owned-router", + }; + const deps = createDeps({ + config: { + agents: { defaults: { model: { primary: "inference/old-model" } } }, + models: { + providers: { + inference: { + baseUrl: binding.endpointUrl, + apiKey: "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + api: "openai-completions", + models: [{ id: "old-model" }], + }, + }, + }, + }, + entry: { + name: "alpha", + agent: "openclaw", + gatewayName: "nemoclaw", + provider: "team-router", + model: "old-model", + endpointUrl: binding.endpointUrl, + preferredInferenceApi: "openai-completions", + nativeLocalProviderAttachment: receipt, + }, + resolveCredentialValue: () => credential, + }); + deps.resolveLocalSwitchRuntime = () => ({ + endpointUrl: binding.endpointUrl, + credentialValue: credential, + authMode: "authenticated", + ownedProxy: false, + }); + const list = vi.spyOn(deps.providerAdapter, "listProviders").mockResolvedValue({ + ok: true, + value: { names: [receipt.providerName] }, + }); + const result = await runInferenceSet( + { provider: "team-router", model: "new-router-model", sandboxName: "alpha" }, + deps, + ); + expect(result.provider).toBe("team-router"); + expect(JSON.stringify(deps.calls.setOpenClawConfigValues.mock.calls)).toContain( + "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + ); + expect(deps.calls.probeSandboxRoute).toHaveBeenCalledWith( + expect.objectContaining({ + nativeProvider: true, + nativeLocalProviderAttachment: receipt, + agentName: "openclaw", + model: "new-router-model", + }), + ); + expect(deps.calls.restartSandboxGateway).toHaveBeenCalled(); + expect( + deps.calls.captureOpenshell.mock.calls.some( + ([args]) => Array.isArray(args) && args[0] === "inference" && args[1] === "set", + ), + ).toBe(false); + + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + provider: "team-router", + model: "new-router-model", + nativeLocalProviderAttachment: receipt, + }), + ); + expect(list).not.toHaveBeenCalled(); + await expect( + runInferenceSet( + { provider: "unconfigured-router", model: "new-router-model", sandboxName: "alpha" }, + deps, + ), + ).rejects.toThrow("Unsupported provider"); + }); + + it("rejects Model Router endpoint overrides before touching provider state", async () => { const deps = fixture(); + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + await expect( + runInferenceSet( + { + provider: "nvidia-router", + model: "nvidia-routed", + sandboxName: "alpha", + endpointUrl: "http://host.openshell.internal:8080/v1", + credentialEnv: "OTHER_KEY", + inferenceApi: "openai-completions", + }, + deps, + ), + ).rejects.toThrow("Model Router uses its configured endpoint"); + expect(create).not.toHaveBeenCalled(); + expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it("reuses the verified native NIM provider without preparing managed vLLM", async () => { + const receipt = nativeLocalTestReceipt("vllm-local", "http://host.openshell.internal:8000/v1", { + agent: "openclaw", + authMode: "sentinel", + }); + const entry = { + name: "alpha", + agent: "openclaw", + gatewayName: "nemoclaw", + provider: "vllm-local", + model: "served-nim-model", + nimContainer: "owned-nim", + endpointUrl: receipt.endpointUrl, + nativeLocalProviderAttachment: receipt, + }; + const deps = createDeps({ config: {}, entry }); + await expect( + resolveOpenClawLocalSwitch({ + agentName: "openclaw", + entry, + provider: "vllm-local", + sandboxName: "alpha", + gatewayName: "nemoclaw", + deps, + }), + ).resolves.toEqual({ selected: receipt, previous: receipt }); + }); + + it("detaches and retires local authority before publishing a hosted selection", async () => { + const deps = fixture(); + deps.resolveCredentialValue = () => "test-hosted-secret"; const detach = vi.spyOn(deps.providerAdapter, "detachProvider"); const deletion = vi.spyOn(deps.providerAdapter, "deleteProvider"); await runInferenceSet( { - provider: "nvidia-nim", + provider: "openai-api", model: "routed-model", sandboxName: "alpha", }, @@ -116,7 +326,7 @@ describe("OpenClaw native local live selection", () => { expect(deps.calls.updateSandbox).toHaveBeenCalledWith( "alpha", expect.objectContaining({ - provider: "nvidia-nim", + provider: "openai-api", nativeLocalProviderAttachment: undefined, }), ); @@ -125,14 +335,20 @@ describe("OpenClaw native local live selection", () => { it("preserves the switch failure when restoring local access also fails", async () => { const deps = fixture(); + deps.resolveCredentialValue = () => "test-hosted-secret"; deps.calls.updateSandbox.mockReturnValue(false); - vi.spyOn(deps.providerAdapter, "attachProvider").mockResolvedValue({ - ok: false, - error: { kind: "command", reason: "failed", message: "reattach rejected" }, - }); + const attachProvider = deps.providerAdapter.attachProvider.bind(deps.providerAdapter); + vi.spyOn(deps.providerAdapter, "attachProvider").mockImplementation((request) => + request.providerName === attachment.providerName + ? Promise.resolve({ + ok: false, + error: { kind: "command", reason: "failed", message: "reattach rejected" }, + }) + : attachProvider(request), + ); await expect( runInferenceSet( - { provider: "nvidia-nim", model: "routed-model", sandboxName: "alpha" }, + { provider: "openai-api", model: "routed-model", sandboxName: "alpha" }, deps, ), ).rejects.toThrow(/Failed to update NemoClaw registry.*restoring.*reattach rejected/s); @@ -141,7 +357,10 @@ describe("OpenClaw native local live selection", () => { it("reselects the same Ollama endpoint with the dual-stack profile (#13544)", async () => { const previous = { ...attachment, ...nativeLocalIdentity(attachment, 1) }; - const { deps, next, isAttached } = switchingFixture(previous, "ollama-local"); + const { deps, next, isAttached } = switchingFixture( + { provider: "ollama-local", endpointUrl: previous.endpointUrl }, + [previous, undefined, undefined, previous], + ); await expect( runInferenceSet({ provider: "ollama-local", model: "old-model", sandboxName: "alpha" }, deps), ).resolves.toMatchObject({ provider: "ollama-local" }); @@ -155,7 +374,10 @@ describe("OpenClaw native local live selection", () => { it("restores the IPv4 attachment when dual-stack selection cannot save the registry (#13544)", async () => { const previous = { ...attachment, ...nativeLocalIdentity(attachment, 1) }; - const { deps, isAttached } = switchingFixture(previous, "ollama-local"); + const { deps, isAttached } = switchingFixture( + { provider: "ollama-local", endpointUrl: previous.endpointUrl }, + [previous, undefined, undefined, previous], + ); deps.calls.updateSandbox.mockReturnValue(false); await expect( runInferenceSet({ provider: "ollama-local", model: "old-model", sandboxName: "alpha" }, deps), @@ -288,59 +510,45 @@ describe("OpenClaw native local live selection", () => { expect(deps.calls.clearNativeLocalProviderAuthority).not.toHaveBeenCalled(); }); - it.each(["ollama-local", "nvidia-router"] as const)( - "changes the %s model without touching the shared gateway route", - async (provider) => { - const selectedBinding = - provider === "nvidia-router" - ? { - ...attachment, - provider, - authMode: "authenticated" as const, - endpointUrl: "http://host.openshell.internal:4000/v1", - } - : attachment; - const selected = { ...selectedBinding, ...nativeLocalIdentity(selectedBinding) }; - const deps = fixture(selected); - const result = await runInferenceSet( - { - provider, - model: "new-model", - sandboxName: "alpha", - }, - deps, - ); - expect(result.provider).toBe(provider); - expect(deps.calls.probeSandboxRoute).toHaveBeenCalledWith( - expect.objectContaining({ - nativeProvider: true, - nativeLocalProviderAttachment: selected, - agentName: "openclaw", - model: "new-model", - }), - ); - expect(deps.calls.updateSandbox).toHaveBeenCalledWith( - "alpha", - expect.objectContaining({ - model: "new-model", - nativeLocalProviderAttachment: selected, - }), - ); - expect(deps.calls.setOpenClawConfigValues).toHaveBeenCalled(); - expect(JSON.stringify(deps.calls.setOpenClawConfigValues.mock.calls)).toContain( - "NEMOCLAW_LOCAL_INFERENCE_TOKEN", - ); - expect(deps.calls.restartSandboxGateway).toHaveBeenCalled(); - expect( - deps.calls.captureOpenshell.mock.calls.some( - ([args]) => Array.isArray(args) && args[0] === "inference" && args[1] === "set", - ), - ).toBe(false); - }, - ); + it("changes the selected model without touching the shared gateway route", async () => { + const deps = fixture(); + deps.resolveCredentialValue = () => "test-hosted-secret"; + const result = await runInferenceSet( + { + provider: "ollama-local", + model: "new-model", + sandboxName: "alpha", + }, + deps, + ); + expect(result.provider).toBe("ollama-local"); + expect(deps.calls.probeSandboxRoute).toHaveBeenCalledWith( + expect.objectContaining({ + nativeProvider: true, + nativeLocalProviderAttachment: attachment, + agentName: "openclaw", + model: "new-model", + }), + ); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + model: "new-model", + nativeLocalProviderAttachment: attachment, + }), + ); + expect(deps.calls.setOpenClawConfigValues).toHaveBeenCalled(); + expect(deps.calls.restartSandboxGateway).toHaveBeenCalled(); + expect( + deps.calls.captureOpenshell.mock.calls.some( + ([args]) => Array.isArray(args) && args[0] === "inference" && args[1] === "set", + ), + ).toBe(false); + }); it("retries exact provider retirement after a committed route switch", async () => { const deps = fixture(); + deps.resolveCredentialValue = () => "test-hosted-secret"; let pending = true; const originalDelete = deps.providerAdapter.deleteProvider.bind(deps.providerAdapter); vi.spyOn(deps.providerAdapter, "deleteProvider").mockImplementation(async (request) => @@ -351,7 +559,7 @@ describe("OpenClaw native local live selection", () => { deps.listNativeLocalProviderAuthorities = () => [attachment]; await expect( runInferenceSet( - { provider: "nvidia-nim", model: "routed-model", sandboxName: "alpha" }, + { provider: "openai-api", model: "routed-model", sandboxName: "alpha" }, deps, ), ).rejects.toThrow(/retry inference set/); @@ -362,15 +570,17 @@ describe("OpenClaw native local live selection", () => { expect(deps.calls.clearNativeLocalProviderAuthority).not.toHaveBeenCalled(); pending = false; + const committed = deps.calls.updateSandbox.mock.calls.at(-1)?.[1]; deps.getSandbox = () => ({ + ...committed, name: "alpha", agent: "openclaw", gatewayName: "nemoclaw", - provider: "nvidia-nim", + provider: "openai-api", model: "routed-model", }); await runInferenceSet( - { provider: "nvidia-nim", model: "routed-model", sandboxName: "alpha" }, + { provider: "openai-api", model: "routed-model", sandboxName: "alpha" }, deps, ); expect(deps.calls.clearNativeLocalProviderAuthority).toHaveBeenCalledWith(attachment); @@ -390,7 +600,7 @@ describe("OpenClaw native local live selection", () => { await expect( runInferenceSet( - { provider: "nvidia-nim", model: "routed-model", sandboxName: "alpha" }, + { provider: "nvidia-router", model: "routed-model", sandboxName: "alpha" }, deps, ), ).rejects.toThrow(/conflicting gateway identity/); @@ -400,3 +610,285 @@ describe("OpenClaw native local live selection", () => { expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); }); }); + +it.each([ + { api: "openai-completions" as const, port: 8000 }, + { api: "openai-responses" as const, port: 8000 }, + { api: "openai-completions" as const, port: 42103 }, + { api: "openai-responses" as const, port: 42103 }, +])( + "switches OpenClaw to a compatible native $api endpoint on port $port without shared route commands", + async ({ api, port }) => { + const { deps } = switchingFixture({ + provider: "compatible-endpoint", + authMode: "authenticated", + endpointUrl: `http://host.openshell.internal:${port}/v1`, + ...(api === "openai-responses" ? { api } : {}), + }); + deps.calls.resolveCredentialValue.mockReturnValue("fixture-only-host-key"); + await expect( + runInferenceSet( + { + provider: "compatible-endpoint", + model: "served-alias", + sandboxName: "alpha", + endpointUrl: `http://host.openshell.internal:${port}/v1`, + credentialEnv: "COMPATIBLE_API_KEY", + inferenceApi: api, + }, + deps, + ), + ).resolves.toMatchObject({ provider: "compatible-endpoint" }); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + preferredInferenceApi: api, + nativeLocalProviderAttachment: expect.objectContaining({ + provider: "compatible-endpoint", + agent: "openclaw", + endpointUrl: `http://host.openshell.internal:${port}/v1`, + }), + }), + ); + expect(deps.calls.resolveCredentialValue).toHaveBeenCalledWith("COMPATIBLE_API_KEY"); + expect( + deps.calls.captureOpenshell.mock.calls.some( + ([args]) => Array.isArray(args) && args[0] === "inference", + ), + ).toBe(false); + }, +); + +it("restores the prior local attachment when a compatible Responses switch cannot publish its registry", async () => { + const { deps, isPresent, isAttached } = switchingFixture({ + provider: "compatible-endpoint", + authMode: "authenticated", + api: "openai-responses", + }); + deps.calls.resolveCredentialValue.mockReturnValue("fixture-only-host-key"); + deps.calls.updateSandbox.mockReturnValue(false); + await expect( + runInferenceSet( + { + provider: "compatible-endpoint", + model: "served-alias", + sandboxName: "alpha", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + inferenceApi: "openai-responses", + }, + deps, + ), + ).rejects.toThrow(/Failed to update NemoClaw registry/); + expect({ present: isPresent(), attached: isAttached() }).toEqual({ + present: false, + attached: false, + }); + expect( + await deps.providerAdapter.listProviderAttachments({ + target: { kind: "named", gatewayName: "nemoclaw" }, + sandboxName: "alpha", + }), + ).toMatchObject({ ok: true, value: { names: [attachment.providerName] } }); +}); + +it.each([ + { api: "openai-completions" as const, explicit: false }, + { api: "openai-responses" as const, explicit: true }, +])( + "reuses an owned no-auth compatible $api proxy when explicit endpoint is $explicit", + async ({ api, explicit }) => { + const deps = fixture( + { + provider: "compatible-endpoint", + authMode: "authenticated", + ...(api === "openai-responses" ? { api } : {}), + }, + "NEMOCLAW_OLLAMA_PROXY_TOKEN", + "http://127.0.0.1:8000/v1", + ); + const prior = deps.getSandbox("alpha")!.nativeLocalProviderAttachment; + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + await runInferenceSet( + { + provider: "compatible-endpoint", + model: "new-model", + sandboxName: "alpha", + ...(explicit ? { endpointUrl: "http://127.0.0.1:8000/v1/" } : {}), + }, + deps, + ); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + nativeLocalProviderAttachment: prior, + preferredInferenceApi: api, + credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN", + }), + ); + expect(deps.calls.probeSandboxRoute).toHaveBeenCalledWith( + expect.objectContaining({ + nativeEndpointUrl: "http://host.openshell.internal:11435/v1", + nativeProvider: true, + }), + ); + expect(create).not.toHaveBeenCalled(); + expect( + deps.calls.captureOpenshell.mock.calls.some( + ([args]) => Array.isArray(args) && args[0] === "inference", + ), + ).toBe(false); + }, +); + +it("changes a no-auth compatible API through the existing owned proxy", async () => { + const { deps, next } = switchingFixture( + { + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:11435/v1", + authMode: "authenticated", + api: "openai-responses", + }, + [ + { provider: "compatible-endpoint", authMode: "authenticated" }, + "NEMOCLAW_OLLAMA_PROXY_TOKEN", + "http://127.0.0.1:8000/v1", + ], + ); + deps.calls.resolveCredentialValue.mockReturnValue("fixture-only-proxy-token"); + await runInferenceSet( + { + provider: "compatible-endpoint", + model: "new-model", + sandboxName: "alpha", + endpointUrl: "http://127.0.0.1:8000/v1", + inferenceApi: "openai-responses", + }, + deps, + ); + expect(deps.calls.resolveCredentialValue).toHaveBeenCalledWith("NEMOCLAW_OLLAMA_PROXY_TOKEN"); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ + nativeLocalProviderAttachment: next, + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN", + preferredInferenceApi: "openai-responses", + }), + ); +}); + +it("rejects a recorded compatible API mismatch before changing sandbox state", async () => { + const deps = fixture( + { + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8000/v1", + authMode: "authenticated", + api: "openai-responses", + }, + "COMPATIBLE_API_KEY", + ); + const entry = deps.getSandbox("alpha")!; + deps.getSandbox = () => ({ ...entry, preferredInferenceApi: "openai-completions" }); + await expect( + runInferenceSet( + { provider: "compatible-endpoint", model: "new-model", sandboxName: "alpha" }, + deps, + ), + ).rejects.toThrow(/API or endpoint does not match/); + expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(deps.calls.setOpenClawConfigValues).not.toHaveBeenCalled(); +}); + +it.each(["http://127.0.0.1:11434/v1", "http://127.0.0.1:8000/other", "http://localhost:8000/v1"])( + "rejects a different no-auth backend %s before native mutation", + async (endpointUrl) => { + const deps = fixture( + { provider: "compatible-endpoint", authMode: "authenticated" }, + "NEMOCLAW_OLLAMA_PROXY_TOKEN", + "http://127.0.0.1:8000/v1", + ); + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + const detach = vi.spyOn(deps.providerAdapter, "detachProvider"); + await expect( + runInferenceSet( + { + provider: "compatible-endpoint", + model: "new-model", + sandboxName: "alpha", + endpointUrl, + inferenceApi: "openai-responses", + }, + deps, + ), + ).rejects.toThrow(/no-auth proxy is bound to the recorded backend/); + expect(create).not.toHaveBeenCalled(); + expect(detach).not.toHaveBeenCalled(); + expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(deps.calls.setOpenClawConfigValues).not.toHaveBeenCalled(); + expect(deps.calls.probeSandboxRoute).not.toHaveBeenCalled(); + }, +); + +it.each([8080, 4000])( + "rejects a compatible switch to reserved control port %s before mutation", + async (port) => { + const { deps } = switchingFixture({ + provider: "compatible-endpoint", + authMode: "authenticated", + endpointUrl: `http://host.openshell.internal:${port}/v1`, + }); + const create = vi.spyOn(deps.providerAdapter, "createProvider"); + const detach = vi.spyOn(deps.providerAdapter, "detachProvider"); + await expect( + runInferenceSet( + { + provider: "compatible-endpoint", + model: "served-alias", + sandboxName: "alpha", + endpointUrl: `http://host.openshell.internal:${port}/v1`, + credentialEnv: "COMPATIBLE_API_KEY", + inferenceApi: "openai-completions", + }, + deps, + ), + ).rejects.toThrow(/protected host port/); + expect(create).not.toHaveBeenCalled(); + expect(detach).not.toHaveBeenCalled(); + expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); + }, +); + +it.each([false, true])( + "migrates a retained compatible profile with owned proxy=%s", + async (ownedProxy) => { + const selected = { + ...attachment, + provider: "compatible-endpoint" as const, + endpointUrl: ownedProxy + ? "http://host.openshell.internal:11435/v1" + : "http://host.openshell.internal:42103/v1", + authMode: "authenticated" as const, + api: "openai-responses" as const, + }; + const previous = { ...selected, ...nativeLocalIdentity(selected, 1) }; + const credentialEnv = ownedProxy ? "NEMOCLAW_OLLAMA_PROXY_TOKEN" : "COMPATIBLE_API_KEY"; + const endpointUrl = ownedProxy ? "http://127.0.0.1:8000/v1" : selected.endpointUrl; + const { deps, next } = switchingFixture(selected, [ + selected, + credentialEnv, + endpointUrl, + previous, + ]); + deps.calls.resolveCredentialValue.mockReturnValue("fixture-only-secret"); + await runInferenceSet( + { provider: selected.provider, model: "old-model", sandboxName: "alpha" }, + deps, + ); + expect(deps.calls.resolveCredentialValue).toHaveBeenCalledWith(credentialEnv); + expect(deps.calls.updateSandbox).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ nativeLocalProviderAttachment: next, endpointUrl }), + ); + }, +); diff --git a/src/lib/actions/inference-set-openclaw-pairing.test.ts b/src/lib/actions/inference-set-openclaw-pairing.test.ts index e0a04c573d0..83fd08ad7da 100644 --- a/src/lib/actions/inference-set-openclaw-pairing.test.ts +++ b/src/lib/actions/inference-set-openclaw-pairing.test.ts @@ -11,6 +11,8 @@ import { settleInferenceSetOpenClawPairing, } from "./inference-set-gateway-restart"; +import { OpenClawPairingQualificationError } from "./sandbox/launch-readiness/openclaw-pairing-qualification"; + const TARGET: InferenceSetOpenClawPairingTarget = { sandboxName: "alpha", gatewayName: "nemoclaw-8080", @@ -107,6 +109,7 @@ describe("settleInferenceSetOpenClawPairing", () => { expect(result).toEqual({ ok: false, failureLayer: "initial-state-unavailable", + failureCategory: "unclassified", }); expect(JSON.stringify(result)).not.toContain("do-not-report"); expect(deps.publishScopeRequest).not.toHaveBeenCalled(); @@ -167,6 +170,7 @@ describe("settleInferenceSetOpenClawPairing", () => { expect(settleInferenceSetOpenClawPairing(TARGET, deps)).toEqual({ ok: false, failureLayer: "final-state-unavailable", + failureCategory: "unclassified", }); expect(deps.approveScopeRequest).toHaveBeenCalledOnce(); }); @@ -219,3 +223,50 @@ describe("settleInferenceSetOpenClawPairing", () => { expect(JSON.stringify(appendAuditEntry.mock.calls)).not.toContain("credential="); }); }); + +it("retains the initial observation category without another approval", () => { + const failure = new OpenClawPairingQualificationError("deadline-exceeded"); + failure.message = "credential=must-not-escape"; + const observePairing = vi.fn(() => { + throw failure; + }); + const deps = pairingDeps({ observePairing }); + expect(settleInferenceSetOpenClawPairing(TARGET, deps)).toEqual({ + ok: false, + failureLayer: "initial-state-unavailable", + failureCategory: "deadline-exceeded", + }); + expect(deps.approveScopeRequest).toHaveBeenCalledTimes(0); +}); + +it("retains the final observation category without another approval", () => { + const failure = new OpenClawPairingQualificationError("deadline-exceeded"); + failure.message = "credential=must-not-escape"; + const observePairing = vi.fn(() => { + throw failure; + }); + observePairing.mockReturnValueOnce(observation("pairing-only")); + const deps = pairingDeps({ observePairing }); + expect(settleInferenceSetOpenClawPairing(TARGET, deps)).toEqual({ + ok: false, + failureLayer: "final-state-unavailable", + failureCategory: "deadline-exceeded", + }); + expect(deps.approveScopeRequest).toHaveBeenCalledTimes(1); +}); + +it("rejects arbitrary categories even on typed observer errors", () => { + const failure = Object.assign(new OpenClawPairingQualificationError(), { + failureCategory: "credential=must-not-escape", + }); + const deps = pairingDeps({ + observePairing: () => { + throw failure; + }, + }); + expect(settleInferenceSetOpenClawPairing(TARGET, deps)).toEqual({ + ok: false, + failureLayer: "initial-state-unavailable", + failureCategory: "unclassified", + }); +}); diff --git a/src/lib/actions/inference-set-openclaw-run.test.ts b/src/lib/actions/inference-set-openclaw-run.test.ts index 60f463c40e0..1912fe10981 100644 --- a/src/lib/actions/inference-set-openclaw-run.test.ts +++ b/src/lib/actions/inference-set-openclaw-run.test.ts @@ -370,31 +370,6 @@ describe("runInferenceSet OpenClaw routing", () => { }); it("detaches native NVIDIA access only after another provider is healthy", async () => { - let attached = true; - const detachProvider = vi.fn(async () => { - attached = false; - return { ok: true, value: { changed: true } }; - }); - const providerAdapter = { - getProvider: vi.fn(async () => ({ - ok: true, - value: { - name: "nemoclaw-nvidia-prod-v1", - type: "nemoclaw-nvidia-inference-v1", - credentialKeys: ["NVIDIA_INFERENCE_API_KEY"], - configKeys: [], - revision: { - id: "11111111-2222-4333-8444-555555555555", - resourceVersion: 1, - }, - }, - })), - detachProvider, - listProviderAttachments: vi.fn(async () => ({ - ok: true, - value: { names: attached ? ["nemoclaw-nvidia-prod-v1"] : [] }, - })), - } as unknown as OpenShellProviderAdapter; const deps = createDeps({ config: { agents: { defaults: { model: { primary: "inference/nvidia/old-model" } } }, @@ -412,10 +387,11 @@ describe("runInferenceSet OpenClaw routing", () => { providerId: "11111111-2222-4333-8444-555555555555", }, }, - providerAdapter, }); - await runInferenceSet({ provider: "nvidia-nim", model: "gpt-5.4", noVerify: true }, deps); + const detachProvider = vi.spyOn(deps.providerAdapter, "detachProvider"); + deps.resolveCredentialValue = () => "test-hosted-secret"; + await runInferenceSet({ provider: "openai-api", model: "gpt-5.4", noVerify: true }, deps); expect(detachProvider).toHaveBeenCalledExactlyOnceWith( expect.objectContaining({ @@ -473,10 +449,11 @@ describe("runInferenceSet OpenClaw routing", () => { resolveCredentialValue: () => "", }); - await runInferenceSet({ provider: "nvidia-nim", model: "gpt-5.4", noVerify: true }, deps); + deps.resolveCredentialValue = () => "test-hosted-secret"; + await runInferenceSet({ provider: "openai-api", model: "gpt-5.4", noVerify: true }, deps); expect(entry).toMatchObject({ - provider: "nvidia-nim", + provider: "openai-api", }); expect(entry.nativeNvidiaProviderAttachment).toBeUndefined(); expect(entry).not.toHaveProperty("nativeNvidiaProviderAuthority"); @@ -543,26 +520,7 @@ describe("runInferenceSet OpenClaw routing", () => { expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); }); - it("does not publish a non-native route when native NVIDIA detach fails", async () => { - const providerAdapter = { - getProvider: vi.fn(async () => ({ - ok: true, - value: { - name: "nemoclaw-nvidia-prod-v1", - type: "nemoclaw-nvidia-inference-v1", - credentialKeys: ["NVIDIA_INFERENCE_API_KEY"], - configKeys: [], - revision: { - id: "11111111-2222-4333-8444-555555555555", - resourceVersion: 1, - }, - }, - })), - detachProvider: vi.fn(async () => ({ - ok: false, - error: { kind: "command", reason: "failed", message: "detach denied" }, - })), - } as unknown as OpenShellProviderAdapter; + it("does not publish another native route when native NVIDIA detach fails", async () => { const config: ConfigObject = { agents: { defaults: { model: { primary: "inference/nvidia/old-model" } } }, models: { providers: {} }, @@ -581,11 +539,20 @@ describe("runInferenceSet OpenClaw routing", () => { providerId: "11111111-2222-4333-8444-555555555555", }, }, - providerAdapter, }); + const detachProvider = deps.providerAdapter.detachProvider.bind(deps.providerAdapter); + vi.spyOn(deps.providerAdapter, "detachProvider").mockImplementation((request) => + request.providerName === "nemoclaw-nvidia-prod-v1" + ? Promise.resolve({ + ok: false, + error: { kind: "command", reason: "failed", message: "detach denied" }, + }) + : detachProvider(request), + ); + deps.resolveCredentialValue = () => "test-hosted-secret"; await expect( - runInferenceSet({ provider: "nvidia-nim", model: "gpt-5.4", noVerify: true }, deps), + runInferenceSet({ provider: "openai-api", model: "gpt-5.4", noVerify: true }, deps), ).rejects.toThrow("Could not detach native NVIDIA provider from sandbox 'alpha'"); expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); @@ -785,7 +752,6 @@ describe("runInferenceSet OpenClaw routing", () => { "nemoclaw", ); expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); - expect(deps.calls.recomputeSandboxConfigHash).not.toHaveBeenCalled(); expect(deps.calls.updateSandbox).toHaveBeenCalledWith( "alpha", expect.objectContaining({ diff --git a/src/lib/actions/inference-set-openrouter.test.ts b/src/lib/actions/inference-set-openrouter.test.ts index 5504f0800ff..1d38ff1926f 100644 --- a/src/lib/actions/inference-set-openrouter.test.ts +++ b/src/lib/actions/inference-set-openrouter.test.ts @@ -156,7 +156,7 @@ describe("OpenRouter model switch verification", () => { it.each([ { model: SUPER, delays: [[6_000]] }, - { model: ULTRA, delays: [] }, + { model: ULTRA, delays: [[6_000]] }, ])( "verifies $model through the sandbox before committing (#12628)", async ({ model, delays }) => { diff --git a/src/lib/actions/inference-set-provider-alias.test.ts b/src/lib/actions/inference-set-provider-alias.test.ts index 48d32874563..1af8daa5e49 100644 --- a/src/lib/actions/inference-set-provider-alias.test.ts +++ b/src/lib/actions/inference-set-provider-alias.test.ts @@ -12,6 +12,7 @@ // restores saved native preferences. import { describe, expect, it, vi } from "vitest"; +import { registeredInferenceFixture } from "../../../test/helpers/registered-inference"; import { shellQuote } from "../core/shell-quote"; // onboard's provider config is the source of truth the local alias map must // stay in sync with. Imported here (test only — not into the inference-set hot @@ -42,7 +43,6 @@ function expectNoInferenceMutation(calls: ReturnType["calls"] expect(calls.updateSandbox).not.toHaveBeenCalled(); expect(calls.writeSandboxConfig).not.toHaveBeenCalled(); expect(calls.updateSession).not.toHaveBeenCalled(); - expect(calls.recomputeSandboxConfigHash).not.toHaveBeenCalled(); expect(calls.restartSandboxGateway).not.toHaveBeenCalled(); } @@ -92,7 +92,7 @@ describe("normalizeInferenceSetProvider — facet 1 provider-name drift (#6321)" it("normalizes underscore spellings of installer alias keys", () => { expect(normalizeInferenceSetProvider("open_router")).toBe("openrouter-api"); - expect(normalizeInferenceSetProvider("nim_local")).toBe("nvidia-nim"); + expect(normalizeInferenceSetProvider("nim_local")).toBe("vllm-local"); expect(normalizeInferenceSetProvider("llama_cpp")).toBe("llama-cpp-local"); expect(normalizeInferenceSetProvider("nous_portal")).toBe("hermes-provider"); }); @@ -199,93 +199,104 @@ describe("runInferenceSet accepts the installer provider name — facet 1 (#6321 expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); expect(deps.calls.updateSession).not.toHaveBeenCalled(); - expect(deps.calls.recomputeSandboxConfigHash).not.toHaveBeenCalled(); expect(deps.calls.restartSandboxGateway).not.toHaveBeenCalled(); }); - it("accepts an additional registered provider without replacing its native config", async () => { - const provider = "native-extra"; - const nativeProviderConfig = { - api: "openai-completions", - apiKey: "native-owned-reference", - baseUrl: "https://native.example/v1", - models: [{ id: "vendor/model-a", name: "vendor/model-a" }], - }; - const output = `nvidia-prod\n${provider}\n`; - const captureOpenshell = vi.fn(() => ({ status: 0, output, stdout: output, stderr: "" })); - const config: ConfigObject = { - agents: { defaults: { model: { primary: "inference/nvidia/model-a" } } }, - models: { - providers: { - inference: { api: "openai-completions", models: [] }, - [provider]: nativeProviderConfig, + it.each([false, true])( + "selects an operator provider through its native configuration (already attached: %s)", + async (attached) => { + const f = registeredInferenceFixture(attached); + const provider = "native-extra"; + const config: ConfigObject = { + agents: { defaults: { model: { primary: "inference/nvidia/model-a" } } }, + models: { + providers: { + inference: { api: "openai-completions", models: [] }, + [provider]: { + api: "openai-completions", + apiKey: "${VENDOR_API_KEY}", + baseUrl: f.binding.endpointUrl, + headers: { "X-Operator-Header": "retained" }, + models: [{ id: "vendor/model-a", name: "vendor/model-a" }], + }, + }, }, - }, - }; - const deps = createDeps({ - config, - entry: { - name: "alpha", - agent: "openclaw", - gatewayName: "nemoclaw-18080", - provider: "nvidia-prod", - model: "nvidia/model-a", - }, - captureOpenshell, - }); - - await expect( - runInferenceSet({ provider, model: "vendor/model-b", noVerify: true }, deps), - ).resolves.toMatchObject({ provider, model: "vendor/model-b" }); - - expect(captureOpenshell).toHaveBeenNthCalledWith( - 1, - ["provider", "list", "-g", "nemoclaw-18080", "--names"], - expect.objectContaining({ ignoreError: true, timeout: 5_000 }), - ); - expect(captureOpenshell).toHaveBeenNthCalledWith( - 2, - [ - "inference", - "set", - "-g", - "nemoclaw-18080", - "--no-verify", - "--provider", - provider, - "--model", - "vendor/model-b", - ], - expect.objectContaining({ ignoreError: true }), - ); - expect(captureOpenshell).toHaveBeenCalledTimes(2); - expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); - expect(deps.calls.setOpenClawConfigValues).toHaveBeenCalledOnce(); - expect(deps.calls.setOpenClawConfigValues).toHaveBeenCalledWith( - "alpha", - expect.arrayContaining([ + }; + const deps = createDeps({ + config, + providerAdapter: f.adapter, + entry: { + name: "alpha", + agent: "openclaw", + gatewayName: f.binding.gatewayName, + lifecycleGeneration: "test-generation", + lifecycleLiveIdentityFingerprint: "a".repeat(64), + provider: "nvidia-prod", + model: "nvidia/model-a", + }, + }); + deps.observeSandboxIdentity = vi.fn(() => ({ + state: "ready" as const, + liveIdentityFingerprint: "a".repeat(64), + })); + deps.readRegisteredInferenceCredentialHandle = vi.fn(async () => f.handle); + await expect( + runInferenceSet({ provider, model: "vendor/model-b", noVerify: true }, deps), + ).resolves.toMatchObject({ provider, model: "vendor/model-b", providerKey: provider }); + expect(f.adapter.listProviders).toHaveBeenCalledExactlyOnceWith( + expect.objectContaining({ target: { kind: "named", gatewayName: f.binding.gatewayName } }), + ); + expect(f.adapter.attachProvider).toHaveBeenCalledTimes(attached ? 0 : 1); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(deps.calls.captureOpenshell).not.toHaveBeenCalled(); + expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + expect(deps.calls.setOpenClawConfigValues).toHaveBeenCalledExactlyOnceWith( + "alpha", + expect.arrayContaining([ + { dotpath: "agents.defaults.model.primary", value: "native-extra/vendor/model-b" }, + expect.objectContaining({ + dotpath: "models.providers.native-extra", + value: expect.objectContaining({ + baseUrl: f.binding.endpointUrl, + api: "openai-completions", + apiKey: f.handle, + headers: expect.objectContaining({ "X-Operator-Header": "retained" }), + models: expect.arrayContaining([ + expect.objectContaining({ id: "vendor/model-b" }), + expect.objectContaining({ id: "vendor/model-a" }), + ]), + }), + }), + ]), + f.binding.gatewayName, + ); + expect(deps.calls.probeSandboxRoute).toHaveBeenCalledWith( expect.objectContaining({ - dotpath: "models.providers.inference", - value: expect.objectContaining({ - models: expect.arrayContaining([expect.objectContaining({ id: "vendor/model-b" })]), + registeredInferenceProviderAttachment: expect.objectContaining({ + providerId: f.binding.providerId, }), }), - ]), - "nemoclaw-18080", - ); - expect( - deps.calls.updateSandbox.mock.calls - .filter(([, fields]) => fields.provider !== undefined) - .at(-1), - ).toEqual([ - "alpha", - expect.objectContaining({ - provider, - endpointUrl: null, - credentialEnv: null, - }), - ]); - }); + ); + expect( + deps.calls.updateSandbox.mock.calls + .filter(([, fields]) => fields.provider !== undefined) + .at(-1), + ).toEqual([ + "alpha", + expect.objectContaining({ + provider, + endpointUrl: f.binding.endpointUrl, + credentialEnv: "VENDOR_API_KEY", + registeredInferenceProviderAttachment: expect.objectContaining({ + attachedBySelection: !attached, + }), + }), + ]); + expect(deps.calls.setNativeNvidiaProviderAuthority).not.toHaveBeenCalled(); + }, + ); it("hands OpenShell the exact `compatible-anthropic-endpoint` name, never the `anthropicCompatible` alias (#6321)", async () => { // The alias must be normalized on the host before any gateway call — the diff --git a/src/lib/actions/inference-set-provider.ts b/src/lib/actions/inference-set-provider.ts index e90336c28c5..d644f0a71de 100644 --- a/src/lib/actions/inference-set-provider.ts +++ b/src/lib/actions/inference-set-provider.ts @@ -111,7 +111,7 @@ export async function readHermesNativeLocalCredentialHandleIfNeeded( return handle; } -async function readNativeCredentialHandle( +export async function readNativeCredentialHandle( sandboxName: string, gatewayName: string, deps: { @@ -225,6 +225,7 @@ export async function probeInferenceSetSandboxRouteUntilConverged( input: SandboxInferenceInvocationInput; previousProvider: string; previousModel: string; + providerAttachmentChanged?: boolean; previousInferenceApi: string | null; targetInferenceApi: string | null; }, @@ -242,6 +243,7 @@ export async function probeInferenceSetSandboxRouteUntilConverged( }, ): Promise { const routeSelectionChanged = + options.providerAttachmentChanged === true || options.previousProvider !== options.input.provider || options.previousModel !== options.input.model; if (routeSelectionChanged) { diff --git a/src/lib/actions/inference-set-recorded-adapter.test.ts b/src/lib/actions/inference-set-recorded-adapter.test.ts index 812d7fa4b8b..544a3a49e76 100644 --- a/src/lib/actions/inference-set-recorded-adapter.test.ts +++ b/src/lib/actions/inference-set-recorded-adapter.test.ts @@ -10,6 +10,7 @@ import { createCompatibleProviderCapture, createDeps, HERMES_TARGET, + nativeLocalTestReceipt, } from "../../../test/helpers/inference-set"; const ADAPTER_URL = buildHttpsPinRouteBaseUrl("a".repeat(64)); @@ -75,6 +76,14 @@ function recordedRouteDeps( gatewayName: GATEWAY_NAME, gatewayPort: 18085, ...metadata, + ...(agent === "hermes" + ? { + nativeLocalProviderAttachment: nativeLocalTestReceipt(route.provider, endpointUrl, { + agent: "hermes", + gatewayName: GATEWAY_NAME, + }), + } + : {}), }, session: baseSession({ agent, @@ -117,6 +126,12 @@ describe("recorded compatible adapter model changes", () => { it("verifies a reused Hermes adapter before writing its model configuration (#13095)", async () => { const route = PROVIDERS[0]; const deps = recordedRouteDeps(route, ADAPTER_URL, "hermes"); + const attachment = deps.getSandbox("alpha")!.nativeLocalProviderAttachment!; + vi.spyOn(deps, "prepareNativeLocalSwitch").mockResolvedValue({ + receipt: attachment, + changed: false, + previousDetached: false, + }); await expect( runInferenceSet({ provider: route.provider, model: TARGET_MODEL }, deps), @@ -133,8 +148,10 @@ describe("recorded compatible adapter model changes", () => { provider: route.provider, model: TARGET_MODEL, preferredInferenceApi: route.inferenceApi, - nativeLocalProviderAttachment: undefined, - nativeEndpointUrl: undefined, + nativeProvider: true, + registeredInferenceProviderAttachment: undefined, + nativeLocalProviderAttachment: attachment, + nativeEndpointUrl: ADAPTER_URL, }); expect(deps.calls.probeSandboxRoute.mock.invocationCallOrder[0]).toBeLessThan( deps.calls.writeSandboxConfig.mock.invocationCallOrder[0], @@ -142,7 +159,14 @@ describe("recorded compatible adapter model changes", () => { expect(deps.calls.writeSandboxConfig).toHaveBeenCalledExactlyOnceWith( "alpha", HERMES_TARGET, - expect.objectContaining({ model: expect.objectContaining({ default: TARGET_MODEL }) }), + expect.objectContaining({ + model: expect.objectContaining({ + default: TARGET_MODEL, + base_url: ADAPTER_URL, + api_key: "openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }), + }), + { deferReload: true, gatewayName: GATEWAY_NAME }, ); expect(deps.calls.updateSandbox).toHaveBeenCalledWith( "alpha", diff --git a/src/lib/actions/inference-set-registered.test.ts b/src/lib/actions/inference-set-registered.test.ts new file mode 100644 index 00000000000..bbdd2b3ad41 --- /dev/null +++ b/src/lib/actions/inference-set-registered.test.ts @@ -0,0 +1,549 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { registeredInferenceFixture } from "../../../test/helpers/registered-inference"; +import { createDeps, HERMES_TARGET, OPENCLAW_TARGET } from "../../../test/helpers/inference-set"; +import { inspectInferenceProviderProfile } from "../adapters/openshell/provider-profile"; +import type { ConfigObject } from "../security/credential-filter"; +import type { SandboxEntry } from "../state/registry"; +import { runInferenceSet, patchOpenClawInferenceConfig } from "./inference-set"; + +function fixture(agent = "openclaw", attached = false) { + const f = registeredInferenceFixture(attached); + const config: ConfigObject = + agent === "hermes" + ? { model: { default: "old-model", provider: "custom" } } + : { + agents: { defaults: { model: { primary: "inference/old-model" } } }, + models: { providers: {} }, + }; + const entry: SandboxEntry = { + name: "alpha", + lifecycleGeneration: "test-generation", + lifecycleLiveIdentityFingerprint: "a".repeat(64), + agent, + gatewayName: f.binding.gatewayName, + provider: "nvidia-prod", + model: "old-model", + }; + const deps = createDeps({ + config, + entry, + providerAdapter: f.adapter, + target: agent === "hermes" ? HERMES_TARGET : OPENCLAW_TARGET, + updateSandbox: (_name, fields) => { + Object.assign(entry, fields); + return true; + }, + }); + const observation = { state: "ready" as const, liveIdentityFingerprint: "a".repeat(64) }; + Object.assign(deps, { observeSandboxIdentity: vi.fn(() => ({ ...observation })) }); + deps.readRegisteredInferenceCredentialHandle = vi.fn(async () => f.handle); + const options = { + provider: "native-extra", + model: "vendor/model-b", + endpointUrl: f.binding.endpointUrl, + credentialEnv: f.binding.credentialEnv, + inferenceApi: f.binding.inferenceApi, + }; + return { ...f, deps, options, config, entry, observation }; +} + +function collisionFixture() { + const f = fixture("openclaw", true); + f.profile.credentials[0].env_vars = ["NVIDIA_API_KEY"]; + const binding = { + ...f.binding, + credentialEnv: "NVIDIA_API_KEY", + profileDigest: inspectInferenceProviderProfile(JSON.stringify(f.profile))!.digest, + attachedBySelection: true, + }; + Object.assign(f.entry, { + provider: binding.providerName, + endpointUrl: binding.endpointUrl, + preferredInferenceApi: binding.inferenceApi, + credentialEnv: binding.credentialEnv, + registeredInferenceProviderAttachment: binding, + }); + f.adapter.getProvider.mockImplementation(async () => ({ + ok: true, + value: { + name: binding.providerName, + type: binding.profileId, + credentialKeys: [binding.credentialEnv], + configKeys: [], + revision: { id: binding.providerId, resourceVersion: f.state.revision }, + }, + })); + f.adapter.inspectProviderProfile.mockImplementation(async () => ({ + ok: true, + value: { + credentialKeys: [binding.credentialEnv], + inference: inspectInferenceProviderProfile(JSON.stringify(f.profile))!, + }, + })); + const native = createDeps({ entry: f.entry, config: f.config }); + const adapter = native.providerAdapter; + const events: string[] = []; + f.deps.providerAdapter = { + ...adapter, + listProviders: f.adapter.listProviders, + getProvider: async (request) => + request.providerName === binding.providerName + ? f.adapter.getProvider() + : adapter.getProvider(request), + inspectProviderProfile: async (request) => + request.profileType === binding.profileId + ? f.adapter.inspectProviderProfile() + : adapter.inspectProviderProfile(request), + listProviderAttachments: async (request) => { + const owned = await adapter.listProviderAttachments(request); + return owned.ok + ? { + ok: true, + value: { + names: [...owned.value.names, ...(f.state.attached ? [binding.providerName] : [])], + }, + } + : owned; + }, + attachProvider: async (request) => { + events.push(`attach:${request.providerName}`); + const current = await f.deps.providerAdapter.listProviderAttachments(request); + return current.ok && current.value.names.length > 0 + ? { + ok: false, + error: { + kind: "command", + reason: "conflict", + message: "credential environment collision", + }, + } + : request.providerName === binding.providerName + ? f.adapter.attachProvider() + : adapter.attachProvider(request); + }, + detachProvider: async (request) => { + events.push(`detach:${request.providerName}`); + return request.providerName === binding.providerName + ? f.adapter.detachProvider() + : adapter.detachProvider(request); + }, + }; + f.deps.calls.probeSandboxRoute.mockResolvedValue({ + ok: false, + httpStatus: 401, + detail: "rejected", + }); + return { ...f, binding, events, ownedAdapter: adapter }; +} + +describe("registered inference selection transaction", () => { + it.each(["openclaw", "hermes"])( + "refuses registered attachment when %s is replaced during provider inspection", + async (agent) => { + const f = fixture(agent); + f.adapter.listProviderAttachments.mockImplementation(async () => { + f.observation.liveIdentityFingerprint = "b".repeat(64); + return { ok: true, value: { names: [] } }; + }); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow(/live identity.*changed/); + expect(f.adapter.ensureProviderPolicyComposition).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.readRegisteredInferenceCredentialHandle).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }, + ); + + it("does not detach a replacement sandbox after a registered route probe fails", async () => { + const f = fixture(); + f.deps.calls.probeSandboxRoute.mockImplementation(async () => { + f.observation.liveIdentityFingerprint = "b".repeat(64); + return { ok: false, httpStatus: 401, detail: "rejected" }; + }); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow(/live identity.*changed/); + expect(f.adapter.attachProvider).toHaveBeenCalledOnce(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it("does not roll back an unverified attachment after sandbox replacement", async () => { + const f = fixture(); + f.adapter.attachProvider.mockImplementation(async () => { + f.state.attached = true; + f.state.revision++; + f.observation.liveIdentityFingerprint = "b".repeat(64); + return { ok: true }; + }); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow(/live identity.*changed/); + expect(f.adapter.attachProvider).toHaveBeenCalledOnce(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it("does not publish a selection after replacement during credential retrieval", async () => { + const f = fixture(); + f.deps.readRegisteredInferenceCredentialHandle = vi.fn(async () => { + f.observation.liveIdentityFingerprint = "b".repeat(64); + return f.handle; + }); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow(/live identity.*changed/); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(f.deps.calls.setOpenClawConfigValues).not.toHaveBeenCalled(); + expect(f.deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); + }); + + it("does not detach the previous registered provider after sandbox replacement", async () => { + const f = collisionFixture(); + const list = f.deps.providerAdapter.listProviderAttachments; + f.deps.providerAdapter.listProviderAttachments = async (request) => { + const result = await list(request); + f.observation.liveIdentityFingerprint = "b".repeat(64); + return result; + }; + await expect( + runInferenceSet({ provider: "nvidia-prod", model: "vendor/model-b" }, f.deps), + ).rejects.toThrow(/live identity.*changed/); + expect(f.events).toEqual([]); + expect(f.state.attached).toBe(true); + }); + + it("does not restore a registered attachment after replacement during owned-provider cleanup", async () => { + const f = collisionFixture(); + const detach = f.ownedAdapter.detachProvider; + f.ownedAdapter.detachProvider = async (request) => { + const result = await detach(request); + f.observation.liveIdentityFingerprint = "b".repeat(64); + return result; + }; + await expect( + runInferenceSet({ provider: "nvidia-prod", model: "vendor/model-b" }, f.deps), + ).rejects.toThrow(/live identity.*changed/); + expect(f.events).toEqual([ + "detach:native-extra", + "attach:nemoclaw-nvidia-prod-v1", + "detach:nemoclaw-nvidia-prod-v1", + ]); + expect(f.state.attached).toBe(false); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it("does not detach owned access after replacement during a switch to registered access", async () => { + const f = collisionFixture(); + f.deps.calls.probeSandboxRoute.mockResolvedValueOnce({ + ok: true, + httpStatus: 200, + detail: "ok", + }); + await runInferenceSet({ provider: "nvidia-prod", model: "vendor/model-b" }, f.deps); + const committed = structuredClone(f.entry); + f.events.length = 0; + vi.clearAllMocks(); + const getProvider = f.ownedAdapter.getProvider; + f.ownedAdapter.getProvider = async (request) => { + const result = await getProvider(request); + f.observation.liveIdentityFingerprint = "b".repeat(64); + return result; + }; + await expect( + runInferenceSet({ ...f.options, credentialEnv: "NVIDIA_API_KEY" }, f.deps), + ).rejects.toThrow(/live identity.*changed/); + expect(f.events).toEqual([]); + expect(f.entry).toEqual(committed); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(f.deps.readRegisteredInferenceCredentialHandle).not.toHaveBeenCalled(); + }); + + it("refuses registered selection without recorded lifecycle identity", async () => { + const f = fixture(); + delete f.entry.lifecycleLiveIdentityFingerprint; + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow(/lifecycle identity/); + expect(f.adapter.ensureProviderPolicyComposition).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it.each([ + { + inventory: "missing", + provider: "native-extra", + list: async () => ({ ok: true as const, value: { names: [] } }), + }, + { + inventory: "messaging", + provider: "alpha-telegram-bridge", + list: async () => ({ ok: true as const, value: { names: ["alpha-telegram-bridge"] } }), + }, + { + inventory: "unavailable", + provider: "native-extra", + list: async () => { + throw new Error("gateway unavailable"); + }, + }, + ])( + "rejects a $inventory provider inventory before reading credentials or changing state", + async ({ provider, list }) => { + const f = fixture(); + f.adapter.listProviders.mockImplementation(list); + await expect(runInferenceSet({ ...f.options, provider }, f.deps)).rejects.toThrow( + `Unsupported provider '${provider}'`, + ); + expect(f.adapter.listProviders).toHaveBeenCalledExactlyOnceWith({ + target: { kind: "named", gatewayName: f.binding.gatewayName }, + timeoutMs: 5_000, + }); + expect(f.deps.readRegisteredInferenceCredentialHandle).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }, + ); + + it("preserves the recorded route and configuration when the operator provider was replaced", async () => { + const f = fixture(); + await runInferenceSet(f.options, f.deps); + const before = structuredClone({ entry: f.entry, config: f.config }); + f.state.id = "replacement-provider"; + vi.clearAllMocks(); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow( + "Recorded registered inference provider identity or profile changed", + ); + expect({ entry: f.entry, config: f.config }).toEqual(before); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(f.deps.calls.setOpenClawConfigValues).not.toHaveBeenCalled(); + expect(f.deps.calls.probeSandboxRoute).not.toHaveBeenCalled(); + expect(f.deps.calls.restartSandboxGateway).not.toHaveBeenCalled(); + }); + + it.each(["openclaw", "hermes"])( + "attaches an explicit operator route for %s without shared inference mutation", + async (agent) => { + const f = fixture(agent); + await expect(runInferenceSet(f.options, f.deps)).resolves.toMatchObject({ + provider: "native-extra", + providerKey: "native-extra", + }); + expect(f.entry.registeredInferenceProviderAttachment).toMatchObject({ + providerId: f.binding.providerId, + attachedBySelection: true, + }); + expect(f.deps.calls.captureOpenshell).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).toHaveBeenCalledOnce(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.restartSandboxGateway).toHaveBeenCalled(); + }, + ); + + it("removes a failed owned selection before restoring a registered attachment with the same credential environment", async () => { + const f = collisionFixture(); + await expect( + runInferenceSet({ provider: "nvidia-prod", model: "vendor/model-b" }, f.deps), + ).rejects.toThrow("rejected"); + expect(f.events).toEqual([ + "detach:native-extra", + "attach:nemoclaw-nvidia-prod-v1", + "detach:nemoclaw-nvidia-prod-v1", + "attach:native-extra", + ]); + expect(f.state.attached).toBe(true); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it("does not restore the registered attachment after uncertain owned cleanup", async () => { + const f = collisionFixture(); + const detach = f.deps.providerAdapter.detachProvider; + f.deps.providerAdapter.detachProvider = async (request) => + request.providerName === f.binding.providerName + ? detach(request) + : { + ok: false, + error: { kind: "command", reason: "uncertain", message: "detach response lost" }, + }; + await expect( + runInferenceSet({ provider: "nvidia-prod", model: "vendor/model-b" }, f.deps), + ).rejects.toThrow("did not confirm removal of native NVIDIA access"); + expect(f.events).toEqual(["detach:native-extra", "attach:nemoclaw-nvidia-prod-v1"]); + expect(f.state.attached).toBe(false); + expect(f.entry.registeredInferenceProviderAttachment).toEqual(f.binding); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it("does not restore an owned provider when removing a new registered attachment is uncertain", async () => { + const f = collisionFixture(); + f.deps.calls.probeSandboxRoute.mockResolvedValueOnce({ + ok: true, + httpStatus: 200, + detail: "ok", + }); + await runInferenceSet({ provider: "nvidia-prod", model: "vendor/model-b" }, f.deps); + const committed = f.entry.nativeNvidiaProviderAttachment; + f.events.length = 0; + f.deps.readRegisteredInferenceCredentialHandle = async () => + f.handle.replace("VENDOR_API_KEY", "NVIDIA_API_KEY"); + const detach = f.deps.providerAdapter.detachProvider; + f.deps.providerAdapter.detachProvider = async (request) => + request.providerName === f.binding.providerName + ? { + ok: false, + error: { kind: "command", reason: "uncertain", message: "detach response lost" }, + } + : detach(request); + await expect( + runInferenceSet({ ...f.options, credentialEnv: "NVIDIA_API_KEY" }, f.deps), + ).rejects.toThrow("attachment removal was not confirmed"); + expect(f.events).toEqual(["detach:nemoclaw-nvidia-prod-v1", "attach:native-extra"]); + expect(f.state.attached).toBe(true); + expect(f.entry.nativeNvidiaProviderAttachment).toEqual(committed); + expect(f.entry.registeredInferenceProviderAttachment).toBeUndefined(); + }); + + it("writes the selected Hermes native endpoint and scoped credential", async () => { + const f = fixture("hermes"); + await runInferenceSet(f.options, f.deps); + expect(f.config.model).toMatchObject({ + base_url: f.binding.endpointUrl, + api_key: f.handle, + default: f.options.model, + }); + }); + + it.each([ + ["endpoint", { endpointUrl: undefined }], + ["API", { inferenceApi: undefined }], + ["credential", { credentialEnv: undefined }], + ])("refuses incomplete %s metadata before attachment", async (_label, change) => { + const f = fixture(); + await expect(runInferenceSet({ ...f.options, ...change }, f.deps)).rejects.toThrow( + "requires explicit", + ); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it.each([false, true])( + "preserves external provider and prior attachment after failed verification (already attached: %s)", + async (attached) => { + const f = fixture("openclaw", attached); + f.deps.calls.probeSandboxRoute.mockResolvedValue({ + ok: false, + httpStatus: 401, + detail: "rejected", + }); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow(); + expect(f.state.attached).toBe(attached); + expect(f.adapter.detachProvider).toHaveBeenCalledTimes(attached ? 0 : 1); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + expect(f.deps.calls.setOpenClawConfigValues).not.toHaveBeenCalled(); + }, + ); + + it("rejects provider changes during handle reading before publication", async () => { + const f = fixture(); + f.deps.readRegisteredInferenceCredentialHandle = async () => { + f.state.revision++; + return f.handle; + }; + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow("changed while reading"); + expect(f.state.attached).toBe(false); + expect(f.deps.calls.updateSandbox).not.toHaveBeenCalled(); + }); + + it("removes its new attachment when registry publication fails", async () => { + const f = fixture(); + f.deps.calls.updateSandbox.mockReturnValue(false); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow("Failed to update"); + expect(f.state.attached).toBe(false); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + }); + + it("retries a committed Hermes selection after its native configuration write fails", async () => { + const f = fixture("hermes"); + const before = structuredClone(f.config); + f.deps.calls.readSandboxConfig + .mockReturnValueOnce(structuredClone(before)) + .mockReturnValueOnce(structuredClone(before)); + f.deps.calls.writeSandboxConfig.mockImplementationOnce(() => { + throw new Error("config write unavailable"); + }); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow( + "Retry the same inference set command", + ); + expect(f.entry.registeredInferenceProviderAttachment).toMatchObject({ + providerId: f.binding.providerId, + attachedBySelection: true, + }); + expect(f.deps.calls.log.mock.calls.flat().join("\n")).not.toContain("rebuild"); + await expect( + runInferenceSet({ provider: f.options.provider, model: f.options.model }, f.deps), + ).resolves.toMatchObject({ inSandboxConfigSynced: true }); + expect(f.adapter.attachProvider).toHaveBeenCalledOnce(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.calls.writeSandboxConfig).toHaveBeenCalledTimes(2); + expect(f.deps.calls.restartSandboxGateway).toHaveBeenCalledOnce(); + }); + + it("retries config publication from the immutable recorded binding without attaching again", async () => { + const f = fixture(); + f.deps.calls.setOpenClawConfigValues.mockImplementationOnce(() => { + throw new Error("config unavailable"); + }); + await expect(runInferenceSet(f.options, f.deps)).rejects.toThrow( + "synchronization did not complete", + ); + expect(f.state.attached).toBe(true); + expect(f.entry.openClawConfigSyncPending).toBe(true); + await expect( + runInferenceSet({ provider: f.options.provider, model: f.options.model }, f.deps), + ).resolves.toMatchObject({ inSandboxConfigSynced: true }); + expect(f.adapter.attachProvider).toHaveBeenCalledOnce(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.entry.openClawConfigSyncPending).toBeUndefined(); + }); +}); + +describe("registered OpenClaw native model configuration", () => { + it("preserves the selected operator model's compatibility, params, and limits", () => { + const f = fixture(); + const model = { + id: "vendor/model-b", + compat: { supportsStore: true }, + params: { extra_body: { reasoning_effort: "high", vendor_option: true } }, + contextWindow: 8192, + maxTokens: 2048, + }; + const config: ConfigObject = { models: { providers: { "native-extra": { models: [model] } } } }; + patchOpenClawInferenceConfig( + config, + "native-extra", + model.id, + f.binding.inferenceApi, + null, + undefined, + undefined, + false, + f.binding.endpointUrl, + true, + f.binding, + f.handle, + ); + expect(config.models).toMatchObject({ + providers: { + "native-extra": { + apiKey: f.handle, + baseUrl: f.binding.endpointUrl, + models: [{ ...model, name: "native-extra/vendor/model-b" }], + }, + }, + }); + }); +}); diff --git a/src/lib/actions/inference-set-route-containment.ts b/src/lib/actions/inference-set-route-containment.ts index 0fd0ec2b14f..5a2de76a634 100644 --- a/src/lib/actions/inference-set-route-containment.ts +++ b/src/lib/actions/inference-set-route-containment.ts @@ -238,12 +238,10 @@ export async function normalizeCustomEndpointUrl( const normalized = normalizeCustomEndpointUrlWithoutDns(value); const shaped = normalizeEndpointUrlShape(normalized); if (isAllowedOpenShellSandboxBridgeUrl(shaped.url)) { - // This is the single sandbox-to-host bridge name that NemoClaw itself - // provisions for local inference. Its supported routes are explicit - // unprivileged HTTP listeners; do not generalize this exemption to HTTPS, - // default/privileged ports, localhost, RFC1918 addresses, or arbitrary - // internal DNS names. - return normalized; + throw new InferenceSetError( + `${ENDPOINT_URL_NOT_ALLOWED_PREFIX} custom inference cannot use an unverified OpenShell host bridge endpoint`, + 2, + ); } // A DNS-backed HTTPS endpoint cannot be pinned by IP substitution alone: the diff --git a/src/lib/actions/inference-set-status-zero-ambiguity.test.ts b/src/lib/actions/inference-set-status-zero-ambiguity.test.ts index a5c05849124..657b6151e96 100644 --- a/src/lib/actions/inference-set-status-zero-ambiguity.test.ts +++ b/src/lib/actions/inference-set-status-zero-ambiguity.test.ts @@ -46,7 +46,7 @@ describe("runInferenceSet status-zero route ambiguity", () => { provider: "compatible-endpoint", model: "mock-model", noVerify: true, - endpointUrl: "http://host.openshell.internal:18767/v1", + endpointUrl: "http://192.168.50.22:18767/v1", credentialEnv: "COMPATIBLE_API_KEY", inferenceApi: "openai-completions", }, diff --git a/src/lib/actions/inference-set.ts b/src/lib/actions/inference-set.ts index 2300bfc9abe..021e331e014 100644 --- a/src/lib/actions/inference-set.ts +++ b/src/lib/actions/inference-set.ts @@ -1,6 +1,23 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + attachRegisteredInferenceProvider, + recordedRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "./inference-set/native"; +import { + resolveRegisteredSelection, + readRegisteredInferenceCredentialHandle, + readVerifiedRegisteredCredentialHandle, + detachPreviousRegisteredSelection, + rollbackRegisteredSelection, + restorePreviousRegisteredSelection, + captureRegisteredSelectionIdentity, + registeredSelectionProviderAdapter, + type SandboxRecreateObservation, +} from "./inference-set/native"; + import type { NativeLocalProviderAttachment } from "../inference/config"; import { requireNativeProviderPolicy, @@ -50,7 +67,6 @@ import { import { type AgentConfigTarget, readSandboxConfig, - recomputeSandboxConfigHash, resolveAgentConfig, rewriteConfigUrlsWithDnsPinning, retireNativeConfigurationTelemetry, @@ -69,6 +85,10 @@ import * as registry from "../state/registry"; import { enforceRemovedImmutabilityMigrationBoundary } from "../state/migrations/removed-immutability"; import { resolveRuntimeInferenceApi } from "./inference-route-api"; import { + isModelRouterProvider, + canonicalOwnedInferenceProviderName, + nativeModelRouterEndpoint, + withRoutedInferencePortLock, assertInferenceModelId, nativeSelectionConfigCredentials, prepareNativeTransitionAuthority, @@ -84,6 +104,7 @@ import { listNativeLocalProviderAuthorities, resolveOpenClawLocalSwitch, resolveLocalSwitchRuntime, + verifyNimSwitchSelection, prepareOpenClawLocalSwitch, rollbackOpenClawLocalSwitch, getNativeLocalProviderAuthority, @@ -215,6 +236,7 @@ function providerCommitFailureAfterSelection(options: { } export interface InferenceSetDeps extends InferenceGatewayRestartDeps { + observeSandboxIdentity?: (sandboxName: string, gatewayName: string) => SandboxRecreateObservation; sandboxPolicyReader?: Pick< import("../adapters/openshell/sandbox-policy").OpenShellSandboxPolicyReader, "readSandboxPolicy" @@ -246,14 +268,9 @@ export interface InferenceSetDeps extends InferenceGatewayRestartDeps { ) => onboardSession.Session; resolveAgentConfig: (sandboxName: string) => AgentConfigTarget; readSandboxConfig: typeof readSandboxConfig; - writeSandboxConfig: ( - sandboxName: string, - target: AgentConfigTarget, - config: ConfigObject, - ) => void; + writeSandboxConfig: typeof writeSandboxConfig; setOpenClawConfigValues: typeof setOpenClawConfigValues; runtimeProviders?: RuntimeProviderBundleRegistry; - recomputeSandboxConfigHash: (sandboxName: string, target: AgentConfigTarget) => void; prepareRunOpenshell: () => void; inferenceRouteMutator: ReturnType; inferenceRouteObserver: OpenShellInferenceRouteObserver; @@ -269,6 +286,7 @@ export interface InferenceSetDeps extends InferenceGatewayRestartDeps { rewriteConfigUrlsWithDnsPinning: (value: ConfigValue) => Promise; resolveCredentialValue: (credentialEnv: string) => string; readNativeNvidiaCredentialHandle: (sandboxName: string, gatewayName: string) => Promise; + readRegisteredInferenceCredentialHandle?: typeof readRegisteredInferenceCredentialHandle; ensureHttpsPinRuntimeAdapter: EnsureHttpsPinRuntimeAdapterFn; revokeHttpsPinRuntimeAdapterRoute: (routeId: string) => Promise; probeSandboxRoute: InferenceSetSandboxRouteProbe; @@ -278,7 +296,6 @@ export interface InferenceSetDeps extends InferenceGatewayRestartDeps { const SUPPORTED_PROVIDER_NAMES = [ "nvidia-prod", - "nvidia-nim", "nvidia-router", "openai-api", "openrouter-api", @@ -329,8 +346,9 @@ const INSTALLER_PROVIDER_ALIASES: Readonly> = { "llama-cpp": "llama-cpp-local", ollama: "ollama-local", vllm: "vllm-local", - nim: "nvidia-nim", - "nim-local": "nvidia-nim", + nim: "vllm-local", + "nim-local": "vllm-local", + "nvidia-nim": "nvidia-prod", routed: "nvidia-router", }; @@ -363,7 +381,7 @@ export function normalizeInferenceSetProvider(provider: string): string { if (hyphenatedAlias) return hyphenatedAlias; if (SUPPORTED_PROVIDER_NAME_SET.has(hyphenated)) return hyphenated; } - return trimmed; + return canonicalOwnedInferenceProviderName(trimmed); } /** Exposed for the alias-sync regression test. */ @@ -392,7 +410,6 @@ function defaultDeps(): InferenceSetDeps { readSandboxConfig, writeSandboxConfig, setOpenClawConfigValues, - recomputeSandboxConfigHash, prepareRunOpenshell: () => { getOpenshellBinary(); }, @@ -443,7 +460,7 @@ async function assertSelectableProvider( gatewayName: string, deps: Pick, ): Promise { - if (getProviderSelectionConfig(provider, model) || provider === "nvidia-router") return; + if (getProviderSelectionConfig(provider, model) || isModelRouterProvider(provider)) return; const registeredProviders = await queryRegisteredGatewayProviders(gatewayName, deps); if (registeredProviders?.includes(provider)) return; const providerGuidance = @@ -723,14 +740,17 @@ function buildProviderConfig( inheritedMaxTokens?: number, upstreamProviderMarker?: string, reasoningEffort: ReasoningEffortRequest = { effort: null, explicit: false }, + registered = false, ): ConfigObject { const existingModels = Array.isArray(existing.models) ? existing.models : []; const selectedIndex = existingModels.findIndex( (entry) => isConfigObject(entry) && entry.id === model, ); - const selectedModel = cloneConfigObject(existingModels[selectedIndex] ?? existingModels[0]); + const selectedModel = cloneConfigObject( + existingModels[selectedIndex] ?? (registered ? undefined : existingModels[0]), + ); const existingModelId = typeof selectedModel.id === "string" ? selectedModel.id : null; - delete selectedModel.compat; + if (!registered) delete selectedModel.compat; if (selectedIndex < 0) { // A different model's limits are not evidence for the selected model. Let // OpenClaw use its own defaults unless this route has an authoritative @@ -746,16 +766,16 @@ function buildProviderConfig( // unchanged while the prior config write still contains the old model. if (typeof contextWindow === "number") { selectedModel.contextWindow = contextWindow; - } else if (contextWindow === null || existingModelId !== model) { + } else if ((contextWindow === null && !registered) || existingModelId !== model) { delete selectedModel.contextWindow; } - if (route.inferenceApi === "anthropic-messages") { + if (!registered && route.inferenceApi === "anthropic-messages") { applyOpenClawAnthropicReplyBudget(selectedModel, inheritedMaxTokens); } if (route.inferenceCompat) { selectedModel.compat = asConfigObject(route.inferenceCompat); } - applyReasoningEffortParams(selectedModel, provider, route, reasoningEffort); + if (!registered) applyReasoningEffortParams(selectedModel, provider, route, reasoningEffort); const providerConfig: ConfigObject = { ...existing, @@ -788,6 +808,8 @@ export function patchOpenClawInferenceConfig( inheritPrimaryReplyBudget = true, nativeEndpointUrl?: string, nativeProvider = usesNativeConfigByDefault(provider, nativeEndpointUrl), + registered?: RegisteredInferenceProviderAttachment, + registeredHandle?: string, ): { changed: boolean; route: SandboxInferenceConfig } { const before = JSON.stringify(config); const route = getSandboxInferenceConfig( @@ -797,6 +819,7 @@ export function patchOpenClawInferenceConfig( nativeProvider, nativeEndpointUrl, "openclaw", + registered, ); const inheritedMaxTokens = inheritPrimaryReplyBudget ? readOpenClawPrimaryReplyBudget(config) @@ -817,8 +840,14 @@ export function patchOpenClawInferenceConfig( inheritedMaxTokens, upstreamProviderMarker, reasoningEffort, + Boolean(registered), ); + if (registered) { + const selected = cloneConfigObject(providers[route.providerKey]); + selected.apiKey = registeredHandle ?? existingProvider.apiKey; + providers[route.providerKey] = selected; + } return { changed: before !== JSON.stringify(config), route }; } @@ -853,7 +882,10 @@ export function writeOpenClawInferenceConfigNatively( if (selectedAgentUpdate) updates.push(selectedAgentUpdate); updates.push( { dotpath: "models.mode", value: "merge" }, - { dotpath: `models.providers.${route.providerKey}`, value: providerConfig }, + { + dotpath: appendOpenClawConfigPathSegment("models.providers", route.providerKey), + value: providerConfig, + }, ); writeValues(sandboxName, updates, gatewayName); } @@ -900,6 +932,7 @@ export function patchHermesInferenceConfig( nativeEndpointUrl?: string, nativeProvider = usesNativeConfigByDefault(provider, nativeEndpointUrl), nativeCredentialHandle?: string, + registered?: RegisteredInferenceProviderAttachment, ): { changed: boolean; route: SandboxInferenceConfig } { const before = JSON.stringify(config); const route = getSandboxInferenceConfig( @@ -909,6 +942,7 @@ export function patchHermesInferenceConfig( nativeProvider, nativeEndpointUrl, "hermes", + registered, ); applyHermesManagedRoute(config, { model, @@ -917,6 +951,7 @@ export function patchHermesInferenceConfig( inferenceApi: route.inferenceApi, contextWindow, nativeCredentialHandle, + ...(registered ? { registeredCredentialEnv: registered.credentialEnv } : {}), }); return { changed: before !== JSON.stringify(config), route }; @@ -927,8 +962,12 @@ function resolveHermesContextWindowForSwitch( model: string, gatewayPort: number, deps: Pick, + nimContextWindow?: number | null, ): number | undefined { - const contextWindow = deps.resolveContextWindowForModel(provider, model, gatewayPort); + const contextWindow = + nimContextWindow === undefined + ? deps.resolveContextWindowForModel(provider, model, gatewayPort) + : nimContextWindow; if (contextWindow != null) { deps.log(` Context window for '${model}': ${contextWindow} tokens`); return contextWindow; @@ -947,14 +986,14 @@ function resolveOpenClawContextWindowForSwitch( sandboxName: string; gatewayPort: number; routeChanged: boolean; + nimContextWindow?: number | null; }, deps: Pick, ): number | null | undefined { - const contextWindow = deps.resolveContextWindowForModel( - options.provider, - options.model, - options.gatewayPort, - ); + const contextWindow = + options.nimContextWindow === undefined + ? deps.resolveContextWindowForModel(options.provider, options.model, options.gatewayPort) + : options.nimContextWindow; if (contextWindow != null) { deps.log(` Context window for '${options.model}': ${contextWindow} tokens`); return contextWindow; @@ -1250,6 +1289,108 @@ function resolveMatchingAgentConfigTarget( return target; } +async function resolveRegisteredConfigSelection(input: { + options: InferenceSetOptions; + deps: InferenceSetDeps; + entry: SandboxEntry; + provider: string; + model: string; + sandboxName: string; + agentName: string; + expectedGatewayName: string; +}) { + const { options, deps, entry, provider, model, sandboxName, agentName, expectedGatewayName } = + input; + const previousRegistered = recordedRegisteredInferenceProviderAttachment(entry); + if (getProviderSelectionConfig(provider, model) || isModelRouterProvider(provider)) + return { previousRegistered, registeredConfig: undefined, registered: undefined }; + const registeredConfig = readInSandboxConfigOrFail( + deps, + sandboxName, + resolveMatchingAgentConfigTarget(deps, sandboxName, agentName), + expectedGatewayName, + ); + const registered = await resolveRegisteredSelection({ + ...options, + provider, + sandboxName, + gatewayName: expectedGatewayName, + agentName, + config: registeredConfig, + previous: previousRegistered, + adapter: deps.providerAdapter, + }); + return { previousRegistered, registeredConfig, registered }; +} + +function assertInferenceSetAgent(agentName: string, sandboxName: string): void { + if (agentName !== "openclaw" && agentName !== "hermes") { + const dcodeHint = + agentName === "langchain-deepagents-code" + ? ` Deep Agents Code has no runtime inference-set path. ${dcodeSelectionChangeGuidance()}` + : ""; + throw new InferenceSetError( + `nemoclaw inference set supports OpenClaw and Hermes sandboxes; '${sandboxName}' uses '${agentName}'.${dcodeHint}`, + 2, + ); + } +} + +function logInferenceProbeRetry( + log: InferenceSetDeps["log"], + result: Awaited>, + delayMs: number, + attempt: number, +): void { + if (result.ok) return; + const retryCause = + result.httpStatus === null + ? "the sandbox probe did not receive an HTTP status" + : `HTTP ${result.httpStatus}`; + log( + ` Waiting ${delayMs / 1_000}s for OpenShell route convergence after ${retryCause} (probe ${attempt}/3)...`, + ); +} + +function unavailableSandboxInferenceProbe( + probeError: unknown, +): Awaited> { + const probeFailureDetail = + probeError instanceof Error && probeError.message + ? (onboardSession.redactSensitiveText(probeError.message)?.trim() ?? "") + : ""; + return { + ok: false, + detail: probeFailureDetail + ? `sandbox inference invocation probe was unavailable: ${probeFailureDetail}` + : "sandbox inference invocation probe was unavailable", + httpStatus: null, + }; +} + +function registeredSelectionDependencies( + deps: InferenceSetDeps, + active: boolean, + revalidateSandboxIdentity: ((operation: string) => void) | undefined, +) { + if (!active) return { deps, revalidateSandboxIdentity: undefined }; + if (!revalidateSandboxIdentity) + throw new InferenceSetError( + "Registered inference selection requires recorded sandbox lifecycle identity.", + ); + revalidateSandboxIdentity("prepare registered inference selection"); + return { + revalidateSandboxIdentity, + deps: { + ...deps, + providerAdapter: registeredSelectionProviderAdapter( + deps.providerAdapter, + revalidateSandboxIdentity, + ), + }, + }; +} + async function runInferenceSetWithoutHostLock( options: InferenceSetOptions, deps: InferenceSetDeps, @@ -1267,20 +1408,28 @@ async function runInferenceSetWithoutHostLock( assertInferenceModelId(model); const { sandboxName, entry, agentName } = resolveTargetSandbox(options.sandboxName, deps); + const capturedRegisteredSandboxIdentity = captureRegisteredSelectionIdentity(entry, deps); const priorHttpsPinRouteId = parseHttpsPinRouteId(entry.endpointUrl); - if (agentName !== "openclaw" && agentName !== "hermes") { - const dcodeHint = - agentName === "langchain-deepagents-code" - ? ` Deep Agents Code has no runtime inference-set path. ${dcodeSelectionChangeGuidance()}` - : ""; - throw new InferenceSetError( - `nemoclaw inference set supports OpenClaw and Hermes sandboxes; '${sandboxName}' uses '${agentName}'.${dcodeHint}`, - 2, - ); - } - // Reject conflicting persisted gateway identity before committing a route. - // The later context-window lookup must use this same validated port. + assertInferenceSetAgent(agentName, sandboxName); const gatewayPort = registryEntryGatewayPort(entry); + const { previousRegistered, registeredConfig, registered } = + await resolveRegisteredConfigSelection({ + options, + deps, + entry, + provider, + model, + sandboxName, + agentName, + expectedGatewayName, + }); + const registeredBoundary = registeredSelectionDependencies( + deps, + Boolean(registered || previousRegistered), + capturedRegisteredSandboxIdentity, + ); + deps = registeredBoundary.deps; + const revalidateRegisteredSandboxIdentity = registeredBoundary.revalidateSandboxIdentity; const session = deps.loadSession(); const explicitInferenceApi = typeof options.inferenceApi === "string" && options.inferenceApi.trim() @@ -1301,9 +1450,17 @@ async function runInferenceSetWithoutHostLock( 2, ); } - const hasExplicitCustomRoute = Boolean( - options.endpointUrl || options.credentialEnv || options.inferenceApi, - ); + const hasExplicitCustomRoute = + !registered && Boolean(options.endpointUrl || options.credentialEnv || options.inferenceApi); + if ( + isModelRouterProvider(provider) && + (hasExplicitCustomRoute || !nativeModelRouterEndpoint(provider)) + ) { + throw new InferenceSetError( + "Model Router uses its configured endpoint, API, and host credential.", + 2, + ); + } const customRoute = hasExplicitCustomRoute ? { ...options, @@ -1312,7 +1469,9 @@ async function runInferenceSetWithoutHostLock( // routes still require the operator to supply a complete identity. inferenceApi: resolveAgentInferenceApi(agentName, provider, explicitOrRecordedInferenceApi), } - : options; + : registered + ? {} + : options; const routeEntry = { ...entry, preferredInferenceApi: resolveAgentInferenceApi( @@ -1349,11 +1508,15 @@ async function runInferenceSetWithoutHostLock( agentName, entry, provider, + endpointUrl: options.endpointUrl, + inferenceApi: explicitOrRecordedInferenceApi, + customRouteChanged: provider === "compatible-endpoint" && hasExplicitCustomRoute, sandboxName, gatewayName: expectedGatewayName, deps, }); const selectingNative = + Boolean(registered) || selectingNativeLocal || Boolean(localSwitch.selected || localSwitch.prepare) || isNativeInferenceSelection(provider, entry, options.endpointUrl, expectedGatewayName, deps); @@ -1374,6 +1537,13 @@ async function runInferenceSetWithoutHostLock( 2, ); } + const nimSelection = verifyNimSwitchSelection( + provider, + model, + preparedRoute.preliminaryRegistryMetadata, + localSwitch.selected ?? previousNativeLocalAttachment, + ["nim", "nim-local"].includes(options.provider.trim().toLowerCase().replaceAll("_", "-")), + ); const preMutationRoute = await observeInferenceRouteForSelection({ nativeProvider: selectingNative, gatewayName: preparedRoute.gatewayName, @@ -1391,7 +1561,7 @@ async function runInferenceSetWithoutHostLock( } const { registryMetadata, - explicitPreferredInferenceApi, + explicitPreferredInferenceApi: finalizedPreferredInferenceApi, directProviderBinding, httpsPinProviderBinding, routeImpactWarning, @@ -1425,6 +1595,15 @@ async function runInferenceSetWithoutHostLock( effectiveInferenceApi: preparedRoute.preliminaryExplicitMetadata?.preferredInferenceApi ?? null, }); + if (registered) + Object.assign(registryMetadata, { + endpointUrl: registered.endpointUrl, + endpointSource: "inference-set", + credentialEnv: registered.credentialEnv, + preferredInferenceApi: registered.inferenceApi, + }); + const explicitPreferredInferenceApi = registered?.inferenceApi ?? finalizedPreferredInferenceApi; + // Local providers (ollama-local, vllm-local) route through the sandbox-facing // host.openshell.internal hostname, which the host-side `openshell inference set` // verify cannot resolve — its default verification is a guaranteed false negative @@ -1452,8 +1631,11 @@ async function runInferenceSetWithoutHostLock( // guaranteed false negative. HTTPS-pin adapters retain their local-health // verification; direct bridge routes are probed from the sandbox below. let effectiveNoVerify = - options.noVerify === true || Boolean(httpsPinProviderBinding) || probeSandboxRoute; - if (deps.isLocalInferenceProvider(provider)) { + options.noVerify === true || + Boolean(nimSelection) || + Boolean(httpsPinProviderBinding) || + probeSandboxRoute; + if (deps.isLocalInferenceProvider(provider) && !nimSelection) { const localValidation = deps.validateLocalProvider(provider); if (localValidation.ok) { effectiveNoVerify = true; @@ -1496,12 +1678,9 @@ async function runInferenceSetWithoutHostLock( // leaving a half-applied switch across the three config layers (#6997). // Route finalization has no side effect when metadata is reused; explicit // custom routes were capability-checked before finalization above. - const config = readInSandboxConfigOrFail( - deps, - sandboxName, - target, - agentName === "openclaw" ? preparedRoute.gatewayName : undefined, - ); + const config = + registeredConfig ?? + readInSandboxConfigOrFail(deps, sandboxName, target, preparedRoute.gatewayName); const preMutationInferenceApi = explicitPreferredInferenceApi ?? resolveRuntimeInferenceApi({ @@ -1530,6 +1709,7 @@ async function runInferenceSetWithoutHostLock( sandboxName, selectingNative, gatewayName: preparedRoute.gatewayName, + endpointUrl: registryMetadata.endpointUrl, deps, }); const rollbackRoute = preMutationRoute.state === "configured" ? preMutationRoute.route : null; @@ -1575,6 +1755,10 @@ async function runInferenceSetWithoutHostLock( let nativeLocalRegistryCommitted = false; let previousNativeLocalDetached = false; let previousNativeLocalDetachCommitted = false; + let registeredAttachment = registered; + let registeredAttachmentChanged = false; + let registeredCommitted = false; + let previousRegisteredDetached = false; let nativeCredentialHandle: string | undefined; let nativeProviderAttachment: NativeProviderAttachment | undefined; let nativeAttachmentChanged = false; @@ -1597,9 +1781,42 @@ async function runInferenceSetWithoutHostLock( return detail; }; try { + previousRegisteredDetached = await detachPreviousRegisteredSelection({ + revalidateSandboxIdentity: revalidateRegisteredSandboxIdentity, + adapter: deps.providerAdapter, + previous: previousRegistered, + next: registered, + }); + if (registered) { + previousNativeDetached = await detachPreviousNativeBeforePublish({ + selectingNative, + selectedProvider: provider, + previousAttachment: previousNativeAttachment, + gatewayName: preparedRoute.gatewayName, + sandboxName, + deps, + }); + previousLocalDetached = await detachPreviousOpenClawLocal({ + previous: localSwitch.previous, + selected: localSwitch.selected, + sandboxName, + deps, + }); + previousNativeLocalDetached = await detachPreviousNativeLocalBeforePublish({ + adapter: deps.providerAdapter, + sandboxName, + detached: false, + previous: previousNativeLocalAttachment, + }); + } await prepareOpenClawLocalSwitch({ selection: localSwitch, + entry, provider, + endpointUrl: registryMetadata.endpointUrl, + credentialEnv: registryMetadata.credentialEnv, + inferenceApi: preMutationInferenceApi, + nimSelection, sandboxName, gatewayName: preparedRoute.gatewayName, deps, @@ -1612,8 +1829,29 @@ async function runInferenceSetWithoutHostLock( }); }, }); + if (registered) { + const selected = await attachRegisteredInferenceProvider({ + revalidateSandboxIdentity: revalidateRegisteredSandboxIdentity, + adapter: deps.providerAdapter, + expected: registered, + }); + registeredAttachment = selected.attachment; + registeredAttachmentChanged = selected.changed; + nativeCredentialHandle = await readVerifiedRegisteredCredentialHandle({ + revalidateSandboxIdentity: revalidateRegisteredSandboxIdentity, + attachment: registeredAttachment, + adapter: deps.providerAdapter, + read: deps.readRegisteredInferenceCredentialHandle, + sleep: deps.sleep, + }); + } const nativeSelection = await prepareNativeSelection({ - selectingNative: selectingNative && !selectingNativeLocal, + selectingNative: + selectingNative && + !registered && + !selectingNativeLocal && + !localSwitch.selected && + !localSwitch.prepare, provider, expectedAttachment: nativeProviderAuthority, previousAttachment: previousNativeAttachment, @@ -1637,7 +1875,7 @@ async function runInferenceSetWithoutHostLock( }); nativeProviderAttachment = nativeSelection.attachment; nativeAttachmentChanged = nativeSelection.attachmentChanged; - nativeCredentialHandle = await readHermesNativeNvidiaCredentialHandleIfNeeded({ + nativeCredentialHandle ??= await readHermesNativeNvidiaCredentialHandleIfNeeded({ agentName, provider, sandboxName, @@ -1653,6 +1891,12 @@ async function runInferenceSetWithoutHostLock( gatewayName: preparedRoute.gatewayName, previous: previousNativeLocalAttachment, binding: directProviderBinding, + entry, + endpointUrl: registryMetadata.endpointUrl, + credentialEnv: registryMetadata.credentialEnv, + inferenceApi: preMutationInferenceApi, + explicitCustomRoute: hasExplicitCustomRoute, + nimSelection, adapter: deps.providerAdapter, resolveCredentialValue: deps.resolveCredentialValue, readAuthority: deps.getNativeLocalProviderAuthority, @@ -1698,7 +1942,7 @@ async function runInferenceSetWithoutHostLock( // not replace the provider binding. providerMutation = null; } - } else if (loopbackNoAuthProxyRoute) { + } else if (loopbackNoAuthProxyRoute && !selectingNative) { // A model-only switch builds no provider binding, so nothing above // inspects the live provider. This route's credential is the local // no-auth proxy token that OpenShell holds and sends on selection, and @@ -1768,6 +2012,7 @@ async function runInferenceSetWithoutHostLock( model, preferredInferenceApi: preMutationInferenceApi, ...(selectingNative ? { nativeProvider: true } : {}), + registeredInferenceProviderAttachment: registeredAttachment, ...openClawLocalProbeFields(nativeSelection.endpointUrl, localSwitch.selected), ...(agentName === "hermes" ? { @@ -1780,35 +2025,18 @@ async function runInferenceSetWithoutHostLock( previousProvider, previousModel, previousInferenceApi, + providerAttachmentChanged: nativeAttachmentChanged || registeredAttachmentChanged, targetInferenceApi: preMutationInferenceApi, }, { probe: deps.probeSandboxRoute, sleep: deps.sleep, - onRetry: (result, delayMs, attempt) => { - if (result.ok) return; - const retryCause = - result.httpStatus === null - ? "the sandbox probe did not receive an HTTP status" - : `HTTP ${result.httpStatus}`; - deps.log( - ` Waiting ${delayMs / 1_000}s for OpenShell route convergence after ${retryCause} (probe ${attempt}/3)...`, - ); - }, + onRetry: (result, delayMs, attempt) => + logInferenceProbeRetry(deps.log, result, delayMs, attempt), }, ); } catch (probeError) { - const probeFailureDetail = - probeError instanceof Error && probeError.message - ? (onboardSession.redactSensitiveText(probeError.message)?.trim() ?? "") - : ""; - probe = { - ok: false, - detail: probeFailureDetail - ? `sandbox inference invocation probe was unavailable: ${probeFailureDetail}` - : "sandbox inference invocation probe was unavailable", - httpStatus: null, - }; + probe = unavailableSandboxInferenceProbe(probeError); } await requireSuccessfulInferenceProbe({ probe, @@ -1881,14 +2109,17 @@ async function runInferenceSetWithoutHostLock( previousNativeLocalDetached, ), ...nativeSelectionRegistryFields(nativeProviderAttachment, previousNativeDetached), + registeredInferenceProviderAttachment: registeredAttachment, ...openClawLocalRegistryFields(localSwitch.selected, previousLocalDetached), ...(nativeLocalProviderAttachment?.endpointUrl ? { endpointUrl: nativeLocalProviderAttachment.endpointUrl } : (nativeSelection.endpointUrl ?? localSwitch.selected?.endpointUrl) && - !nativeProviderAttachment?.custom + !nativeProviderAttachment?.custom && + !loopbackNoAuthProxyRoute ? { endpointUrl: nativeSelection.endpointUrl ?? localSwitch.selected?.endpointUrl } : {}), }); + revalidateRegisteredSandboxIdentity?.("publish registered inference selection"); if ( !deps.updateSandbox( sandboxName, @@ -1905,6 +2136,7 @@ async function runInferenceSetWithoutHostLock( `Failed to update NemoClaw registry for sandbox '${sandboxName}'.`, ); } + registeredCommitted = true; nativeLocalRegistryCommitted = Boolean(nativeLocalProviderAttachment); previousNativeLocalDetachCommitted = previousNativeLocalDetached; nativeRegistryCommitted = Boolean(nativeProviderAttachment); @@ -1961,7 +2193,13 @@ async function runInferenceSetWithoutHostLock( let patched: { changed: boolean; route: SandboxInferenceConfig }; if (agentName === "hermes") { - const contextWindow = resolveHermesContextWindowForSwitch(provider, model, gatewayPort, deps); + const contextWindow = resolveHermesContextWindowForSwitch( + provider, + model, + gatewayPort, + deps, + nimSelection?.contextWindow, + ); patched = patchHermesInferenceConfig( config, provider, @@ -1973,6 +2211,7 @@ async function runInferenceSetWithoutHostLock( localSwitch.selected?.endpointUrl, selectingNative, nativeCredentialHandle, + registeredAttachment, ); } else { // Recompute the context window for the model being switched to, so it does @@ -1984,6 +2223,7 @@ async function runInferenceSetWithoutHostLock( sandboxName, gatewayPort, routeChanged: openClawConfigSyncPending, + nimContextWindow: nimSelection?.contextWindow, }, deps, ); @@ -2000,6 +2240,8 @@ async function runInferenceSetWithoutHostLock( nativeSelection.endpointUrl ?? localSwitch.selected?.endpointUrl, selectingNative, + registeredAttachment, + nativeCredentialHandle, ); } @@ -2010,7 +2252,10 @@ async function runInferenceSetWithoutHostLock( ); // In-sandbox config is the last, crash-prone layer (gateway + registry already consistent). // Both agents return an error after a failed sync so automation cannot accept partial - // convergence. OpenClaw retries the atomic native update; Hermes rebuilds its managed config. + // convergence. Registered selections retry without replacing restored agent configuration. + const registeredConfigRecovery = registeredAttachment + ? "Retry the same inference set command to finish applying the recorded selection." + : undefined; let inSandboxConfigSynced = false; try { if (agentName === "openclaw") { @@ -2023,17 +2268,14 @@ async function runInferenceSetWithoutHostLock( ); inSandboxConfigSynced = true; } else { - deps.writeSandboxConfig(sandboxName, target, config); - try { - deps.recomputeSandboxConfigHash(sandboxName, target); - } catch (hashError) { - const detail = - hashError instanceof Error && hashError.message ? hashError.message : String(hashError); - deps.log( - ` Warning: wrote the Hermes config for '${sandboxName}', but failed to refresh its integrity hash: ${detail}.`, - ); - throw hashError; - } + deps.writeSandboxConfig(sandboxName, target, config, { + gatewayName: preparedRoute.gatewayName, + // Native attachments require the existing restart owner to activate + // refreshed process credentials after all route writes complete. + ...(selectingNative || previousNativeAttachment || previousNativeLocalAttachment + ? { deferReload: true as const } + : {}), + }); inSandboxConfigSynced = true; } } catch (writeError) { @@ -2096,7 +2338,9 @@ async function runInferenceSetWithoutHostLock( throw new InferenceSetError( `Hermes inference route synchronization did not complete for '${sandboxName}'. ` + `The OpenShell route and NemoClaw registry remain committed, but the in-sandbox ` + - `Hermes configuration did not fully converge. Review the preceding configuration warning. ` + + `Hermes configuration did not fully converge. ` + + (registeredConfigRecovery ? `${registeredConfigRecovery} ` : "") + + `Review the preceding configuration warning. ` + `If integrity metadata is invalid, back up needed data and restore matching configuration ` + `and integrity files from a trusted backup or create a fresh Hermes sandbox. ` + `Rebuild preserves the native home and may retain the mismatch.`, @@ -2133,12 +2377,20 @@ async function runInferenceSetWithoutHostLock( openClawConfigSyncPending: inSandboxConfigSynced && openClawConfigSyncPending, }; } catch (error) { + revalidateRegisteredSandboxIdentity?.("recover registered inference selection"); recordInferenceSetFailure( sandboxName, Boolean(appliedInferenceSelection || appliedProvider || providerCommitResidual), Boolean(ambiguousInferenceSelection || providerMutation), expectedGatewayName, ); + await rollbackRegisteredSelection({ + revalidateSandboxIdentity: revalidateRegisteredSandboxIdentity, + adapter: deps.providerAdapter, + committed: registeredCommitted, + attachment: registeredAttachment, + changed: registeredAttachmentChanged, + }); if (error instanceof OpenClawInferenceConfigSyncError) throw error; const hermesLocalFailure = await rollbackNativeLocalSelection({ adapter: deps.providerAdapter, @@ -2182,6 +2434,15 @@ async function runInferenceSetWithoutHostLock( deps, error: hermesLocalFailure, }); + await restorePreviousRegisteredSelection({ + revalidateSandboxIdentity: revalidateRegisteredSandboxIdentity, + adapter: deps.providerAdapter, + committed: registeredCommitted, + previous: previousRegistered, + previousDetached: previousRegisteredDetached, + cleanupError: localRollbackFailure, + originalError: error, + }); const failure = await restorePreviousOpenClawLocal({ previous: localSwitch.previous, detached: previousLocalDetached, @@ -2281,22 +2542,30 @@ export async function runInferenceSet( lockedSelection.entry, deps.runtimeProviders, ); - // Validate both persisted binding fields before pending provider cleanup. const gatewayName = resolveGatewayName(registryEntryGatewayPort(lockedSelection.entry)); - const mutation = await deps.withGatewayRouteMutationLock(gatewayName, async () => { - await reconcilePreviousOpenClawLocal({ - entry: lockedSelection.entry, - sandboxName: selected.sandboxName, - gatewayName, - deps, - }); - return await runInferenceSetWithoutHostLock( - { ...options, sandboxName: selected.sandboxName }, - deps, - gatewayName, - runtimeProvider, - ); - }); + const mutation = await deps.withGatewayRouteMutationLock(gatewayName, () => + withRoutedInferencePortLock( + Boolean( + nativeModelRouterEndpoint(options.provider) || + nativeModelRouterEndpoint(lockedSelection.entry.provider), + ), + async () => { + await reconcilePreviousOpenClawLocal({ + entry: lockedSelection.entry, + sandboxName: selected.sandboxName, + gatewayName, + deps, + }); + return await runInferenceSetWithoutHostLock( + { ...options, sandboxName: selected.sandboxName }, + deps, + gatewayName, + runtimeProvider, + ); + }, + {}, + ), + ); // Retain the outer sandbox lifecycle lock so another process cannot replace // this sandbox between the committed write, an optional restart, and // device-scope convergence. diff --git a/src/lib/actions/inference-set/local-runtime.test.ts b/src/lib/actions/inference-set/local-runtime.test.ts index 5502bea2539..f031a85cb50 100644 --- a/src/lib/actions/inference-set/local-runtime.test.ts +++ b/src/lib/actions/inference-set/local-runtime.test.ts @@ -2,7 +2,13 @@ // SPDX-License-Identifier: Apache-2.0 import { beforeEach, expect, it, vi } from "vitest"; -import { resolveHermesLocalSwitchRuntime, resolveLocalSwitchRuntime } from "./local-runtime"; +import * as nim from "../../inference/nim"; +import { nativeLocalTestReceipt } from "../../../../test/helpers/inference-set"; +import { + resolveHermesLocalSwitchRuntime, + resolveLocalSwitchRuntime, + verifyNimSwitchSelection, +} from "./local-runtime"; const mocks = vi.hoisted(() => ({ managed: vi.fn(), @@ -154,3 +160,38 @@ it("rejects an unavailable Hermes endpoint before reading credentials", () => { ).toThrow("local inference endpoint is unavailable"); expect(mocks.credential).not.toHaveBeenCalled(); }); + +it.each(["authenticated", "sentinel"] as const)( + "rejects a recorded %s attachment outside the selected NIM boundary", + (authMode) => { + const getServed = vi.spyOn(nim, "getServedModelId"); + const endpointUrl = nim.getNimInferenceEndpoints().baseUrl; + const previous = nativeLocalTestReceipt( + "vllm-local", + authMode === "sentinel" ? "http://host.openshell.internal:18000/v1" : endpointUrl, + { agent: "openclaw", authMode }, + ); + expect(() => + verifyNimSwitchSelection( + "vllm-local", + "nim/model", + { nimContainer: "owned-nim", endpointUrl }, + previous, + ), + ).toThrow("recorded NIM route does not match"); + expect(getServed).not.toHaveBeenCalled(); + expect(mocks.managed).not.toHaveBeenCalled(); + }, +); + +it("rejects an explicit credential binding for a verified NIM selection", () => { + expect(() => + resolveHermesLocalSwitchRuntime("vllm-local", hermesBinding, mocks.credential, { + nimContainer: "owned-nim", + endpointUrl: nim.getNimInferenceEndpoints().baseUrl, + contextWindow: null, + }), + ).toThrow("Local NIM cannot use an explicit provider credential binding"); + expect(mocks.managed).not.toHaveBeenCalled(); + expect(mocks.credential).not.toHaveBeenCalled(); +}); diff --git a/src/lib/actions/inference-set/local-runtime.ts b/src/lib/actions/inference-set/local-runtime.ts index 2707dbb9a10..c254b465336 100644 --- a/src/lib/actions/inference-set/local-runtime.ts +++ b/src/lib/actions/inference-set/local-runtime.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { nativeModelRouterEndpoint } from "../../inference/native-local/model-router"; +import { loadBlueprintProfile } from "../../core/model-router-port"; import { getLocalProviderBaseUrl, getManagedVllmProviderBinding, @@ -11,16 +13,87 @@ import { LLAMA_CPP_CREDENTIAL_ENV, LLAMA_CPP_GATEWAY_BASE_URL, } from "../../inference/llama-cpp/contract"; +import { + getNimInferenceEndpoints, + getServedModelId, + getNimContextWindow, +} from "../../inference/nim"; +import type { NativeLocalProviderAttachment } from "../../inference/native-local/profile"; +import type { RegistryInferenceMetadata } from "../inference-set-route-containment"; import type { InferenceSetProviderBinding } from "../inference-set-route-containment"; import { InferenceSetError } from "../inference-set-error"; export { validateLocalProvider, type ValidationResult } from "../../inference/local"; +export type NimSwitchSelection = { + nimContainer: string; + endpointUrl: string; + contextWindow: number | null; +}; + +/** Verify the NIM owner selected by the registry or matching onboard session. */ +export function verifyNimSwitchSelection( + provider: string, + model: string, + metadata: RegistryInferenceMetadata, + previous?: NativeLocalProviderAttachment, + requireNimSelection = false, +): NimSwitchSelection | undefined { + if (requireNimSelection && !metadata.nimContainer) + throw new InferenceSetError( + "Local NVIDIA NIM requires its recorded container and endpoint. Re-run onboarding with NEMOCLAW_PROVIDER=nim-local for this sandbox.", + 2, + ); + if (provider !== "vllm-local" || !metadata.nimContainer) return; + const endpointUrl = getNimInferenceEndpoints().baseUrl; + if ( + metadata.endpointUrl !== endpointUrl || + (previous?.provider === provider && + (previous.endpointUrl !== endpointUrl || previous.authMode !== "sentinel")) + ) + throw new InferenceSetError( + "The recorded NIM route does not match its local runtime. Re-run onboarding to repair it.", + 2, + ); + if (getServedModelId() !== model) + throw new InferenceSetError( + "Local NVIDIA NIM does not serve the selected model. The inference selection was not changed.", + ); + return { + nimContainer: metadata.nimContainer, + endpointUrl, + contextWindow: getNimContextWindow(model), + }; +} + /** Read the endpoint and credential from the same runtime owners used by onboarding. */ export function resolveLocalSwitchRuntime( provider: string, resolveCredential: (name: string) => string | null, + nimSelection?: NimSwitchSelection, ) { + const routerEndpoint = nativeModelRouterEndpoint(provider); + if (routerEndpoint) { + const profile = loadBlueprintProfile("routed"); + const credentialValue = resolveCredential( + profile?.router.credential_env || profile?.credential_env || "NVIDIA_INFERENCE_API_KEY", + ); + if (!credentialValue) + throw new InferenceSetError("The existing Model Router credential is unavailable."); + return { + endpointUrl: routerEndpoint, + credentialValue, + authMode: "authenticated" as const, + ownedProxy: false, + }; + } + if (provider === "vllm-local" && nimSelection) + return { + endpointUrl: nimSelection.endpointUrl, + credentialValue: "dummy", + authMode: "sentinel" as const, + ownedProxy: false, + }; if (provider === "vllm-local") { const managed = getManagedVllmProviderBinding(); return { @@ -65,7 +138,26 @@ export function resolveHermesLocalSwitchRuntime( provider: string, binding: InferenceSetProviderBinding | null, resolveCredentialValue: (name: string) => string, + nimSelection?: NimSwitchSelection, ) { + if (nativeModelRouterEndpoint(provider)) { + if (binding) + throw new InferenceSetError("Model Router uses its configured endpoint and host credential."); + return resolveLocalSwitchRuntime(provider, resolveCredentialValue); + } + if (provider === "vllm-local" && nimSelection) { + if (binding) + throw new InferenceSetError( + "Local NIM cannot use an explicit provider credential binding.", + 2, + ); + return { + endpointUrl: nimSelection.endpointUrl, + credentialValue: "dummy", + authMode: "sentinel" as const, + ownedProxy: false, + }; + } const vllm = provider === "vllm-local" ? getManagedVllmProviderBinding() : null; const proxy = provider === "ollama-local" && shouldFrontOllamaWithProxy(); const endpointUrl = diff --git a/src/lib/actions/inference-set/native-custom.ts b/src/lib/actions/inference-set/native-custom.ts new file mode 100644 index 00000000000..600a7c67b4d --- /dev/null +++ b/src/lib/actions/inference-set/native-custom.ts @@ -0,0 +1,78 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import type { InferenceSetDeps } from "../inference-set"; +import { InferenceSetError } from "../inference-set-error"; +import { redactInferenceSetRouteDiagnostic } from "../inference-set-provider-diagnostics"; +import { cliOpenShellSandboxPolicyReader } from "../../adapters/openshell/sandbox-policy-cli"; +import type { NativeProviderAttachment } from "../../inference/native-provider/contract"; +import { prepareCustomNativeProvider } from "../../inference/native-provider/custom-setup"; +import { customNativeCredentialEnv } from "../../inference/native-provider/hosted"; +import { fixedNativeProviderForAttachment } from "../../inference/native-provider/fixed"; +import { nativeProviderLifecycle } from "../../inference/native-provider"; + +export async function prepareCustomNativeSelection(input: { + provider: string; + expectedAttachment?: NativeProviderAttachment; + gatewayName: string; + sandboxName: string; + endpointUrl?: string | null; + credentialEnv?: string | null; + inferenceApi?: string | null; + beforeAttach?: (receipt: NativeProviderAttachment) => Promise; + deps: InferenceSetDeps; +}): Promise<{ + attachment: NativeProviderAttachment; + attachmentChanged: boolean; + endpointUrl?: string; +}> { + if (!input.deps.getNativeHostedProviderAuthority || !input.deps.setNativeHostedProviderAuthority) + throw new InferenceSetError("Native hosted provider ownership storage is unavailable"); + const credentialEnv = input.credentialEnv || customNativeCredentialEnv(input.provider); + const policy = await ( + input.deps.sandboxPolicyReader ?? cliOpenShellSandboxPolicyReader + ).readSandboxPolicy({ + target: { kind: "named", gatewayName: input.gatewayName }, + sandboxName: input.sandboxName, + // Exclude the previous attachment's generated rule, which is detached + // before replacement. The sandbox's base rules remain in place. + scope: "base", + }); + if (!policy.ok) { + throw new InferenceSetError( + `Cannot inspect the sandbox policy before selecting a custom endpoint: ${redactInferenceSetRouteDiagnostic(policy.error.message)} Resolve the policy-read error before retrying. No provider was changed.`, + ); + } + const receipt = await prepareCustomNativeProvider({ + gatewayName: input.gatewayName, + provider: input.provider, + endpointUrl: input.endpointUrl || "", + api: + input.inferenceApi || + (input.provider === "compatible-anthropic-endpoint" + ? "anthropic-messages" + : "openai-completions"), + credentialEnv, + credentialValue: input.deps.resolveCredentialValue(credentialEnv) || null, + adapter: input.deps.providerAdapter, + sandboxPolicyDocument: policy.value.document, + recordedAttachment: input.expectedAttachment, + recordedSandboxName: input.sandboxName, + readAuthority: input.deps.getNativeHostedProviderAuthority, + writeAuthority: input.deps.setNativeHostedProviderAuthority, + }); + await input.beforeAttach?.(receipt); + const attached = await nativeProviderLifecycle( + fixedNativeProviderForAttachment(receipt), + ).ensureNativeProviderAttached({ + adapter: input.deps.providerAdapter, + target: { kind: "named", gatewayName: input.gatewayName }, + sandboxName: input.sandboxName, + expected: receipt, + }); + return { + attachment: attached.receipt, + attachmentChanged: attached.changed, + endpointUrl: receipt.endpointUrl, + }; +} diff --git a/src/lib/actions/inference-set/native-local.ts b/src/lib/actions/inference-set/native-local.ts index 2444fbbe941..9cb0c84d723 100644 --- a/src/lib/actions/inference-set/native-local.ts +++ b/src/lib/actions/inference-set/native-local.ts @@ -1,9 +1,18 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { nativeModelRouterEndpoint } from "../../inference/native-local/model-router"; +import { + assertNativeModelRouterOwner, + prepareNativeModelRouterSelection, +} from "../../onboard/routed-inference"; +import type { NimSwitchSelection } from "./local-runtime"; import type { InferenceSetDeps } from "../inference-set"; import { InferenceSetError } from "../inference-set-error"; -import { isLocalInferenceProvider } from "../../inference/native-local/agent-config"; +import { + usesNativeLocalInferenceForAgent, + gatewayReachableCompatibleEndpointUrl, +} from "../../inference/native-local/eligibility"; import { detachNativeLocalProvider, ensureNativeLocalProviderAttached, @@ -15,6 +24,8 @@ import { } from "../../inference/native-local/profile"; import { prepareNativeLocalSelection } from "../../inference/native-local/selection"; import type { SandboxEntry } from "../../state/registry"; +import { usesLoopbackNoAuthProxyRoute } from "../inference-set-route-containment"; +import { OLLAMA_PROXY_PORT } from "../../core/ollama-proxy-port"; export type OpenClawLocalSwitch = { selected?: NativeLocalProviderAttachment; @@ -27,6 +38,9 @@ export async function resolveOpenClawLocalSwitch(input: { agentName: string; entry: SandboxEntry; provider: string; + endpointUrl?: string | null; + inferenceApi?: string | null; + customRouteChanged?: boolean; sandboxName: string; gatewayName: string; deps: InferenceSetDeps; @@ -58,31 +72,86 @@ export async function resolveOpenClawLocalSwitch(input: { expected: previous, }); } - // A model-only change must preserve the native router selected during onboarding. - if (provider === "nvidia-router") { - if (previous?.provider === provider) return { selected: previous, previous }; + const router = Boolean(nativeModelRouterEndpoint(provider)); + if ( + previous && + input.endpointUrl && + usesLoopbackNoAuthProxyRoute(entry, provider) && + new URL(input.endpointUrl).href.replace(/\/+$/, "") !== + new URL(entry.endpointUrl!).href.replace(/\/+$/, "") + ) { throw new InferenceSetError( - "Model Router requires a verified native provider attachment. Back up this sandbox. " + - `Run nemoclaw onboard --fresh --name ${sandboxName} --recreate-sandbox to select Model Router.`, + "The no-auth proxy is bound to the recorded backend. Reuse that endpoint; inference set cannot change its host, port, or path.", 2, ); } - if (!isLocalInferenceProvider(provider) || (provider === "vllm-local" && entry.nimContainer)) + const endpointUrl = input.endpointUrl ?? (entry.provider === provider ? entry.endpointUrl : null); + if ( + !usesNativeLocalInferenceForAgent( + "openclaw", + provider, + endpointUrl, + Boolean(entry.nimContainer), + ) + ) return { previous }; - // A retained OpenClaw row may still own the historical shared route. - if (!previous && entry.provider === provider) return {}; + // Only onboarding establishes loopback rewrites. New live selections keep + // the existing reserved-alias admission rule; recorded proxy reuse is separate. if ( - !previous || - entry.provider !== provider || - previous.profileId !== nativeLocalIdentity(previous).profileId + provider === "compatible-endpoint" && + input.endpointUrl && + new URL(input.endpointUrl).hostname !== "host.openshell.internal" && + !usesLoopbackNoAuthProxyRoute(entry, provider) && + (!previous || + gatewayReachableCompatibleEndpointUrl(provider, input.endpointUrl) !== previous.endpointUrl) ) + return { previous }; + // A retained OpenClaw row may still own the historical shared route. + if (!previous && entry.provider === provider) { + if (router) + throw new InferenceSetError( + "Recreate this legacy Model Router sandbox before changing inference.", + 2, + ); + return {}; + } + if (!previous || entry.provider !== provider) return { previous, prepare: true }; + const expectedEndpoint = usesLoopbackNoAuthProxyRoute(entry, provider) + ? `http://host.openshell.internal:${OLLAMA_PROXY_PORT}${new URL(entry.endpointUrl!).pathname.replace(/\/+$/, "")}` + : gatewayReachableCompatibleEndpointUrl(provider, endpointUrl); + if ( + provider === "compatible-endpoint" && + (previous.endpointUrl !== expectedEndpoint || + (previous.api ?? "openai-completions") !== (input.inferenceApi ?? "openai-completions")) + ) { + if (input.customRouteChanged) return { previous, prepare: true }; + throw new InferenceSetError( + "Recorded compatible provider API or endpoint does not match its native attachment.", + ); + } + if (router) { + const runtime = deps.resolveLocalSwitchRuntime?.(provider, deps.resolveCredentialValue); + if ( + !runtime?.endpointUrl || + !runtime.credentialValue || + runtime.endpointUrl !== previous.endpointUrl + ) + throw new InferenceSetError("Model Router ownership requires its configured credential.", 2); + assertNativeModelRouterOwner(runtime.endpointUrl, runtime.credentialValue); + } + if (previous.profileId !== nativeLocalIdentity(previous).profileId) return { previous, prepare: true }; return { selected: previous, previous }; } export async function prepareOpenClawLocalSwitch(input: { selection: OpenClawLocalSwitch; + entry: SandboxEntry; provider: string; + endpointUrl?: string | null; + credentialEnv?: string | null; + inferenceApi?: string | null; + nimSelection?: NimSwitchSelection; sandboxName: string; gatewayName: string; deps: InferenceSetDeps; @@ -96,16 +165,38 @@ export async function prepareOpenClawLocalSwitch(input: { !deps.setNativeLocalProviderAuthority ) throw new InferenceSetError("Native local provider ownership storage is unavailable."); - const runtime = deps.resolveLocalSwitchRuntime(input.provider, deps.resolveCredentialValue); + const ownedProxy = usesLoopbackNoAuthProxyRoute(input.entry, input.provider); + const runtime = + input.provider === "compatible-endpoint" + ? { + endpointUrl: ownedProxy + ? input.selection.previous?.endpointUrl + : gatewayReachableCompatibleEndpointUrl(input.provider, input.endpointUrl), + credentialValue: deps.resolveCredentialValue(input.credentialEnv || "COMPATIBLE_API_KEY"), + authMode: "authenticated" as const, + ownedProxy, + } + : deps.resolveLocalSwitchRuntime( + input.provider, + deps.resolveCredentialValue, + input.nimSelection, + ); if (!runtime.endpointUrl) throw new InferenceSetError("The local runtime endpoint is unavailable."); - input.selection.selected = await prepareNativeLocalSelection({ + input.selection.selected = await ( + nativeModelRouterEndpoint(input.provider) + ? prepareNativeModelRouterSelection + : prepareNativeLocalSelection + )({ adapter: deps.providerAdapter, binding: { agent: "openclaw", provider: input.provider as NativeLocalProviderAttachment["provider"], endpointUrl: runtime.endpointUrl, authMode: runtime.authMode, + ...(input.provider === "compatible-endpoint" && input.inferenceApi === "openai-responses" + ? { api: "openai-responses" as const } + : {}), gatewayName: input.gatewayName, sandboxName: input.sandboxName, }, diff --git a/src/lib/actions/inference-set/native-policy.test.ts b/src/lib/actions/inference-set/native-policy.test.ts index c2e375634b9..79489436880 100644 --- a/src/lib/actions/inference-set/native-policy.test.ts +++ b/src/lib/actions/inference-set/native-policy.test.ts @@ -218,6 +218,9 @@ describe("custom native provider policy admission", () => { { host: "api-us.example.com", endpoint: { host: "api-*.example.com", port: 443 } }, { host: "deep.api.example.com", endpoint: { host: "**.example.com", port: 443 } }, { host: "api.example.com", endpoint: { host: "api.*.com", port: 443 } }, + { host: "api1.example.com", endpoint: { host: "api?.example.com", port: 443 } }, + { host: "apia.example.com", endpoint: { host: "api[ab].example.com", port: 443 } }, + { host: "api1.example.com", endpoint: { host: "api[!a-z].example.com", port: 443 } }, { host: "api.example.com", endpoint: { host: "", port: 443, allowed_ips: ["1.1.1.1"] } }, { host: "api.example.com", endpoint: { port: 443, allowed_ips: ["1.1.1.1"] } }, ])("refuses overlapping $endpoint before changing providers", async ({ host, endpoint }) => { diff --git a/src/lib/actions/inference-set/native.ts b/src/lib/actions/inference-set/native.ts index 0916bd3af67..5a577c7de58 100644 --- a/src/lib/actions/inference-set/native.ts +++ b/src/lib/actions/inference-set/native.ts @@ -1,15 +1,21 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { nativeLocalCredentialReference } from "../../inference/native-local/model-router"; +export { + nativeModelRouterEndpoint, + isModelRouterProvider, +} from "../../inference/native-local/model-router"; +export { withRoutedInferencePortLock } from "../../onboard/routed-inference"; import { managedInferenceApiKey, NVIDIA_INFERENCE_PLACEHOLDER } from "../../inference-credential"; import type { InferenceSetDeps } from "../inference-set"; import type { ConfigObject } from "../../security/credential-filter"; -import { redactInferenceSetRouteDiagnostic } from "../inference-set-provider-diagnostics"; import { nativeHostedAgentConfig } from "../../inference/native-provider/agent-config"; import { isLocalInferenceProvider, - nativeLocalCredentialReference, -} from "../../inference/native-local/agent-config"; + usesNativeLocalInference, +} from "../../inference/native-local/eligibility"; +import { normalizeNativeLocalProviderAttachment } from "../../inference/native-local/profile"; export { isLocalInferenceProvider }; export { resolveOpenClawLocalSwitch, @@ -30,6 +36,7 @@ export { export { reconcilePreviousOpenClawLocal } from "./native-local"; export { resolveLocalSwitchRuntime, + verifyNimSwitchSelection, validateLocalProvider, type ValidationResult, } from "./local-runtime"; @@ -47,15 +54,15 @@ import { } from "../../inference/native-nvidia"; import { hostedNativeProvider, + HOSTED_NATIVE_PROVIDERS, usesNativeHermesEndpoint, isCustomHostedProvider, - customNativeCredentialEnv, } from "../../inference/native-provider/hosted"; import { recordedNativeProviderAttachment } from "../../inference/native-provider/recorded-selection"; import type { SandboxEntry } from "../../state/registry"; -import { prepareCustomNativeProvider } from "../../inference/native-provider/custom-setup"; -import { cliOpenShellSandboxPolicyReader } from "../../adapters/openshell/sandbox-policy-cli"; -import { isHostedCustomEndpoint } from "../../inference/native-provider/custom"; +import { isCustomNativeEndpoint } from "../../inference/native-provider/custom"; +import { prepareCustomNativeSelection } from "./native-custom"; +import { prepareHostedNativeProvider } from "../../inference/native-provider/setup"; /** Preserve shared-route config unless a custom native endpoint is explicit. */ export function usesNativeConfigByDefault(provider: string, nativeEndpointUrl?: string): boolean { @@ -71,8 +78,15 @@ export function isNativeInferenceSelection( ): boolean { if (isCustomHostedProvider(provider)) { const endpoint = endpointUrl ?? (entry.provider === provider ? entry.endpointUrl : undefined); - if (!endpoint || !recordedNativeProviderAttachment(entry)) return false; - return isHostedCustomEndpoint(provider, endpoint); + const local = normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment); + const nativeLocal = + local && + local.gatewayName === gatewayName && + local.sandboxName === entry.name && + local.provider === entry.provider && + (local.agent ?? "langchain-deepagents-code") === entry.agent; + if (!endpoint || (!recordedNativeProviderAttachment(entry) && !nativeLocal)) return false; + return isCustomNativeEndpoint(provider, endpoint); } if (!fixedNativeProvider(provider)) return false; if (provider !== "hermes-provider") return true; @@ -113,6 +127,7 @@ export function resolveNativeSelectionAuthority(input: { provider: string; gatewayName: string; previousAttachment?: NativeProviderAttachment; + endpointUrl?: string | null; deps: InferenceSetDeps; }): NativeProviderAttachment | undefined { if (!input.selectingNative) return undefined; @@ -130,7 +145,9 @@ export function resolveNativeSelectionAuthority(input: { const access = nativeAuthorityAccess( input.provider, input.deps, - previous ? hostedNativeProvider(input.provider, previous.endpointUrl)?.endpoint : undefined, + input.provider === "hermes-provider" + ? hostedNativeProvider(input.provider, input.endpointUrl ?? previous?.endpointUrl)?.endpoint + : undefined, ); const gatewayAuthority = access.read(input.gatewayName); const definition = previous @@ -205,42 +222,29 @@ export async function prepareNativeSelection(input: { endpointUrl?: string; }> { if (!input.selectingNative) return { attachmentChanged: false }; - if (isCustomHostedProvider(input.provider)) { + if (isCustomHostedProvider(input.provider)) return prepareCustomNativeSelection(input); + if ( + input.provider === "hermes-provider" && + (input.endpointUrl || input.expectedAttachment?.endpointUrl) + ) { if ( !input.deps.getNativeHostedProviderAuthority || !input.deps.setNativeHostedProviderAuthority ) throw new InferenceSetError("Native hosted provider ownership storage is unavailable"); - const credentialEnv = input.credentialEnv || customNativeCredentialEnv(input.provider); - const policy = await ( - input.deps.sandboxPolicyReader ?? cliOpenShellSandboxPolicyReader - ).readSandboxPolicy({ - target: { kind: "named", gatewayName: input.gatewayName }, - sandboxName: input.sandboxName, - // Exclude the previous attachment's generated rule, which is detached - // before replacement. The sandbox's base rules remain in place. - scope: "base", - }); - if (!policy.ok) { - throw new InferenceSetError( - `Cannot inspect the sandbox policy before selecting a custom endpoint: ${redactInferenceSetRouteDiagnostic(policy.error.message)} Resolve the policy-read error before retrying. No provider was changed.`, - ); - } - const receipt = await prepareCustomNativeProvider({ - gatewayName: input.gatewayName, + if (input.inferenceApi && input.inferenceApi !== "openai-completions") + throw new InferenceSetError("Hermes Provider requires its recorded Chat Completions API"); + const receipt = await prepareHostedNativeProvider({ provider: input.provider, - endpointUrl: input.endpointUrl || "", - api: - input.inferenceApi || - (input.provider === "compatible-anthropic-endpoint" - ? "anthropic-messages" - : "openai-completions"), - credentialEnv, - credentialValue: input.deps.resolveCredentialValue(credentialEnv) || null, + gatewayName: input.gatewayName, + endpointUrl: input.endpointUrl ?? input.expectedAttachment?.endpointUrl, + retainedHermesEndpoint: true, adapter: input.deps.providerAdapter, - sandboxPolicyDocument: policy.value.document, - recordedAttachment: input.expectedAttachment, - recordedSandboxName: input.sandboxName, + credentialValue: input.expectedAttachment + ? null + : input.deps.resolveCredentialValue("NOUS_API_KEY") || null, + reuseExistingCredential: Boolean(input.expectedAttachment), + recordedSandbox: input.deps.getSandbox(input.sandboxName), readAuthority: input.deps.getNativeHostedProviderAuthority, writeAuthority: input.deps.setNativeHostedProviderAuthority, }); @@ -403,13 +407,14 @@ export function nativeSelectionConfigCredentials( existingApiKey: unknown, existingHeaders: ConfigObject, ) { - const native = nativeHostedAgentConfig(provider, baseUrl); + const local = usesNativeLocalInference(provider, baseUrl) + ? nativeLocalCredentialReference(provider, baseUrl) + : null; + const native = local ? undefined : nativeHostedAgentConfig(provider, baseUrl); return { apiKey: + local ?? native?.apiKey ?? - (isLocalInferenceProvider(provider) || provider === "nvidia-router" - ? nativeLocalCredentialReference(provider, baseUrl, "openclaw") - : null) ?? managedInferenceApiKey( baseUrl, typeof existingApiKey === "string" && @@ -432,6 +437,7 @@ export function prepareNativeTransitionAuthority(input: { sandboxName: string; selectingNative: boolean; gatewayName: string; + endpointUrl?: string | null; deps: InferenceSetDeps; }) { const previousNativeAttachment = recordedNativeProviderAttachment(input.entry); @@ -450,3 +456,38 @@ export function assertInferenceModelId(model: string): void { 2, ); } + +/** Keep owned resource names on their existing logical-provider lifecycle. */ +export function canonicalOwnedInferenceProviderName(name: string): string { + if (name === fixedNativeProvider("nvidia-prod")!.providerName) return "nvidia-prod"; + const fixed = HOSTED_NATIVE_PROVIDERS.find((definition) => definition.providerName === name); + if (fixed) return fixed.logicalProvider; + if ( + /^nemoclaw-(?:local-v1-[a-f0-9]{24}|custom-[a-f0-9]{32}-v1|hermes-[a-f0-9]{20}-v1)$/u.test(name) + ) + throw new InferenceSetError( + "Use the recorded logical provider name for this NemoClaw-owned attachment", + 2, + ); + return name; +} + +export { + attachRegisteredInferenceProvider, + detachRegisteredInferenceProvider, + verifyRegisteredInferenceProviderAttachment, + registeredInferenceProviderRevision, + recordedRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "./registered"; +export { + resolveRegisteredSelection, + readRegisteredInferenceCredentialHandle, + readVerifiedRegisteredCredentialHandle, + detachPreviousRegisteredSelection, + rollbackRegisteredSelection, + restorePreviousRegisteredSelection, + captureRegisteredSelectionIdentity, + registeredSelectionProviderAdapter, + type SandboxRecreateObservation, +} from "./registered"; diff --git a/src/lib/actions/inference-set/registered.test.ts b/src/lib/actions/inference-set/registered.test.ts new file mode 100644 index 00000000000..b61a11f9cb5 --- /dev/null +++ b/src/lib/actions/inference-set/registered.test.ts @@ -0,0 +1,251 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { spawnSync } from "node:child_process"; +import { captureSanitizedResolvedOpenshellAsync } from "../../adapters/openshell/sanitized-capture"; +import type { ConfigObject } from "../../security/credential-filter"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; +import { recordingProbeShell } from "../../../../test/helpers/native-local-probe-shell"; +import { probeSandboxInferenceInvocation } from "../sandbox/inference-invocation-probe"; +import { resolveRegisteredSelection, readRegisteredInferenceCredentialHandle } from "./registered"; +vi.mock("../../adapters/openshell/sanitized-capture", () => ({ + captureSanitizedResolvedOpenshellAsync: vi.fn(), +})); + +function selection(config: ConfigObject) { + const f = registeredInferenceFixture(); + return { + ...f, + input: { + provider: f.binding.providerName, + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + agentName: "openclaw", + config, + adapter: f.adapter, + }, + }; +} + +function nativeConfig(extra: Record = {}) { + return { + models: { + providers: { + "native-extra": { + baseUrl: "https://native.example/v1", + api: "openai-completions", + apiKey: { source: "env", provider: "default", id: "VENDOR_API_KEY" }, + ...extra, + }, + }, + }, + }; +} + +describe("registered provider native projection", () => { + it("uses an exact environment secret reference without reading a host secret", async () => { + const f = selection(nativeConfig()); + await expect(resolveRegisteredSelection(f.input)).resolves.toEqual(f.binding); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it.each([ + ["endpoint", { endpointUrl: "https://other.example/v1" }], + ["API", { inferenceApi: "openai-responses" }], + ["credential", { credentialEnv: "OTHER_API_KEY" }], + ])("refuses disagreeing explicit %s metadata before mutation", async (_label, fields) => { + const f = selection(nativeConfig()); + await expect(resolveRegisteredSelection({ ...f.input, ...fields })).rejects.toThrow( + "disagrees", + ); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it.each([ + { apiKey: "raw-credential-canary" }, + { headers: { Authorization: "raw-credential-canary" } }, + { default_headers: { "x-api-key": "raw-credential-canary" } }, + ])("refuses raw credential or authentication header configuration", async (fields) => { + const f = selection(nativeConfig(fields)); + await expect(resolveRegisteredSelection(f.input)).rejects.toThrow(/credential|authentication/); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it("recovers all metadata from the same-provider recorded binding", async () => { + const f = selection({}); + await expect(resolveRegisteredSelection({ ...f.input, previous: f.binding })).resolves.toEqual( + f.binding, + ); + }); + + it("reads the published Hermes URL and transport on retry", async () => { + const f = selection({ + providers: { + "native-extra": { + api: "https://native.example/v1", + api_mode: "chat_completions", + api_key: "${VENDOR_API_KEY}", + }, + }, + }); + await expect( + resolveRegisteredSelection({ ...f.input, agentName: "hermes", previous: f.binding }), + ).resolves.toEqual(f.binding); + }); +}); + +describe("registered provider invocation shell", () => { + it("rejects retained HTTP before running a credential-bearing probe", async () => { + const f = registeredInferenceFixture(true); + const shell = recordingProbeShell("200", "{}", { VENDOR_API_KEY: f.handle }); + await expect( + probeSandboxInferenceInvocation( + { + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + provider: f.binding.providerName, + model: "vendor/model", + preferredInferenceApi: f.binding.inferenceApi, + nativeProvider: true, + registeredInferenceProviderAttachment: { + ...f.binding, + endpointUrl: "http://native.example:443/v1", + }, + }, + { execute: shell.execute }, + ), + ).rejects.toThrow("HTTPS endpoint"); + expect(shell.requests).toEqual([]); + }); + + it("sends the scoped handle only to the recorded endpoint", async () => { + const f = registeredInferenceFixture(true); + const shell = recordingProbeShell( + "200", + JSON.stringify({ choices: [{ message: { content: "ready" } }] }), + { VENDOR_API_KEY: f.handle }, + ); + await expect( + probeSandboxInferenceInvocation( + { + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + provider: f.binding.providerName, + model: "vendor/model", + preferredInferenceApi: f.binding.inferenceApi, + nativeProvider: true, + registeredInferenceProviderAttachment: f.binding, + }, + { execute: shell.execute }, + ), + ).resolves.toEqual({ ok: true }); + expect(shell.destinations()).toEqual(["https://native.example/v1/chat/completions"]); + expect(shell.requests[0]).toContain("Authorization: Bearer " + f.handle); + }); + + it.each([ + "raw-credential-canary", + "openshell:resolve:env:VENDOR_API_KEY", + "openshell:resolve:env:v7_OTHER_API_KEY", + "openshell:resolve:env:v7_VENDOR_API_KEY\nraw-credential-canary", + ])("rejects an invalid credential before curl (%s)", async (credential) => { + const f = registeredInferenceFixture(true); + const shell = recordingProbeShell("200", "{}", { VENDOR_API_KEY: credential }); + const result = await probeSandboxInferenceInvocation( + { + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + provider: f.binding.providerName, + model: "vendor/model", + preferredInferenceApi: f.binding.inferenceApi, + nativeProvider: true, + registeredInferenceProviderAttachment: f.binding, + }, + { execute: shell.execute }, + ); + expect(result.ok).toBe(false); + expect(JSON.stringify(result)).not.toContain(credential); + expect(shell.requests).toEqual([]); + }); +}); + +describe("registered credential handle read", () => { + it.each([ + ["scoped", "openshell:resolve:env:v7_VENDOR_API_KEY", true], + ["raw", "raw-credential-canary", false], + ["unscoped", "openshell:resolve:env:VENDOR_API_KEY", false], + ["wrong environment", "openshell:resolve:env:v7_OTHER_KEY", false], + ["multiline", "openshell:resolve:env:v7_VENDOR_API_KEY\nraw-credential-canary", false], + ])("reads the actual shell result for %s", async (_label, value, valid) => { + const f = registeredInferenceFixture(); + vi.mocked(captureSanitizedResolvedOpenshellAsync).mockImplementation(async (args) => { + expect(args.slice(0, 9)).toEqual([ + "sandbox", + "exec", + "--name", + f.binding.sandboxName, + "-g", + f.binding.gatewayName, + "--", + "sh", + "-c", + ]); + const result = spawnSync("/bin/sh", ["-c", String(args[9])], { + encoding: "utf8", + env: { PATH: process.env.PATH, VENDOR_API_KEY: String(value) }, + }); + return { + status: result.status ?? 1, + output: result.stdout, + stdout: result.stdout, + stderr: result.stderr, + }; + }); + const observed = await readRegisteredInferenceCredentialHandle(f.binding, async () => {}).then( + (handle) => ({ ok: true, handle }), + (error) => ({ ok: false, message: String(error) }), + ); + expect(observed.ok).toBe(valid); + expect(observed).toEqual( + valid + ? { ok: true, handle: value } + : { ok: false, message: expect.stringContaining("did not expose a provider-bound") }, + ); + }); +}); + +describe("registered native API dialects", () => { + it.each([ + { + api: "openai-responses" as const, + path: "responses", + body: { output: [{ type: "message", content: [{ type: "output_text", text: "ready" }] }] }, + header: "Authorization: Bearer ", + }, + { + api: "anthropic-messages" as const, + path: "messages", + body: { type: "message", content: [{ type: "text", text: "ready" }] }, + header: "x-api-key: ", + }, + ])("uses $api native request and response semantics", async ({ api, path, body, header }) => { + const f = registeredInferenceFixture(true); + const binding = { ...f.binding, inferenceApi: api }; + const shell = recordingProbeShell("200", JSON.stringify(body), { VENDOR_API_KEY: f.handle }); + const result = await probeSandboxInferenceInvocation( + { + sandboxName: binding.sandboxName, + gatewayName: binding.gatewayName, + provider: binding.providerName, + model: "vendor/model", + preferredInferenceApi: api, + registeredInferenceProviderAttachment: binding, + }, + { execute: shell.execute }, + ); + expect(result).toEqual({ ok: true }); + expect(shell.destinations()).toEqual(["https://native.example/v1/" + path]); + expect(shell.requests[0]).toContain(header + f.handle); + }); +}); diff --git a/src/lib/actions/inference-set/registered.ts b/src/lib/actions/inference-set/registered.ts new file mode 100644 index 00000000000..f5849e84c2d --- /dev/null +++ b/src/lib/actions/inference-set/registered.ts @@ -0,0 +1,264 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; +import { + normalizeRegisteredInferenceProviderAttachment, + attachRegisteredInferenceProvider, + detachRegisteredInferenceProvider, + verifyRegisteredInferenceProviderAttachment, + registeredInferenceProviderRevision, + resolveRegisteredInferenceProvider, +} from "../../inference/native-provider/registered"; +import type { RegisteredInferenceProviderAttachment } from "../../inference/native-provider/registered"; +import { readNativeCredentialHandle } from "../inference-set-provider"; +import { createRegisteredSandboxIdentityRevalidation } from "../../onboard/sandbox-recreate-transaction"; +import { observeSandboxOnGateway } from "../../onboard/sandbox-recreate-probe"; +import type { InferenceSetDeps } from "../inference-set"; +import type { SandboxEntry } from "../../state/registry"; +export type { SandboxRecreateObservation } from "../../onboard/sandbox-recreate-transaction"; + +export function captureRegisteredSelectionIdentity(entry: SandboxEntry, deps: InferenceSetDeps) { + return createRegisteredSandboxIdentityRevalidation(entry, { + readRegistration: deps.getSandbox, + observe: + deps.observeSandboxIdentity ?? + ((sandboxName, gatewayName) => + observeSandboxOnGateway({ sandboxName, gatewayName }, undefined, undefined, 15_000)), + }); +} + +// Registered transitions also detach and restore NemoClaw-owned provider access. +export function registeredSelectionProviderAdapter( + adapter: OpenShellProviderAdapter, + revalidateSandboxIdentity: (operation: string) => void, +): OpenShellProviderAdapter { + return { + ...adapter, + async ensureProviderPolicyComposition(request) { + revalidateSandboxIdentity("compose provider policy for registered inference selection"); + const result = await adapter.ensureProviderPolicyComposition(request); + revalidateSandboxIdentity("verify provider policy for registered inference selection"); + return result; + }, + async attachProvider(request) { + revalidateSandboxIdentity("attach provider for registered inference selection"); + const result = await adapter.attachProvider(request); + revalidateSandboxIdentity("verify provider attachment for registered inference selection"); + return result; + }, + async detachProvider(request) { + revalidateSandboxIdentity("detach provider for registered inference selection"); + const result = await adapter.detachProvider(request); + revalidateSandboxIdentity("verify provider detachment for registered inference selection"); + return result; + }, + }; +} + +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : {}; +} + +function referencedEnvironment(value: unknown): string | undefined { + const ref = record(value); + if (ref.source === "env" && ref.provider === "default" && typeof ref.id === "string") + return ref.id; + if (typeof value !== "string") return undefined; + return ( + /^\$\{([A-Z][A-Z0-9_]{0,255})\}$/u.exec(value)?.[1] ?? + /^openshell:resolve:env:(?:(?:v[0-9]{1,20}|s[a-f0-9]{64})_)?([A-Z][A-Z0-9_]{0,255})$/u.exec( + value, + )?.[1] + ); +} + +function selectMetadata( + explicit: string | null | undefined, + configured: unknown, + label: string, +): string { + if (explicit && configured && explicit !== configured) + throw new Error(`Registered provider ${label} disagrees with its native configuration`); + const value = explicit || configured; + if (typeof value !== "string" || !value) + throw new Error( + `Registered provider requires explicit ${label} metadata in its native configuration or inference set options`, + ); + return value; +} + +export async function resolveRegisteredSelection(input: { + provider: string; + sandboxName: string; + gatewayName: string; + agentName: string; + config: Record; + endpointUrl?: string | null; + inferenceApi?: string | null; + credentialEnv?: string | null; + previous?: unknown; + adapter: OpenShellProviderAdapter; +}): Promise { + const previous = normalizeRegisteredInferenceProviderAttachment(input.previous); + if (input.previous !== undefined && !previous) + throw new Error("Recorded registered provider binding is invalid"); + if ( + previous && + (previous.sandboxName !== input.sandboxName || previous.gatewayName !== input.gatewayName) + ) + throw new Error("Recorded registered provider belongs to another sandbox or gateway"); + const models = record(input.config.models); + const hermesCustom = Array.isArray(input.config.custom_providers) + ? input.config.custom_providers.find((entry) => record(entry).name === input.provider) + : undefined; + const configured = + input.agentName === "openclaw" + ? record(record(models.providers)[input.provider]) + : record(record(input.config.providers)[input.provider] ?? hermesCustom); + const retained = previous?.providerName === input.provider ? previous : undefined; + const endpoint = + configured.baseUrl ?? + configured.base_url ?? + (input.agentName === "hermes" ? configured.api : undefined) ?? + retained?.endpointUrl; + const mode = configured.api_mode ?? configured.transport; + const hermesApi = + mode === "anthropic_messages" + ? "anthropic-messages" + : mode === "codex_responses" + ? "openai-responses" + : (mode === "chat_completions" || mode === undefined) && endpoint + ? "openai-completions" + : undefined; + const api = (input.agentName === "hermes" ? hermesApi : configured.api) ?? retained?.inferenceApi; + const headers = { ...record(configured.headers), ...record(configured.default_headers) }; + if ( + Object.keys(headers).some((key) => ["authorization", "x-api-key"].includes(key.toLowerCase())) + ) + throw new Error( + "Registered native configuration must use its provider credential reference instead of an authentication header override", + ); + const key = configured.apiKey ?? configured.api_key; + const env = referencedEnvironment(key) ?? retained?.credentialEnv; + if (key !== undefined && !referencedEnvironment(key) && !retained) + throw new Error( + "Registered provider configuration must name its OpenShell credential environment variable; raw credentials are not imported", + ); + return resolveRegisteredInferenceProvider({ + adapter: input.adapter, + gatewayName: input.gatewayName, + sandboxName: input.sandboxName, + providerName: input.provider, + endpointUrl: selectMetadata(input.endpointUrl, endpoint, "endpoint URL"), + inferenceApi: selectMetadata(input.inferenceApi, api, "API family"), + credentialEnv: selectMetadata(input.credentialEnv, env, "credential environment"), + previous: retained, + }); +} + +/** Read only the selected OpenShell-issued handle, never a host or raw credential. */ +export async function readRegisteredInferenceCredentialHandle( + attachment: RegisteredInferenceProviderAttachment, + sleep: (milliseconds: number) => Promise, +): Promise { + const env = attachment.credentialEnv; + const pattern = new RegExp(`^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_${env}$`, "u"); + return readNativeCredentialHandle(attachment.sandboxName, attachment.gatewayName, { + script: `if [ -z "\${${env}:-}" ]; then printf "%s\\n" NEMOCLAW_NATIVE_PROVIDER_HANDLE_PENDING >&2; exit 75; fi; case "$${env}" in *[!a-zA-Z0-9:_]*) exit 2;; esac; printf "%s" "$${env}" | LC_ALL=C grep -E '^openshell:resolve:env:(v[0-9]{1,20}|s[a-f0-9]{64})_${env}$'`, + pattern, + providerLabel: "registered inference", + sleep, + }); +} + +export { + attachRegisteredInferenceProvider, + detachRegisteredInferenceProvider, + verifyRegisteredInferenceProviderAttachment, + registeredInferenceProviderRevision, + recordedRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; + +export async function readVerifiedRegisteredCredentialHandle(input: { + revalidateSandboxIdentity?: (operation: string) => void; + attachment: RegisteredInferenceProviderAttachment; + adapter: OpenShellProviderAdapter; + read?: typeof readRegisteredInferenceCredentialHandle; + sleep: (milliseconds: number) => Promise; +}): Promise { + const expected = input.attachment; + const revision = await registeredInferenceProviderRevision({ adapter: input.adapter, expected }); + input.revalidateSandboxIdentity?.("read registered inference credential handle"); + const handle = await (input.read ?? readRegisteredInferenceCredentialHandle)( + expected, + input.sleep, + ); + input.revalidateSandboxIdentity?.("verify registered inference credential handle"); + await verifyRegisteredInferenceProviderAttachment({ ...input, expected }); + if ( + revision !== (await registeredInferenceProviderRevision({ adapter: input.adapter, expected })) + ) + throw new Error("Registered provider changed while reading its credential handle"); + return handle; +} + +export async function detachPreviousRegisteredSelection(input: { + revalidateSandboxIdentity?: (operation: string) => void; + adapter: OpenShellProviderAdapter; + previous?: RegisteredInferenceProviderAttachment; + next?: RegisteredInferenceProviderAttachment; +}): Promise { + if (!input.previous || input.previous.providerId === input.next?.providerId) return false; + return detachRegisteredInferenceProvider({ ...input, expected: input.previous }); +} + +export async function rollbackRegisteredSelection(input: { + revalidateSandboxIdentity?: (operation: string) => void; + adapter: OpenShellProviderAdapter; + committed: boolean; + attachment?: RegisteredInferenceProviderAttachment; + changed: boolean; +}): Promise { + if (input.committed) return; + if (input.changed && input.attachment) + await detachRegisteredInferenceProvider({ ...input, expected: input.attachment }); +} + +export async function restorePreviousRegisteredSelection(input: { + revalidateSandboxIdentity?: (operation: string) => void; + adapter: OpenShellProviderAdapter; + committed: boolean; + previous?: RegisteredInferenceProviderAttachment; + previousDetached: boolean; + cleanupError: unknown; + originalError: unknown; +}): Promise { + if (input.committed || !input.previousDetached || !input.previous) return; + // A failed cleanup cannot establish that its credential environment is free. + if (input.cleanupError !== input.originalError) throw input.cleanupError; + await attachRegisteredInferenceProvider({ ...input, expected: input.previous }); +} + +/** Capture only the selected operator model; prior model limits are not defaults. */ +export function registeredOpenclawModelConfiguration( + config: Record, + binding: RegisteredInferenceProviderAttachment, + model: string, +): { model: Record; compatibility: Record | null } { + const selected = record(record(record(config.models).providers)[binding.providerName]); + if (selected.baseUrl !== binding.endpointUrl || selected.api !== binding.inferenceApi) + throw new Error( + "Selected registered OpenClaw configuration does not match its recorded provider binding.", + ); + const models = Array.isArray(selected.models) ? selected.models : []; + const entry = record(models.find((value) => record(value).id === model)); + const compatibility = + entry.compat && typeof entry.compat === "object" && !Array.isArray(entry.compat) + ? record(entry.compat) + : null; + return { model: entry, compatibility }; +} diff --git a/src/lib/actions/inference/native-local.ts b/src/lib/actions/inference/native-local.ts index debcae15ce9..e1746db5004 100644 --- a/src/lib/actions/inference/native-local.ts +++ b/src/lib/actions/inference/native-local.ts @@ -1,11 +1,19 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { nativeModelRouterEndpoint } from "../../inference/native-local/model-router"; +import { + assertNativeModelRouterOwner, + prepareNativeModelRouterSelection, +} from "../../onboard/routed-inference"; import { requireNativeProviderPolicy } from "../../adapters/openshell/provider-policy"; import { InferenceSetError } from "../inference-set-error"; import type { SandboxEntry } from "../../state/registry"; import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; -import { resolveHermesLocalSwitchRuntime } from "../inference-set/local-runtime"; +import { + resolveHermesLocalSwitchRuntime, + type NimSwitchSelection, +} from "../inference-set/local-runtime"; import { normalizeNativeLocalProviderAttachment, nativeLocalIdentity, @@ -18,8 +26,12 @@ import { ensureNativeLocalProviderAttached, detachNativeLocalProvider, } from "../../inference/native-local/profile"; +import { gatewayReachableCompatibleEndpointUrl } from "../../inference/native-local/eligibility"; import { prepareNativeLocalSelection } from "../../inference/native-local/selection"; -import type { InferenceSetProviderBinding } from "../inference-set-route-containment"; +import { + usesLoopbackNoAuthProxyRoute, + type InferenceSetProviderBinding, +} from "../inference-set-route-containment"; export async function prepareNativeLocalSwitch(input: { provider: string; @@ -28,31 +40,110 @@ export async function prepareNativeLocalSwitch(input: { sandboxName: string; previous?: NativeLocalProviderAttachment; binding: InferenceSetProviderBinding | null; + entry?: Pick; + endpointUrl?: string | null; + credentialEnv?: string | null; + inferenceApi?: string | null; + explicitCustomRoute?: boolean; + nimSelection?: NimSwitchSelection; adapter: OpenShellProviderAdapter; resolveCredentialValue: (name: string) => string; readAuthority?: Parameters[0]["readAuthority"]; writeAuthority?: Parameters[0]["writeAuthority"]; }) { + let binding = input.binding; + const ownedProxy = Boolean( + input.previous && input.entry && usesLoopbackNoAuthProxyRoute(input.entry, input.provider), + ); if ( - input.previous?.provider === input.provider && - !input.binding && - input.previous.profileId === nativeLocalIdentity(input.previous).profileId + !binding && + input.provider === "compatible-endpoint" && + (!input.previous || input.explicitCustomRoute) && + input.endpointUrl && + input.credentialEnv ) { - return { - ...(await ensureNativeLocalProviderAttached({ - adapter: input.adapter, - expected: input.previous, - sandboxName: input.sandboxName, - })), - previousDetached: false, + const recordedMapping = + input.previous && + input.entry?.endpointUrl && + new URL(input.endpointUrl).href.replace(/\/+$/, "") === + new URL(input.entry.endpointUrl).href.replace(/\/+$/, "") && + gatewayReachableCompatibleEndpointUrl(input.provider, input.endpointUrl) === + input.previous.endpointUrl; + const reuseRecorded = + input.previous && + (ownedProxy || recordedMapping) && + (input.previous.api ?? "openai-completions") === (input.inferenceApi ?? "openai-completions"); + if (!reuseRecorded) + binding = { + baseUrl: ownedProxy || recordedMapping ? input.previous!.endpointUrl : input.endpointUrl, + credentialEnv: input.credentialEnv, + token: input.resolveCredentialValue(input.credentialEnv), + providerType: "openai", + }; + } + if (input.previous?.provider === input.provider && !binding) { + if ( + input.provider === "compatible-endpoint" && + (input.previous.api ?? "openai-completions") !== (input.inferenceApi ?? "openai-completions") + ) + throw new InferenceSetError( + "Recorded compatible provider API does not match its native attachment.", + ); + if (nativeModelRouterEndpoint(input.provider)) { + const runtime = resolveHermesLocalSwitchRuntime( + input.provider, + null, + input.resolveCredentialValue, + ); + if (runtime.endpointUrl !== input.previous.endpointUrl || !runtime.credentialValue) + throw new InferenceSetError( + "Model Router endpoint ownership changed. Recreate this sandbox before changing inference.", + 2, + ); + assertNativeModelRouterOwner(runtime.endpointUrl, runtime.credentialValue); + } + + if (input.previous.profileId === nativeLocalIdentity(input.previous).profileId) + return { + ...(await ensureNativeLocalProviderAttached({ + adapter: input.adapter, + expected: input.previous, + sandboxName: input.sandboxName, + })), + previousDetached: false, + }; + } + if ( + !binding && + input.provider === "compatible-endpoint" && + input.previous?.provider === input.provider + ) { + const credentialEnv = input.credentialEnv ?? input.entry?.credentialEnv; + const token = credentialEnv ? input.resolveCredentialValue(credentialEnv) : ""; + if (!credentialEnv || !token) + throw new InferenceSetError( + "The recorded compatible endpoint credential is required to replace its IPv4-only profile.", + 2, + ); + binding = { + baseUrl: input.previous.endpointUrl, + credentialEnv, + token, + providerType: "openai", }; } const runtime = resolveHermesLocalSwitchRuntime( input.provider, - input.binding, + binding, input.resolveCredentialValue, + input.nimSelection, ); - const receipt = await prepareNativeLocalSelection({ + if (!runtime.endpointUrl) throw new Error("Native local inference endpoint is missing."); + const receipt = await ( + nativeModelRouterEndpoint(input.provider) + ? prepareNativeModelRouterSelection + : prepareNativeLocalSelection + )({ adapter: input.adapter, binding: { provider: input.provider as NativeLocalProvider, @@ -61,9 +152,14 @@ export async function prepareNativeLocalSwitch(input: { endpointUrl: runtime.endpointUrl, ...(input.agentName === "hermes" ? { agent: "hermes" as const } : {}), authMode: runtime.authMode, + ...(input.agentName === "hermes" && + input.provider === "compatible-endpoint" && + input.inferenceApi === "openai-responses" + ? { api: "openai-responses" as const } + : {}), }, credentialValue: runtime.credentialValue || null, - ownedProxy: runtime.ownedProxy, + ownedProxy: ownedProxy || runtime.ownedProxy, readAuthority: input.readAuthority, writeAuthority: input.writeAuthority, }); @@ -205,6 +301,19 @@ export async function prepareNativeLocalSelectionBoundary(input: { 2, ); } + if ( + selectingNativeLocal && + previous && + input.endpointUrl && + usesLoopbackNoAuthProxyRoute(input.entry, input.provider) && + new URL(input.endpointUrl).href.replace(/\/+$/, "") !== + new URL(input.entry.endpointUrl!).href.replace(/\/+$/, "") + ) { + throw new InferenceSetError( + "The no-auth proxy is bound to the recorded backend. Reuse that endpoint; inference set cannot change its host, port, or path.", + 2, + ); + } if (selectingNativeLocal) await (input.requirePolicy ?? requireNativeProviderPolicy)(input.gatewayName); return { previous, selectingNativeLocal }; diff --git a/src/lib/actions/sandbox/agents/managed-workload-rebuild-profile.ts b/src/lib/actions/sandbox/agents/managed-workload-rebuild-profile.ts index 6fb31e8d6e7..4bd7f22df87 100644 --- a/src/lib/actions/sandbox/agents/managed-workload-rebuild-profile.ts +++ b/src/lib/actions/sandbox/agents/managed-workload-rebuild-profile.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredOpenclawModelConfiguration } from "../../inference-set/registered"; +import { readSandboxConfig, resolveAgentConfig } from "../../../sandbox/config"; import { recordedNativeProviderAttachment } from "../../../inference/native-provider/recorded-selection"; import { rebindLoopbackDashboardUrlPort } from "../../../dashboard/url"; import { resolveContextWindowForModel } from "../../../inference/context-window"; @@ -20,6 +22,8 @@ import type { RebuildRecreateOnboardOpts } from "../rebuild-gpu-opt-out"; import type { RebuildTargetConfig } from "../rebuild-target-preflight"; export const managedRebuildProfileDependencies = { + readSandboxConfig, + resolveAgentConfig, resolveContextWindowForModel, resolveManagedStartupInferenceRoute, }; @@ -123,13 +127,26 @@ export function prepareManagedRebuildProfileHandoff(input: { recordedExternalUrl: input.recordedDashboardExternalUrl, }); const nativeAttachment = recordedNativeProviderAttachment(resumeConfig); + const registered = resumeConfig.registeredInferenceProviderAttachment; + let selected: ReturnType | undefined; + if (registered && agent === "openclaw") { + const config = managedRebuildProfileDependencies.readSandboxConfig( + registered.sandboxName, + managedRebuildProfileDependencies.resolveAgentConfig(registered.sandboxName), + registered.gatewayName, + ); + selected = registeredOpenclawModelConfiguration(config, registered, resumeConfig.model); + } const inference = managedRebuildProfileDependencies.resolveManagedStartupInferenceRoute( agent, resumeConfig.provider, resumeConfig.model, resumeConfig.preferredInferenceApi, - Boolean(nativeAttachment || resumeConfig.nativeLocalProviderAttachment), - resumeConfig.nativeLocalProviderAttachment?.endpointUrl ?? nativeAttachment?.endpointUrl, + Boolean(registered || nativeAttachment || resumeConfig.nativeLocalProviderAttachment), + registered?.endpointUrl ?? + resumeConfig.nativeLocalProviderAttachment?.endpointUrl ?? + nativeAttachment?.endpointUrl, + registered, ); const upstreamProvider = agent === "hermes" && resumeConfig.provider === "hermes-provider" @@ -147,7 +164,7 @@ export function prepareManagedRebuildProfileHandoff(input: { input.gatewayPort, ) : null; - if (openClawRouteChanged && currentOpenClawContextWindow === null) { + if (openClawRouteChanged && !selected && currentOpenClawContextWindow === null) { throw new Error( `Cannot determine a context window for the current OpenClaw target '${resumeConfig.provider}/${resumeConfig.model}'.`, ); @@ -165,7 +182,10 @@ export function prepareManagedRebuildProfileHandoff(input: { agent === "langchain-deepagents-code" ? resumeConfig.endpointUrl : null, api: requireManagedStartupInferenceApi(inference.inferenceApi), primaryModelRef: agent === "openclaw" ? inference.primaryModelRef : null, - compatibility: agent === "openclaw" ? (inference.inferenceCompat ?? {}) : null, + compatibility: + agent === "openclaw" + ? (selected?.compatibility ?? inference.inferenceCompat ?? {}) + : null, }, chatUiUrl, effectiveDashboardPort, @@ -186,6 +206,7 @@ export function prepareManagedRebuildProfileHandoff(input: { input.environment, agent === "openclaw" ? { + ...(selected ? { openClawSelectedModel: selected.model } : {}), ...(currentOpenClawContextWindow === null ? {} : { openClawContextWindow: currentOpenClawContextWindow }), diff --git a/src/lib/actions/sandbox/connect.ts b/src/lib/actions/sandbox/connect.ts index 16c971d8799..ccdb1407ee6 100644 --- a/src/lib/actions/sandbox/connect.ts +++ b/src/lib/actions/sandbox/connect.ts @@ -3192,7 +3192,10 @@ async function prepareConnectSandboxWithinLifecycleFence( ); if (publication.diagnostic) { console.error( - ` Readiness evidence: stage=${publication.diagnostic.stage} reason=${publication.diagnostic.reason}`, + ` Readiness evidence: stage=${publication.diagnostic.stage} reason=${publication.diagnostic.reason}` + + (publication.diagnostic.pairingFailureCategory + ? ` pairing=${publication.diagnostic.pairingFailureCategory}` + : ""), ); } process.exit(1); diff --git a/src/lib/actions/sandbox/destroy-container-identity.test.ts b/src/lib/actions/sandbox/destroy-container-identity.test.ts index 44fc8d6060e..bfc1d85d194 100644 --- a/src/lib/actions/sandbox/destroy-container-identity.test.ts +++ b/src/lib/actions/sandbox/destroy-container-identity.test.ts @@ -8,6 +8,7 @@ import { classifyDestroyContainerIdentity, type DestroyContainerIdentityVerdict, formatAmbiguousDestroyIdentity, + isSameDestroyContainerIdentityProof, } from "./destroy-presence"; type Row = { @@ -15,12 +16,15 @@ type Row = { managedBy?: string; workspace?: string; sandboxId?: string; + namespace?: string; + isolationRole?: string; }; function observeRows(rows: Row[], malformedRows = 0) { return { status: "observed" as const, rows: rows.map((row) => ({ + ...row, id: row.id, managedBy: row.managedBy ?? "", workspace: row.workspace ?? "", @@ -44,6 +48,14 @@ const FOREIGN = { sandboxId: "", } as const; +const WORKLOAD = { ...MANAGED, namespace: "default", isolationRole: "sandbox" }; +const SUPERVISOR = { + ...WORKLOAD, + id: "bbbb000000000000", + workspace: "", + isolationRole: "supervisor", +}; + function expectAmbiguous( verdict: DestroyContainerIdentityVerdict, ): Extract { @@ -70,6 +82,58 @@ describe("classifyDestroyContainerIdentity", () => { }); }); + it("qualifies the workload and its supervisor with the workload workspace first", () => { + expect( + classifyDestroyContainerIdentity("destroytest", observeRows([SUPERVISOR, WORKLOAD])), + ).toEqual({ status: "recovery", identities: [WORKLOAD, SUPERVISOR] }); + }); + + it.each([ + ["another sandbox ID", { ...SUPERVISOR, sandboxId: "other" }], + ["another namespace", { ...SUPERVISOR, namespace: "other" }], + ["a missing namespace", { ...SUPERVISOR, namespace: "" }], + ["a foreign owner", { ...SUPERVISOR, managedBy: "foreign" }], + ["a duplicate workload role", { ...SUPERVISOR, isolationRole: "sandbox" }], + ["an unknown role", { ...SUPERVISOR, isolationRole: "staging" }], + ["a duplicate container ID", { ...SUPERVISOR, id: WORKLOAD.id }], + ["an unexpected supervisor workspace", { ...SUPERVISOR, workspace: "default" }], + ])("refuses the supervisor with %s", (_label, supervisor) => { + expect( + classifyDestroyContainerIdentity("destroytest", observeRows([WORKLOAD, supervisor])).status, + ).toBe("ambiguous"); + }); + + it("refuses a paired runtime that conflicts with retained recovery authority", () => { + expect( + classifyDestroyContainerIdentity( + "destroytest", + observeRows([WORKLOAD, SUPERVISOR]), + fingerprintOpenShellSandboxId("other")!, + ).status, + ).toBe("ambiguous"); + }); + + it("refuses a second supervisor beside a qualified pair", () => { + expect( + classifyDestroyContainerIdentity( + "destroytest", + observeRows([WORKLOAD, SUPERVISOR, { ...SUPERVISOR, id: "cccc000000000000" }]), + ).status, + ).toBe("ambiguous"); + }); + + it.each([{ namespace: "changed" }, { isolationRole: "sandbox" }])( + "detects role or namespace changes in a pinned container proof: %j", + (change) => { + expect( + isSameDestroyContainerIdentityProof( + { identities: [WORKLOAD, SUPERVISOR] }, + { identities: [WORKLOAD, { ...SUPERVISOR, ...change }] }, + ), + ).toBe(false); + }, + ); + it("does not treat native Podman ownership as Docker compatibility", () => { const podmanManaged = { ...MANAGED, managedBy: "true" }; diff --git a/src/lib/actions/sandbox/destroy-deepagents-state-cleanup.test.ts b/src/lib/actions/sandbox/destroy-deepagents-state-cleanup.test.ts index c4642c13331..76a5669c613 100644 --- a/src/lib/actions/sandbox/destroy-deepagents-state-cleanup.test.ts +++ b/src/lib/actions/sandbox/destroy-deepagents-state-cleanup.test.ts @@ -12,6 +12,24 @@ import { resetDestroyModuleCache, } from "../../../../test/helpers/destroy-flow-test-harness"; +import type { SandboxWorkloadReceipt } from "../../state/registry"; + +const managedHermesWorkload = { + schemaVersion: 1, + kind: "managed-image", + reference: `ghcr.io/nvidia/nemoclaw/hermes@sha256:${"a".repeat(64)}`, + platform: "linux/amd64", + release: "v0.0.0", + sourceRevision: "a".repeat(40), + sourceCohort: "test-cohort", + capabilityContractVersion: 1, + startupProfileContractVersion: 1, + encodedProfile: "e30", + startupProfileSha256: "b".repeat(64), + credentialProxyReplayRequired: true, + shared: true, +} satisfies SandboxWorkloadReceipt; + describe("native agent destroy state cleanup", { timeout: 15_000 }, () => { let testHome: string; @@ -187,4 +205,47 @@ describe("native agent destroy state cleanup", { timeout: 15_000 }, () => { expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); expect(harness.errorSpy.mock.calls.flat().join("\n")).toContain("registry entry was preserved"); }); + it("preserves the registry when owned managed Hermes state-volume cleanup fails", async () => { + const harness = createDestroyHarness({ + agent: "hermes", + openshellDriver: "docker", + workload: managedHermesWorkload, + managedAgentStateVolumeCleanupResults: [ + { + status: "failed", + detail: "volume is still in use", + volumeName: "nemoclaw-hermes-state-v1-alpha", + }, + ], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow("process.exit(1)"); + + expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); + expect(harness.errorSpy.mock.calls.map((call) => String(call[0])).join("\n")).toContain( + "volume is still in use", + ); + }); + + it("leaves a foreign same-name Hermes volume untouched and completes registry cleanup", async () => { + const harness = createDestroyHarness({ + agent: "hermes", + openshellDriver: "docker", + workload: managedHermesWorkload, + managedAgentStateVolumeCleanupResults: [ + { + status: "not-owned", + detail: "the exact NemoClaw ownership labels are absent or changed", + volumeName: "nemoclaw-hermes-state-v1-alpha", + }, + ], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); + + expect(harness.warnSpy.mock.calls.map((call) => String(call[0])).join("\n")).toContain( + "Left managed state volume 'nemoclaw-hermes-state-v1-alpha' untouched", + ); + expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha"); + }); }); diff --git a/src/lib/actions/sandbox/destroy-execution.ts b/src/lib/actions/sandbox/destroy-execution.ts index 42557d00194..430a06484d0 100644 --- a/src/lib/actions/sandbox/destroy-execution.ts +++ b/src/lib/actions/sandbox/destroy-execution.ts @@ -213,6 +213,7 @@ export function wipeAgentNativeHome( ' for keep in "$@"; do [ "${keep#"$candidate"/}" = "$keep" ] || return 0; done', " return 1", "}", + "cleanup_failed=0", "clean_dir() {", " local directory entry", ' directory="$1"; shift', @@ -226,7 +227,7 @@ export function wipeAgentNativeHome( " fi", ' clean_dir "$entry" "$@"', " else", - ' rm -rf -- "$entry"', + ' if ! rm -rf -- "$entry"; then cleanup_failed=1; fi', " fi", " done", "}", @@ -245,6 +246,7 @@ export function wipeAgentNativeHome( " done", "}", 'clean_dir "$root" "$@"', + 'if [ "$cleanup_failed" != 0 ]; then exit 1; fi', 'if ! verify_dir "$root" "$@"; then', ' echo "agent native root retains sandbox-owned state" >&2', " exit 22", @@ -719,7 +721,10 @@ export async function executeSandboxDestroy({ try { runtimeProvider = requireRuntimeProviderDestructiveCleanupAuthority( sandboxName, - sandbox, + { + ...sandbox, + openshellDriver: sandbox.openshellDriver ?? expectedRuntimeProviderIdentity?.providerId, + }, runtimeProviders, ).provider; hostLocalInferenceAuthority = prepareSandboxHostLocalInferenceDestroyAuthority( @@ -1084,7 +1089,7 @@ export async function executeSandboxDestroy({ return { ok: false as const, deleteOutput: - `OpenShell deleted sandbox '${sandboxName}', but its host forward ports did not release. ` + + `OpenShell deleted sandbox '${sandboxName}', but release of its host forward ports could not be proved. ` + "The local sandbox record was preserved for recovery.", exitCode: 1, gatewayUnreachable: false, diff --git a/src/lib/actions/sandbox/destroy-flow.test.ts b/src/lib/actions/sandbox/destroy-flow.test.ts index 48289399c2b..1e25ac706da 100644 --- a/src/lib/actions/sandbox/destroy-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-flow.test.ts @@ -26,7 +26,6 @@ import { } from "../../../../test/helpers/destroy-flow-test-harness"; import { serializedLlamaCppHostLocalInferenceReceipt } from "../../../../test/helpers/host-local-inference-receipt"; import { createSandboxHostLocalInferenceProvenance } from "../../state/registry/host-local-inference"; -import type { SandboxWorkloadReceipt } from "../../state/registry"; import * as dockerLlamaCppOperation from "../../onboard/runtime-provider/docker-llama-cpp-operation"; import { createDockerLlamaCppPrivateBridgeController } from "../../onboard/runtime-provider/docker-llama-cpp-private-bridge"; import { prepareManagedLlamaCppRuntimeCleanupForSandbox } from "../../inference/local-model-profile/cleanup"; @@ -40,22 +39,6 @@ import { const enforceRemovedImmutabilityMigrationBoundaryReal = enforceRemovedImmutabilityMigrationBoundary; -const managedHermesWorkload = { - schemaVersion: 1, - kind: "managed-image", - reference: `ghcr.io/nvidia/nemoclaw/hermes@sha256:${"a".repeat(64)}`, - platform: "linux/amd64", - release: "v0.0.0", - sourceRevision: "a".repeat(40), - sourceCohort: "test-cohort", - capabilityContractVersion: 1, - startupProfileContractVersion: 1, - encodedProfile: "e30", - startupProfileSha256: "b".repeat(64), - credentialProxyReplayRequired: true, - shared: true, -} satisfies SandboxWorkloadReceipt; - describe("destroySandbox flow", () => { let exitSpy: MockInstance; let originalGatewayEnv: string | undefined; @@ -554,7 +537,7 @@ describe("destroySandbox flow", () => { it("revalidates schema-4 Portable identity at every destroy checkpoint without Docker preflight (#9189)", async () => { const harness = createDestroyHarness({ - dockerRunResult: { status: 0, stdout: `${"f".repeat(64)}\t\tforeign\t` }, + dockerRunResult: { status: 0, stdout: `${"f".repeat(64)}\t\tforeign\t\t\t` }, portableDestroyAuthority: true, }); @@ -669,78 +652,6 @@ describe("destroySandbox flow", () => { expect(harness.retirePortableLifecycleReceiptSpy).not.toHaveBeenCalled(); }); - it( - "removes the owned managed Hermes state volume after confirmed sandbox deletion", - { timeout: 30_000 }, - async () => { - const harness = createDestroyHarness({ - agent: "hermes", - openshellDriver: "docker", - workload: managedHermesWorkload, - managedAgentStateVolumeCleanupResults: [{ status: "removed" }], - }); - - await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); - - expect(harness.removeManagedAgentStateVolumesSpy).toHaveBeenCalledWith( - { - agentName: "hermes", - runtimeProviderId: "docker", - sandboxName: "alpha", - workloadKind: "managed-image", - }, - { runtimeProviders: expect.any(Object) }, - ); - expect(harness.removeManagedAgentStateVolumesSpy.mock.invocationCallOrder[0]).toBeLessThan( - harness.removeSandboxSpy.mock.invocationCallOrder[0], - ); - }, - ); - - it("preserves the registry when owned managed Hermes state-volume cleanup fails", async () => { - const harness = createDestroyHarness({ - agent: "hermes", - openshellDriver: "docker", - workload: managedHermesWorkload, - managedAgentStateVolumeCleanupResults: [ - { - status: "failed", - detail: "volume is still in use", - volumeName: "nemoclaw-hermes-state-v1-alpha", - }, - ], - }); - - await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow("process.exit(1)"); - - expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); - expect(harness.errorSpy.mock.calls.map((call) => String(call[0])).join("\n")).toContain( - "volume is still in use", - ); - }); - - it("leaves a foreign same-name Hermes volume untouched and completes registry cleanup", async () => { - const harness = createDestroyHarness({ - agent: "hermes", - openshellDriver: "docker", - workload: managedHermesWorkload, - managedAgentStateVolumeCleanupResults: [ - { - status: "not-owned", - detail: "the exact NemoClaw ownership labels are absent or changed", - volumeName: "nemoclaw-hermes-state-v1-alpha", - }, - ], - }); - - await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); - - expect(harness.warnSpy.mock.calls.map((call) => String(call[0])).join("\n")).toContain( - "Left managed state volume 'nemoclaw-hermes-state-v1-alpha' untouched", - ); - expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha"); - }); - it("runs routed teardown under the gateway and host router-port locks (#9098)", async () => { const harness = createDestroyHarness({ provider: "nvidia-router" }); @@ -912,8 +823,8 @@ describe("destroySandbox flow", () => { it("refuses before destructive work when multiple Docker identities share the name", async () => { const rows = [ - "aaaa000000000000\topenshell\tdefault\tsb-real", - "ffff000000000000\t\tforeign\t", + "aaaa000000000000\topenshell\tdefault\tsb-real\t\t", + "ffff000000000000\t\tforeign\t\t\t", ].join("\n"); const harness = createDestroyHarness({ dockerRunResult: { status: 0, stdout: rows }, @@ -941,8 +852,8 @@ describe("destroySandbox flow", () => { }); it("refuses identity drift after read-only destroy preflight (#8999)", async () => { - const managed = "aaaa000000000000\topenshell\tdefault\tsb-alpha"; - const foreign = "ffff000000000000\t\tforeign\t"; + const managed = "aaaa000000000000\topenshell\tdefault\tsb-alpha\t\t"; + const foreign = "ffff000000000000\t\tforeign\t\t\t"; const harness = createDestroyHarness({ dockerRunResultSequence: [ { status: 0, stdout: managed }, @@ -971,12 +882,12 @@ describe("destroySandbox flow", () => { it.each([ [ "a replacement identity", - { status: 0, stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta" }, + { status: 0, stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta\t\t" }, ], ["container absence", { status: 0, stdout: "" }], ["a failed revalidation probe", { status: 1, stderr: "daemon unavailable" }], ])("refuses %s before sandbox mutation", async (_scenario, changedIdentity) => { - const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha" }; + const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha\t\t" }; const harness = createDestroyHarness({ dockerRunResultSequence: [managed, managed, changedIdentity], }); @@ -996,7 +907,7 @@ describe("destroySandbox flow", () => { const absent = { status: 0, stdout: "" }; const replacement = { status: 0, - stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta", + stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta\t\t", }; const harness = createDestroyHarness({ sandboxPresent: false, @@ -1012,10 +923,10 @@ describe("destroySandbox flow", () => { }); it("restores MCP preparation when identity changes before wipe", async () => { - const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha" }; + const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha\t\t" }; const replacement = { status: 0, - stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta", + stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta\t\t", }; const harness = createDestroyHarness({ mcpServers: ["github"], @@ -1061,7 +972,7 @@ describe("destroySandbox flow", () => { it.each([ [ "a replacement identity", - { status: 0, stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta" }, + { status: 0, stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta\t\t" }, "Container identity changed after managed inference cleanup", ], [ @@ -1072,7 +983,7 @@ describe("destroySandbox flow", () => { ])( "restores MCP preparation and refuses deletion after %s", async (_scenario, changedIdentity, expectedMessage) => { - const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha" }; + const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha\t\t" }; const harness = createDestroyHarness({ mcpServers: ["github"], dockerRunResultSequence: [managed, managed, managed, managed, changedIdentity], @@ -1094,10 +1005,10 @@ describe("destroySandbox flow", () => { ); it("revalidates immediately before delete and reports partial provider preparation", async () => { - const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha" }; + const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha\t\t" }; const replacement = { status: 0, - stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta", + stdout: "bbbb000000000000\topenshell\tdefault\tsb-beta\t\t", }; const harness = createDestroyHarness({ detachedProviders: ["provider-a"], @@ -1128,7 +1039,7 @@ describe("destroySandbox flow", () => { }); it("keeps the final exact probe immediately adjacent to sandbox delete", async () => { - const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha" }; + const managed = { status: 0, stdout: "aaaa000000000000\topenshell\tdefault\tsb-alpha\t\t" }; const trace: string[] = []; let identityProbeCalls = 0; const harness = createDestroyHarness({ @@ -1258,7 +1169,7 @@ describe("destroySandbox flow", () => { dockerOrphanIds: [containerId], dockerRunResult: { status: 0, - stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha\t\t`, }, }); @@ -1271,6 +1182,49 @@ describe("destroySandbox flow", () => { expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha"); }); + it("deletes a sandbox with its verified supervisor and removes both residual IDs", async () => { + const workloadId = "a".repeat(64); + const supervisorId = "b".repeat(64); + const harness = createDestroyHarness({ + dockerOrphanIds: [supervisorId, workloadId], + dockerRunResult: { + status: 0, + stdout: [ + `${supervisorId}\topenshell\t\tsb-alpha\tdefault\tsupervisor`, + `${workloadId}\topenshell\tdefault\tsb-alpha\tdefault\tsandbox`, + ].join("\n"), + }, + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); + + expect(harness.events).toContain("delete"); + expect( + harness.dockerRunSpy.mock.calls.filter(([args]) => args[0] === "rm").map(([args]) => args), + ).toEqual([ + ["rm", "-f", workloadId], + ["rm", "-f", supervisorId], + ]); + expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha"); + }); + + it("refuses supervisor identity drift before deleting a paired sandbox", async () => { + const workload = `${"a".repeat(64)}\topenshell\tdefault\tsb-alpha\tdefault\tsandbox`; + const supervisor = `${"b".repeat(64)}\topenshell\t\tsb-alpha\tdefault\tsupervisor`; + const harness = createDestroyHarness({ + dockerRunResultSequence: [ + { status: 0, stdout: `${workload}\n${supervisor}` }, + { status: 0, stdout: `${workload}\n${supervisor.replace("sb-alpha", "sb-other")}` }, + ], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow("process.exit(1)"); + + expect(harness.events).not.toContain("delete"); + expect(harness.dockerRunSpy.mock.calls.filter(([args]) => args[0] === "rm")).toEqual([]); + expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); + }); + it("preserves registry state when the absent sandbox has a replacement Docker identity (#9073)", async () => { const expectedContainerId = "a".repeat(64); const replacementContainerId = "b".repeat(64); @@ -1279,7 +1233,7 @@ describe("destroySandbox flow", () => { dockerOrphanIds: [replacementContainerId], dockerRunResult: { status: 0, - stdout: `${expectedContainerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${expectedContainerId}\topenshell\tdefault\tsb-alpha\t\t`, }, }); @@ -1301,7 +1255,7 @@ describe("destroySandbox flow", () => { dockerRemoveStatus: 1, dockerRunResult: { status: 0, - stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha\t\t`, }, }; const harness = createDestroyHarness(options); @@ -1330,7 +1284,7 @@ describe("destroySandbox flow", () => { dockerOrphanQueryStatus: 1, dockerRunResult: { status: 0, - stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha\t\t`, }, }); @@ -1352,7 +1306,7 @@ describe("destroySandbox flow", () => { dockerOrphanIds: [containerId], dockerRunResult: { status: 0, - stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha\t\t`, }, }); diff --git a/src/lib/actions/sandbox/destroy-native-home-fallback.test.ts b/src/lib/actions/sandbox/destroy-native-home-fallback.test.ts index 09548b7231b..56b5b156fc8 100644 --- a/src/lib/actions/sandbox/destroy-native-home-fallback.test.ts +++ b/src/lib/actions/sandbox/destroy-native-home-fallback.test.ts @@ -1,7 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it, vi } from "vitest"; +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it, onTestFinished, vi } from "vitest"; vi.mock("./mcp-bridge", async (importOriginal) => ({ ...(await importOriginal()), @@ -51,7 +55,7 @@ describe("native-home destroy fallback", () => { ), )( "completes $agent cleanup as the sandbox owner after $initial", - ({ openshellResult, agent }) => { + ({ agent, openshellResult }) => { const runOpenshell = vi.fn(() => openshellResult); const runPrivileged = vi.fn((_command: readonly string[]) => ({ status: 1, @@ -208,15 +212,27 @@ describe("native-home destroy fallback", () => { }); it.each([ - { agent: "langchain-deepagents-code", sandboxUser: DEEP_AGENTS_CODE_SANDBOX_USER }, - { agent: "openclaw", sandboxUser: "998:998" }, - { agent: "hermes", sandboxUser: "998:999" }, - { agent: undefined, sandboxUser: "998:998" }, - { agent: null, sandboxUser: "998:998" }, - { agent: "", sandboxUser: "998:998" }, + { + agent: "langchain-deepagents-code", + sandboxUser: DEEP_AGENTS_CODE_SANDBOX_USER, + state: "registered", + openshellDriver: "mxc", + }, + { + agent: "langchain-deepagents-code", + sandboxUser: DEEP_AGENTS_CODE_SANDBOX_USER, + state: "pending", + openshellDriver: undefined, + }, + { agent: "hermes", sandboxUser: "998:999", state: "registered", openshellDriver: "mxc" }, + { agent: "openclaw", sandboxUser: "998:998", state: "registered", openshellDriver: "mxc" }, + { agent: undefined, sandboxUser: "998:998", state: "registered", openshellDriver: "mxc" }, + { agent: null, sandboxUser: "998:998", state: "registered", openshellDriver: "mxc" }, + { agent: "", sandboxUser: "998:998", state: "registered", openshellDriver: "mxc" }, + { agent: "hermes", sandboxUser: "998:999", state: "pending", openshellDriver: undefined }, ])( - "destroys $agent through the pinned sandbox user after both cleanup passes fail", - async ({ agent, sandboxUser }) => { + "destroys a $state $agent sandbox through its provider's pinned sandbox user", + async ({ agent, sandboxUser, openshellDriver }) => { const events: string[] = []; const base = createInMemoryRuntimeProviderBundle({ providerId: "mxc", @@ -245,7 +261,11 @@ describe("native-home destroy fallback", () => { }; const bundle = { ...base, - cleanup: { ...base.cleanup, captureDestroyIdentity: () => identity }, + cleanup: { + ...base.cleanup, + captureDestroyIdentity: () => identity, + captureDestroyIdentityByName: () => identity, + }, lifecycle: { ...base.lifecycle, privilegedSandboxControl: { ...control, execute, executeAsSandboxUser }, @@ -254,7 +274,8 @@ describe("native-home destroy fallback", () => { const sandbox: SandboxEntry = { name: "alpha", agent, - openshellDriver: "mxc", + openshellDriver, + ...(openshellDriver === undefined ? { pendingRouteReservation: true as const } : {}), gatewayName: "nemoclaw", lifecycleGeneration: "alpha-generation-1", }; @@ -278,8 +299,10 @@ describe("native-home destroy fallback", () => { const runOpenshell = vi.fn( (args: string[]) => handlers[`${String(args[0])}:${String(args[1])}`]?.() ?? ok, ); + const stopInferenceResources = vi.fn(); + const onMutationStarted = vi.fn(); - await executeSandboxDestroy({ + const result = await executeSandboxDestroy({ force: false, deleteGatewayName: "nemoclaw", getSandbox: () => current, @@ -288,13 +311,15 @@ describe("native-home destroy fallback", () => { sandbox, sandboxConfirmedAbsent: false, sandboxName: "alpha", - stopInferenceResources: vi.fn(), - onMutationStarted: vi.fn(), + stopInferenceResources, + onMutationStarted, runtimeProviders: { mxc: bundle }, expectedRuntimeProviderIdentity: identity, deps: { deleteConvergence: { now: () => 0, sleep: () => undefined } }, }); + expect(result.ok, result.deleteOutput).toBe(true); + expect(sandbox.openshellDriver).toBe(openshellDriver); expect(events.slice(0, 4)).toEqual([ "openshell-exec", "privileged", @@ -311,4 +336,159 @@ describe("native-home destroy fallback", () => { ); }, ); + + it.each([ + { condition: "provider changed", observe: () => ({ providerId: "other" }) }, + { condition: "resource changed", observe: () => ({ resourceHandle: "replacement" }) }, + { condition: "ownership changed", observe: () => ({ ownershipSha256: "replacement" }) }, + { + condition: "probe failed", + observe: () => { + throw new Error("runtime identity probe unavailable"); + }, + }, + ])( + "preserves a pending sandbox without mutation when its identity $condition", + async ({ observe }) => { + const identity = { + schemaVersion: 1 as const, + providerId: "mxc", + resourceHandle: "in-memory:alpha", + ownershipSha256: null, + }; + const base = createInMemoryRuntimeProviderBundle({ + providerId: "mxc", + workloadProfile: { + support: null, + hostArchitectures: ["x64"], + managedImageSelectionPolicy: "prefer-managed", + legacyDockerfileBuilds: false, + }, + }); + const captureDestroyIdentityByName = vi.fn(() => ({ ...identity, ...observe() })); + const sandbox: SandboxEntry = Object.freeze({ name: "alpha", pendingRouteReservation: true }); + const runOpenshell = vi.fn(); + const stopInferenceResources = vi.fn(); + const onMutationStarted = vi.fn(); + const result = await executeSandboxDestroy({ + force: false, + deleteGatewayName: "nemoclaw", + getSandbox: () => sandbox, + runOpenshell, + sandbox, + sandboxConfirmedAbsent: false, + sandboxName: "alpha", + stopInferenceResources, + onMutationStarted, + runtimeProviders: { + mxc: { ...base, cleanup: { ...base.cleanup, captureDestroyIdentityByName } }, + }, + expectedRuntimeProviderIdentity: identity, + }); + expect(result.ok).toBe(false); + expect(result.deleteOutput).toContain("before destroy preparation"); + expect(captureDestroyIdentityByName).toHaveBeenCalledExactlyOnceWith("alpha"); + expect(runOpenshell).not.toHaveBeenCalled(); + expect(stopInferenceResources).not.toHaveBeenCalled(); + expect(onMutationStarted).not.toHaveBeenCalled(); + }, + ); +}); + +function cleanupFixture() { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "native-home-script-")); + onTestFinished(() => fs.rmSync(directory, { recursive: true, force: true })); + const root = path.join(directory, "sandbox"); + fs.mkdirSync(root); + let command: readonly string[] = []; + wipeAgentNativeHome( + "alpha", + "langchain-deepagents-code", + (args) => { + command = args.slice(args.indexOf("--") + 1); + return { status: 0, stdout: "", stderr: "" }; + }, + undefined, + ); + // Execute the production shell with only its filesystem root relocated. + const script = command[2]!.replace("root='/sandbox'", 'root="$NATIVE_HOME_FIXTURE"'); + const protectedPaths = command.slice(4).map((entry) => entry.replace("/sandbox", root)); + const run = (rmMode = "normal") => + spawnSync( + "sh", + [ + "-c", + [ + "rm() {", + ' case "$3" in', + ' */.a-denied) case "$NATIVE_HOME_RM_MODE" in denied) return 1;; retained) return 0;; esac;;', + " esac", + ' /bin/rm "$@"', + "}", + script, + ].join("\n"), + command[3]!, + ...protectedPaths, + ], + { + encoding: "utf8", + timeout: 5_000, + env: { ...process.env, NATIVE_HOME_FIXTURE: root, NATIVE_HOME_RM_MODE: rmMode }, + }, + ); + return { directory, root, run }; +} + +describe("native-home cleanup shell", () => { + it.each([ + { mode: "normal", status: 0, remains: false }, + { mode: "denied", status: 1, remains: true }, + { mode: "retained", status: 22, remains: true }, + ])("finishes eligible entries and verifies remaining state after $mode removal", (testCase) => { + const { root, run } = cleanupFixture(); + fs.mkdirSync(path.join(root, ".deepagents")); + const protectedFile = path.join(root, ".deepagents", ".env"); + fs.writeFileSync(protectedFile, "synthetic protected configuration"); + fs.writeFileSync(path.join(root, ".a-denied"), "first entry"); + fs.writeFileSync(path.join(root, "z-owned"), "later entry"); + + const result = run(testCase.mode); + + expect(result.error).toBeUndefined(); + expect(result.status).toBe(testCase.status); + expect(fs.existsSync(path.join(root, "z-owned"))).toBe(false); + expect(fs.existsSync(path.join(root, ".a-denied"))).toBe(testCase.remains); + expect(fs.readFileSync(protectedFile, "utf8")).toBe("synthetic protected configuration"); + expect(run().status).toBe(0); + expect(fs.existsSync(path.join(root, ".a-denied"))).toBe(false); + expect(fs.readFileSync(protectedFile, "utf8")).toBe("synthetic protected configuration"); + }); + + it.each([ + { + target: "root", + status: 20, + linkPath: (root: string) => { + fs.rmdirSync(root); + return root; + }, + }, + { + target: "protected ancestor", + status: 21, + linkPath: (root: string) => path.join(root, ".deepagents"), + }, + ])("refuses a symlink at the $target without following it", ({ linkPath, status }) => { + const { directory, root, run } = cleanupFixture(); + const outside = path.join(directory, "outside"); + fs.mkdirSync(outside); + fs.writeFileSync(path.join(outside, "retained"), "outside state"); + fs.symlinkSync(outside, linkPath(root)); + + const result = run(); + + expect(result.error).toBeUndefined(); + expect(result.status).toBe(status); + expect(fs.readFileSync(path.join(outside, "retained"), "utf8")).toBe("outside state"); + }); }); diff --git a/src/lib/actions/sandbox/destroy-preflight.ts b/src/lib/actions/sandbox/destroy-preflight.ts index a4ef1f75366..922cde0b9c8 100644 --- a/src/lib/actions/sandbox/destroy-preflight.ts +++ b/src/lib/actions/sandbox/destroy-preflight.ts @@ -30,7 +30,7 @@ import { listHostGatewayRegistryEntries, registryEntryGatewayPort, } from "../../state/gateway-registry"; -import { resolveNemoclawStateGatewayPort } from "../../state/paths"; +import { resolveNemoclawStateDir, resolveNemoclawStateGatewayPort } from "../../state/paths"; import type { acquireOnboardLock, compareAndSwapSession, @@ -154,6 +154,7 @@ export type SandboxDestroyPreflight = { export type SandboxDestroyRegistryAuthority = { entry: SandboxEntry | null; gatewayPort: number; + stateDir: string; getSandbox: typeof registry.getSandbox; listSandboxes: typeof registry.listSandboxes; listGatewayCleanupOwners: typeof registry.listSandboxes; @@ -176,6 +177,7 @@ export function resolveSandboxDestroyRegistryAuthority( return { entry: registry.getSandbox(sandboxName), gatewayPort: resolveNemoclawStateGatewayPort(), + stateDir: resolveNemoclawStateDir(), getSandbox: registry.getSandbox, listSandboxes: registry.listSandboxes, listGatewayCleanupOwners, @@ -185,6 +187,10 @@ export function resolveSandboxDestroyRegistryAuthority( return { entry: hit.entry, gatewayPort: hit.registryGatewayPort ?? sandboxGatewayPort(hit.entry), + stateDir: resolveNemoclawStateDir( + undefined, + hit.registryGatewayPort ?? sandboxGatewayPort(hit.entry), + ), getSandbox: getSandboxAcrossGatewayRoots, listSandboxes: () => ({ sandboxes: listPublishedSandboxesAcrossGatewayRoots(), diff --git a/src/lib/actions/sandbox/destroy-presence.ts b/src/lib/actions/sandbox/destroy-presence.ts index b58ddef511d..0dccf601797 100644 --- a/src/lib/actions/sandbox/destroy-presence.ts +++ b/src/lib/actions/sandbox/destroy-presence.ts @@ -5,6 +5,8 @@ import { OPENSHELL_SANDBOX_ID_LABEL, OPENSHELL_SANDBOX_NAME_LABEL, OPENSHELL_SANDBOX_WORKSPACE_LABEL, + OPENSHELL_SANDBOX_NAMESPACE_LABEL, + OPENSHELL_ISOLATION_ROLE_LABEL, inspectDockerSandboxNameLabeledContainers, resolveOpenShellSandboxOwnershipLabel, } from "../../onboard/openshell-docker-sandbox-containers"; @@ -31,6 +33,8 @@ export type SandboxNameLabeledContainer = { managedBy: string; workspace: string; sandboxId: string; + namespace?: string; + isolationRole?: string; }; /** Verdict for whether destroy resolved one complete managed container identity. */ @@ -124,6 +128,40 @@ export function classifyDestroyContainerIdentity( managed, }; } + if (managed.some((row) => row.isolationRole)) { + const workload = managed.find((row) => row.isolationRole === "sandbox"); + const supervisor = managed.find((row) => row.isolationRole === "supervisor"); + const workloadComplete = Boolean( + workload?.workspace && workload.sandboxId && workload.namespace, + ); + const retainedIdentityMatches = + retainedSandboxIdentityFingerprint === undefined || + fingerprintOpenShellSandboxId(workload?.sandboxId ?? "") === + retainedSandboxIdentityFingerprint; + if ( + workloadComplete && + retainedIdentityMatches && + managed.length === 2 && + workload && + supervisor && + supervisor.id !== workload.id && + supervisor.workspace === "" && + supervisor.sandboxId === workload.sandboxId && + supervisor.namespace === workload.namespace + ) { + // The workload stays first because state-volume cleanup uses its workspace. + return { status: "recovery", identities: [workload, supervisor] }; + } + if (!workloadComplete || !retainedIdentityMatches || managed.length !== 1) { + return { + status: "ambiguous", + sandboxName, + reason: "container isolation roles do not identify one sandbox and its matching supervisor", + foreign, + managed, + }; + } + } if (managed.length > 0 && retainedSandboxIdentityFingerprint !== undefined) { const identityMatches = managed.every( (row) => @@ -189,7 +227,9 @@ export function formatAmbiguousDestroyIdentity( const describe = (row: SandboxNameLabeledContainer): string => `${display(row.id).slice(0, 12)} (${ownership.label}=${displayLabel(row.managedBy)}, ` + `${OPENSHELL_SANDBOX_WORKSPACE_LABEL}=${displayLabel(row.workspace)}, ` + - `${OPENSHELL_SANDBOX_ID_LABEL}=${displayLabel(row.sandboxId)})`; + `${OPENSHELL_SANDBOX_ID_LABEL}=${displayLabel(row.sandboxId)}, ` + + `${OPENSHELL_SANDBOX_NAMESPACE_LABEL}=${displayLabel(row.namespace ?? "")}, ` + + `${OPENSHELL_ISOLATION_ROLE_LABEL}=${displayLabel(row.isolationRole ?? "")})`; const sandboxName = display(verdict.sandboxName); const lines = [ `Refusing to destroy sandbox '${sandboxName}': ${sanitizeReadinessText(verdict.reason, IDENTITY_DIAGNOSTIC_MAX_LENGTH)}.`, @@ -347,7 +387,9 @@ export function isSameDestroyContainerIdentityProof( identity.id === candidate.id && identity.managedBy === candidate.managedBy && identity.workspace === candidate.workspace && - identity.sandboxId === candidate.sandboxId + identity.sandboxId === candidate.sandboxId && + identity.namespace === candidate.namespace && + identity.isolationRole === candidate.isolationRole ); }); } diff --git a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts index cbd031e637b..d2c65f299f4 100644 --- a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts @@ -369,7 +369,7 @@ describe("destroySandbox retained recovery flow", () => { const sandboxContainerId = "a".repeat(64); const bootstrapContainerId = "b".repeat(64); const identityRows = [sandboxContainerId, bootstrapContainerId] - .map((id) => `${id}\topenshell\tdefault\tsb-alpha`) + .map((id) => `${id}\topenshell\tdefault\tsb-alpha\t\t`) .join("\n"); const harness = createDestroyHarness({ sandboxPresent: false, @@ -425,7 +425,7 @@ describe("destroySandbox retained recovery flow", () => { const bootstrapContainerId = "b".repeat(64); const foreignContainerId = "e".repeat(64); const identityRows = [sandboxContainerId, bootstrapContainerId] - .map((id) => `${id}\topenshell\tdefault\tsb-alpha`) + .map((id) => `${id}\topenshell\tdefault\tsb-alpha\t\t`) .join("\n"); const harness = createDestroyHarness({ sandboxPresent: false, @@ -498,7 +498,7 @@ describe("destroySandbox retained recovery flow", () => { liveListOutput: "Id: sb-alpha", dockerRunResult: { status: 0, - stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha\t\t`, }, registryEntryOverrides: { lifecycleGeneration: recovery.lifecycleGeneration!, @@ -720,7 +720,7 @@ describe("destroySandbox retained recovery flow", () => { sandboxPresent: false, dockerRunResult: { status: 0, - stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha\t\t`, }, registryEntryOverrides: { lifecycleGeneration: recovery.lifecycleGeneration!, @@ -766,7 +766,7 @@ describe("destroySandbox retained recovery flow", () => { dockerOrphanIds: [bootstrapContainerId], dockerRunResult: { status: 0, - stdout: `${bootstrapContainerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${bootstrapContainerId}\topenshell\tdefault\tsb-alpha\t\t`, }, registryEntryOverrides: { lifecycleGeneration: recovery.lifecycleGeneration!, @@ -798,7 +798,7 @@ describe("destroySandbox retained recovery flow", () => { dockerOrphanIds: [bootstrapContainerId], dockerRunResult: { status: 0, - stdout: `${bootstrapContainerId}\topenshell\tdefault\tsb-alpha`, + stdout: `${bootstrapContainerId}\topenshell\tdefault\tsb-alpha\t\t`, }, registryEntryOverrides: { lifecycleGeneration: recovery.lifecycleGeneration!, @@ -876,7 +876,7 @@ describe("destroySandbox retained recovery flow", () => { const sandboxContainerId = "a".repeat(64); const bootstrapContainerId = "b".repeat(64); const identityRows = [sandboxContainerId, bootstrapContainerId] - .map((id) => `${id}\topenshell\tdefault\tsb-alpha`) + .map((id) => `${id}\topenshell\tdefault\tsb-alpha\t\t`) .join("\n"); const harness = createDestroyHarness({ registryEntryPresent: false, @@ -922,7 +922,7 @@ describe("destroySandbox retained recovery flow", () => { registryEntryPresent: false, dockerRunResult: { status: 0, - stdout: `${"a".repeat(64)}\topenshell\tdefault\tsb-alpha`, + stdout: `${"a".repeat(64)}\topenshell\tdefault\tsb-alpha\t\t`, }, retainedRecoveryRecords: [firstRecovery, secondRecovery], }); diff --git a/src/lib/actions/sandbox/destroy.ts b/src/lib/actions/sandbox/destroy.ts index 28c3256a246..e0f3c06a6c3 100644 --- a/src/lib/actions/sandbox/destroy.ts +++ b/src/lib/actions/sandbox/destroy.ts @@ -1249,6 +1249,8 @@ async function destroySandboxUnlocked( }, { runtimeProviders: CURRENT_RUNTIME_PROVIDER_BUNDLES, + workspace: + destroyRuntimeSelection?.workspace ?? initialContainerIdentities?.[0]?.workspace, }, ), ); diff --git a/src/lib/actions/sandbox/doctor-flow.test.ts b/src/lib/actions/sandbox/doctor-flow.test.ts index e8ddc144c16..1202a345cb9 100644 --- a/src/lib/actions/sandbox/doctor-flow.test.ts +++ b/src/lib/actions/sandbox/doctor-flow.test.ts @@ -7,6 +7,7 @@ import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, type MockInstance, vi } from "vitest"; import { testTimeoutOptions } from "../../../../test/helpers/timeouts"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; type RunSandboxDoctor = (typeof import("./doctor"))["runSandboxDoctor"]; type PortableAgentReceiptDisposition = ReturnType< @@ -748,6 +749,51 @@ describe("runSandboxDoctor flow", () => { }, ); + it("keeps a registered provider on its recorded native route during doctor", async () => { + const { binding } = registeredInferenceFixture(true); + const expected = { ...binding, gatewayName: "nemoclaw-19080" }; + const harness = createDoctorHarness("native-extra", { + registryOverrides: { + registeredInferenceProviderAttachment: expected, + endpointUrl: expected.endpointUrl, + preferredInferenceApi: expected.inferenceApi, + credentialEnv: expected.credentialEnv, + }, + }); + const verify = vi + .spyOn(requireDist("./inference-route-health.js"), "verifyRegisteredStatusAttachment") + .mockResolvedValue(undefined); + const probe = vi + .spyOn(requireDist("./inference-invocation-probe.js"), "probeSandboxInferenceInvocation") + .mockResolvedValue({ ok: true }); + const report = await harness.runSandboxDoctor("alpha", ["--json"], { quietJson: true }); + expect(harness.captureOpenShellSpy).not.toHaveBeenCalledWith( + expect.arrayContaining(["inference", "get"]), + expect.anything(), + ); + expect(harness.probeSandboxInferenceGatewayHealthSpy).not.toHaveBeenCalled(); + expect(verify).toHaveBeenCalledExactlyOnceWith({ + sandboxName: "alpha", + gatewayName: expected.gatewayName, + expected, + }); + expect(probe).toHaveBeenCalledWith( + expect.objectContaining({ + nativeProvider: true, + provider: "native-extra", + registeredInferenceProviderAttachment: expected, + }), + {}, + 95_000, + ); + expect(report?.checks).toContainEqual( + expect.objectContaining({ + label: "Inference route (registered native)", + status: "ok", + }), + ); + }); + it("uses hosted attachment authority and never reads the shared gateway route", async () => { const receipt = { schemaVersion: 1, diff --git a/src/lib/actions/sandbox/doctor-inference.test.ts b/src/lib/actions/sandbox/doctor-inference.test.ts index f8b8fe1c446..020ca051810 100644 --- a/src/lib/actions/sandbox/doctor-inference.test.ts +++ b/src/lib/actions/sandbox/doctor-inference.test.ts @@ -8,9 +8,90 @@ import { import { describe, expect, it, vi } from "vitest"; import type { ProviderHealthStatus } from "../../inference/health"; import { collectInferenceChecks, collectManagedLlamaCppDoctorChecks } from "./doctor-inference"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; +import { verifyRegisteredInferenceProviderAttachment } from "../../inference/config"; const endpoint = "https://inference.local/v1/models"; +describe("registered provider doctor", () => { + it.each([ + "served", + "request-failed", + "replaced", + "detached", + "wrong-gateway", + "wrong-endpoint", + "malformed", + "unreachable", + ])("checks registered identity and invocation without a shared route: %s", async (scenario) => { + const fixture = registeredInferenceFixture(true); + const changes: Record> = { + replaced: { id: "replacement-id" }, + detached: { attached: false }, + }; + Object.assign(fixture.state, changes[scenario]); + const shared = vi.fn(); + const probe = vi.fn(async () => + scenario === "request-failed" + ? { ok: false as const, detail: "request failed", httpStatus: 503 } + : { ok: true as const }, + ); + const verify = vi.fn(async ({ expected }: { expected: typeof fixture.binding }) => { + await verifyRegisteredInferenceProviderAttachment({ adapter: fixture.adapter, expected }); + }); + const checks = await collectInferenceChecks( + "alpha", + { + provider: "native-extra", + model: "selected-model", + agentName: "hermes", + recordedEndpointUrl: + scenario === "wrong-endpoint" ? "https://other.example/v1" : fixture.binding.endpointUrl, + preferredInferenceApi: fixture.binding.inferenceApi, + credentialEnv: fixture.binding.credentialEnv, + registeredInferenceProviderAttachment: scenario === "malformed" ? {} : fixture.binding, + }, + scenario !== "unreachable", + { + gatewayName: scenario === "wrong-gateway" ? "other" : fixture.binding.gatewayName, + verifyRegisteredStatusAttachmentImpl: verify, + probeSandboxInferenceInvocationImpl: probe, + probeSandboxInferenceGatewayHealthImpl: shared, + probeProviderHealthImpl: () => null, + includeServingProcessCheck: false, + }, + ); + expect(checks).toContainEqual( + expect.objectContaining({ + label: "Inference route (registered native)", + status: scenario === "served" ? "ok" : "fail", + }), + ); + expect(shared).not.toHaveBeenCalled(); + expect(probe.mock.calls).toEqual( + ["served", "request-failed"].includes(scenario) + ? [ + [ + expect.objectContaining({ + sandboxName: "alpha", + gatewayName: fixture.binding.gatewayName, + provider: "native-extra", + model: "selected-model", + nativeProvider: true, + preferredInferenceApi: fixture.binding.inferenceApi, + registeredInferenceProviderAttachment: fixture.binding, + }), + {}, + expect.any(Number), + ], + ] + : [], + ); + expect(fixture.adapter.attachProvider).not.toHaveBeenCalled(); + expect(fixture.adapter.detachProvider).not.toHaveBeenCalled(); + }); +}); + function gateway(ok: boolean, httpStatus = ok ? 200 : 0) { return { ok, diff --git a/src/lib/actions/sandbox/doctor-inference.ts b/src/lib/actions/sandbox/doctor-inference.ts index 0ac1d58554f..e4d3e865795 100644 --- a/src/lib/actions/sandbox/doctor-inference.ts +++ b/src/lib/actions/sandbox/doctor-inference.ts @@ -7,8 +7,10 @@ import { } from "../../inference/native-local/contract"; import { verifyNativeLocalStatusAttachment, + verifyRegisteredStatusAttachment, runSandboxInferenceInvocationProbe, } from "./inference-route-health"; +import { recordedRegisteredInferenceProviderAttachment } from "../../inference/native-provider/registered"; import { recordedNativeProviderAttachment } from "../../inference/native-provider/recorded-selection"; import { hostedNativeProviderForAttachment } from "../../inference/native-provider/hosted-attachment"; @@ -51,6 +53,8 @@ export type DoctorInferenceRoute = { nativeNvidiaProviderAttachment?: NativeNvidiaProviderAttachment; nativeHostedProviderAttachment?: unknown; preferredInferenceApi?: string | null; + credentialEnv?: string | null; + registeredInferenceProviderAttachment?: unknown; }; type ManagedLlamaCppDoctorDeps = { @@ -98,6 +102,7 @@ export function collectManagedLlamaCppDoctorChecks( type DoctorInferenceDeps = { gatewayName?: string | null; + verifyRegisteredStatusAttachmentImpl?: typeof verifyRegisteredStatusAttachment; verifyNativeHostedStatusAttachmentImpl?: typeof verifyNativeHostedStatusAttachment; probeSandboxInferenceInvocationImpl?: typeof probeSandboxInferenceInvocation; probeProviderHealthImpl?: typeof probeProviderHealth; @@ -110,6 +115,71 @@ type DoctorInferenceDeps = { includeServingProcessCheck?: boolean; }; +async function collectRegisteredRouteProbe( + sandboxName: string, + route: DoctorInferenceRoute, + sandboxReachable: boolean, + deps: DoctorInferenceDeps, +): Promise { + const failure = (detail: string): ProviderHealthStatus => ({ + ok: false, + probed: false, + providerLabel: "Registered native provider", + endpoint: "recorded native endpoint", + detail, + failureLabel: "unreachable", + }); + if (!sandboxReachable) + return failure("skipped because the sandbox is not reachable through its named gateway"); + try { + const expected = recordedRegisteredInferenceProviderAttachment({ + ...route, + name: sandboxName, + gatewayName: deps.gatewayName, + endpointUrl: route.recordedEndpointUrl, + }); + if (!expected || !deps.gatewayName) + return failure("Registered provider attachment or gateway binding is missing"); + await (deps.verifyRegisteredStatusAttachmentImpl ?? verifyRegisteredStatusAttachment)({ + sandboxName, + gatewayName: deps.gatewayName, + expected, + }); + const result = await ( + deps.probeSandboxInferenceInvocationImpl ?? probeSandboxInferenceInvocation + )( + { + sandboxName, + gatewayName: deps.gatewayName, + provider: route.provider, + model: route.model, + agentName: route.agentName, + preferredInferenceApi: expected.inferenceApi, + registeredInferenceProviderAttachment: expected, + nativeProvider: true, + }, + {}, + READINESS_INFERENCE_INVOCATION_TIMEOUT_MS, + ); + return { + ok: result.ok, + probed: true, + providerLabel: "Registered native provider", + endpoint: expected.endpointUrl, + detail: result.ok + ? "The attached provider served a native inference request." + : formatUntrustedProbeDetail(result.detail), + ...(result.ok + ? {} + : { failureLabel: classifyInferenceRouteFailureLabel(result.httpStatus ?? 0) }), + }; + } catch (error) { + return failure( + formatUntrustedProbeDetail(error instanceof Error ? error.message : String(error)), + ); + } +} + async function collectNativeLocalRouteProbe( sandboxName: string, route: DoctorInferenceRoute, @@ -458,26 +528,31 @@ export async function collectInferenceChecks( if (effortCheck) checks.push(effortCheck); const nativeNvidia = isNativeNvidiaProvider(route.provider); const nativeLocal = route.nativeLocalProviderAttachment !== undefined; - const routeProbe = nativeLocal - ? await collectNativeLocalRouteProbe(sandboxName, route, sandboxReachable, deps) - : route.nativeHostedProviderAttachment !== undefined - ? await collectNativeHostedRouteProbe(sandboxName, route, sandboxReachable, deps) - : nativeNvidia - ? await collectNativeNvidiaRouteProbe(sandboxName, route, sandboxReachable, deps) - : await collectInferenceRouteProbe( - sandboxName, - sandboxReachable, - deps.probeSandboxInferenceGatewayHealthImpl ?? probeSandboxInferenceGatewayHealth, - deps.gatewayName, - ); - pushInferenceHealthCheck(checks, routeProbe, { - label: nativeLocal - ? "Inference route (native local)" + const registered = route.registeredInferenceProviderAttachment !== undefined; + const routeProbe = registered + ? await collectRegisteredRouteProbe(sandboxName, route, sandboxReachable, deps) + : nativeLocal + ? await collectNativeLocalRouteProbe(sandboxName, route, sandboxReachable, deps) : route.nativeHostedProviderAttachment !== undefined - ? "Inference route (native hosted)" + ? await collectNativeHostedRouteProbe(sandboxName, route, sandboxReachable, deps) : nativeNvidia - ? "Inference route (native NVIDIA)" - : "Inference route (gateway)", + ? await collectNativeNvidiaRouteProbe(sandboxName, route, sandboxReachable, deps) + : await collectInferenceRouteProbe( + sandboxName, + sandboxReachable, + deps.probeSandboxInferenceGatewayHealthImpl ?? probeSandboxInferenceGatewayHealth, + deps.gatewayName, + ); + pushInferenceHealthCheck(checks, routeProbe, { + label: registered + ? "Inference route (registered native)" + : nativeLocal + ? "Inference route (native local)" + : route.nativeHostedProviderAttachment !== undefined + ? "Inference route (native hosted)" + : nativeNvidia + ? "Inference route (native NVIDIA)" + : "Inference route (gateway)", }); for (const diagnostic of collectProviderHealthDiagnostics( route, diff --git a/src/lib/actions/sandbox/doctor.ts b/src/lib/actions/sandbox/doctor.ts index 6623e214893..a1a8ac17d61 100644 --- a/src/lib/actions/sandbox/doctor.ts +++ b/src/lib/actions/sandbox/doctor.ts @@ -428,6 +428,7 @@ async function resolveInferenceRoute( !recordedNativeNvidia && !recordedNativeLocal && sb?.nativeHostedProviderAttachment === undefined && + sb?.registeredInferenceProviderAttachment === undefined && openshellBin && openshellConnected && gatewayName @@ -450,6 +451,10 @@ async function resolveInferenceRoute( ? { nativeLocalProviderAttachment: sb?.nativeLocalProviderAttachment } : {}), preferredInferenceApi: sb?.preferredInferenceApi, + credentialEnv: sb?.credentialEnv, + ...(sb?.registeredInferenceProviderAttachment !== undefined + ? { registeredInferenceProviderAttachment: sb.registeredInferenceProviderAttachment } + : {}), ...(sb?.nativeHostedProviderAttachment !== undefined ? { nativeHostedProviderAttachment: sb.nativeHostedProviderAttachment } : {}), diff --git a/src/lib/actions/sandbox/exec-policy-hint-detection.test.ts b/src/lib/actions/sandbox/exec-policy-hint-detection.test.ts index a4a69a51245..c624853b1ed 100644 --- a/src/lib/actions/sandbox/exec-policy-hint-detection.test.ts +++ b/src/lib/actions/sandbox/exec-policy-hint-detection.test.ts @@ -16,6 +16,9 @@ const DENIED_GIT_LINE = "[1783046885.833] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/lib/git-core/git-remote-http(3973) -> github.com:443 [policy:- engine:opa] [reason:endpoint github.com:443 is not allowed by any policy]"; const SSH_RELAY_INFO_LINE = "[1783046565.338] [sandbox] [OCSF ] [ocsf] NET:OPEN [INFO] [msg:ssh relay open (channel_id=8e95bfe4, target=unix:/run/openshell/ssh.sock)]"; +// OpenShell 0.1.3 policy_dns::build_dns_denial_event and OCSF shorthand format. +const DNS_POLICY_DENIAL_LINE = + "[1783046573.602] [sandbox] [OCSF ] [ocsf] NET:REFUSE [MED] DENIED host.openshell.internal [reason:policy_dns_ineligible]"; // Exact proxy JSON body captured alongside the OCSF lines above. Keep this // fixture strict: speculative wording variants would widen false positives. const PROXY_JSON_LINE = @@ -23,6 +26,31 @@ const PROXY_JSON_LINE = const INTERLEAVED_CURL_PROXY_JSON_LINE = 'curl: (22) The reque{"detail":"POST host.openshell.internal:4318/v1/traces not permitted by policy","error":"policy_denied"}sted URL returned error: 403'; +// Captured from the OpenShell 0.1.3 Deep Agents Code observability denial in run 39474622328. +const REST_DENIAL = { + error: "policy_denied", + policy: "observability-otlp-local", + rule: "POST /not-traces", + detail: "POST /not-traces not permitted by policy", + layer: "l7", + protocol: "rest", + method: "POST", + path: "/not-traces", + host: "host.openshell.internal", + port: 4318, + binary: "/opt/venv/bin/python3", + rule_missing: { + type: "rest_allow", + layer: "l7", + method: "POST", + path: "/not-traces", + host: "host.openshell.internal", + port: 4318, + binary: "/opt/venv/bin/python3", + }, + next_steps: [], +}; + // A denial timestamp of 1783046573.602s parses to 1783046573602ms. Anchor the // command-start stamps around it to exercise the recency window. const START_BEFORE_DENIAL = 1783046573000; @@ -31,6 +59,24 @@ const START_AFTER_DENIAL = 1783046800000; describe("isPolicyDenialLine (#5978)", () => { it.each([ ["OCSF NET:OPEN DENIED audit line", DENIED_CURL_LINE, true], + ["OCSF policy DNS refusal", DNS_POLICY_DENIAL_LINE, true], + ...[ + "policy_dns_upstream_nxdomain", + "policy_dns_upstream_no_data", + "policy_dns_upstream_timeout", + "policy_dns_trusted_gateway_unavailable", + "policy_dns_ineligible_extra", + ].map( + (reason) => + [reason, DNS_POLICY_DENIAL_LINE.replace("policy_dns_ineligible", reason), false] as const, + ), + [ + "DNS error without a policy event", + "FAILED:URLError:", + false, + ], + ["allowed DNS event", DNS_POLICY_DENIAL_LINE.replace("DENIED", "ALLOWED"), false], + ["DNS policy text in a message", `[msg:${DNS_POLICY_DENIAL_LINE}]`, false], [ "bracketed OCSF NET:OPEN DENIED audit line", "[policy:-] [NET:OPEN] DENIED [reason:example.com:443 is not allowed by any policy]", @@ -41,6 +87,43 @@ describe("isPolicyDenialLine (#5978)", () => { "[1783046573.602] [sandbox] NET:OPEN [MED] DENIED /usr/bin/curl(9) -> [2001:db8::1]:443 [reason:not allowed by any policy]", true, ], + ["OpenShell 0.1.3 REST denial", JSON.stringify(REST_DENIAL), true], + [ + "REST denial after Python HTTPError text", + `FAILED:HTTPError:HTTP Error 403: Forbidden:${JSON.stringify(REST_DENIAL)}`, + true, + ], + ...[ + { error: "middleware_denied" }, + { policy: "" }, + { layer: "l4" }, + { protocol: "mcp" }, + { method: "GET" }, + { method: "post" }, + { path: "/other" }, + { path: "/bad\npath" }, + { rule: "POST /other" }, + { detail: "arbitrary denial prose" }, + { detail: "POST /other not permitted by policy" }, + { detail: "x".repeat(1025) }, + { rule_missing: null }, + { rule_missing: [] }, + { rule_missing: { ...REST_DENIAL.rule_missing, type: "mcp_allow" } }, + { rule_missing: { ...REST_DENIAL.rule_missing, layer: "l4" } }, + { rule_missing: { ...REST_DENIAL.rule_missing, method: "GET" } }, + { rule_missing: { ...REST_DENIAL.rule_missing, path: "/other" } }, + { rule_missing: { ...REST_DENIAL.rule_missing, host: "other.example" } }, + { next_steps: null }, + { extra: true }, + ].map( + (change) => + [ + `REST denial with inconsistent ${JSON.stringify(change)}`, + JSON.stringify({ ...REST_DENIAL, ...change }), + false, + ] as const, + ), + ["truncated REST denial", JSON.stringify(REST_DENIAL).slice(0, -1), false], ["proxy JSON policy_denied body", PROXY_JSON_LINE, true], ["proxy JSON interleaved with curl stderr", INTERLEAVED_CURL_PROXY_JSON_LINE, true], [ @@ -210,13 +293,36 @@ describe("isPolicyDenialLine (#5978)", () => { ], ["unrelated log line", "[123.0] [sandbox] [INFO ] flushed activity summary", false], ["empty line", "", false], - ])("classifies %s", (_label, line, expected) => { + ] as const)("classifies %s", (_label, line, expected) => { expect(isPolicyDenialLine(line)).toBe(expected); }); }); describe("extractDeniedEndpoint (#5978)", () => { it.each([ + ...( + [ + ["host.openshell.internal", 4318, "host.openshell.internal:4318"], + [undefined, undefined, null], + ["unsafe\nhost", 4318, null], + ["host.openshell.internal", 99999, null], + ["host.openshell.internal", "4318", null], + ] as const + ).map( + ([host, port, endpoint]) => + [ + `REST endpoint fields ${String(host)}:${String(port)} with unrelated remediation URLs`, + JSON.stringify({ + ...REST_DENIAL, + host, + port, + rule_missing: { ...REST_DENIAL.rule_missing, host, port }, + next_steps: ["https://other.example:443"], + agent_guidance: "-> misleading.example:8080", + }), + endpoint, + ] as const, + ), ["arrow target of a curl denial", DENIED_CURL_LINE, "example.com:443"], ["arrow target of a git denial", DENIED_GIT_LINE, "github.com:443"], [ @@ -244,7 +350,7 @@ describe("extractDeniedEndpoint (#5978)", () => { "2026-07-03T04:00:00Z proxy CONNECT [2001:db8::1]:443 policy_denied", "[2001:db8::1]:443", ], - ])("extracts the safe host:port from %s", (_label, line, expected) => { + ] as const)("extracts the safe host:port from %s", (_label, line, expected) => { expect(extractDeniedEndpoint(line)).toBe(expected); }); @@ -278,6 +384,16 @@ describe("extractDeniedEndpoint (#5978)", () => { }); describe("findRecentPolicyDenial (#5978)", () => { + it.each([ + [1783046573602, { endpoint: null }], + [1783046573603, null], + ])( + "correlates DNS policy denial with command start %i without inventing a port", + (startedAt, expected) => { + expect(findRecentPolicyDenial(DNS_POLICY_DENIAL_LINE, startedAt)).toEqual(expected); + }, + ); + it("matches a denial logged after the command started and returns its endpoint", () => { const match = findRecentPolicyDenial( [SSH_RELAY_INFO_LINE, DENIED_CURL_LINE].join("\n"), diff --git a/src/lib/actions/sandbox/exec-policy-hint-detection.ts b/src/lib/actions/sandbox/exec-policy-hint-detection.ts index cd3ebdc5903..cff3f8f751c 100644 --- a/src/lib/actions/sandbox/exec-policy-hint-detection.ts +++ b/src/lib/actions/sandbox/exec-policy-hint-detection.ts @@ -25,6 +25,10 @@ function isSafeEndpoint(candidate: string): boolean { // Require DENIED in the OCSF decision slot. This rejects allowed events whose // unrelated metadata happens to contain the word DENIED. const OCSF_NETWORK_DENIAL_RE = /\bNET:OPEN\b\]?(?:\s+\[[^\]\r\n]*\])*\s+DENIED(?=\s|$)/; +// OpenShell 0.1.3 refuses ineligible host aliases during policy DNS lookup. +// Other DNS refusals include upstream failures and do not prove policy denial. +const OCSF_DNS_POLICY_DENIAL_RE = + /(?:^|\]\s+)NET:REFUSE \[MED\] DENIED [a-z0-9][a-z0-9._-]{0,252} \[reason:policy_dns_ineligible\]$/u; const PROXY_DENIAL_DETAIL_RE = /^CONNECT\s+(\[[^\]\s]+\]:\d{1,5}|[^\s:]+:\d{1,5})\s+not\s+(?:allowed|permitted)\s+by\s+(?:any\s+)?policy$/i; const PROXY_HTTP_DENIAL_DETAIL_RE = @@ -81,6 +85,57 @@ function isStructuredProxyPolicyDenialDetail(detail: string): boolean { return isStructuredForwardProxyPolicyDenial(detail); } +function isStructuredRestPolicyDenial(payload: Record): boolean { + const { method, path, detail, rule_missing: missing } = payload; + // OpenShell 0.1.3 rest::deny_response_body separates the method and path + // from the endpoint. Require its denial fields to agree; never render its + // optional remediation text or use that text as evidence of a denial. + const fields = [ + "error", + "policy", + "rule", + "detail", + "layer", + "protocol", + "method", + "path", + "host", + "port", + "binary", + "rule_missing", + "next_steps", + "agent_guidance", + ]; + if ( + Object.keys(payload).some((key) => !fields.includes(key)) || + payload.layer !== "l7" || + payload.protocol !== "rest" || + typeof payload.policy !== "string" || + !SAFE_DYNAMIC_PROXY_REASON_RE.test(payload.policy) || + typeof method !== "string" || + !/^[A-Z][A-Z0-9!#$%&'*+.^_`|~-]{0,31}$/.test(method) || + typeof path !== "string" || + !SAFE_PROXY_PATH_RE.test(path) || + payload.rule !== `${method} ${path}` || + typeof detail !== "string" || + detail.length > MAX_STRUCTURED_PROXY_DETAIL_LENGTH || + typeof missing !== "object" || + missing === null || + Array.isArray(missing) || + !Array.isArray(payload.next_steps) + ) + return false; + const rule = missing as Record; + return ( + rule.type === "rest_allow" && + rule.layer === "l7" && + rule.method === method && + rule.path === path && + ["host", "port", "binary"].every((key) => rule[key] === payload[key]) && + isStructuredL7PolicyDenialReason(method, path, detail) + ); +} + // Source-of-truth for structured proxy JSON: // - Invalid state: OpenShell reports some policy refusals only in structured // CONNECT or forward-HTTP 403 JSON while the child tool receives opaque text. @@ -126,20 +181,22 @@ function firstJsonObject(line: string): string | null { return null; } -function isStructuredJsonPolicyDenial(line: string): boolean { - if (line.length > MAX_STRUCTURED_PROXY_LINE_LENGTH) return false; +function structuredJsonPolicyDenial(line: string): Record | null { + if (line.length > MAX_STRUCTURED_PROXY_LINE_LENGTH) return null; const jsonObject = firstJsonObject(line); - if (jsonObject === null) return false; + if (jsonObject === null) return null; try { const parsed: unknown = JSON.parse(jsonObject); - if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return false; + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return null; const payload = parsed as Record; - const keys = Object.keys(payload); - if (keys.length !== 2 || !keys.includes("detail") || !keys.includes("error")) return false; - if (payload.error !== "policy_denied" || typeof payload.detail !== "string") return false; - return isStructuredProxyPolicyDenialDetail(payload.detail); + if (payload.error !== "policy_denied" || typeof payload.detail !== "string") return null; + const denied = + Object.keys(payload).length === 2 + ? isStructuredProxyPolicyDenialDetail(payload.detail) + : isStructuredRestPolicyDenial(payload); + return denied ? payload : null; } catch { - return false; + return null; } } @@ -151,8 +208,8 @@ function isStructuredJsonPolicyDenial(line: string): boolean { * exec cannot inherit a misleading breadcrumb. */ export function isPolicyDenialLine(line: string): boolean { - if (OCSF_NETWORK_DENIAL_RE.test(line)) return true; - return isStructuredJsonPolicyDenial(line); + if (OCSF_NETWORK_DENIAL_RE.test(line) || OCSF_DNS_POLICY_DENIAL_RE.test(line)) return true; + return structuredJsonPolicyDenial(line) !== null; } const LEADING_EPOCH_TIMESTAMP_RE = /^\s*\[\d+(?:\.\d+)?\]\s*/; @@ -164,6 +221,14 @@ const LEADING_ISO_TIMESTAMP_RE = * Returns null when no candidate passes the terminal-output allowlist. */ export function extractDeniedEndpoint(line: string): string | null { + const payload = structuredJsonPolicyDenial(line); + if (payload?.protocol === "rest") { + // REST responses contain remediation URLs. Only their endpoint fields + // identify the denied destination; never scan the surrounding response. + if (typeof payload.host !== "string" || typeof payload.port !== "number") return null; + const endpoint = `${payload.host}:${payload.port}`; + return isSafeEndpoint(endpoint) ? endpoint : null; + } const candidates: string[] = []; const arrow = line.match(/->\s*(\[[^\]\s]+\]:\d{1,5}|[^\s\]]+:\d{1,5})(?:\b|$)/); if (arrow) candidates.push(arrow[1]); diff --git a/src/lib/actions/sandbox/exec-policy-hint-rendering.ts b/src/lib/actions/sandbox/exec-policy-hint-rendering.ts index 10d1e3e47ac..70bac2dca81 100644 --- a/src/lib/actions/sandbox/exec-policy-hint-rendering.ts +++ b/src/lib/actions/sandbox/exec-policy-hint-rendering.ts @@ -21,7 +21,7 @@ export function buildPolicyDenialExecHint( const target = endpoint ? ` for ${endpoint}` : ""; return [ `${cliName}: recent network policy denial detected${target} inside sandbox '${sandboxName}'.`, - " The sandbox's egress policy blocked this request; the tool above only saw the proxy's 403.", + " Match the denial's destination and time to the failed request before changing policy.", ` See the denied flow: ${cliName} ${sandboxName} logs --tail 50`, ` Review applied presets: ${cliName} ${sandboxName} policy list`, ` Allow the host: ${cliName} ${sandboxName} policy add `, diff --git a/src/lib/actions/sandbox/forward-recovery-declared-ports.test.ts b/src/lib/actions/sandbox/forward-recovery-declared-ports.test.ts index 869913ac4b4..b93c8904a80 100644 --- a/src/lib/actions/sandbox/forward-recovery-declared-ports.test.ts +++ b/src/lib/actions/sandbox/forward-recovery-declared-ports.test.ts @@ -23,6 +23,7 @@ import type { ObserveOpenShellForwardsRequest, OpenShellForwardIdentity, OpenShellForwardObservation, + OpenShellForwardReleaseResult, OpenShellLegacyForwardRetirementResult, RetireLegacyOpenShellForwardRequest, StartOpenShellForwardRequest, @@ -597,6 +598,49 @@ describe("declared and cleanup forward sets", () => { ).toEqual([18_790, 3_001, 8_643]); expect(mocks.verifyForwardRelease.mock.calls[0]?.[0].forwards[0]?.localHost).toBe("0.0.0.0"); }); + + it.each(["bound", "timeout", "authority", "validation"] as const)( + "reports forward release %s without including the error message", + async (outcome) => { + mocks.getRegisteredAgent.mockReturnValue(null); + const errors = { + timeout: { kind: "timeout", message: "The OpenShell forward operation timed out." }, + authority: { + kind: "authority", + message: "NemoClaw could not prove current OpenShell forward authority.", + }, + validation: { kind: "validation", message: "The OpenShell forward request is invalid." }, + } as const; + mocks.verifyForwardRelease.mockImplementationOnce( + async ( + request: VerifyOpenShellForwardReleaseRequest, + ): Promise => + outcome === "bound" + ? { state: "bound", forwards: request.forwards } + : outcome === "validation" + ? { state: "indeterminate", error: errors.validation } + : { + state: "indeterminate", + forwards: request.forwards, + error: errors[outcome], + }, + ); + const consoleError = vi.spyOn(console, "error").mockImplementation(() => undefined); + const { teardownSandboxDashboardForward } = await import("./forward-recovery"); + + await expect(teardownSandboxDashboardForward("box")).resolves.toBe(false); + const classification = outcome === "bound" ? "bound" : `indeterminate/${outcome}`; + expect(consoleError).toHaveBeenCalledExactlyOnceWith( + " ForwardTcp port release was not verified for registered host port(s): 18789. " + + `[forward-release ${classification}]`, + ); + expect(mocks.verifyForwardRelease).toHaveBeenCalledExactlyOnceWith( + expect.objectContaining({ timeoutMs: 5_000, assertCurrent: expect.any(Function) }), + ); + expect(mocks.startForward).not.toHaveBeenCalled(); + expect(mocks.retireLegacyForward).not.toHaveBeenCalled(); + }, + ); }); it.each([ diff --git a/src/lib/actions/sandbox/forward-recovery.ts b/src/lib/actions/sandbox/forward-recovery.ts index 332a9cf0bf7..c0cf6de3bf0 100644 --- a/src/lib/actions/sandbox/forward-recovery.ts +++ b/src/lib/actions/sandbox/forward-recovery.ts @@ -547,8 +547,11 @@ export async function teardownSandboxDashboardForward( "forwards" in release ? release.forwards.map((forward) => forward.port) : forwards.map((forward) => forward.port); + const classification = + release.state === "bound" ? "bound" : `indeterminate/${release.error.kind}`; console.error( - ` ForwardTcp cleanup did not release registered host port(s): ${unreleasedPorts.join(", ")}.`, + ` ForwardTcp port release was not verified for registered host port(s): ${unreleasedPorts.join(", ")}. ` + + `[forward-release ${classification}]`, ); return false; } diff --git a/src/lib/actions/sandbox/gateway-restart.test.ts b/src/lib/actions/sandbox/gateway-restart.test.ts index 5654d98679e..c2f121bc3f1 100644 --- a/src/lib/actions/sandbox/gateway-restart.test.ts +++ b/src/lib/actions/sandbox/gateway-restart.test.ts @@ -76,6 +76,221 @@ describe("restartSandboxGateway native lifecycle", () => { }; } + function readinessDiagnostic() { + const prefix = " OpenClaw restart readiness: "; + const line = vi + .mocked(console.error) + .mock.calls.map(([value]) => String(value)) + .find((value) => value.startsWith(prefix)); + return line ? JSON.parse(line.slice(prefix.length)) : null; + } + + it.each([ + { + stdout: '{"ready":true}\n302', + status: 0, + httpStatus: 302, + jsonParsed: true, + ready: true, + decision: false, + }, + { + stdout: '{"ready":true}\n401', + status: 0, + httpStatus: 401, + jsonParsed: true, + ready: true, + decision: false, + }, + { + stdout: '{"ready":false}\n503', + status: 0, + httpStatus: 503, + jsonParsed: true, + ready: false, + decision: false, + }, + { + stdout: '{"ready":false}\n200', + status: 0, + httpStatus: 200, + jsonParsed: true, + ready: false, + decision: false, + }, + { + stdout: '{"ready":"true"}\n200', + status: 0, + httpStatus: 200, + jsonParsed: true, + ready: null, + decision: false, + }, + { + stdout: '{"ok":true}\n200', + status: 0, + httpStatus: 200, + jsonParsed: true, + ready: null, + decision: false, + }, + { + stdout: "not-json\n200", + status: 0, + httpStatus: 200, + jsonParsed: false, + ready: null, + decision: false, + }, + { + stdout: '{"ready":true}', + status: 0, + httpStatus: null, + jsonParsed: false, + ready: null, + decision: false, + }, + { + stdout: '{"ready":true}\n200', + status: 28, + httpStatus: 200, + jsonParsed: true, + ready: true, + decision: false, + }, + { + stdout: "\n000", + status: 7, + httpStatus: null, + jsonParsed: false, + ready: null, + decision: false, + }, + ])( + "reports readiness evidence without changing HTTP $httpStatus command $status outcome ($stdout)", + async (sample) => { + silenceConsole(); + const execute = vi + .fn() + .mockResolvedValueOnce({ status: 0, stdout: "", stderr: "" }) + .mockResolvedValue({ + status: sample.status, + stdout: sample.stdout, + stderr: "secret-stderr", + }); + const deps = baseDeps({ + executeSandboxExecCommand: execute, + waitForRecoveredSandboxGateway: async ( + name: string, + options: Parameters[1], + ) => { + expect(await options!.probeImpl!(name)).toBe(sample.decision); + return sample.decision; + }, + }); + const result = await restartSandboxGateway("alpha", { quiet: true, deps }); + expect(result.ok).toBe(sample.decision); + expect(execute).toHaveBeenCalledTimes(2); + expect(execute.mock.calls[0]?.[2]).toBe(210000); + expect(execute.mock.calls[1]?.[2]).toBe(10000); + expect(readinessDiagnostic()).toEqual({ + nativeRestartStatus: 0, + totalProbes: 1, + droppedSamples: 0, + samples: [ + { + ordinal: 1, + commandStatus: sample.status, + transportErrorKind: null, + httpStatus: sample.httpStatus, + jsonParsed: sample.jsonParsed, + ready: sample.ready, + probeResult: false, + }, + ], + }); + expect(deps.ensureSandboxPortForward).not.toHaveBeenCalled(); + expect(vi.mocked(console.error).mock.calls.join("\n")).not.toContain("secret-stderr"); + }, + ); + + it("retains the last 32 readiness observations without response bodies or extra probes", async () => { + silenceConsole(); + const secret = "private-token-".repeat(10000); + const execute = vi + .fn() + .mockResolvedValueOnce({ status: 0, stdout: secret, stderr: secret }) + .mockResolvedValue({ + status: 0, + stdout: `${JSON.stringify({ ready: false, token: secret, url: "https://user:password@example.com" })}\n503`, + stderr: secret, + }); + const deps = baseDeps({ + executeSandboxExecCommand: execute, + waitForRecoveredSandboxGateway: async ( + name: string, + options: Parameters[1], + ) => + waitForRecoveredSandboxGateway(name, { + ...options, + timeoutSeconds: 102, + sleepImpl: () => undefined, + }), + }); + await expect(restartSandboxGateway("alpha", { quiet: true, deps })).resolves.toMatchObject({ + ok: false, + failureLayer: "health timeout", + }); + const diagnostic = readinessDiagnostic(); + expect(diagnostic).toMatchObject({ + nativeRestartStatus: 0, + totalProbes: 35, + droppedSamples: 3, + }); + expect(diagnostic.samples.map((sample: { ordinal: number }) => sample.ordinal)).toEqual( + Array.from({ length: 32 }, (_, index) => index + 4), + ); + expect(execute).toHaveBeenCalledTimes(36); + const printed = vi.mocked(console.error).mock.calls.join("\n"); + expect(printed.length).toBeLessThan(6000); + expect(printed).not.toMatch(/private-token|password|https:\/\/|example\.com/); + }); + + it.each(["cancelled", "capture", "invocation", "timeout", "unavailable", "malformed"] as const)( + "records unavailable readiness evidence for %s without retrying the command", + async (kind) => { + silenceConsole(); + const execute = vi.fn().mockResolvedValueOnce({ status: 0, stdout: "", stderr: "" }); + execute.mockRejectedValue(new SandboxCommandTransportError(kind)); + const deps = baseDeps({ + executeSandboxExecCommand: execute, + waitForRecoveredSandboxGateway: async ( + name: string, + options: Parameters[1], + ) => { + expect(await options!.probeImpl!(name)).toBeNull(); + return false; + }, + }); + await expect(restartSandboxGateway("alpha", { quiet: true, deps })).resolves.toMatchObject({ + ok: false, + failureLayer: "health timeout", + }); + expect(readinessDiagnostic().samples).toEqual([ + { + ordinal: 1, + commandStatus: null, + transportErrorKind: kind, + httpStatus: null, + jsonParsed: false, + ready: null, + probeResult: null, + }, + ]); + expect(execute).toHaveBeenCalledTimes(2); + }, + ); + it.each(["cancelled", "capture", "invocation", "timeout", "unavailable", "malformed"] as const)( "reports %s transport failure without retrying restart", async (kind) => { @@ -196,6 +411,77 @@ describe("restartSandboxGateway native lifecycle", () => { }, ); + it.each([ + { settled: true, observations: null }, + { settled: false, observations: [true, false] }, + ])( + "preserves the readiness settle result $settled and its probe count", + async ({ settled, observations }) => { + silenceConsole(); + vi.stubEnv("NEMOCLAW_GATEWAY_RECOVERY_SETTLE_SECONDS", "25"); + const execute = vi + .fn() + .mockResolvedValueOnce({ status: 0, stdout: "", stderr: "" }) + .mockResolvedValueOnce({ status: 0, stdout: '{"ready":true}\n200', stderr: "" }) + .mockResolvedValueOnce({ + status: 0, + stdout: `${JSON.stringify({ ready: settled })}\n200`, + stderr: "", + }); + const sleep = vi.fn(); + const deps = baseDeps({ + executeSandboxExecCommand: execute, + waitForRecoveredSandboxGateway: ( + name: string, + options: Parameters[1], + ) => + waitForRecoveredSandboxGateway(name, { ...options, timeoutSeconds: 0, sleepImpl: sleep }), + }); + expect((await restartSandboxGateway("alpha", { quiet: true, deps })).ok).toBe(settled); + expect(execute).toHaveBeenCalledTimes(3); + expect(sleep).toHaveBeenCalledExactlyOnceWith(25); + expect( + readinessDiagnostic()?.samples.map( + (sample: { probeResult: boolean }) => sample.probeResult, + ) ?? null, + ).toEqual(observations); + }, + ); + + it("records a missing readiness command result without inventing a transport cause", async () => { + silenceConsole(); + const execute = vi + .fn() + .mockResolvedValueOnce({ status: 0, stdout: "", stderr: "" }) + .mockResolvedValue(null); + const deps = baseDeps({ + executeSandboxExecCommand: execute, + waitForRecoveredSandboxGateway: async ( + name: string, + options: Parameters[1], + ) => { + expect(await options!.probeImpl!(name)).toBeNull(); + return false; + }, + }); + await expect(restartSandboxGateway("alpha", { quiet: true, deps })).resolves.toMatchObject({ + ok: false, + failureLayer: "health timeout", + }); + expect(readinessDiagnostic().samples).toEqual([ + { + ordinal: 1, + commandStatus: null, + transportErrorKind: null, + httpStatus: null, + jsonParsed: false, + ready: null, + probeResult: null, + }, + ]); + expect(execute).toHaveBeenCalledTimes(2); + }); + it("treats lost OpenClaw readiness transport as unavailable evidence", async () => { silenceConsole(); const execute = vi diff --git a/src/lib/actions/sandbox/gateway-restart.ts b/src/lib/actions/sandbox/gateway-restart.ts index 2588017bd51..27460af383d 100644 --- a/src/lib/actions/sandbox/gateway-restart.ts +++ b/src/lib/actions/sandbox/gateway-restart.ts @@ -371,24 +371,50 @@ function failedAuxiliaryRecoveryDetail(results: RestartAuxiliaryRecoveryResult[] return `gateway health passed but ${failed.join(", ")} could not be re-established`; } -function openClawRestartReady(result: GatewayRestartCommandResult | null): boolean | null { - if (result === null) return null; - const output = result.stdout.trimEnd(); +function openClawRestartObservation( + result: GatewayRestartCommandResult | null, + transportError: SandboxCommandTransportError | null = null, +) { + const output = result?.stdout.trimEnd() ?? ""; const separator = output.lastIndexOf("\n"); - if (result.status !== 0 || separator < 0 || output.slice(separator + 1).trim() !== "200") { - return false; - } + const statusLine = separator < 0 ? "" : output.slice(separator + 1).trim(); + let jsonParsed = false; + let ready: boolean | null = null; try { const document: unknown = JSON.parse(output.slice(0, separator)); - return ( + jsonParsed = separator >= 0; + if ( + jsonParsed && document !== null && typeof document === "object" && "ready" in document && - document.ready === true - ); + typeof document.ready === "boolean" + ) { + ready = document.ready; + } } catch { - return false; + // Only fixed fields leave this parser; response bodies can contain secrets. } + const knownTransportKinds = [ + "cancelled", + "capture", + "invocation", + "timeout", + "unavailable", + "malformed", + ]; + return { + commandStatus: result && Number.isSafeInteger(result.status) ? result.status : null, + transportErrorKind: + transportError && knownTransportKinds.includes(transportError.kind) + ? transportError.kind + : null, + httpStatus: /^[1-5][0-9]{2}$/.test(statusLine) ? Number(statusLine) : null, + jsonParsed, + ready, + probeResult: + result === null ? null : result.status === 0 && statusLine === "200" && ready === true, + }; } export async function restartSandboxGatewayWithDeps( @@ -500,6 +526,18 @@ export async function restartSandboxGatewayWithDeps( return { ok: false, failureLayer: "health timeout", detail }; } + let readinessProbeCount = 0; + const readinessSamples: (ReturnType & { ordinal: number })[] = + []; + const observeReadiness = ( + result: GatewayRestartCommandResult | null, + error: SandboxCommandTransportError | null = null, + ) => { + const observation = openClawRestartObservation(result, error); + readinessSamples.push({ ordinal: ++readinessProbeCount, ...observation }); + if (readinessSamples.length > 32) readinessSamples.shift(); + return observation.probeResult; + }; if ( !(await deps.waitForRecoveredSandboxGateway(sandboxName, { quiet, @@ -516,10 +554,10 @@ export async function restartSandboxGatewayWithDeps( deps.buildOpenClawReadinessProbeCommand(name), 10_000, ); - return openClawRestartReady(result); + return observeReadiness(result); } catch (error) { if (!(error instanceof SandboxCommandTransportError)) throw error; - return null; + return observeReadiness(null, error); } }, } @@ -528,6 +566,16 @@ export async function restartSandboxGatewayWithDeps( ) { const detail = `gateway process restarted but ${agentName === "openclaw" ? "readiness" : "health"} did not pass before timeout`; printGatewayRestartFailure(sandboxName, "health timeout", detail); + if (agentName === "openclaw") { + console.error( + ` OpenClaw restart readiness: ${JSON.stringify({ + nativeRestartStatus: restartResult.status, + totalProbes: readinessProbeCount, + droppedSamples: readinessProbeCount - readinessSamples.length, + samples: readinessSamples, + })}`, + ); + } await deps.printGatewayWedgeDiagnostics(sandboxName, deps.executeSandboxExecCommand); return { ok: false, failureLayer: "health timeout", detail }; } diff --git a/src/lib/actions/sandbox/inference-invocation-probe.test.ts b/src/lib/actions/sandbox/inference-invocation-probe.test.ts index 025466a15c8..fb066af7bf1 100644 --- a/src/lib/actions/sandbox/inference-invocation-probe.test.ts +++ b/src/lib/actions/sandbox/inference-invocation-probe.test.ts @@ -944,24 +944,68 @@ describe("native hosted inference probe credential boundary", () => { it.each([ undefined, + "", "sk-not-a-supervisor-placeholder", "openshell:resolve:env:OTHER_API_KEY", "openshell:resolve:env:OPENAI_API_KEY;evil", - ])("rejects absent, raw, or wrong native credentials before curl [case %#]", (credential) => { - const command = buildSandboxInferenceInvocationCommand({ - ...input, - provider: "openai-api", - nativeProvider: true, - }); - const result = spawnSync("/bin/sh", ["-c", command], { - env: { PATH: process.env.PATH, ...(credential ? { OPENAI_API_KEY: credential } : {}) }, - encoding: "utf8", - timeout: 5000, - }); - expect(result.status).toBe(2); - expect(result.stdout).toBe(""); - expect(result.stderr).toBe(""); - }); + "openshell:resolve:env:OPENAI_API_KEY\nsecret-canary", + ])( + "reports absent or invalid native credentials without exposing values [case %#]", + async (credential) => { + const selected = { + ...input, + provider: "compatible-endpoint", + nativeProvider: true, + nativeEndpointUrl: "https://custom.example/v1", + }; + const command = buildSandboxInferenceInvocationCommand(selected); + const result = spawnSync("/bin/sh", ["-c", command], { + env: { PATH: process.env.PATH, ...(credential ? { COMPATIBLE_API_KEY: credential } : {}) }, + encoding: "utf8", + timeout: 5000, + }); + expect(result.status).toBe(2); + expect(result.stdout).toBe(""); + expect(result.stderr).toBe( + `native-hosted-credential:${credential ? "invalid" : "missing"}\n`, + ); + expect( + await probeSandboxInferenceInvocation(selected, { + execute: async () => ({ + status: result.status!, + stdout: result.stdout, + stderr: result.stderr, + }), + }), + ).toEqual({ + ok: false, + detail: `sandbox inference invocation probe exited with status 2: the native inference credential handle is ${credential ? "invalid" : "missing"}`, + httpStatus: null, + endpoint: "https://custom.example/v1/chat/completions", + }); + }, + ); + + it.each([ + [2, "", "native-hosted-credential:missing secret-canary", true], + [2, "", "secret-canary\nnative-hosted-credential:invalid", true], + [7, "", "native-hosted-credential:missing", true], + [2, "curl-error:2\n", "native-hosted-credential:missing", true], + [2, "", "native-hosted-credential:missing", false], + ] as const)( + "ignores credential diagnostics outside the native rejection result [case %#]", + async (status, stdout, stderr, nativeProvider) => { + const result = await probeSandboxInferenceInvocation( + { ...input, nativeProvider, nativeEndpointUrl: "https://custom.example/v1" }, + { execute: async () => ({ status, stdout, stderr }) }, + ); + expect(result).toMatchObject({ + ok: false, + detail: `sandbox inference invocation probe exited with status ${status}`, + httpStatus: null, + }); + }, + ); }); it.each([true, undefined])( diff --git a/src/lib/actions/sandbox/inference-invocation-probe.ts b/src/lib/actions/sandbox/inference-invocation-probe.ts index 70e975f5091..bf4dd220f36 100644 --- a/src/lib/actions/sandbox/inference-invocation-probe.ts +++ b/src/lib/actions/sandbox/inference-invocation-probe.ts @@ -1,8 +1,17 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + normalizeRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "../../inference/config"; + import { managedDcodeRequiresNonemptyContent } from "../../inference/managed-dcode/identity"; -import { nativeHostedAuthHeaderScript } from "../../inference/native-provider/contract"; +import { + nativeHostedAuthHeaderScript, + NATIVE_HOSTED_CREDENTIAL_MISSING, + NATIVE_HOSTED_CREDENTIAL_INVALID, +} from "../../inference/native-provider/contract"; import { NATIVE_LOCAL_AUTH_HEADER_SCRIPT } from "../../inference/native-local/agent-config"; import { NATIVE_NVIDIA_AUTH_HEADER_SCRIPT } from "../../inference/native-nvidia/contract"; import { @@ -63,6 +72,7 @@ export type SandboxInferenceInvocationInput = { preferredInferenceApi: string | null; nativeProvider?: boolean; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; nativeEndpointUrl?: string; }; @@ -92,6 +102,27 @@ export const READINESS_INFERENCE_INVOCATION_TIMEOUT_MS = 95_000; const INFERENCE_INVOCATION_MAX_RESPONSE_BYTES = 64 * 1024; function nativeProbeDefinition(input: SandboxInferenceInvocationInput) { + if (input.registeredInferenceProviderAttachment) { + const binding = normalizeRegisteredInferenceProviderAttachment( + input.registeredInferenceProviderAttachment, + ); + if ( + !binding || + binding.providerName !== input.provider || + binding.sandboxName !== input.sandboxName || + binding.gatewayName !== input.gatewayName || + binding.inferenceApi !== input.preferredInferenceApi + ) + throw new Error("Registered inference probe does not match its selected attachment"); + if (new URL(binding.endpointUrl).protocol !== "https:") + throw new Error("Registered inference requires its explicit native HTTPS endpoint"); + return { + logicalProvider: binding.providerName, + endpoint: binding.endpointUrl, + credentialEnv: binding.credentialEnv, + api: binding.inferenceApi, + }; + } if (input.nativeLocalProviderAttachment) return undefined; if (input.nativeProvider && isCustomHostedProvider(input.provider)) { if (!input.nativeEndpointUrl) @@ -132,10 +163,14 @@ function buildProbeRequest(input: SandboxInferenceInvocationInput): { input.nativeProvider ?? Boolean(local), local?.endpointUrl ?? input.nativeEndpointUrl ?? null, input.agentName, + input.registeredInferenceProviderAttachment, ); const useNativeNvidia = input.nativeProvider === true && isNativeNvidiaProvider(input.provider); const hosted = nativeProbeDefinition(input); - const nativeHosted = input.nativeProvider === true ? hosted : undefined; + const nativeHosted = + input.nativeProvider === true || input.registeredInferenceProviderAttachment + ? hosted + : undefined; if (input.nativeProvider === true && !useNativeNvidia && !nativeHosted && !local) { throw new Error("Unsupported native inference probe selection"); } @@ -216,7 +251,10 @@ export function buildSandboxInferenceInvocationCommand( const useNativeLocal = Boolean( normalizeNativeLocalProviderAttachment(input.nativeLocalProviderAttachment), ); - const nativeHosted = input.nativeProvider === true ? nativeProbeDefinition(input) : undefined; + const nativeHosted = + input.nativeProvider === true || input.registeredInferenceProviderAttachment + ? nativeProbeDefinition(input) + : undefined; const nativeAuth = useNativeNvidia ? NATIVE_NVIDIA_AUTH_HEADER_SCRIPT : useNativeLocal @@ -232,6 +270,11 @@ export function buildSandboxInferenceInvocationCommand( const endpoint = shellQuote(request.endpoint); return [ "umask 077", + ...(input.registeredInferenceProviderAttachment && nativeHosted + ? [ + `printf '%s' "$${nativeHosted.credentialEnv}" | LC_ALL=C grep -Eq '^openshell:resolve:env:(v[0-9]{1,20}|s[a-f0-9]{64})_${nativeHosted.credentialEnv}$' || exit 2`, + ] + : []), ...(nativeAuth ? [nativeAuth] : []), "body=$(mktemp /tmp/nemoclaw-inference-invocation.XXXXXX) || exit 1", "trap 'rm -f \"$body\"' EXIT HUP INT TERM", @@ -348,6 +391,7 @@ export async function probeSandboxInferenceInvocation( input.nativeEndpointUrl ?? null, input.agentName, + input.registeredInferenceProviderAttachment, ).inferenceApi; if (httpStatus !== null && validateInferenceResponseBody(inferenceApi, body).ok) { return { ok: true }; @@ -374,6 +418,22 @@ export async function probeSandboxInferenceInvocation( : httpStatus ? `sandbox inference invocation probe returned HTTP ${httpStatus}` : `sandbox inference invocation probe exited with status ${result.status}`; + if (result.status === 2 && !result.stdout && input.nativeProvider === true) { + const credentialError = + result.stderr.trim() === NATIVE_HOSTED_CREDENTIAL_MISSING + ? "the native inference credential handle is missing" + : result.stderr.trim() === NATIVE_HOSTED_CREDENTIAL_INVALID + ? "the native inference credential handle is invalid" + : null; + if (credentialError) { + return { + ok: false, + detail: `${detail}: ${credentialError}`, + httpStatus: null, + endpoint: resolveSandboxInferenceInvocationEndpoint(input), + }; + } + } const legacyDnsFailure = input.nativeProvider !== true && result.stdout.split("\n").some((line) => line.trim() === "curl-error:6") && diff --git a/src/lib/actions/sandbox/inference-route-health.test.ts b/src/lib/actions/sandbox/inference-route-health.test.ts index 5ff04b28fc3..80dda7b0a4c 100644 --- a/src/lib/actions/sandbox/inference-route-health.test.ts +++ b/src/lib/actions/sandbox/inference-route-health.test.ts @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it, vi } from "vitest"; +import { beforeEach, describe, expect, it, vi } from "vitest"; import type { OpenShellSandboxBufferedCommandExecutor } from "../../adapters/openshell/sandbox-command"; import { DCODE_MANAGED_EXEC_LAUNCHER, @@ -583,3 +583,276 @@ it("offers conditional recreation guidance when the legacy route cannot be reach expect(health.ok).toBe(false); expect(health.detail).toContain("If this legacy route is no longer available"); }); + +vi.mock("../../inference/native-local/profile", async (importOriginal) => ({ + ...(await importOriginal()), + verifyRecordedNativeLocalProviderAttachment: vi.fn(async () => undefined), +})); +vi.mock("../../adapters/openshell/provider-adapter-cli", () => ({ + createCliOpenShellProviderAdapter: vi.fn(() => ({})), +})); +vi.mock("../../state/registry/cross-port", () => ({ findSandboxAcrossGatewayRoots: vi.fn() })); +vi.mock("../../onboard/sandbox-recreate-probe", () => ({ observeSandboxOnGateway: vi.fn() })); +vi.mock("../../sandbox/config", async (importOriginal) => ({ + ...(await importOriginal()), + readSandboxConfig: vi.fn(), + resolveAgentConfig: vi.fn(), + setOpenClawConfigValues: vi.fn(), +})); + +const { nativeLocalIdentity } = await import("../../inference/native-local/contract"); +const { verifyRecordedNativeLocalProviderAttachment } = + await import("../../inference/native-local/profile"); +const { findSandboxAcrossGatewayRoots } = await import("../../state/registry/cross-port"); +const { observeSandboxOnGateway } = await import("../../onboard/sandbox-recreate-probe"); +const { observeBlueprintNativeInference, prepareBlueprintNativeInference } = + await import("../../blueprint-runner"); +const { readSandboxConfig, resolveAgentConfig, setOpenClawConfigValues } = + await import("../../sandbox/config"); +const localBinding = { + provider: "compatible-endpoint", + agent: "openclaw" as const, + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated" as const, + gatewayName: "nemoclaw", + sandboxName: "alpha", +}; +const nativeEntry = { + name: "alpha", + agent: "openclaw" as const, + gatewayName: "nemoclaw", + provider: "compatible-endpoint", + model: "test-model", + preferredInferenceApi: "openai-completions", + endpointUrl: "http://localhost:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + lifecycleGeneration: "generation-one", + lifecycleLiveIdentityFingerprint: "identity-one", + nativeLocalProviderAttachment: { + ...localBinding, + ...nativeLocalIdentity(localBinding), + schemaVersion: 1 as const, + providerId: "provider-id", + }, +}; +const nativeRequest = { + gatewayName: "nemoclaw", + sandboxName: "alpha", + provider: "compatible-endpoint", + model: "test-model", + api: "openai-completions", +}; + +describe("blueprint native inference authority", () => { + beforeEach(() => { + vi.mocked(verifyRecordedNativeLocalProviderAttachment) + .mockReset() + .mockResolvedValue(nativeEntry.nativeLocalProviderAttachment); + vi.mocked(findSandboxAcrossGatewayRoots) + .mockReset() + .mockReturnValue({ + entry: nativeEntry, + registryFile: "/test-state/sandboxes.json", + gatewayPort: 8080, + } as ReturnType); + vi.mocked(observeSandboxOnGateway) + .mockReset() + .mockReturnValue({ state: "ready", liveIdentityFingerprint: "identity-one" }); + vi.mocked(resolveAgentConfig).mockReturnValue({ + agentName: "openclaw", + configPath: "/sandbox/.openclaw/openclaw.json", + configDir: "/sandbox/.openclaw", + configFile: "openclaw.json", + format: "json", + }); + vi.mocked(readSandboxConfig).mockReturnValue({ + agents: { defaults: { model: { primary: "inference/test-model" } } }, + models: { providers: { inference: { api: "openai-completions", timeoutSeconds: 180 } } }, + }); + }); + + it("updates the selected timeout while registered sandbox authority still matches", async () => { + await prepareBlueprintNativeInference({ ...nativeRequest, timeoutSeconds: 300 }); + expect(setOpenClawConfigValues).toHaveBeenCalledExactlyOnceWith( + "alpha", + [{ dotpath: "models.providers.inference.timeoutSeconds", value: 300 }], + "nemoclaw", + ); + }); + + it.each([ + { + name: "registration", + replace: () => { + vi.mocked(findSandboxAcrossGatewayRoots).mockReturnValue({ + entry: { ...nativeEntry, lifecycleGeneration: "replacement" }, + registryFile: "/test-state/sandboxes.json", + gatewayPort: 8080, + } as ReturnType); + }, + }, + { + name: "live sandbox", + replace: () => { + vi.mocked(observeSandboxOnGateway).mockReturnValue({ + state: "ready", + liveIdentityFingerprint: "replacement", + }); + }, + }, + ])( + "refuses a timeout write after $name replacement at the final observer return", + async ({ replace }) => { + let replaced = false; + const original = { state: "ready" as const, liveIdentityFingerprint: "identity-one" }; + // Each observer checks live authority three times; replacement follows the second observer. + vi.mocked(observeSandboxOnGateway) + .mockReturnValueOnce(original) + .mockReturnValueOnce(original) + .mockReturnValueOnce(original) + .mockReturnValueOnce(original) + .mockReturnValueOnce(original) + .mockImplementationOnce(() => { + queueMicrotask(() => { + replaced = true; + replace(); + }); + return original; + }); + await expect( + prepareBlueprintNativeInference({ ...nativeRequest, timeoutSeconds: 300 }), + ).rejects.toThrow(/identity.*changed/); + expect(replaced).toBe(true); + expect(setOpenClawConfigValues).not.toHaveBeenCalled(); + }, + ); + + it.each([ + "http://localhost:8000/v1", + "http://127.0.0.1:8000/v1/", + "http://[::1]:8000/v1", + "http://host.openshell.internal:8000/v1", + ])("verifies the recorded native attachment for selected URL %s", async (endpointUrl) => { + vi.mocked(findSandboxAcrossGatewayRoots).mockReturnValue({ + entry: { ...nativeEntry, endpointUrl }, + registryFile: "/test-state/sandboxes.json", + gatewayPort: 8080, + } as ReturnType); + const identity = await observeBlueprintNativeInference(nativeRequest); + expect(JSON.parse(identity)).toMatchObject({ + provider: "compatible-endpoint", + model: "test-model", + attachment: { providerId: "provider-id" }, + }); + expect(verifyRecordedNativeLocalProviderAttachment).toHaveBeenCalledWith( + expect.objectContaining({ + sandboxName: "alpha", + gatewayName: "nemoclaw", + expected: nativeEntry.nativeLocalProviderAttachment, + agent: "openclaw", + provider: "compatible-endpoint", + }), + ); + expect(observeSandboxOnGateway).toHaveBeenCalledWith( + { sandboxName: "alpha", gatewayName: "nemoclaw" }, + undefined, + undefined, + 15_000, + ); + }); + + it.each([ + { gatewayName: "other" }, + { provider: "other" }, + { model: "other" }, + { api: "openai-responses" }, + { endpointUrl: "http://host.openshell.internal:9000/v1" }, + ])("refuses a changed requested selection %j before attachment lookup", async (change) => { + await expect(observeBlueprintNativeInference({ ...nativeRequest, ...change })).rejects.toThrow( + /matching registered native inference/, + ); + expect(verifyRecordedNativeLocalProviderAttachment).not.toHaveBeenCalled(); + }); + + it.each([ + { nativeLocalProviderAttachment: undefined }, + { + nativeLocalProviderAttachment: { + ...nativeEntry.nativeLocalProviderAttachment, + providerName: "forged", + }, + }, + { nativeHostedProviderAttachment: {} }, + { agent: "hermes" }, + { endpointUrl: "http://host.openshell.internal:9000/v1" }, + { + nativeLocalProviderAttachment: { + ...nativeEntry.nativeLocalProviderAttachment, + credentialEnv: "OTHER_KEY", + ...nativeLocalIdentity({ ...localBinding, credentialEnv: "OTHER_KEY" }), + }, + }, + { + nativeLocalProviderAttachment: { + ...nativeEntry.nativeLocalProviderAttachment, + api: "openai-responses", + ...nativeLocalIdentity({ ...localBinding, api: "openai-responses" }), + }, + }, + ])("refuses invalid recorded native authority %j", async (change) => { + vi.mocked(findSandboxAcrossGatewayRoots).mockReturnValue({ + entry: { ...nativeEntry, ...change }, + registryFile: "/test-state/sandboxes.json", + gatewayPort: 8080, + } as ReturnType); + await expect(observeBlueprintNativeInference(nativeRequest)).rejects.toThrow( + /matching registered native inference/, + ); + expect(verifyRecordedNativeLocalProviderAttachment).not.toHaveBeenCalled(); + }); + + it("propagates actual provider identity and profile refusal", async () => { + vi.mocked(verifyRecordedNativeLocalProviderAttachment).mockRejectedValue( + new Error("provider profile or identity changed"), + ); + await expect(observeBlueprintNativeInference(nativeRequest)).rejects.toThrow( + /profile or identity changed/, + ); + }); + + it("refuses missing recorded lifecycle authority", async () => { + vi.mocked(findSandboxAcrossGatewayRoots).mockReturnValue(null); + await expect(observeBlueprintNativeInference(nativeRequest)).rejects.toThrow( + /recorded sandbox lifecycle authority/, + ); + }); + + it("refuses replacement live sandbox identity", async () => { + vi.mocked(observeSandboxOnGateway).mockReturnValue({ + state: "ready", + liveIdentityFingerprint: "replacement", + }); + await expect(observeBlueprintNativeInference(nativeRequest)).rejects.toThrow( + /live identity.*changed/, + ); + expect(verifyRecordedNativeLocalProviderAttachment).not.toHaveBeenCalled(); + }); + + it.each([{ model: "replacement" }, { credentialEnv: "OTHER_KEY" }])( + "refuses selection changes during provider verification %j", + async (change) => { + vi.mocked(verifyRecordedNativeLocalProviderAttachment).mockImplementation(async () => { + vi.mocked(findSandboxAcrossGatewayRoots).mockReturnValue({ + entry: { ...nativeEntry, ...change }, + registryFile: "/test-state/sandboxes.json", + gatewayPort: 8080, + } as ReturnType); + return nativeEntry.nativeLocalProviderAttachment; + }); + await expect(observeBlueprintNativeInference(nativeRequest)).rejects.toThrow( + /matching registered native inference|selection changed/, + ); + }, + ); +}); diff --git a/src/lib/actions/sandbox/inference-route-health.ts b/src/lib/actions/sandbox/inference-route-health.ts index 780525e6a11..6136f6a9910 100644 --- a/src/lib/actions/sandbox/inference-route-health.ts +++ b/src/lib/actions/sandbox/inference-route-health.ts @@ -2,10 +2,21 @@ // SPDX-License-Identifier: Apache-2.0 import { + recordedRegisteredInferenceProviderAttachment, + verifyRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; +import { + gatewayReachableCompatibleEndpointUrl, + NATIVE_LOCAL_CREDENTIAL_ENV, + normalizeNativeLocalBinding, normalizeNativeLocalProviderAttachment, type NativeLocalProviderAttachment, } from "../../inference/native-local/contract"; -import { verifyNativeLocalProviderAttachment } from "../../inference/native-local/profile"; +import { + verifyNativeLocalProviderAttachment, + verifyRecordedNativeLocalProviderAttachment, +} from "../../inference/native-local/profile"; import type { SandboxEntry } from "../../state/registry/types"; import { recordedNativeProviderAttachment } from "../../inference/native-provider/recorded-selection"; import { hostedNativeProvider } from "../../inference/native-provider/hosted"; @@ -16,7 +27,6 @@ import { } from "../../inference/native-provider/fixed"; import { nativeProviderLifecycle } from "../../inference/native-provider"; import type { NativeProviderAttachment } from "../../inference/native-provider/contract"; -import { NATIVE_NVIDIA_AUTH_HEADER_SCRIPT } from "../../inference/native-nvidia/contract"; import { buildSandboxCommandEnvironment } from "../../adapters/sandbox/command-transport"; import type { OpenShellSandboxBufferedCommandExecutor } from "../../adapters/openshell/sandbox-command"; import { createCliOpenShellSandboxCommandExecutor } from "../../adapters/openshell/sandbox-command-cli"; @@ -26,6 +36,7 @@ import * as agentRuntime from "../../agent/runtime"; import { REPOSITORY_ROOT } from "../../core/repository-root"; import type { ProviderHealthStatus } from "../../inference/health"; import { + NATIVE_NVIDIA_AUTH_HEADER_SCRIPT, NVIDIA_HOSTED_NATIVE_ENDPOINT, isNativeNvidiaProvider, verifyNativeNvidiaProviderAttachment, @@ -76,6 +87,24 @@ export async function verifyNativeNvidiaStatusAttachment(input: { }); } +export type VerifyRegisteredStatusAttachment = (input: { + gatewayName: string; + sandboxName: string; + expected: RegisteredInferenceProviderAttachment; +}) => Promise; + +export const verifyRegisteredStatusAttachment: VerifyRegisteredStatusAttachment = async (input) => { + if ( + input.expected.gatewayName !== input.gatewayName || + input.expected.sandboxName !== input.sandboxName + ) + throw new Error("Registered provider sandbox or gateway binding changed."); + await verifyRegisteredInferenceProviderAttachment({ + adapter: createCliOpenShellProviderAdapter(), + expected: input.expected, + }); +}; + export type VerifyNativeLocalStatusAttachment = (input: { gatewayName: string; sandboxName: string; @@ -417,6 +446,7 @@ export type SandboxInferenceRouteHealthContext = { nativeLocalEndpoint?: string; nativeProvider?: boolean; nativeEndpointUrl?: string; + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; model?: string | null; preferredInferenceApi?: string | null; }; @@ -473,6 +503,9 @@ export function buildSandboxInferenceRouteHealth( }; } const definition = + (context.registeredInferenceProviderAttachment + ? { logicalProvider: context.provider ?? "", label: "registered" } + : undefined) ?? fixedNativeProvider(context.provider) ?? (context.nativeProvider && (context.provider === "compatible-endpoint" || @@ -484,12 +517,14 @@ export function buildSandboxInferenceRouteHealth( invocation && !invocation.ok && invocation.endpoint ? invocation.endpoint : resolveSandboxInferenceInvocationEndpoint({ - sandboxName: "", + sandboxName: context.registeredInferenceProviderAttachment?.sandboxName ?? "", provider: definition.logicalProvider, model: context.model ?? "", preferredInferenceApi: context.preferredInferenceApi ?? null, nativeProvider: true, nativeEndpointUrl: context.nativeEndpointUrl, + gatewayName: context.registeredInferenceProviderAttachment?.gatewayName, + registeredInferenceProviderAttachment: context.registeredInferenceProviderAttachment, }); const diagnostics = providerHealthDiagnostics(providerHealth, Boolean(invocation?.ok)); if ( @@ -516,7 +551,9 @@ export function buildSandboxInferenceRouteHealth( endpoint, detail: invocation ? `The native ${definition.label} route did not serve an inference request: ${invocation.detail}.` - : `Could not probe the native ${definition.label} route from inside the sandbox. Recreate legacy beta sandboxes before using this route.`, + : context.registeredInferenceProviderAttachment + ? "Could not probe the registered provider from inside the sandbox. Check its binding and selected model, then retry." + : `Could not probe the native ${definition.label} route from inside the sandbox. Recreate legacy beta sandboxes before using this route.`, failureLabel: classifyInferenceInvocationFailureLabel(invocation?.httpStatus ?? null), }; return diagnostics.length > 0 ? { ...nativeHealth, subprobes: diagnostics } : nativeHealth; @@ -617,9 +654,12 @@ export async function runSandboxInferenceInvocationProbe( export function readNativeStatusSelection(entry: SandboxEntry | null | undefined) { let attachment: NativeProviderAttachment | undefined; + let registeredAttachment: RegisteredInferenceProviderAttachment | undefined; let failure: string | null = null; try { - if (entry?.nativeLocalProviderAttachment !== undefined) { + if (entry?.registeredInferenceProviderAttachment !== undefined) { + registeredAttachment = recordedRegisteredInferenceProviderAttachment(entry); + } else if (entry?.nativeLocalProviderAttachment !== undefined) { attachment = normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment); if (!attachment) failure = @@ -628,8 +668,9 @@ export function readNativeStatusSelection(entry: SandboxEntry | null | undefined attachment = entry ? recordedNativeProviderAttachment(entry) : undefined; } } catch { - failure = "Recorded native provider ownership is invalid; no inference request was sent."; + failure = "Recorded native provider binding is invalid; no inference request was sent."; } + const registered = entry?.registeredInferenceProviderAttachment !== undefined; const local = entry?.nativeLocalProviderAttachment !== undefined; const definition = local ? null @@ -638,14 +679,97 @@ export function readNativeStatusSelection(entry: SandboxEntry | null | undefined : hostedNativeProvider(entry?.provider); return { attachment, + registeredAttachment, failure, - label: local ? "local" : (definition?.label ?? "NVIDIA"), - endpoint: local - ? (attachment?.endpointUrl ?? "") - : (definition?.endpoint ?? NVIDIA_HOSTED_NATIVE_ENDPOINT), + label: registered ? "registered" : local ? "local" : (definition?.label ?? "NVIDIA"), + endpoint: registered + ? (registeredAttachment?.endpointUrl ?? "") + : local + ? (attachment?.endpointUrl ?? "") + : (definition?.endpoint ?? NVIDIA_HOSTED_NATIVE_ENDPOINT), }; } +/** Verify an existing inference selection before attaching an independent runtime identity. */ +export async function verifyBlueprintNativeInferenceSelection( + entry: SandboxEntry, + requested: Readonly<{ + gatewayName: string; + sandboxName: string; + provider: string; + model: string; + api: string; + endpointUrl?: string; + }>, +): Promise { + const selection = readNativeStatusSelection(entry); + const attachment = selection.registeredAttachment ?? selection.attachment; + const local = + entry.nativeLocalProviderAttachment === undefined + ? undefined + : normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment); + // The selection keeps the user's URL and source credential name. The + // attachment uses the admitted host bridge and the sandbox credential handle. + const selectedLocal = local + ? normalizeNativeLocalBinding({ + ...local, + endpointUrl: gatewayReachableCompatibleEndpointUrl(entry.provider ?? "", entry.endpointUrl), + }) + : undefined; + if ( + entry.name !== requested.sandboxName || + entry.gatewayName !== requested.gatewayName || + entry.provider !== requested.provider || + entry.model !== requested.model || + (entry.agent ?? "openclaw") !== "openclaw" || + (entry.preferredInferenceApi ?? "openai-completions") !== requested.api || + (requested.endpointUrl && + entry.endpointUrl?.replace(/\/+$/, "") !== requested.endpointUrl.replace(/\/+$/, "")) || + selection.failure || + (local && + ((local.api ?? "openai-completions") !== requested.api || + local.endpointUrl !== selectedLocal?.endpointUrl || + local.credentialEnv !== NATIVE_LOCAL_CREDENTIAL_ENV)) || + !attachment || + [ + entry.nativeLocalProviderAttachment, + entry.nativeHostedProviderAttachment, + entry.nativeNvidiaProviderAttachment, + entry.registeredInferenceProviderAttachment, + ].filter((value) => value !== undefined).length !== 1 + ) + throw new Error( + "Runtime identity apply requires the matching registered native inference selection.", + ); + const target = { gatewayName: requested.gatewayName, sandboxName: requested.sandboxName }; + if (selection.registeredAttachment) { + await verifyRegisteredStatusAttachment({ ...target, expected: selection.registeredAttachment }); + } else if (entry.nativeLocalProviderAttachment !== undefined) { + await verifyRecordedNativeLocalProviderAttachment({ + adapter: createCliOpenShellProviderAdapter(), + ...target, + expected: entry.nativeLocalProviderAttachment, + agent: entry.agent ?? "openclaw", + provider: entry.provider, + }); + } else { + await verifyNativeHostedStatusAttachment({ ...target, expected: attachment }); + } + return JSON.stringify({ + sandboxName: entry.name, + gatewayName: entry.gatewayName, + agent: entry.agent ?? "openclaw", + provider: entry.provider, + model: entry.model, + api: entry.preferredInferenceApi ?? "openai-completions", + endpointUrl: entry.endpointUrl, + credentialEnv: entry.credentialEnv, + lifecycleGeneration: entry.lifecycleGeneration, + lifecycleLiveIdentityFingerprint: entry.lifecycleLiveIdentityFingerprint, + attachment, + }); +} + export class HermesNvidiaCredentialRejectedError extends Error { constructor() { super("Hermes NVIDIA credential is invalid or stale."); diff --git a/src/lib/actions/sandbox/launch-readiness-gateway-health.test.ts b/src/lib/actions/sandbox/launch-readiness-gateway-health.test.ts index 1fdf9975206..b2f9bc7bec7 100644 --- a/src/lib/actions/sandbox/launch-readiness-gateway-health.test.ts +++ b/src/lib/actions/sandbox/launch-readiness-gateway-health.test.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; + import { execFileSync } from "node:child_process"; import { describe, expect, it, vi } from "vitest"; @@ -592,3 +594,33 @@ describe("OpenClaw native-local readiness identity (#13338)", () => { }, ); }); + +it("requires the recorded external identity before invoking its native route", async () => { + const f = registeredInferenceFixture(true); + const verifyRegisteredAttachment = vi.fn(async () => {}); + const inferenceInvocationProbe = vi.fn(async () => ({ ok: true }) as const); + const input = { + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + agentName: "openclaw", + entry: { + name: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + provider: f.binding.providerName, + model: "model-a", + endpointUrl: f.binding.endpointUrl, + credentialEnv: f.binding.credentialEnv, + preferredInferenceApi: f.binding.inferenceApi, + registeredInferenceProviderAttachment: f.binding, + }, + deps: { verifyRegisteredAttachment, inferenceInvocationProbe }, + }; + await expect(requireNativeInferenceHealth(input)).resolves.toBe(true); + expect(inferenceInvocationProbe).toHaveBeenCalledWith( + expect.objectContaining({ registeredInferenceProviderAttachment: f.binding }), + ); + verifyRegisteredAttachment.mockRejectedValue(new Error("replaced")); + inferenceInvocationProbe.mockClear(); + await expect(requireNativeInferenceHealth(input)).rejects.toThrow("replaced"); + expect(inferenceInvocationProbe).not.toHaveBeenCalled(); +}); diff --git a/src/lib/actions/sandbox/launch-readiness.test.ts b/src/lib/actions/sandbox/launch-readiness.test.ts index 99724ed72b1..06207d7c221 100644 --- a/src/lib/actions/sandbox/launch-readiness.test.ts +++ b/src/lib/actions/sandbox/launch-readiness.test.ts @@ -26,6 +26,8 @@ import { withLaunchReadinessMutationGate, } from "./launch-readiness"; +import { OpenClawPairingQualificationError } from "./launch-readiness/openclaw-pairing-qualification"; + const SANDBOX = "alpha"; const GATEWAY_NAME = "nemoclaw"; const GATEWAY_PORT = 8080; @@ -363,7 +365,11 @@ describe("launch readiness validation", () => { publishLaunchReadiness(publicationFromDecision(SANDBOX, first), currentDeps), ).resolves.toEqual({ kind: "evidence-failed", - diagnostic: { stage: "publication-validation", reason: "pairing-observation-failed" }, + diagnostic: { + stage: "publication-validation", + reason: "pairing-observation-failed", + pairingFailureCategory: "unclassified", + }, }); expect(currentDeps.commandExecutor!.runBuffered).not.toHaveBeenCalled(); expect(publishedIdentity).toBeNull(); @@ -1334,7 +1340,24 @@ describe("launch readiness validation", () => { }; expect(await publishLaunchReadiness(publication, pairingObservationUnavailable)).toEqual({ kind: "evidence-failed", - diagnostic: { stage: "publication-validation", reason: "pairing-observation-failed" }, + diagnostic: { + stage: "publication-validation", + reason: "pairing-observation-failed", + pairingFailureCategory: "unclassified", + }, + }); + + const pairingDeadline = deps(); + pairingDeadline.observeOpenClawPairingQualification = () => { + throw new OpenClawPairingQualificationError("deadline-exceeded"); + }; + expect(await publishLaunchReadiness(publication, pairingDeadline)).toEqual({ + kind: "evidence-failed", + diagnostic: { + stage: "publication-validation", + reason: "pairing-observation-failed", + pairingFailureCategory: "deadline-exceeded", + }, }); const hashUnavailable = deps(); diff --git a/src/lib/actions/sandbox/launch-readiness.ts b/src/lib/actions/sandbox/launch-readiness.ts index 0603f68af16..0b29a401116 100644 --- a/src/lib/actions/sandbox/launch-readiness.ts +++ b/src/lib/actions/sandbox/launch-readiness.ts @@ -76,6 +76,7 @@ import { observeOpenClawPairingSettlement, OpenClawPairingObservationRetryableError, OpenClawPairingQualificationError, + type OpenClawPairingObservationFailureCategory, OPENCLAW_ONBOARDING_PAIRING_FINAL_OBSERVATION_TIMEOUT_MS, OPENCLAW_ONBOARDING_PAIRING_POLL_MS, OPENCLAW_ONBOARDING_PAIRING_SETTLEMENT_TIMEOUT_MS, @@ -183,6 +184,7 @@ export type LaunchReadinessPublicationResult = | "missing-authority" | "pairing-observation-failed" | "runtime-observation-failed"; + pairingFailureCategory?: OpenClawPairingObservationFailureCategory; }; }; @@ -831,7 +833,8 @@ async function captureLaunchIdentity( const inference = registry.getSandboxEntryInference(entry); const nativeProvider = Boolean(getNativeProviderAttachment(entry)) || - entry.nativeLocalProviderAttachment !== undefined; + entry.nativeLocalProviderAttachment !== undefined || + entry.registeredInferenceProviderAttachment !== undefined; let liveInference: { provider: string; model: string } | null = null; if (!nativeProvider) { const inferenceGetStartedAt = performance.now(); @@ -919,7 +922,8 @@ async function captureLaunchIdentity( openclawVersion, stateDirectory, ); - } catch { + } catch (error) { + if (error instanceof OpenClawPairingQualificationError) throw error; throw new OpenClawPairingQualificationError(); } } @@ -1502,6 +1506,9 @@ export async function publishLaunchReadiness( kind: "evidence-failed", diagnostic: { stage: "publication-validation", + ...(error instanceof OpenClawPairingQualificationError + ? { pairingFailureCategory: error.failureCategory } + : {}), reason: error instanceof OpenClawPairingQualificationError ? "pairing-observation-failed" diff --git a/src/lib/actions/sandbox/launch-readiness/health.ts b/src/lib/actions/sandbox/launch-readiness/health.ts index eae80fe8c73..95ba77d5a8e 100644 --- a/src/lib/actions/sandbox/launch-readiness/health.ts +++ b/src/lib/actions/sandbox/launch-readiness/health.ts @@ -1,8 +1,12 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { recordedRegisteredInferenceProviderAttachment } from "../../../inference/native-provider/registered"; import { normalizeNativeLocalProviderAttachment } from "../../../inference/native-local/contract"; -import { verifyNativeLocalStatusAttachment } from "../inference-route-health"; +import { + verifyNativeLocalStatusAttachment, + verifyRegisteredStatusAttachment, +} from "../inference-route-health"; import { captureOpenshell } from "../../../adapters/openshell/runtime"; import { createCliOpenShellProviderAdapter } from "../../../adapters/openshell/provider-adapter-cli"; @@ -89,6 +93,7 @@ export interface LaunchReadinessHealthDeps { ) => Promise>; inferenceInvocationProbe?: typeof runSandboxInferenceInvocationProbe; verifyNativeLocalAttachment?: typeof verifyNativeLocalStatusAttachment; + verifyRegisteredAttachment?: typeof verifyRegisteredStatusAttachment; verifyNativeNvidiaAttachment?: (input: { sandboxName: string; gatewayName: string; @@ -240,6 +245,33 @@ export async function requireNativeInferenceHealth(input: { deps: LaunchReadinessHealthDeps; allowTransientNvidiaInvocation?: boolean; }): Promise { + const registered = recordedRegisteredInferenceProviderAttachment(input.entry); + if (registered) { + if ( + !input.entry.model || + registered.sandboxName !== input.sandboxName || + registered.gatewayName !== input.gatewayName + ) + throw new LaunchReadinessEvidenceError(); + await (input.deps.verifyRegisteredAttachment ?? verifyRegisteredStatusAttachment)({ + sandboxName: input.sandboxName, + gatewayName: input.gatewayName, + expected: registered, + }); + const result = await ( + input.deps.inferenceInvocationProbe ?? runSandboxInferenceInvocationProbe + )({ + sandboxName: input.sandboxName, + gatewayName: input.gatewayName, + agentName: input.agentName, + provider: registered.providerName, + model: input.entry.model, + preferredInferenceApi: registered.inferenceApi, + registeredInferenceProviderAttachment: registered, + }); + if (!result.ok) throw new LaunchReadinessObservationError("health", "inference request"); + return true; + } if (input.entry.nativeLocalProviderAttachment !== undefined) { const expected = normalizeNativeLocalProviderAttachment( input.entry.nativeLocalProviderAttachment, @@ -437,7 +469,11 @@ export async function requireLaunchSemanticHealth( } if (inferenceConfigured) { const inferenceStartedAt = performance.now(); - if (getNativeProviderAttachment(entry) || entry.nativeLocalProviderAttachment !== undefined) { + if ( + getNativeProviderAttachment(entry) || + entry.nativeLocalProviderAttachment !== undefined || + entry.registeredInferenceProviderAttachment !== undefined + ) { try { await requireNativeInferenceHealth({ sandboxName, diff --git a/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-observation-parsing.test.ts b/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-observation-parsing.test.ts index fb4d7e6e211..e864efa43f7 100644 --- a/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-observation-parsing.test.ts +++ b/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-observation-parsing.test.ts @@ -1,9 +1,17 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import type { spawnSync, SpawnSyncReturns } from "node:child_process"; +import { describe, expect, it, vi } from "vitest"; import { buildOpenClawPairingObservationScript, + observeOpenClawPairingSettlement, + observeOpenClawPairingRepairSettlement, + observeOrdinaryOpenClawPairingSettlement, + observeOpenClawPairingQualification, + OpenClawPairingQualificationError, + OpenClawPairingObservationRetryableError, + OpenClawPairingIncompatibleError, parseOpenClawPairingObservation, parseOpenClawPairingRepairObservation, parseOpenClawPairingSettlementObservation, @@ -67,3 +75,90 @@ describe("OpenClaw pairing observation parsing", () => { expect(script).toContain(Buffer.from(rawStateDirectory, "utf8").toString("base64")); }); }); + +describe.each([ + observeOpenClawPairingSettlement, + observeOpenClawPairingRepairSettlement, + observeOrdinaryOpenClawPairingSettlement, + observeOpenClawPairingQualification, +])("pairing observation failure categories: %s", (observe) => { + it.each([ + [ + "deadline-exceeded", + { error: Object.assign(new Error(TOKEN), { code: "ETIMEDOUT" }), signal: "SIGTERM" }, + ], + ["output-limit-exceeded", { error: Object.assign(new Error(TOKEN), { code: "ENOBUFS" }) }], + ["child-error", { error: Object.assign(new Error(TOKEN), { code: TOKEN }) }], + ["child-signal", { signal: "SIGTERM" }], + ["child-exit-failed", { status: 1 }], + ["child-exit-failed", { status: null }], + ["state-changing", { status: 3 }], + ["incompatible-state", { status: 4 }], + ["invalid-output", { status: 0 }], + ])("reports %s without child output or credentials", (category, overrides) => { + const child = vi.fn().mockReturnValue({ + pid: 123, + status: 0, + signal: null, + output: [], + stdout: TOKEN, + stderr: TOKEN, + ...overrides, + } as SpawnSyncReturns); + const log = vi.spyOn(console, "error").mockImplementation(() => {}); + let caught: unknown; + try { + observe("alpha", "gateway", "2026.9.2", "/sandbox/.openclaw", { + getOpenshellBinary: () => "openshell", + readApprovalPolicy: () => "test-policy", + // This fixture returns UTF-8 output for the observer's fixed encoding. + spawnSync: child as unknown as typeof spawnSync, + }); + } catch (error) { + caught = error; + } + expect(caught).toBeInstanceOf(OpenClawPairingQualificationError); + expect(caught).toMatchObject({ failureCategory: category }); + expect(caught instanceof OpenClawPairingObservationRetryableError).toBe( + category === "state-changing", + ); + expect(caught instanceof OpenClawPairingIncompatibleError).toBe( + category === "incompatible-state", + ); + expect( + `${String(caught)}${JSON.stringify(caught)}${JSON.stringify(log.mock.calls)}`, + ).not.toContain(TOKEN); + expect(child).toHaveBeenCalledOnce(); + expect(child.mock.calls[0]?.[2]).toMatchObject({ + timeout: 3000, + maxBuffer: 4096, + stdio: ["pipe", "pipe", "ignore"], + }); + }); + + it("classifies unavailable input before spawning a child", () => { + const child = vi.fn(); + expect(() => + observe("alpha", "gateway", "2026.9.2", "/sandbox/.openclaw", { + readApprovalPolicy: () => null, + spawnSync: child, + }), + ).toThrow(expect.objectContaining({ failureCategory: "input-unavailable" })); + expect(child).not.toHaveBeenCalled(); + }); + + it("redacts unexpected observer exceptions", () => { + expect(() => + observe("alpha", "gateway", "2026.9.2", "/sandbox/.openclaw", { + readApprovalPolicy: () => { + throw new Error(TOKEN); + }, + }), + ).toThrow( + expect.objectContaining({ + failureCategory: "unclassified", + message: "OpenClaw pairing qualification is unavailable.", + }), + ); + }); +}); diff --git a/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.test.ts b/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.test.ts index 4b7f6712cc4..28f5949ea02 100644 --- a/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.test.ts +++ b/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.test.ts @@ -9,7 +9,11 @@ import { performance } from "node:perf_hooks"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { createOpenClawPairingFixture } from "../../../../../test/helpers/openclaw-pairing-fixture"; +import { + createOpenClawPairingFixture, + SQLITE_PAIRING_FIXTURE_SCRIPT, +} from "../../../../../test/helpers/openclaw-pairing-fixture"; +import { redactString } from "../../../../../test/e2e/fixtures/redaction.ts"; import { buildAutoPairApprovalScript, @@ -110,7 +114,7 @@ function writeSqlitePairingState( token, publicKeyPem: publicKeyPem(ED25519_SPKI_PREFIX, Buffer.from(fixture.publicKey, "base64url")), }; - const result = spawnSync("python3", ["-c", SQLITE_FIXTURE_SCRIPT, stateDirectory], { + const result = spawnSync("python3", ["-c", SQLITE_PAIRING_FIXTURE_SCRIPT, stateDirectory], { encoding: "utf8", input: JSON.stringify(payload), }); @@ -167,86 +171,6 @@ function checkpointSqlitePairingState(stateDirectory: string): string { return database; } -const SQLITE_FIXTURE_SCRIPT = String.raw` -import json -import os -import sqlite3 -import sys - -state_dir = sys.argv[1] -value = json.load(sys.stdin) -os.umask(0o007) -sqlite_state_dir = os.path.join(state_dir, 'state') -os.makedirs(sqlite_state_dir, mode=0o770, exist_ok=True) -database = os.path.join(sqlite_state_dir, 'openclaw.sqlite') -connection = sqlite3.connect(database) -connection.execute('PRAGMA journal_mode = WAL') -connection.execute('PRAGMA wal_autocheckpoint = 0') -connection.executescript(''' -CREATE TABLE device_identities ( - identity_key TEXT PRIMARY KEY, device_id TEXT NOT NULL, public_key_pem TEXT NOT NULL, - private_key_pem TEXT NOT NULL, created_at_ms INTEGER NOT NULL, updated_at_ms INTEGER NOT NULL -); -CREATE TABLE device_pairing_paired ( - device_id TEXT PRIMARY KEY, public_key TEXT NOT NULL, display_name TEXT, - operator_label TEXT, platform TEXT, device_family TEXT, client_id TEXT, client_mode TEXT, - browser_origin TEXT, role TEXT, roles_json TEXT, scopes_json TEXT, - approved_scopes_json TEXT, remote_ip TEXT, tokens_json TEXT, approved_via TEXT, - node_surface_json TEXT, pending_node_surface_json TEXT, created_at_ms INTEGER NOT NULL, - approved_at_ms INTEGER NOT NULL, last_seen_at_ms INTEGER, last_seen_reason TEXT -); -CREATE TABLE device_pairing_pending ( - request_id TEXT PRIMARY KEY, device_id TEXT NOT NULL, public_key TEXT NOT NULL, - display_name TEXT, platform TEXT, device_family TEXT, client_id TEXT, client_mode TEXT, - browser_origin TEXT, role TEXT, roles_json TEXT, scopes_json TEXT, remote_ip TEXT, - silent INTEGER, is_repair INTEGER, ts INTEGER NOT NULL, refreshed_at_ms INTEGER -); -CREATE TABLE device_auth_tokens ( - device_id TEXT NOT NULL, role TEXT NOT NULL, token TEXT NOT NULL, scopes_json TEXT NOT NULL, - updated_at_ms INTEGER NOT NULL, - PRIMARY KEY (device_id, role) -); -''') -connection.execute(f"PRAGMA user_version = {value.get('userVersion', 15)}") -connection.execute( - 'INSERT INTO device_identities VALUES (?, ?, ?, ?, ?, ?)', - ('primary', value['deviceId'], value['publicKeyPem'], value['privateKeyPem'], 1, 1), -) -operator = { - 'operator': { - 'token': value['token'], - 'role': 'operator', - 'scopes': value['pairedTokenScopes'], - }, -} -connection.execute( - 'INSERT INTO device_pairing_paired VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', - ( - value['deviceId'], value['publicKey'], None, None, None, None, 'cli', 'cli', None, - 'operator', json.dumps(['operator']), json.dumps(value['pairedScopes']), - json.dumps(value['pairedScopes']), None, json.dumps(operator), None, None, None, - 1, 1, None, None, - ), -) -connection.execute( - 'INSERT INTO device_auth_tokens VALUES (?, ?, ?, ?, ?)', - (value['deviceId'], 'operator', value['token'], json.dumps(value['authScopes']), 1), -) -for request in value.get('pending') or []: - connection.execute( - 'INSERT INTO device_pairing_pending VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', - ( - request['requestId'], request['deviceId'], request['publicKey'], None, None, None, - request.get('clientId'), request.get('clientMode'), None, request.get('role'), - json.dumps(request['roles']) if 'roles' in request else None, - json.dumps(request['scopes']) if 'scopes' in request else None, - None, None, int(request['isRepair']) if 'isRepair' in request else None, 1, None, - ), - ) -connection.commit() -os._exit(0) -`; - function localScriptSpawn( _binary: string, _args: readonly string[], @@ -298,7 +222,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }); } - function observeOrdinarySettlement(approvalPolicy = POLICY) { + function observeOrdinarySettlement(approvalPolicy = POLICY, spawn = localScriptSpawn) { return observeOrdinaryOpenClawPairingSettlement( "alpha", "nemoclaw-8080", @@ -307,11 +231,15 @@ describe("OpenClaw launch-readiness pairing qualification", () => { { getOpenshellBinary: () => "openshell", readApprovalPolicy: () => approvalPolicy, - spawnSync: localScriptSpawn as typeof spawnSync, + spawnSync: spawn as typeof spawnSync, }, ); } + function writeCanonicalSqliteState() { + writeSqlitePairingState(stateDirectory, { deviceId, publicKey, privateKeyPem }); + } + function observeRepairSettlement(approvalPolicy = POLICY) { return observeOpenClawPairingRepairSettlement( "alpha", @@ -342,6 +270,11 @@ describe("OpenClaw launch-readiness pairing qualification", () => { ).toContain(adapter); }); + function readFixture(directory: string, name: string): [string, T] { + const file = path.join(stateDirectory, directory, name); + return [file, JSON.parse(fs.readFileSync(file, "utf8")) as T]; + } + function writePairingOnlyState(): void { const pairedPath = path.join(stateDirectory, "devices", "paired.json"); const authPath = path.join(stateDirectory, "identity", "device-auth.json"); @@ -359,11 +292,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { const sqliteIt = it.skipIf(!OPENSSL_ED25519_AVAILABLE); it("rejects a database swapped only while sqlite3.connect reopens its pathname", () => { - writeSqlitePairingState(stateDirectory, { - deviceId, - publicKey, - privateKeyPem, - }); + writeCanonicalSqliteState(); const databasePath = checkpointSqlitePairingState(stateDirectory); const attackerPath = path.join(root, "attacker.sqlite"); fs.copyFileSync(databasePath, attackerPath); @@ -403,11 +332,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }); it("rejects WAL and SHM swapped only while SQLite performs its first read", () => { - writeSqlitePairingState(stateDirectory, { - deviceId, - publicKey, - privateKeyPem, - }); + writeCanonicalSqliteState(); const databasePath = path.join(stateDirectory, "state", "openclaw.sqlite"); const walPath = `${databasePath}-wal`; const sharedMemoryPath = `${databasePath}-shm`; @@ -462,11 +387,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { sqliteIt( "qualifies committed WAL state without consulting legacy JSON or mutating DB/WAL", () => { - writeSqlitePairingState(stateDirectory, { - deviceId, - publicKey, - privateKeyPem, - }); + writeCanonicalSqliteState(); const database = path.join(stateDirectory, "state", "openclaw.sqlite"); const walProof = spawnSync( "python3", @@ -500,6 +421,29 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }, ); + sqliteIt( + "waits for the SQLite client token without accepting absent or mismatched credentials", + () => { + writeCanonicalSqliteState(); + updateSqlitePairingState(stateDirectory, "DELETE FROM device_auth_tokens"); + const diagnostic = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(() => observeOrdinarySettlement()).toThrow(OpenClawPairingObservationRetryableError); + expect(diagnostic).not.toHaveBeenCalled(); + expect(() => observe()).toThrow(OpenClawPairingIncompatibleError); + expect(() => observeSettlement()).toThrow(OpenClawPairingIncompatibleError); + updateSqlitePairingState( + stateDirectory, + "INSERT INTO device_auth_tokens SELECT device_id, 'operator', json_extract(tokens_json, '$.operator.token'), json_extract(tokens_json, '$.operator.scopes'), 1 FROM device_pairing_paired", + ); + expect(observeOrdinarySettlement()).toMatchObject({ state: "settled" }); + updateSqlitePairingState( + stateDirectory, + "UPDATE device_auth_tokens SET token = 'mismatched'", + ); + expect(() => observeOrdinarySettlement()).toThrow(OpenClawPairingIncompatibleError); + }, + ); + sqliteIt("qualifies the exact persisted admin-upgraded CLI state", () => { writeSqlitePairingState(stateDirectory, { deviceId, @@ -532,11 +476,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }); it("rejects invalid native identity timestamps", () => { - writeSqlitePairingState(stateDirectory, { - deviceId, - publicKey, - privateKeyPem, - }); + writeCanonicalSqliteState(); updateSqlitePairingState(stateDirectory, "UPDATE device_identities SET created_at_ms = -1"); expect(() => observe()).toThrow("OpenClaw pairing qualification is unavailable"); @@ -557,11 +497,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }); sqliteIt("treats an unavailable OpenSSL binary as a terminal qualification failure", () => { - writeSqlitePairingState(stateDirectory, { - deviceId, - publicKey, - privateKeyPem, - }); + writeCanonicalSqliteState(); const spawnWithoutOpenSsl = ( _binary: string, _args: readonly string[], @@ -591,11 +527,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }); it("retries a nonempty WAL without SHM instead of creating the missing sidecar", () => { - writeSqlitePairingState(stateDirectory, { - deviceId, - publicKey, - privateKeyPem, - }); + writeCanonicalSqliteState(); const sqliteStateDirectory = path.join(stateDirectory, "state"); const sharedMemoryPath = path.join(sqliteStateDirectory, "openclaw.sqlite-shm"); fs.rmSync(sharedMemoryPath); @@ -630,11 +562,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }); it("rejects an unsafe authoritative SQLite database instead of reading legacy state", () => { - writeSqlitePairingState(stateDirectory, { - deviceId, - publicKey, - privateKeyPem, - }); + writeCanonicalSqliteState(); fs.chmodSync(path.join(stateDirectory, "state", "openclaw.sqlite"), 0o666); expect(() => observe()).toThrow("OpenClaw pairing qualification is unavailable"); @@ -776,8 +704,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { expect(() => observeSettlement()).toThrow("OpenClaw pairing qualification is unavailable"); writeJson(path.join(stateDirectory, "devices", "pending.json"), {}); - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.scopes = ["operator.pairing", "operator.read"]; writeJson(pairedPath, paired); expect(() => observeSettlement()).toThrow("OpenClaw pairing qualification is unavailable"); @@ -812,12 +739,19 @@ describe("OpenClaw launch-readiness pairing qualification", () => { "rejects a probe baseline with inconsistent %s permissions (#13100)", (scope) => { writePairingOnlyState(); - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.clientMode = "probe"; paired[deviceId]!.scopes.push(scope); writeJson(pairedPath, paired); + const diagnostic = vi.spyOn(console, "error").mockImplementation(() => {}); expect(() => observeOrdinarySettlement()).toThrow(OpenClawPairingIncompatibleError); + expect(() => observeOrdinarySettlement()).toThrow( + expect.objectContaining({ + field: scope === "operator.write" ? "scope.consistency" : "paired.scopes", + failureCategory: "incompatible-state", + }), + ); + expect(diagnostic).not.toHaveBeenCalled(); }, ); @@ -866,18 +800,76 @@ describe("OpenClaw launch-readiness pairing qualification", () => { }); it.each(["ui", "unknown"])("rejects %s metadata on the local device (#13100)", (clientMode) => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.clientMode = clientMode; writeJson(pairedPath, paired); - expect(() => observeOrdinarySettlement()).toThrow(OpenClawPairingQualificationError); + const diagnostic = vi.spyOn(console, "error").mockImplementation(() => {}); + const spawn = vi.fn(localScriptSpawn); + expect(() => observeOrdinarySettlement(POLICY, spawn)).toThrow( + OpenClawPairingIncompatibleError, + ); + expect(spawn.mock.results[0]?.value.status).toBe(4); + expect(spawn.mock.results[0]?.value.stdout).toBe( + '__NEMOCLAW_OPENCLAW_PAIRING_MISMATCH__={"field":"paired.clientMode"}\n', + ); + expect(diagnostic.mock.calls).toEqual([]); + }); + + it.each([ + ["auth.tokens", [], {}], + ["auth.tokens", ["operator", "observer"], {}], + ["auth.tokens.operator.token", ["operator"], { token: `${TOKEN}-mismatch` }], + ["auth.tokens.operator.role", ["operator"], { role: "observer" }], + ] as const)("reports only the rejected client credential field %s", (reason, roles, fields) => { + const [authPath, auth] = readFixture("identity", "device-auth.json"); + const invalidAuth = { + ...auth, + tokens: Object.fromEntries( + roles.map((role) => [role, { ...auth.tokens.operator, ...fields }]), + ), + }; + writeJson(authPath, invalidAuth); + const diagnostic = vi.spyOn(console, "error").mockImplementation(() => {}); + const spawn = vi.fn(localScriptSpawn); + expect(() => observeOrdinarySettlement(POLICY, spawn)).toThrow( + OpenClawPairingIncompatibleError, + ); + expect({ + status: spawn.mock.results[0]?.value.status, + stdout: spawn.mock.results[0]?.value.stdout, + diagnostic: diagnostic.mock.calls, + redactedDiagnostic: diagnostic.mock.calls.map(([message]) => redactString(message)), + auth: JSON.parse(fs.readFileSync(authPath, "utf8")), + }).toEqual({ + status: 4, + stdout: `__NEMOCLAW_OPENCLAW_PAIRING_MISMATCH__={"field":"${reason}"}\n`, + diagnostic: [], + redactedDiagnostic: [], + auth: invalidAuth, + }); + }); + + it.each([ + TOKEN, + "__NEMOCLAW_OPENCLAW_PAIRING_MISMATCH__=" + TOKEN, + '__NEMOCLAW_OPENCLAW_PAIRING_MISMATCH__={"field":"auth.tokens"}\n' + TOKEN, + `__NEMOCLAW_OPENCLAW_PAIRING_MISMATCH__={"field":"${TOKEN}"}`, + ])("does not log unrecognized pairing output %j", (stdout) => { + const diagnostic = vi.spyOn(console, "error").mockImplementation(() => {}); + const spawn = vi.fn(() => ({ status: 4, stdout })) as unknown as typeof localScriptSpawn; + expect(() => observeOrdinarySettlement(POLICY, spawn)).toThrow( + OpenClawPairingIncompatibleError, + ); + expect(diagnostic.mock.calls).toEqual([]); + expect(() => observeOrdinarySettlement(POLICY, spawn)).toThrow( + expect.objectContaining({ field: null, failureCategory: "incompatible-state" }), + ); }); it.each( [OPENCLAW_PAIRING_REQUEST_SCOPES, OPENCLAW_PAIRING_REQUIRED_SCOPES].map((scopes) => [scopes]), )("preserves qualification across cli/probe/cli metadata with %j grants (#13100)", (scopes) => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.scopes = [...scopes]; paired[deviceId]!.approvedScopes = [...scopes]; writeJson(pairedPath, paired); @@ -905,8 +897,7 @@ describe("OpenClaw launch-readiness pairing qualification", () => { ); it("accepts native approval preserving an explicitly granted read scope (#13100)", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.scopes = [...OPENCLAW_PAIRING_REQUIRED_SCOPES]; paired[deviceId]!.approvedScopes = [...OPENCLAW_PAIRING_REQUIRED_SCOPES]; writeJson(pairedPath, paired); @@ -1246,8 +1237,7 @@ process.stdout.write("{}\\n"); [ "mismatched client credential", () => { - const authPath = path.join(stateDirectory, "identity", "device-auth.json"); - const auth = JSON.parse(fs.readFileSync(authPath, "utf8")) as AuthFixture; + const [authPath, auth] = readFixture("identity", "device-auth.json"); auth.tokens.operator.token = "different-token"; writeJson(authPath, auth); }, @@ -1255,8 +1245,7 @@ process.stdout.write("{}\\n"); [ "changed paired request scopes", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.scopes = ["operator.pairing", "operator.read", "operator.write"]; writeJson(pairedPath, paired); }, @@ -1264,8 +1253,7 @@ process.stdout.write("{}\\n"); [ "changed approved request scopes", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.approvedScopes = ["operator.pairing", "operator.read"]; writeJson(pairedPath, paired); }, @@ -1273,8 +1261,7 @@ process.stdout.write("{}\\n"); [ "changed paired token scopes", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.tokens.operator.scopes = ["operator.pairing", "operator.write"]; writeJson(pairedPath, paired); }, @@ -1282,8 +1269,7 @@ process.stdout.write("{}\\n"); [ "changed client-auth token scopes", () => { - const authPath = path.join(stateDirectory, "identity", "device-auth.json"); - const auth = JSON.parse(fs.readFileSync(authPath, "utf8")) as AuthFixture; + const [authPath, auth] = readFixture("identity", "device-auth.json"); auth.tokens.operator.scopes = ["operator.pairing", "operator.write"]; writeJson(authPath, auth); }, @@ -1291,8 +1277,7 @@ process.stdout.write("{}\\n"); [ "changed canonical client ID", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.clientId = "unknown-client"; writeJson(pairedPath, paired); }, @@ -1300,8 +1285,7 @@ process.stdout.write("{}\\n"); [ "changed canonical client mode", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.clientMode = "unknown-mode"; writeJson(pairedPath, paired); }, @@ -1309,8 +1293,7 @@ process.stdout.write("{}\\n"); [ "duplicate operator identity roles", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.roles = ["operator", "operator"]; writeJson(pairedPath, paired); }, @@ -1318,8 +1301,7 @@ process.stdout.write("{}\\n"); [ "missing canonical operator role", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); delete (paired[deviceId]! as unknown as Record).role; writeJson(pairedPath, paired); }, @@ -1327,8 +1309,7 @@ process.stdout.write("{}\\n"); [ "missing canonical operator roles", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); delete (paired[deviceId]! as unknown as Record).roles; writeJson(pairedPath, paired); }, @@ -1367,8 +1348,7 @@ process.stdout.write("{}\\n"); >; identity.publicKey = `${publicKey}=`; writeJson(identityPath, identity); - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.publicKey = `${publicKey}=`; writeJson(pairedPath, paired); }, @@ -1376,12 +1356,10 @@ process.stdout.write("{}\\n"); [ "whitespace-padded operator token", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.tokens.operator.token = ` ${TOKEN} `; writeJson(pairedPath, paired); - const authPath = path.join(stateDirectory, "identity", "device-auth.json"); - const auth = JSON.parse(fs.readFileSync(authPath, "utf8")) as AuthFixture; + const [authPath, auth] = readFixture("identity", "device-auth.json"); auth.tokens.operator.token = ` ${TOKEN} `; writeJson(authPath, auth); }, @@ -1400,8 +1378,7 @@ process.stdout.write("{}\\n"); [ "alternate scopes on the paired device", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.requestedScopes = ["operator.admin"]; writeJson(pairedPath, paired); }, @@ -1409,8 +1386,7 @@ process.stdout.write("{}\\n"); [ "alternate public key on the paired device", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired[deviceId]!.publicKeyPem = publicKeyPem(ED25519_SPKI_PREFIX, Buffer.alloc(32, 8)); writeJson(pairedPath, paired); }, @@ -1418,8 +1394,7 @@ process.stdout.write("{}\\n"); [ "alternate roles on the paired operator token", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); (paired[deviceId]!.tokens.operator as unknown as Record).roles = [ "admin", ]; @@ -1429,8 +1404,7 @@ process.stdout.write("{}\\n"); [ "alternate scopes on the client authorization", () => { - const authPath = path.join(stateDirectory, "identity", "device-auth.json"); - const auth = JSON.parse(fs.readFileSync(authPath, "utf8")) as AuthFixture; + const [authPath, auth] = readFixture("identity", "device-auth.json"); (auth.tokens.operator as unknown as Record).approvedScopes = [ "operator.admin", ]; @@ -1440,8 +1414,7 @@ process.stdout.write("{}\\n"); [ "ambiguous local device state", () => { - const pairedPath = path.join(stateDirectory, "devices", "paired.json"); - const paired = JSON.parse(fs.readFileSync(pairedPath, "utf8")) as PairedFixture; + const [pairedPath, paired] = readFixture("devices", "paired.json"); paired.duplicate = { ...paired[deviceId]!, deviceId: "different-device", diff --git a/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.ts b/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.ts index 51418aa2744..3073bc289a0 100644 --- a/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.ts +++ b/src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification.ts @@ -111,8 +111,25 @@ interface OpenClawPairingQualificationDeps { spawnSync: typeof spawnSync; } +export const OPENCLAW_PAIRING_OBSERVATION_FAILURE_CATEGORIES = [ + "input-unavailable", + "deadline-exceeded", + "output-limit-exceeded", + "child-error", + "child-signal", + "child-exit-failed", + "state-changing", + "incompatible-state", + "invalid-output", + "unclassified", +] as const; +export type OpenClawPairingObservationFailureCategory = + (typeof OPENCLAW_PAIRING_OBSERVATION_FAILURE_CATEGORIES)[number]; + export class OpenClawPairingQualificationError extends Error { - constructor() { + constructor( + readonly failureCategory: OpenClawPairingObservationFailureCategory = "unclassified", + ) { super("OpenClaw pairing qualification is unavailable."); this.name = "OpenClawPairingQualificationError"; } @@ -120,7 +137,7 @@ export class OpenClawPairingQualificationError extends Error { export class OpenClawPairingIncompatibleError extends OpenClawPairingQualificationError { constructor(readonly field: string | null = null) { - super(); + super("incompatible-state"); this.name = "OpenClawPairingIncompatibleError"; } } @@ -147,7 +164,7 @@ function parseOpenClawPairingMismatchField(output: unknown): string | null { export class OpenClawPairingObservationRetryableError extends OpenClawPairingQualificationError { constructor() { - super(); + super("state-changing"); this.name = "OpenClawPairingObservationRetryableError"; } } @@ -266,16 +283,16 @@ export function buildOpenClawPairingObservationScript( ): string { const pairingStateModule = readOpenClawPairingStateModule(); if (!path.posix.isAbsolute(stateDirectory)) { - throw new OpenClawPairingQualificationError(); + throw new OpenClawPairingQualificationError("input-unavailable"); } if ( !approvalPolicyModuleB64 || Buffer.from(approvalPolicyModuleB64, "base64").toString("base64") !== approvalPolicyModuleB64 ) { - throw new OpenClawPairingQualificationError(); + throw new OpenClawPairingQualificationError("input-unavailable"); } if (!pairingStateModule) { - throw new OpenClawPairingQualificationError(); + throw new OpenClawPairingQualificationError("input-unavailable"); } const stateDirectoryB64 = Buffer.from(stateDirectory, "utf8").toString("base64"); const marker = mode === "qualification" ? QUALIFICATION_MARKER : SETTLEMENT_MARKER; @@ -756,6 +773,15 @@ try: if not valid: reject(field) auth_tokens = auth.get('tokens') + # OpenClaw commits approval before the client stores the hello response token. + # Ordinary onboarding can observe that gap; qualification still requires the token. + if ( + ORDINARY_SETTLEMENT and first.get('layout') == 'sqlite' + and type(auth.get('version')) is int and auth.get('version') == 1 + and auth.get('deviceId') == device_id + and isinstance(auth_tokens, dict) and not auth_tokens + ): + retry_observation() auth_operator = auth_tokens.get('operator') if isinstance(auth_tokens, dict) and set(auth_tokens) == {'operator'} else None if type(auth.get('version')) is not int or auth.get('version') != 1: reject('auth.version') @@ -980,7 +1006,7 @@ function runOpenClawPairingObservation( normalizedVersion.length > 128 || (mode === "qualification" && !normalizedVersion) ) { - throw new OpenClawPairingQualificationError(); + throw new OpenClawPairingQualificationError("input-unavailable"); } const approvalPolicyModuleB64 = Buffer.from(approvalPolicy, "utf8").toString("base64"); const script = buildOpenClawPairingObservationScript( @@ -1013,9 +1039,18 @@ function runOpenClawPairingObservation( mode === "ordinary-settlement" ? parseOpenClawPairingMismatchField(result.stdout) : null, ); } - if (result.error || result.signal || result.status !== 0) { - throw new OpenClawPairingQualificationError(); + if (result.error) { + const code = (result.error as NodeJS.ErrnoException).code; + throw new OpenClawPairingQualificationError( + code === "ETIMEDOUT" + ? "deadline-exceeded" + : code === "ENOBUFS" + ? "output-limit-exceeded" + : "child-error", + ); } + if (result.signal) throw new OpenClawPairingQualificationError("child-signal"); + if (result.status !== 0) throw new OpenClawPairingQualificationError("child-exit-failed"); return { output: String(result.stdout ?? ""), policy: approvalPolicy }; } @@ -1036,7 +1071,7 @@ export function observeOpenClawPairingSettlement( execDeps, ); const observation = parseOpenClawPairingSettlementObservation(executed.output); - if (!observation) throw new OpenClawPairingQualificationError(); + if (!observation) throw new OpenClawPairingQualificationError("invalid-output"); return observation; } catch (error) { if (error instanceof OpenClawPairingQualificationError) throw error; @@ -1061,7 +1096,7 @@ export function observeOpenClawPairingRepairSettlement( execDeps, ); const observation = parseOpenClawPairingRepairObservation(executed.output); - if (!observation) throw new OpenClawPairingQualificationError(); + if (!observation) throw new OpenClawPairingQualificationError("invalid-output"); return observation; } catch (error) { if (error instanceof OpenClawPairingQualificationError) throw error; @@ -1086,7 +1121,7 @@ export function observeOrdinaryOpenClawPairingSettlement( execDeps, ); const observation = parseOpenClawPairingSettlementObservation(executed.output); - if (!observation) throw new OpenClawPairingQualificationError(); + if (!observation) throw new OpenClawPairingQualificationError("invalid-output"); return observation; } catch (error) { if (error instanceof OpenClawPairingQualificationError) throw error; @@ -1113,7 +1148,7 @@ export function observeOpenClawPairingQualification( execDeps, ); const projection = parseOpenClawPairingObservation(executed.output); - if (!projection) throw new OpenClawPairingQualificationError(); + if (!projection) throw new OpenClawPairingQualificationError("invalid-output"); return { schemaVersion: 1, kind: "openclaw-pairing", diff --git a/src/lib/actions/sandbox/mcp-bridge-adapter-teardown.test.ts b/src/lib/actions/sandbox/mcp-bridge-adapter-teardown.test.ts index ed69542b15b..01934c5c8ad 100644 --- a/src/lib/actions/sandbox/mcp-bridge-adapter-teardown.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-adapter-teardown.test.ts @@ -93,6 +93,24 @@ import { restoreMcpBridgesAfterRebuild, } from "./mcp-bridge-rebuild"; +const capturedPolicy = JSON.stringify({ + version: 1, + network_policies: { + mcp_bridge_github: { + name: "MCP GitHub", + binaries: [{ path: "/usr/bin/node" }], + endpoints: [ + { + host: "api.githubcopilot.com", + port: 443, + protocol: "mcp", + mcp: { versions: ["2025-11-25"], allow_all_known_mcp_methods: true }, + }, + ], + }, + }, +}); + const sandbox = { agent: "hermes" } as SandboxEntry; const runtimeSelection = { gatewayName: "nemoclaw-8091", @@ -116,9 +134,7 @@ describe("MCP adapter teardown rollback", () => { mocks.getMcpProviderInspectionRuntimeSelection.mockReset().mockReturnValue(runtimeSelection); mocks.getBridgeAdapter.mockReset().mockReturnValue("hermes-config"); mocks.getSandboxAgent.mockReset().mockReturnValue("hermes"); - mocks.captureRecordedSandboxBasePolicy - .mockReset() - .mockResolvedValue("version: 1\nnetwork_policies:\n mcp_bridge_github: {}\n"); + mocks.captureRecordedSandboxBasePolicy.mockReset().mockResolvedValue(capturedPolicy); mocks.getSandboxOrThrow.mockReset().mockReturnValue(sandbox); mocks.inspectExactMcpDestroyProvider.mockReset().mockReturnValue({ credentialKeys: ["GITHUB_TOKEN"], @@ -254,10 +270,22 @@ describe("MCP adapter teardown rollback", () => { expect(mocks.restoreExistingMcpBridgeRuntime).not.toHaveBeenCalled(); }); + it("rejects an invalid policy before scrubbing or detaching the source", async () => { + mocks.captureRecordedSandboxBasePolicy.mockResolvedValue( + "version: 1\nnetwork_policies:\n mcp_bridge_github: {}\n", + ); + await expect(prepareMcpBridgesForRebuild("alpha", [entry])).rejects.toThrow( + "does not satisfy the shipped sandbox policy schema", + ); + expect(mocks.unregisterAgentAdapter).not.toHaveBeenCalled(); + expect(mocks.removeGeneratedPolicy).not.toHaveBeenCalled(); + expect(mocks.detachProvider).not.toHaveBeenCalled(); + }); + it("preserves an existing Hermes entry through legacy teardown and replacement restore", async () => { mocks.captureRecordedSandboxBasePolicy .mockReset() - .mockResolvedValueOnce("version: 1\nnetwork_policies:\n mcp_bridge_github: {}\n") + .mockResolvedValueOnce(capturedPolicy) .mockResolvedValueOnce("version: 1\nnetwork_policies: {}\n"); mocks.removeGeneratedPolicy.mockReset().mockResolvedValue(undefined); mocks.detachProvider.mockReset().mockResolvedValue("detached"); @@ -278,7 +306,7 @@ describe("MCP adapter teardown rollback", () => { entries: [entry], detachedProviderEntries: [entry], scrubbedAdapterEntries: [expect.objectContaining(entry)], - policyHandoff: "version: 1\nnetwork_policies:\n mcp_bridge_github: {}\n", + policyHandoff: capturedPolicy, runtimeSelection, }); diff --git a/src/lib/actions/sandbox/mcp-bridge-add-restart.ts b/src/lib/actions/sandbox/mcp-bridge-add-restart.ts index ae05519d5e2..f8350d5d6b9 100644 --- a/src/lib/actions/sandbox/mcp-bridge-add-restart.ts +++ b/src/lib/actions/sandbox/mcp-bridge-add-restart.ts @@ -46,7 +46,7 @@ import { getMcpProviderInspectionRuntimeSelection, inspectMcpProvider, inspectMcpProviderAttachments, - MCP_BRIDGE_PROVIDER_TYPE, + mcpBridgeProviderType, type McpCredentialRevisionObservation, observeMcpCredentialRevision, providerMatchesCredential, @@ -384,7 +384,7 @@ async function inspectMcpAddRecovery( (!providerInspection.id || (entry.providerId !== undefined && providerInspection.id !== entry.providerId) || providerInspection.resourceVersion === null || - providerInspection.type !== MCP_BRIDGE_PROVIDER_TYPE || + providerInspection.type !== mcpBridgeProviderType(entry.env[0]) || providerInspection.credentialKeys?.length !== 1 || providerInspection.credentialKeys[0] !== entry.env[0]) ) { @@ -902,7 +902,7 @@ async function addMcpBridgeUnlocked( // again after provider creation to close the intervening race. await assertNoProviderCredentialCollisions(sandboxName, [entry], providerRuntimeSelection); const providerAdapter = createMcpProviderAdapterBoundary(providerRuntimeSelection).adapter; - await ensureMcpBridgeProviderProfile(providerRuntimeSelection, providerAdapter); + await ensureMcpBridgeProviderProfile(providerRuntimeSelection, entry.env[0], providerAdapter); // Load the real protocol:mcp policy without a credential binding before // provider mutation. OpenShell requires the endpointless provider to be // attached before it accepts credential_binding.provider, and withholds diff --git a/src/lib/actions/sandbox/mcp-bridge-hermes-add-finality.test.ts b/src/lib/actions/sandbox/mcp-bridge-hermes-add-finality.test.ts index 9a1f6f03488..6bc079af240 100644 --- a/src/lib/actions/sandbox/mcp-bridge-hermes-add-finality.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-hermes-add-finality.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { beforeEach, describe, expect, it, vi } from "vitest"; +import { mcpBridgeProviderType } from "./mcp-bridge/provider-profile"; const mocks = vi.hoisted(() => { const state = { @@ -136,7 +137,7 @@ vi.mock("./mcp-bridge-provider", async (importOriginal) => ({ ? "aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee" : "11111111-2222-4333-8444-555555555555", resourceVersion: 7, - type: "nemoclaw-mcp-v1", + type: mcpBridgeProviderType("GITHUB_TOKEN"), } : { credentialKeys: null, @@ -179,7 +180,7 @@ function configureUpsertMcpProviderMock(): void { exists: true, id: "11111111-2222-4333-8444-555555555555", resourceVersion: 7, - type: "nemoclaw-mcp-v1", + type: mcpBridgeProviderType("GITHUB_TOKEN"), }, }; }, @@ -290,7 +291,7 @@ describe("Hermes MCP add reload finality", () => { ? "aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee" : "11111111-2222-4333-8444-555555555555", resourceVersion: 7, - type: "nemoclaw-mcp-v1", + type: mcpBridgeProviderType("GITHUB_TOKEN"), } : { credentialKeys: null, @@ -479,7 +480,7 @@ describe("Hermes MCP add reload finality", () => { exists: true, id: "11111111-2222-4333-8444-555555555555", resourceVersion: 7, - type: "nemoclaw-mcp-v1", + type: mcpBridgeProviderType("GITHUB_TOKEN"), } : { credentialKeys: null, diff --git a/src/lib/actions/sandbox/mcp-bridge-input-targets.test.ts b/src/lib/actions/sandbox/mcp-bridge-input-targets.test.ts index 12d53182784..5d48c82d3ff 100644 --- a/src/lib/actions/sandbox/mcp-bridge-input-targets.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-input-targets.test.ts @@ -201,7 +201,7 @@ replace(provider, "upsertMcpProvider", () => ({ action: "created", inspection: { credentialKeys: ["LOCAL_MCP_TOKEN"], exists: true, - id: "11111111-2222-4333-8444-555555555555", resourceVersion: 1, type: "nemoclaw-mcp-v1", + id: "11111111-2222-4333-8444-555555555555", resourceVersion: 1, type: provider.mcpBridgeProviderType("LOCAL_MCP_TOKEN"), }, })); replace(provider, "attachProvider", () => {}); @@ -317,7 +317,7 @@ replace(provider, "getMcpProviderInspectionRuntimeSelection", () => ({ replace(provider, "inspectMcpProvider", () => ({ credentialKeys: ["GITHUB_TOKEN"], exists: true, id: "11111111-2222-4333-8444-555555555555", resourceVersion: 7, - type: "nemoclaw-mcp-v1", + type: provider.mcpBridgeProviderType("GITHUB_TOKEN"), })); replace(provider, "inspectMcpProviderAttachments", () => ({ attachments: [] })); replace(policies, "getPresetContentGatewayState", () => "absent"); @@ -433,7 +433,7 @@ replace(validation, "assertMcpCredentialBoundaryRuntimeVersion", () => {}); replace(provider, "getMcpProviderInspectionRuntimeSelection", () => ({ gatewayName: "nemoclaw", workspace: "default" })); replace(provider, "inspectMcpProvider", () => state.provider ? ({ exists: true, id: providerId, resourceVersion: 7, - type: "nemoclaw-mcp-v1", credentialKeys: ["GITHUB_TOKEN"], + type: provider.mcpBridgeProviderType("GITHUB_TOKEN"), credentialKeys: ["GITHUB_TOKEN"], }) : ({ exists: false, id: null, resourceVersion: null, type: null, credentialKeys: null })); replace(provider, "inspectMcpProviderAttachments", () => ({ attachments: state.attachment ? [{ name: "alpha-mcp-github", providerId, credentialKeys: ["GITHUB_TOKEN"] }] : [], diff --git a/src/lib/actions/sandbox/mcp-bridge-input-validation.test.ts b/src/lib/actions/sandbox/mcp-bridge-input-validation.test.ts index f8bd9d54cd9..49056e4d24d 100644 --- a/src/lib/actions/sandbox/mcp-bridge-input-validation.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-input-validation.test.ts @@ -11,7 +11,7 @@ import { resolveCredentialEnv, validateMcpCredentialEnvName, } from "./mcp-bridge"; -import childVisibleCredentialManifest from "./openshell-child-visible-credentials.v0.0.116.json"; +import childVisibleCredentialManifest from "./openshell-child-visible-credentials.v0.1.7.json"; const CHILD_VISIBLE_CREDENTIAL_CASES = [ { diff --git a/src/lib/actions/sandbox/mcp-bridge-policy-render.ts b/src/lib/actions/sandbox/mcp-bridge-policy-render.ts index d9362054e79..938b54e96ed 100644 --- a/src/lib/actions/sandbox/mcp-bridge-policy-render.ts +++ b/src/lib/actions/sandbox/mcp-bridge-policy-render.ts @@ -112,6 +112,8 @@ function renderMcpBridgePolicyYaml( allowed_ips: allowedIps, ...(providerName ? { credential_binding: { provider: providerName } } : {}), mcp: { + // Match the pinned runtime's default so policy readback preserves equality. + versions: ["2025-11-25"], max_body_bytes: MCP_BRIDGE_POLICY_MAX_BODY_BYTES, strict_tool_names: true, allow_all_known_mcp_methods: false, diff --git a/src/lib/actions/sandbox/mcp-bridge-policy.test.ts b/src/lib/actions/sandbox/mcp-bridge-policy.test.ts index 2202017b343..24a0a3d4062 100644 --- a/src/lib/actions/sandbox/mcp-bridge-policy.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-policy.test.ts @@ -351,6 +351,7 @@ describe("generated MCP policy", () => { allowed_ips: ["2606:4700:4700::1111", "8.8.8.8"], credential_binding: { provider: "alpha-mcp-bound-provider" }, mcp: { + versions: ["2025-11-25"], max_body_bytes: MCP_BRIDGE_POLICY_MAX_BODY_BYTES, strict_tool_names: true, allow_all_known_mcp_methods: false, diff --git a/src/lib/actions/sandbox/mcp-bridge-provider-inspection.ts b/src/lib/actions/sandbox/mcp-bridge-provider-inspection.ts index 1bc8c24de0d..4908f7bcec9 100644 --- a/src/lib/actions/sandbox/mcp-bridge-provider-inspection.ts +++ b/src/lib/actions/sandbox/mcp-bridge-provider-inspection.ts @@ -39,8 +39,9 @@ import { preflightMcpServerUrlResolvedTarget, } from "./mcp-bridge-validation"; import { normalizeRecordedMcpServerUrl } from "./mcp-bridge/recorded-url"; +import { MCP_BRIDGE_PROVIDER_TYPE, mcpBridgeProviderType } from "./mcp-bridge/provider-profile"; -export const MCP_BRIDGE_PROVIDER_TYPE = "nemoclaw-mcp-v1"; +export { MCP_BRIDGE_PROVIDER_TYPE, mcpBridgeProviderType } from "./mcp-bridge/provider-profile"; export type McpProviderInspection = { exists: boolean | null; @@ -363,8 +364,8 @@ export function providerMatchesCredential( expectedProviderId !== undefined && inspection.id === expectedProviderId && inspection.resourceVersion !== null && - inspection.type === MCP_BRIDGE_PROVIDER_TYPE && expectedCredential !== undefined && + inspection.type === mcpBridgeProviderType(expectedCredential) && inspection.credentialKeys?.length === 1 && inspection.credentialKeys[0] === expectedCredential ); @@ -383,7 +384,7 @@ export function providerMatchesManagedCredential( expectedProviderId !== undefined && inspection.id === expectedProviderId && inspection.resourceVersion !== null && - inspection.type === "generic" && + (inspection.type === "generic" || inspection.type === MCP_BRIDGE_PROVIDER_TYPE) && expectedCredential !== undefined && inspection.credentialKeys?.length === 1 && inspection.credentialKeys[0] === expectedCredential @@ -417,7 +418,7 @@ export function providerShapeDetail( allowLegacyGeneric: true, }) ) { - return `Provider type 'generic' predates the OpenShell 0.0.106 endpoint-binding contract. Remove this MCP server, then add it again with '${expectedCredential ?? ""}' exported.`; + return `Provider type '${inspection.type}' does not declare its MCP credential for OpenShell 0.1.7. Remove this MCP server, then add it again with '${expectedCredential ?? ""}' exported.`; } const type = inspection.type ?? "unparseable"; const keys = inspection.credentialKeys?.join(", ") || "none or unparseable"; @@ -443,13 +444,13 @@ export async function assertMcpProviderRecoverable( } if (inspection.exists) { if ( - inspection.type === "generic" && + (inspection.type === "generic" || inspection.type === MCP_BRIDGE_PROVIDER_TYPE) && providerMatchesManagedCredential(inspection, expectedCredential, entry.providerId, { allowLegacyGeneric: true, }) ) { throw new McpBridgeError( - `OpenShell provider '${entry.providerName}' uses the legacy generic profile, which OpenShell 0.0.106 cannot bind to an MCP endpoint. Run mcp remove for '${entry.server}', then add it again with '${expectedCredential}' exported.`, + `OpenShell provider '${entry.providerName}' uses the legacy ${inspection.type} profile, which cannot bind its credential to an MCP endpoint on OpenShell 0.1.7. Run mcp remove for '${entry.server}', then add it again with '${expectedCredential}' exported.`, ); } if (!providerMatchesCredential(inspection, expectedCredential, entry.providerId)) { diff --git a/src/lib/actions/sandbox/mcp-bridge-provider-mutation.ts b/src/lib/actions/sandbox/mcp-bridge-provider-mutation.ts index aa08b513952..1bd26d4df6c 100644 --- a/src/lib/actions/sandbox/mcp-bridge-provider-mutation.ts +++ b/src/lib/actions/sandbox/mcp-bridge-provider-mutation.ts @@ -16,14 +16,12 @@ */ import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; -import { endpointlessProviderProfilePath } from "../../adapters/openshell/provider-profile"; -import { REPOSITORY_ROOT } from "../../core/repository-root"; +import { mcpBridgeProviderType, withMcpBridgeProviderProfile } from "./mcp-bridge/provider-profile"; import type { McpSourceEntry } from "./mcp-bridge-contracts"; import { McpBridgeError, type ParsedEnvReference } from "./mcp-bridge-contracts"; import { createMcpProviderAdapterBoundary, inspectMcpProvider, - MCP_BRIDGE_PROVIDER_TYPE, type McpProviderInspectionRuntimeSelection, type McpProviderInspection, providerMatchesCredential, @@ -46,20 +44,21 @@ export { /** Ensure the endpointless profile required by OpenShell static credential binding. */ export async function ensureMcpBridgeProviderProfile( runtimeSelection: McpProviderInspectionRuntimeSelection, + credential: string, providerAdapter?: OpenShellProviderAdapter, ): Promise { const boundary = createMcpProviderAdapterBoundary(runtimeSelection, providerAdapter); - const result = await boundary.adapter.importProviderProfile({ - profilePath: endpointlessProviderProfilePath(REPOSITORY_ROOT, MCP_BRIDGE_PROVIDER_TYPE), - target: boundary.target, - }); + const profileType = mcpBridgeProviderType(credential); + const result = await withMcpBridgeProviderProfile(credential, (profilePath) => + boundary.adapter.importProviderProfile({ profilePath, target: boundary.target }), + ); if (result.ok) return; throw new McpBridgeError( result.error.kind === "command" && result.error.reason === "profile_incompatible" - ? `OpenShell provider profile '${MCP_BRIDGE_PROVIDER_TYPE}' already exists but does not match NemoClaw's endpointless credential contract. Refusing to attach MCP credentials to it.` + ? `OpenShell provider profile '${profileType}' already exists but does not match NemoClaw's endpointless credential contract. Refusing to attach MCP credentials to it.` : result.operation === "import" - ? `Could not import OpenShell provider profile '${MCP_BRIDGE_PROVIDER_TYPE}'.` - : `OpenShell provider profile '${MCP_BRIDGE_PROVIDER_TYPE}' could not be exported for validation. Refusing to attach MCP credentials to it.`, + ? `Could not import OpenShell provider profile '${profileType}'.` + : `OpenShell provider profile '${profileType}' could not be exported for validation. Refusing to attach MCP credentials to it.`, ); } @@ -184,7 +183,7 @@ export async function upsertMcpProvider( action === "create" ? await boundary.adapter.createProvider({ name: providerName, - type: MCP_BRIDGE_PROVIDER_TYPE, + type: mcpBridgeProviderType(envNames[0]), credentials, config: [], fromExisting: false, diff --git a/src/lib/actions/sandbox/mcp-bridge-provider-profile.test.ts b/src/lib/actions/sandbox/mcp-bridge-provider-profile.test.ts index fd3c4f0d844..388be58364b 100644 --- a/src/lib/actions/sandbox/mcp-bridge-provider-profile.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-provider-profile.test.ts @@ -2,9 +2,13 @@ // SPDX-License-Identifier: Apache-2.0 import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { existsSync, readFileSync } from "node:fs"; import { setProviderCommandRuntimeHooksForTest } from "../../adapters/openshell/provider-command"; -import { ensureMcpBridgeProviderProfile, MCP_BRIDGE_PROVIDER_TYPE } from "./mcp-bridge-provider"; +import { ensureMcpBridgeProviderProfile, mcpBridgeProviderType } from "./mcp-bridge-provider"; + +const CREDENTIAL = "GITHUB_TOKEN"; +const MCP_BRIDGE_PROVIDER_TYPE = mcpBridgeProviderType(CREDENTIAL); const runtimeSelection = { gatewayName: "nemoclaw-8080", workspace: "default" }; @@ -19,7 +23,16 @@ afterEach(() => { function exportedEndpointlessProfile(id: string, inferenceCapable: boolean): string { return JSON.stringify({ id, - credentials: [], + credentials: [ + { + name: CREDENTIAL, + env_vars: [CREDENTIAL], + required: true, + auth_style: "bearer", + header_name: "Authorization", + query_param: "", + }, + ], endpoints: [], binaries: [], inference_capable: inferenceCapable, @@ -28,10 +41,18 @@ function exportedEndpointlessProfile(id: string, inferenceCapable: boolean): str describe("OpenShell MCP provider profile", () => { it("imports the endpointless profile before managed provider use", async () => { + let generatedPath = ""; const runOpenshell = vi .fn() .mockReturnValueOnce({ status: 1, stdout: "", stderr: "provider profile not found" }) - .mockReturnValueOnce({ status: 0, stdout: "Imported", stderr: "" }) + .mockImplementationOnce((args: string[]) => { + generatedPath = args[4]; + const profile = JSON.parse(readFileSync(generatedPath, "utf8")); + expect(profile).toMatchObject( + JSON.parse(exportedEndpointlessProfile(MCP_BRIDGE_PROVIDER_TYPE, false)), + ); + return { status: 0, stdout: "Imported", stderr: "" }; + }) .mockReturnValueOnce({ status: 0, stdout: exportedEndpointlessProfile(MCP_BRIDGE_PROVIDER_TYPE, false), @@ -39,10 +60,19 @@ describe("OpenShell MCP provider profile", () => { }); setProviderCommandRuntimeHooksForTest({ runOpenshell: runOpenshell as never }); - await expect(ensureMcpBridgeProviderProfile(runtimeSelection)).resolves.toBeUndefined(); + await expect( + ensureMcpBridgeProviderProfile(runtimeSelection, CREDENTIAL), + ).resolves.toBeUndefined(); expect(runOpenshell).toHaveBeenCalledTimes(3); + expect(existsSync(generatedPath)).toBe(false); expect(runOpenshell).toHaveBeenCalledWith( - ["provider", "profile", "import", "--file", expect.stringMatching(/nemoclaw-mcp-v1\.yaml$/)], + [ + "provider", + "profile", + "import", + "--file", + expect.stringMatching(/nemoclaw-mcp-profile-[^/]+\/profile\.json$/), + ], expect.any(Object), ); }); @@ -55,43 +85,55 @@ describe("OpenShell MCP provider profile", () => { }); setProviderCommandRuntimeHooksForTest({ runOpenshell: runOpenshell as never }); - await expect(ensureMcpBridgeProviderProfile(runtimeSelection)).resolves.toBeUndefined(); + await expect( + ensureMcpBridgeProviderProfile(runtimeSelection, CREDENTIAL), + ).resolves.toBeUndefined(); expect(runOpenshell).toHaveBeenCalledWith( ["provider", "profile", "export", MCP_BRIDGE_PROVIDER_TYPE, "--output", "json"], expect.any(Object), ); }); - it("rejects an existing profile that can supply its own endpoint authority", async () => { + it.each([ + ["endpoint authority", { endpoints: [{ host: "other.example", port: 443 }] }], + ["missing credential declaration", { credentials: [] }], + ["inference authority", { inference_capable: true }], + [ + "workload files", + { files: [{ path: "client.yaml", content: "unowned", env_var: "CLIENT_CONFIG" }] }, + ], + ] as const)("rejects an existing profile that supplies unowned %s", async (_name, unowned) => { const runOpenshell = vi.fn().mockReturnValueOnce({ status: 0, stdout: JSON.stringify({ - id: MCP_BRIDGE_PROVIDER_TYPE, - credentials: [], - endpoints: [{ host: "other.example", port: 443 }], - binaries: [], - inference_capable: false, + ...JSON.parse(exportedEndpointlessProfile(MCP_BRIDGE_PROVIDER_TYPE, false)), + ...unowned, }), stderr: "", }); setProviderCommandRuntimeHooksForTest({ runOpenshell: runOpenshell as never }); - await expect(ensureMcpBridgeProviderProfile(runtimeSelection)).rejects.toThrow( + await expect(ensureMcpBridgeProviderProfile(runtimeSelection, CREDENTIAL)).rejects.toThrow( /does not match NemoClaw's endpointless credential contract/, ); + expect(runOpenshell).toHaveBeenCalledOnce(); }); it("suppresses MCP profile import output at the bridge error boundary", async () => { const secret = "mcp-import-secret-must-not-leak"; + let generatedPath = ""; const runOpenshell = vi .fn() .mockReturnValueOnce({ status: 1, stdout: "", stderr: "provider profile not found" }) - .mockReturnValueOnce({ status: 1, stdout: "", stderr: `import rejected: ${secret}` }); + .mockImplementationOnce((args: string[]) => { + generatedPath = args[4]; + return { status: 1, stdout: "", stderr: `import rejected: ${secret}` }; + }); setProviderCommandRuntimeHooksForTest({ runOpenshell: runOpenshell as never }); let message = ""; try { - await ensureMcpBridgeProviderProfile(runtimeSelection); + await ensureMcpBridgeProviderProfile(runtimeSelection, CREDENTIAL); } catch (error) { message = error instanceof Error ? error.message : String(error); } @@ -100,6 +142,8 @@ describe("OpenShell MCP provider profile", () => { ); expect(message).not.toContain(secret); expect(message).not.toContain("import rejected"); + expect(generatedPath).not.toBe(""); + expect(existsSync(generatedPath)).toBe(false); }); it("fails closed when the MCP profile cannot be exported", async () => { @@ -110,7 +154,7 @@ describe("OpenShell MCP provider profile", () => { }); setProviderCommandRuntimeHooksForTest({ runOpenshell: runOpenshell as never }); - await expect(ensureMcpBridgeProviderProfile(runtimeSelection)).rejects.toThrow( + await expect(ensureMcpBridgeProviderProfile(runtimeSelection, CREDENTIAL)).rejects.toThrow( /nemoclaw-mcp-v1.*could not be exported for validation/u, ); expect(runOpenshell).toHaveBeenCalledOnce(); diff --git a/src/lib/actions/sandbox/mcp-bridge-provider.test.ts b/src/lib/actions/sandbox/mcp-bridge-provider.test.ts index cc1e0aa5048..646a670fc9f 100644 --- a/src/lib/actions/sandbox/mcp-bridge-provider.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-provider.test.ts @@ -21,6 +21,7 @@ import { detachProvider, ensureMcpBridgeProviderProfile, MCP_BRIDGE_PROVIDER_TYPE, + mcpBridgeProviderType, observeMcpCredentialRevision, refreshMcpProviderEnvironment, upsertMcpProvider, @@ -45,7 +46,7 @@ function providerMetadataOutput( return [ `Name: ${name}`, `Id: ${id}`, - `Type: ${type}`, + `Type: ${type === MCP_BRIDGE_PROVIDER_TYPE ? mcpBridgeProviderType(credentialKey) : type}`, `Resource version: ${resourceVersion}`, `Credential keys: ${credentialKey}`, "Config keys: ", @@ -118,7 +119,7 @@ describe("OpenShell MCP provider state", () => { }; await expect(assertMcpProviderRecoverable(entry, runtimeSelection)).rejects.toThrow( - /legacy generic profile.*cannot bind to an MCP endpoint/, + /legacy generic profile.*cannot bind its credential to an MCP endpoint/, ); }); @@ -209,7 +210,16 @@ describe("OpenShell MCP provider state", () => { const profileOutput = (id: string, inferenceCapable: boolean) => JSON.stringify({ id, - credentials: [], + credentials: [ + { + name: "EXPECTED_TOKEN", + env_vars: ["EXPECTED_TOKEN"], + required: true, + auth_style: "bearer", + header_name: "Authorization", + query_param: "", + }, + ], endpoints: [], binaries: [], inference_capable: inferenceCapable, @@ -289,7 +299,7 @@ describe("OpenShell MCP provider state", () => { providerCommand.setProviderCommandRuntimeHooksForTest({ runOpenshell: runOpenshell as never }); const providerAdapter = createMcpProviderAdapterBoundary(runtimeSelection).adapter; - await ensureMcpBridgeProviderProfile(runtimeSelection, providerAdapter); + await ensureMcpBridgeProviderProfile(runtimeSelection, "EXPECTED_TOKEN", providerAdapter); const created = await upsertMcpProvider("alpha-mcp-fake", [{ name: "EXPECTED_TOKEN" }], { allowExisting: false, runtimeSelection, diff --git a/src/lib/actions/sandbox/mcp-bridge-provider.ts b/src/lib/actions/sandbox/mcp-bridge-provider.ts index b9e3ad25b0d..d2482624633 100644 --- a/src/lib/actions/sandbox/mcp-bridge-provider.ts +++ b/src/lib/actions/sandbox/mcp-bridge-provider.ts @@ -16,6 +16,7 @@ export { inspectMcpProvider, inspectMcpProviderAttachments, MCP_BRIDGE_PROVIDER_TYPE, + mcpBridgeProviderType, preflightMcpEntryTargets, providerAttached, providerMatchesCredential, diff --git a/src/lib/actions/sandbox/mcp-bridge-rebuild.ts b/src/lib/actions/sandbox/mcp-bridge-rebuild.ts index 633dfc612bd..004e1fec12a 100644 --- a/src/lib/actions/sandbox/mcp-bridge-rebuild.ts +++ b/src/lib/actions/sandbox/mcp-bridge-rebuild.ts @@ -8,7 +8,10 @@ import YAML from "yaml"; import type { McpSourceEntry } from "./mcp-bridge-contracts"; import type { PreparedStoppedNativeState } from "../../state/state-directory-restore"; import * as policies from "../../policy"; -import { isSandboxPolicyCredentialFree } from "../../policy/sandbox-policy-validation"; +import { + isSandboxPolicyCredentialFree, + parseAndValidateSandboxPolicy, +} from "../../policy/sandbox-policy-validation"; import { rollbackScrubbedMcpAdapters, scrubManagedMcpAdapterOrThrow, @@ -199,6 +202,7 @@ export async function prepareMcpBridgesForStoppedSandboxRebuild( if (!policyHandoff || !isSandboxPolicyCredentialFree(policyHandoff)) { throw new McpBridgeError("Stopped MCP preservation requires a credential-free live policy."); } + parseAndValidateSandboxPolicy(policyHandoff); return { ...prepared, policyHandoff, @@ -264,6 +268,7 @@ export async function prepareMcpBridgesForRebuild( `Cannot prepare the MCP rebuild policy handoff for sandbox '${sandboxName}' because its live OpenShell policy contains a literal credential value. Replace literal credentials with supported OpenShell credential bindings or resolver placeholders, then retry the rebuild.`, ); } + parseAndValidateSandboxPolicy(policyHandoff); const expectedTeardownPolicy = policyWithoutManagedMcpEntries(policyHandoff, entries); const detached: McpSourceEntry[] = []; const scrubbedAdapters: McpScrubbedAdapterEntry[] = []; diff --git a/src/lib/actions/sandbox/mcp-bridge-restart.ts b/src/lib/actions/sandbox/mcp-bridge-restart.ts index 67141bdef21..dce3af4d5b0 100644 --- a/src/lib/actions/sandbox/mcp-bridge-restart.ts +++ b/src/lib/actions/sandbox/mcp-bridge-restart.ts @@ -262,7 +262,7 @@ export async function restoreExistingMcpBridgeRuntime( providerRuntimeSelection, ); const providerAdapter = createMcpProviderAdapterBoundary(providerRuntimeSelection).adapter; - await ensureMcpBridgeProviderProfile(providerRuntimeSelection, providerAdapter); + await ensureMcpBridgeProviderProfile(providerRuntimeSelection, entry.env[0], providerAdapter); if (options.applyPolicy !== false) { await applyGeneratedPolicy( sandboxName, diff --git a/src/lib/actions/sandbox/mcp-bridge-runtime-validation.test.ts b/src/lib/actions/sandbox/mcp-bridge-runtime-validation.test.ts index cbd276b0194..7d492935c3d 100644 --- a/src/lib/actions/sandbox/mcp-bridge-runtime-validation.test.ts +++ b/src/lib/actions/sandbox/mcp-bridge-runtime-validation.test.ts @@ -14,7 +14,7 @@ function matchingOpenshellRuntime() { resolveOpenshell: () => "/test/openshell", runVersionCommand: () => ({ status: 0, - stdout: `openshell ${MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION}\n`, + stdout: "openshell 0.1.7\n", stderr: "", }), }; @@ -42,16 +42,16 @@ describe("MCP credential-boundary runtime validation", () => { ...matchingOpenshellRuntime(), runVersionCommand: () => ({ status: 0, - stdout: "openshell 0.0.73\n", + stdout: "openshell 0.1.2\n", stderr: "", }), }); const error = captureRuntimeVersionError(mismatch); expect(error.message).toContain( - `expected ${MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION}, actual 0.0.73 (version mismatch). Install OpenShell ${MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION}, or point NEMOCLAW_OPENSHELL_BIN to that version, then retry.`, + `expected ${MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION}, actual 0.1.2 (version mismatch). Install OpenShell ${MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION}, or point NEMOCLAW_OPENSHELL_BIN to that version, then retry.`, ); expect(error).toMatchObject({ - actualVersion: "0.0.73", + actualVersion: "0.1.2", detail: "version mismatch", reason: "version-mismatch", }); diff --git a/src/lib/actions/sandbox/mcp-bridge-url-validation.ts b/src/lib/actions/sandbox/mcp-bridge-url-validation.ts index f4e9d718491..d70e184480e 100644 --- a/src/lib/actions/sandbox/mcp-bridge-url-validation.ts +++ b/src/lib/actions/sandbox/mcp-bridge-url-validation.ts @@ -51,12 +51,12 @@ function rejectUnsupportedOpenShellMcpHostAlias(hostname: string): void { // invalidState: a host alias is accepted without an attested gateway address, // forcing broad private-range policy instead of an exact destination pin. // sourceBoundary: the pinned OpenShell release owns gateway-address discovery. - // whyNotSourceFix: v0.0.116 exposes no attested driver gateway address. + // whyNotSourceFix: this integration has no attested driver gateway address. // regressionTest: URL validation and all three live adapters reject aliases. // removalCondition: remove only after a reviewed OpenShell capability exposes // an attested address; a future version number alone is not that capability. throw new McpBridgeError( - `Authenticated MCP OpenShell host alias '${hostname}' is unavailable with OpenShell v0.0.116 because that release does not expose an attested driver gateway address for exact policy pinning. Use a normal HTTPS DNS endpoint with public address records.`, + `Authenticated MCP OpenShell host alias '${hostname}' is unavailable because this integration does not expose an attested driver gateway address for exact policy pinning. Use a normal HTTPS DNS endpoint with public address records.`, 2, ); } diff --git a/src/lib/actions/sandbox/mcp-bridge-validation.ts b/src/lib/actions/sandbox/mcp-bridge-validation.ts index e676ae0c187..6db8a69cdcf 100644 --- a/src/lib/actions/sandbox/mcp-bridge-validation.ts +++ b/src/lib/actions/sandbox/mcp-bridge-validation.ts @@ -27,7 +27,7 @@ import { normalizeMcpServerUrl } from "./mcp-bridge-url-validation"; // must reject a missing or malformed security manifest instead of letting the // CLI start with a weakened credential-name denylist. Input, package, image, // and workflow contracts pin its structure, installed path, and version. -import childVisibleCredentialManifest from "./openshell-child-visible-credentials.v0.0.116.json"; +import childVisibleCredentialManifest from "./openshell-child-visible-credentials.v0.1.7.json"; export { MCP_SERVER_URL_MAX_LENGTH, @@ -153,7 +153,7 @@ export function assertMcpCredentialBoundaryRuntimeVersion( // key and exposes or executes the provider value outside the intended request. // sourceBoundary: the versioned JSON manifest pins OpenShell-owned keys to the // shipped source commit; NemoClaw owns host and agent runtime-control rejects. -// whyNotSourceFix: v0.0.116 intentionally projects bound credential placeholders +// whyNotSourceFix: OpenShell intentionally projects bound credential placeholders // and does not advertise safe credential-name capabilities at runtime. // regressionTest: the mcp-bridge-input validation/runtime suites check every // pinned and runtime key; package contracts require version alignment. diff --git a/src/lib/actions/sandbox/mcp-bridge/provider-profile.ts b/src/lib/actions/sandbox/mcp-bridge/provider-profile.ts new file mode 100644 index 00000000000..f3e53fdfd57 --- /dev/null +++ b/src/lib/actions/sandbox/mcp-bridge/provider-profile.ts @@ -0,0 +1,51 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import YAML from "yaml"; + +import { endpointlessProviderProfilePath } from "../../../adapters/openshell/provider-profile"; +import { REPOSITORY_ROOT } from "../../../core/repository-root"; +import { validateMcpCredentialEnvName } from "../mcp-bridge-validation"; + +export const MCP_BRIDGE_PROVIDER_TYPE = "nemoclaw-mcp-v1"; + +/** A profile declares one credential name without deriving identity from its secret. */ +export function mcpBridgeProviderType(credential: string): string { + return `${MCP_BRIDGE_PROVIDER_TYPE}-${createHash("sha256").update(credential).digest("hex").slice(0, 32)}`; +} + +export async function withMcpBridgeProviderProfile( + credential: string, + use: (profilePath: string) => T | Promise, +): Promise { + validateMcpCredentialEnvName(credential); + const profile = YAML.parse( + fs.readFileSync( + endpointlessProviderProfilePath(REPOSITORY_ROOT, MCP_BRIDGE_PROVIDER_TYPE), + "utf8", + ), + ) as Record; + profile.id = mcpBridgeProviderType(credential); + profile.credentials = [ + { + name: credential, + env_vars: [credential], + required: true, + auth_style: "bearer", + header_name: "Authorization", + query_param: "", + }, + ]; + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-mcp-profile-")); + try { + const profilePath = path.join(directory, "profile.json"); + fs.writeFileSync(profilePath, JSON.stringify(profile), { mode: 0o600, flag: "wx" }); + return await use(profilePath); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} diff --git a/src/lib/actions/sandbox/mcp-bridge/status-resolution-test-fixture.ts b/src/lib/actions/sandbox/mcp-bridge/status-resolution-test-fixture.ts index f6e0f11d0a5..9ba81bfe75c 100644 --- a/src/lib/actions/sandbox/mcp-bridge/status-resolution-test-fixture.ts +++ b/src/lib/actions/sandbox/mcp-bridge/status-resolution-test-fixture.ts @@ -115,7 +115,7 @@ providerCommands.runOpenshellProviderCommand = (args) => { const credentialKey = providerName === "alpha-mcp-slack" ? "SLACK_TOKEN" : providerCredentialKey; return { status: 0, - stdout: "Name: " + providerName + "\nId: 11111111-2222-4333-8444-555555555555\nType: nemoclaw-mcp-v1\nResource version: 4\nCredential keys: " + credentialKey + "\nConfig keys: \n", + stdout: "Name: " + providerName + "\nId: 11111111-2222-4333-8444-555555555555\nType: " + providerInspection.mcpBridgeProviderType(credentialKey) + "\nResource version: 4\nCredential keys: " + credentialKey + "\nConfig keys: \n", stderr: "", }; } diff --git a/src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.5.json b/src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.5.json new file mode 100644 index 00000000000..79acb51fb2e --- /dev/null +++ b/src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.5.json @@ -0,0 +1,143 @@ +{ + "openshellVersion": "0.1.5", + "openshellCommit": "bdb956f2cd65a635f1294fccf1d138d263bf650f", + "sources": [ + "crates/openshell-core/src/google_cloud.rs", + "crates/openshell-core/src/provider_credentials.rs", + "crates/openshell-core/src/secrets.rs" + ], + "nemoclawSources": [ + "src/lib/subprocess-env.ts", + "src/lib/actions/sandbox/mcp-bridge-validation.ts", + "agents/hermes/mcp-config-transaction.py" + ], + "generation": { + "method": "openshell-static-config-and-child-env-source-review-v1", + "upstreamSourceEvidence": [ + { + "gitObjectId": "efc15a65270ce9afee65b1170f277ff44109c0bf", + "path": "crates/openshell-core/src/google_cloud.rs", + "sha256": "6b9ac82967d06a4d496ad2ebc29e3bf8c68c1b1e8c12a2ab9c30e7155a8d6fba" + }, + { + "gitObjectId": "d71f34b7c6229e5430fbf16f76af6ce2d6a430b9", + "path": "crates/openshell-core/src/provider_credentials.rs", + "sha256": "863eac733e4b114c841dee790aaf56dc897c66c53cceca50eafc546f1aea4723" + }, + { + "gitObjectId": "2e51c9de28e1a153f1ca30decdd1e4744a749e74", + "path": "crates/openshell-core/src/secrets.rs", + "sha256": "16b3f771048a8337e453acbc1006ec9d744434bea34252f088b3acbac0dd5f02" + } + ], + "nemoclawSourceEvidence": [ + { + "path": "src/lib/subprocess-env.ts", + "sha256": "0e41a8298494c8578709d012cdb8b3e1df7172c55d2883b6a732835200acffcf" + }, + { + "path": "src/lib/actions/sandbox/mcp-bridge-validation.ts", + "sha256": "f86d2ef3666f4e5074288b343db644d1ff5b089b602d2c063e375398c5cf7e44" + }, + { + "path": "agents/hermes/mcp-config-transaction.py", + "sha256": "4195ed298e2f58e25bddd66c38c37a30454d6949ba4c1293b61dd9ff8e1b8f96" + } + ] + }, + "rawChildValueKeys": [ + "GCP_PROJECT_ID", + "GOOGLE_CLOUD_PROJECT", + "CLOUD_ML_REGION", + "GCP_LOCATION", + "GCP_SERVICE_ACCOUNT_EMAIL", + "GOOSE_PROVIDER", + "ANTHROPIC_VERTEX_PROJECT_ID", + "VERTEX_LOCATION" + ], + "rewrittenChildValueKeys": [ + "GCE_METADATA_HOST", + "GCE_METADATA_IP", + "METADATA_SERVER_DETECTION" + ], + "runtimeControlKeys": [ + "_JAVA_OPTIONS", + "ALL_PROXY", + "all_proxy", + "API_SERVER_KEY", + "BASH_ENV", + "BASHOPTS", + "CDPATH", + "CLASSPATH", + "CONDA_PREFIX", + "CURL_CA_BUNDLE", + "DENO_CERT", + "DOCKER_HOST", + "ENV", + "GCONV_PATH", + "GIT_SSL_CAINFO", + "GIT_SSL_CAPATH", + "GLOBIGNORE", + "grpc_proxy", + "HOME", + "HOSTNAME", + "HTTP_PROXY", + "http_proxy", + "HTTPS_PROXY", + "https_proxy", + "IFS", + "KUBECONFIG", + "LANG", + "LOCPATH", + "LOGNAME", + "NLSPATH", + "NODE_ENV", + "NODE_EXTRA_CA_CERTS", + "NO_PROXY", + "no_proxy", + "PATH", + "PROMPT_COMMAND", + "PS4", + "REQUESTS_CA_BUNDLE", + "RUST_BACKTRACE", + "RUST_LOG", + "SHELL", + "SHELLOPTS", + "SSH_AUTH_SOCK", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + "TEMP", + "TERM", + "TMP", + "TMPDIR", + "USER", + "VIRTUAL_ENV", + "ZDOTDIR" + ], + "runtimeControlPrefixes": [ + "DEEPAGENTS_", + "DYLD_", + "GATEWAY_", + "GLIBC_", + "GRPC_", + "HERMES_", + "JAVA_", + "JDK_", + "LANGCHAIN_", + "LANGGRAPH_", + "LANGSMITH_", + "LC_", + "LD_", + "MALLOC_", + "NEMOCLAW_", + "NODE_", + "OPENAI_", + "OPENCLAW_", + "OPENSHELL_", + "PERL", + "PYTHON", + "RUBY", + "UV_", + "XDG_" + ] +} diff --git a/src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json b/src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json new file mode 100644 index 00000000000..808d36fda84 --- /dev/null +++ b/src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json @@ -0,0 +1,161 @@ +{ + "openshellVersion": "0.1.7", + "openshellCommit": "5d25211b25a8fc887e8d5c606d57d6ab22f1d76c", + "sources": [ + "crates/openshell-core/src/google_cloud.rs", + "crates/openshell-core/src/provider_credentials.rs", + "crates/openshell-core/src/secrets.rs", + "crates/openshell-sandbox/src/child_env.rs", + "crates/openshell-sandbox/src/process.rs", + "crates/openshell-sandbox/src/boundary_exec.rs" + ], + "nemoclawSources": [ + "src/lib/subprocess-env.ts", + "src/lib/actions/sandbox/mcp-bridge-validation.ts", + "agents/hermes/mcp-config-transaction.py" + ], + "generation": { + "method": "openshell-static-config-and-child-env-source-review-v1", + "upstreamSourceEvidence": [ + { + "path": "crates/openshell-core/src/google_cloud.rs", + "gitObjectId": "efc15a65270ce9afee65b1170f277ff44109c0bf", + "sha256": "6b9ac82967d06a4d496ad2ebc29e3bf8c68c1b1e8c12a2ab9c30e7155a8d6fba" + }, + { + "path": "crates/openshell-core/src/provider_credentials.rs", + "gitObjectId": "d71f34b7c6229e5430fbf16f76af6ce2d6a430b9", + "sha256": "863eac733e4b114c841dee790aaf56dc897c66c53cceca50eafc546f1aea4723" + }, + { + "path": "crates/openshell-core/src/secrets.rs", + "gitObjectId": "2e51c9de28e1a153f1ca30decdd1e4744a749e74", + "sha256": "16b3f771048a8337e453acbc1006ec9d744434bea34252f088b3acbac0dd5f02" + }, + { + "path": "crates/openshell-sandbox/src/child_env.rs", + "gitObjectId": "5d9c90a20f8be6bde9d00e37e5342747bdfe9e31", + "sha256": "62c20a49adca18d73f5f4ebc2615a9ffee73659b0464b8c1f3ce8e81b0223e95" + }, + { + "path": "crates/openshell-sandbox/src/process.rs", + "gitObjectId": "a8995bc7da9f09ee82a38aa2508974d668ffe6b3", + "sha256": "3d65602e474b01ea2f7ff503accbec1628572d94c2b0b692f102bcd5d79e4d76" + }, + { + "path": "crates/openshell-sandbox/src/boundary_exec.rs", + "gitObjectId": "d823fef977bedd0ba3309aeeebef206507d474ec", + "sha256": "6cb999db3775f6449ffbc7b6d1deda2934f7cb1db6eceb7f9e6c4c2838e9815f" + } + ], + "nemoclawSourceEvidence": [ + { + "path": "src/lib/subprocess-env.ts", + "sha256": "0e41a8298494c8578709d012cdb8b3e1df7172c55d2883b6a732835200acffcf" + }, + { + "path": "src/lib/actions/sandbox/mcp-bridge-validation.ts", + "sha256": "33416f45a7e0176ab0d2b825956554344db94da9dae661ff652bb9cadbad8196" + }, + { + "path": "agents/hermes/mcp-config-transaction.py", + "sha256": "64c7abd559795d6be761f61be484fbc654b31546bd1bc231f265571b6493160a" + } + ] + }, + "rawChildValueKeys": [ + "GCP_PROJECT_ID", + "GOOGLE_CLOUD_PROJECT", + "CLOUD_ML_REGION", + "GCP_LOCATION", + "GCP_SERVICE_ACCOUNT_EMAIL", + "GOOSE_PROVIDER", + "ANTHROPIC_VERTEX_PROJECT_ID", + "VERTEX_LOCATION" + ], + "rewrittenChildValueKeys": [ + "GCE_METADATA_HOST", + "GCE_METADATA_IP", + "METADATA_SERVER_DETECTION" + ], + "runtimeControlKeys": [ + "_JAVA_OPTIONS", + "ALL_PROXY", + "all_proxy", + "API_SERVER_KEY", + "BASH_ENV", + "BASHOPTS", + "CDPATH", + "CLASSPATH", + "CONDA_PREFIX", + "CURL_CA_BUNDLE", + "DENO_CERT", + "DOCKER_HOST", + "ENV", + "GCONV_PATH", + "GIT_SSL_CAINFO", + "GIT_SSL_CAPATH", + "GLOBIGNORE", + "grpc_proxy", + "HOME", + "HOSTNAME", + "HTTP_PROXY", + "http_proxy", + "HTTPS_PROXY", + "https_proxy", + "IFS", + "KUBECONFIG", + "LANG", + "LOCPATH", + "LOGNAME", + "NLSPATH", + "NODE_ENV", + "NODE_EXTRA_CA_CERTS", + "NO_PROXY", + "no_proxy", + "PATH", + "PROMPT_COMMAND", + "PS4", + "REQUESTS_CA_BUNDLE", + "RUST_BACKTRACE", + "RUST_LOG", + "SHELL", + "SHELLOPTS", + "SSH_AUTH_SOCK", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + "TEMP", + "TERM", + "TMP", + "TMPDIR", + "USER", + "VIRTUAL_ENV", + "ZDOTDIR" + ], + "runtimeControlPrefixes": [ + "DEEPAGENTS_", + "DYLD_", + "GATEWAY_", + "GLIBC_", + "GRPC_", + "HERMES_", + "JAVA_", + "JDK_", + "LANGCHAIN_", + "LANGGRAPH_", + "LANGSMITH_", + "LC_", + "LD_", + "MALLOC_", + "NEMOCLAW_", + "NODE_", + "OPENAI_", + "OPENCLAW_", + "OPENSHELL_", + "PERL", + "PYTHON", + "RUBY", + "UV_", + "XDG_" + ] +} diff --git a/src/lib/actions/sandbox/process-recovery-openclaw-doctor.test.ts b/src/lib/actions/sandbox/process-recovery-openclaw-doctor.test.ts index 60352d938dc..d9b487f7c9a 100644 --- a/src/lib/actions/sandbox/process-recovery-openclaw-doctor.test.ts +++ b/src/lib/actions/sandbox/process-recovery-openclaw-doctor.test.ts @@ -528,26 +528,21 @@ describe("OpenClaw post-upgrade recovery doctor", () => { expect(execute.mock.invocationCallOrder[0]).toBeLessThan(capture.mock.invocationCallOrder[0]!); }); - it("keeps filesystem gates direct and probes health through the OpenShell network namespace", async () => { + it("uses owner SSH for maintenance and native exec for final health (#12603)", async () => { const execute = vi.fn(async () => ({ status: 0, stdout: "", stderr: "" })); - const executePrivileged = vi.fn( - (_sandboxName: string, _command: readonly string[], _timeout: number) => ({ - status: 0, - stdout: "", - stderr: "", - }), - ); + const maintenance = vi.fn(async () => ({ status: 0, stdout: "", stderr: "" })); const capture = vi.fn(() => ({ status: 0, output: "" })); + const runtimeSelection = { gatewayName: "recorded-gateway", workspace: "default" as const }; const deps = { captureOpenshell: capture as never, - executePrivilegedSandboxCommand: executePrivileged, executeSandboxExecCommand: execute, + executeMaintenanceCommand: maintenance, now: () => 0, sleep: vi.fn(async () => undefined), }; - const begun = await beginOpenClawPostRestoreDoctor("alpha", undefined, deps); - expect(begun).toEqual({ ok: true, window: { sandboxName: "alpha" } }); + const begun = await beginOpenClawPostRestoreDoctor("alpha", runtimeSelection, deps); + expect(begun).toEqual({ ok: true, window: { sandboxName: "alpha", runtimeSelection } }); await expect( finishOpenClawPostRestoreDoctor( (begun as { ok: true; window: Parameters[0] }) @@ -556,38 +551,17 @@ describe("OpenClaw post-upgrade recovery doctor", () => { ), ).resolves.toEqual({ ok: true }); - expect(execute).toHaveBeenNthCalledWith( - 1, - "alpha", - expect.not.stringContaining("curl"), - 30_000, - {}, - ); - expect(execute).toHaveBeenNthCalledWith( - 2, - "alpha", - expect.stringContaining("curl"), - expect.any(Number), - {}, - ); - expect(executePrivileged).toHaveBeenCalledTimes(5); - const expectedDirectCall = [ - "alpha", - ["/bin/sh", "-lc", expect.stringContaining('/usr/bin/setpriv --reuid="$uid" --regid="$gid"')], - expect.any(Number), - ]; - expect(executePrivileged.mock.calls).toEqual([ - expectedDirectCall, - expectedDirectCall, - expectedDirectCall, - expectedDirectCall, - expectedDirectCall, + expect(execute.mock.calls).toEqual([ + ["alpha", expect.not.stringContaining("curl"), 30_000, { runtimeSelection }], + ["alpha", expect.stringContaining("curl"), expect.any(Number), { runtimeSelection }], + ]); + expect(maintenance.mock.calls).toEqual([ + ["alpha", expect.stringContaining("curl"), expect.any(Number), { runtimeSelection }], + ["alpha", expect.not.stringContaining("curl"), 30_000, { runtimeSelection }], + ["alpha", expect.not.stringContaining("curl"), expect.any(Number), { runtimeSelection }], + ["alpha", expect.not.stringContaining("curl"), expect.any(Number), { runtimeSelection }], + ["alpha", expect.not.stringContaining("curl"), expect.any(Number), { runtimeSelection }], ]); - expect(executePrivileged.mock.calls[0]?.[1][2]).toContain("curl"); - expect(executePrivileged.mock.calls[1]?.[1][2]).not.toContain("curl"); - expect(executePrivileged.mock.calls[2]?.[1][2]).not.toContain("curl"); - expect(executePrivileged.mock.calls[3]?.[1][2]).not.toContain("curl"); - expect(executePrivileged.mock.calls[4]?.[1][2]).not.toContain("curl"); }); it("does not restart when the one-shot marker cannot be persisted", async () => { @@ -954,23 +928,20 @@ describe.each([ "unavailable", "malformed", ] as const)("doctor maintenance transport failure: %s", (kind) => { - it("reports an unverified marker without restarting or trying privilege", async () => { + it("reports an unverified marker without restarting", async () => { const error = new SandboxCommandTransportError(kind); const execute = vi.fn().mockRejectedValue(error); const capture = vi.fn(); - const privileged = vi.fn(); await expect( beginOpenClawPostRestoreDoctor("alpha", undefined, { captureOpenshell: capture as never, executeSandboxExecCommand: execute, - executePrivilegedSandboxCommand: privileged, now: () => 0, sleep: vi.fn(), }), ).resolves.toEqual({ ok: false, stage: "mark", detail: error.message }); expect(execute).toHaveBeenCalledOnce(); expect(capture).not.toHaveBeenCalled(); - expect(privileged).not.toHaveBeenCalled(); }); it("reports an unverified native release without polling or retrying", async () => { diff --git a/src/lib/actions/sandbox/process-recovery.ts b/src/lib/actions/sandbox/process-recovery.ts index 049e4b813a2..6e44e356fbe 100644 --- a/src/lib/actions/sandbox/process-recovery.ts +++ b/src/lib/actions/sandbox/process-recovery.ts @@ -24,6 +24,7 @@ import { } from "../../adapters/openshell/runtime"; import { DEFAULT_SANDBOX_EXEC_TIMEOUT_MS, + executeSandboxMaintenanceCommand, executeSandboxExecCommand, buildSandboxCommandEnvironment, SandboxCommandTransportError, @@ -255,8 +256,8 @@ interface OpenClawPostRestoreDoctorDeps { sandboxName: string, runtimeSelection?: OpenShellRuntimeSelection, ) => Promise; - executePrivilegedSandboxCommand?: typeof executePrivilegedSandboxCommand; executeSandboxExecCommand: typeof executeSandboxExecCommand; + executeMaintenanceCommand?: typeof executeSandboxMaintenanceCommand; lookupSandbox?: CliOpenShellSandboxLookup; now: () => number; sleep: typeof sleepSeconds; @@ -264,8 +265,8 @@ interface OpenClawPostRestoreDoctorDeps { const OPENCLAW_POST_RESTORE_DOCTOR_DEPS: OpenClawPostRestoreDoctorDeps = { captureOpenshell, - executePrivilegedSandboxCommand, executeSandboxExecCommand, + executeMaintenanceCommand: executeSandboxMaintenanceCommand, now: Date.now, sleep: sleepSeconds, }; @@ -277,29 +278,13 @@ async function executeOpenClawDoctorGateCommand( timeout: number, runtimeSelection?: OpenShellRuntimeSelection, ): Promise { - if (!deps.executePrivilegedSandboxCommand) { - try { - return await deps.executeSandboxExecCommand(sandboxName, command, timeout, { - ...(runtimeSelection ? { runtimeSelection } : {}), - }); - } catch (error) { - if (!(error instanceof SandboxCommandTransportError)) throw error; - return null; - } - } - const ownerCommand = [ - "set -e", - "uid=\"$(stat -c '%u' /sandbox/.openclaw)\"", - "gid=\"$(stat -c '%g' /sandbox/.openclaw)\"", - `exec /usr/bin/setpriv --reuid="$uid" --regid="$gid" --clear-groups -- /bin/sh -lc ${shellQuote(command)}`, - ].join("; "); try { - return deps.executePrivilegedSandboxCommand( - sandboxName, - ["/bin/sh", "-lc", ownerCommand], - timeout, - ); - } catch { + const execute = deps.executeMaintenanceCommand ?? deps.executeSandboxExecCommand; + return await execute(sandboxName, command, timeout, { + ...(runtimeSelection ? { runtimeSelection } : {}), + }); + } catch (error) { + if (!(error instanceof SandboxCommandTransportError)) throw error; return null; } } @@ -1145,6 +1130,7 @@ export async function finishOpenClawPostRestoreDoctor( const OPENCLAW_UNREGISTERED_POST_RESTORE_DOCTOR_DEPS: OpenClawPostRestoreDoctorDeps = { captureOpenshell, executeSandboxExecCommand, + executeMaintenanceCommand: executeSandboxMaintenanceCommand, now: Date.now, sleep: sleepSeconds, }; @@ -1152,9 +1138,9 @@ const OPENCLAW_UNREGISTERED_POST_RESTORE_DOCTOR_DEPS: OpenClawPostRestoreDoctorD /** * Enter the same startup-owned maintenance window before a recreated sandbox * has published its replacement registry row. The caller must independently - * revalidate its prepared-create identity around this operation; every - * in-sandbox command remains routed through OpenShell without a container - * fallback. + * revalidate its prepared-create identity around this operation. Maintenance + * commands use OpenShell SSH as the sandbox owner; health probes use native + * exec after release. */ export function beginUnregisteredOpenClawBackupQuiesce( sandboxName: string, @@ -2535,7 +2521,7 @@ async function checkAndRecoverSandboxProcessesWithoutHostLock( checked: true, wasRunning: true, recovered: false, - forwardRecovered: forwardRecovered || anyAuxiliaryRecovered(auxiliaryResults), + forwardRecovered: true, }; } const dashboardForwardRecovered = await measureAsync("forward", () => diff --git a/src/lib/actions/sandbox/rebuild-backup-phase.test.ts b/src/lib/actions/sandbox/rebuild-backup-phase.test.ts index 66e4fc6e3d9..cc835ed948a 100644 --- a/src/lib/actions/sandbox/rebuild-backup-phase.test.ts +++ b/src/lib/actions/sandbox/rebuild-backup-phase.test.ts @@ -15,8 +15,13 @@ const mocks = vi.hoisted(() => ({ retireOpenClawPostRestoreDoctorForDelete: vi.fn(), recordRebuildRecoveryBackup: vi.fn(), secureTempFile: vi.fn(), + nativeRestoreTransportError: vi.fn(), })); +vi.mock("../../state/sandbox", async (importOriginal) => ({ + ...(await importOriginal()), + nativeRestoreTransportError: mocks.nativeRestoreTransportError, +})); vi.mock("../../policy", async (importOriginal) => ({ ...(await importOriginal()), captureRecordedSandboxBasePolicy: mocks.captureRecordedSandboxBasePolicy, @@ -46,6 +51,7 @@ import { const temporaryDirectories: string[] = []; beforeEach(() => { + mocks.nativeRestoreTransportError.mockReset().mockReturnValue(null); mocks.abortOpenClawPostRestoreDoctor.mockReset().mockResolvedValue({ ok: true }); mocks.beginOpenClawBackupQuiesce.mockReset().mockResolvedValue({ ok: true, @@ -100,6 +106,20 @@ describe("rebuild policy handoff", () => { expect(mocks.finishOpenClawBackupQuiesce).not.toHaveBeenCalled(); }); + it("preserves the source when the SFTP client is unavailable", async () => { + mocks.nativeRestoreTransportError.mockReturnValue("Install the OpenSSH client tools"); + const backup = vi.fn(); + + await expect(runRebuildBackupPhase(input(), backup)).rejects.toThrow( + "Install the OpenSSH client tools", + ); + + expect(backup).not.toHaveBeenCalled(); + expect(mocks.beginOpenClawBackupQuiesce).not.toHaveBeenCalled(); + expect(mocks.captureRecordedSandboxBasePolicy).not.toHaveBeenCalled(); + expect(mocks.recordRebuildRecoveryBackup).not.toHaveBeenCalled(); + }); + it("releases an unchanged source backup without requesting doctor", async () => { const window = { sandboxName: "alpha", kind: "backup" as const }; diff --git a/src/lib/actions/sandbox/rebuild-backup-phase.ts b/src/lib/actions/sandbox/rebuild-backup-phase.ts index 6c3f33cda68..bc66ebd83a2 100644 --- a/src/lib/actions/sandbox/rebuild-backup-phase.ts +++ b/src/lib/actions/sandbox/rebuild-backup-phase.ts @@ -15,7 +15,11 @@ import type { WebSearchConfig } from "../../inference/web-search"; import type { SandboxMessagingPlan } from "../../messaging"; import { resolveDockerSnapshotRecreateGpuDevice } from "../../onboard/runtime-provider/snapshot"; import { secureTempFile } from "../../onboard/temp-files"; -import { readRebuildPolicyHandoff, writeRebuildPolicyHandoff } from "../../state/sandbox"; +import { + nativeRestoreTransportError, + readRebuildPolicyHandoff, + writeRebuildPolicyHandoff, +} from "../../state/sandbox"; import { captureRecordedSandboxBasePolicy } from "../../policy"; import { isSandboxPolicyCredentialFree } from "../../policy/sandbox-policy-validation"; import type { RebuildBail, RebuildLog } from "./rebuild-credential-preflight"; @@ -232,6 +236,8 @@ export async function runRebuildBackupPhase( input: RebuildBackupPhaseInput, backupStateForRebuild: typeof backupSandboxStateForRebuild = backupSandboxStateForRebuild, ): Promise { + const transportError = nativeRestoreTransportError(); + if (transportError) return input.bail(transportError); const preparedRecoveryManifest = input.preparedRecoveryManifest; const preparedRetainedPolicy = preparedRecoveryManifest ? readRebuildPolicyHandoff(preparedRecoveryManifest) diff --git a/src/lib/actions/sandbox/rebuild-base-image-resolution-flow.test.ts b/src/lib/actions/sandbox/rebuild-base-image-resolution-flow.test.ts index f78bb8bbfe3..eda8ad1b62f 100644 --- a/src/lib/actions/sandbox/rebuild-base-image-resolution-flow.test.ts +++ b/src/lib/actions/sandbox/rebuild-base-image-resolution-flow.test.ts @@ -42,7 +42,8 @@ describe("rebuildSandbox base-image resolution flow", () => { const harness = createRebuildFlowHarness({ sandboxEntry: { agent: "hermes", - provider: "nvidia-router", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", imageTag: "nemoclaw-hermes:recorded", nemoclawVersion: "0.1.0", }, diff --git a/src/lib/actions/sandbox/rebuild-credential-preflight.ts b/src/lib/actions/sandbox/rebuild-credential-preflight.ts index 923ab6866ac..c474dc76599 100644 --- a/src/lib/actions/sandbox/rebuild-credential-preflight.ts +++ b/src/lib/actions/sandbox/rebuild-credential-preflight.ts @@ -1,6 +1,14 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + validateRebuildHostInferenceCredential, + type HostCredentialTarget, +} from "./rebuild/host-credential-preflight"; +import { + recordedRegisteredInferenceProviderAttachment, + verifyRegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; import { runOpenshell } from "../../adapters/openshell/runtime"; import { fixedNativeProviderForAttachment } from "../../inference/native-provider/fixed"; import { recordedNativeProviderAttachment } from "../../inference/native-provider/recorded-selection"; @@ -10,13 +18,14 @@ import { resolveSandboxGatewayName, type RebuildSandboxEntry } from "./rebuild-f import { rebuildOnboardDependencies } from "./rebuild-onboard-dependencies"; import { checkRebuildGatewayProviderOrBail, - validateRebuildHostInferenceCredential, shouldVerifyRebuildGatewayProvider, - type HostCredentialTarget, } from "./rebuild-provider-preflight"; import { getRebuildCredentialEnvFromRegistry } from "./rebuild-resume-config"; import { createManagedProviderAdapter } from "../../adapters/openshell/managed-provider-adapter"; -import { verifyRecordedNativeLocalProviderAttachment } from "../../inference/native-local/profile"; +import { + NativeProviderCredentialExpiredError, + verifyRecordedNativeLocalProviderAttachment, +} from "../../inference/native-local/profile"; const hermesProviderAuth = require("../../hermes-provider-auth") as { HERMES_PROVIDER_NAME: string; @@ -193,6 +202,20 @@ export async function preflightRebuildCredentials( bail: RebuildBail, options: RebuildCredentialPreflightOptions = {}, ): Promise { + if (sb.registeredInferenceProviderAttachment !== undefined) { + try { + const expected = recordedRegisteredInferenceProviderAttachment(sb); + if (!expected) throw new Error("Missing registered provider binding"); + await verifyRegisteredInferenceProviderAttachment({ + adapter: createManagedProviderAdapter(), + expected, + }); + return true; + } catch { + bail("Could not verify registered provider binding; sandbox is untouched"); + return false; + } + } const rebuildCredentialEnv = getRebuildCredentialEnvFromRegistry( sb.provider, sb.credentialEnv, @@ -209,7 +232,9 @@ export async function preflightRebuildCredentials( agent: sb.agent, provider: rebuildProvider, }); - } catch { + } catch (error) { + if (error instanceof NativeProviderCredentialExpiredError) + bail("Native provider credential is expired; sandbox is untouched"); bail("Could not verify native local provider credentials; sandbox is untouched"); return false; } diff --git a/src/lib/actions/sandbox/rebuild-dcode-preflight.test.ts b/src/lib/actions/sandbox/rebuild-dcode-preflight.test.ts index 36b06c78506..ba5030d1fdc 100644 --- a/src/lib/actions/sandbox/rebuild-dcode-preflight.test.ts +++ b/src/lib/actions/sandbox/rebuild-dcode-preflight.test.ts @@ -4,7 +4,7 @@ import { describe, expect, it, vi } from "vitest"; import { managedWorkloadRebuild, - rebuildProviderPreflight, + rebuildHostCredentialPreflight, } from "../../../../test/helpers/rebuild-flow-harness"; import { expectNoSandboxDelete } from "../../../../test/helpers/rebuild-delete-assertions"; import { @@ -27,7 +27,7 @@ describe("rebuildSandbox DCode flow: preflight", () => { async (provider) => { const harness = createRebuildFlowHarness({ agentName: "langchain-deepagents-code", - sandboxEntry: { ...makeDcodeSandboxEntry(), provider }, + sandboxEntry: { ...makeDcodeSandboxEntry(false), provider }, }); configureDcodeSession(harness); @@ -47,7 +47,7 @@ describe("rebuildSandbox DCode flow: preflight", () => { const harness = createRebuildFlowHarness({ agentName: "langchain-deepagents-code", sandboxEntry: { - ...makeDcodeSandboxEntry(), + ...makeDcodeSandboxEntry(false), provider: "nvidia-prod", credentialEnv: "NVIDIA_INFERENCE_API_KEY", nativeNvidiaProviderAttachment: { @@ -60,7 +60,9 @@ describe("rebuildSandbox DCode flow: preflight", () => { hydrateCredentialEnv: () => "invalid-rebuild-test-credential", }); configureDcodeSession(harness); - vi.mocked(rebuildProviderPreflight.validateRebuildHostInferenceCredential).mockRestore(); + vi.mocked( + rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential, + ).mockRestore(); const catalog = vi .spyOn(managedWorkloadRebuild, "prepareManagedWorkloadRebuildHandoff") .mockRejectedValue(new Error("managed image catalog is unavailable")); @@ -82,14 +84,16 @@ describe("rebuildSandbox DCode flow: preflight", () => { const harness = createRebuildFlowHarness({ agentName: "langchain-deepagents-code", sandboxEntry: { - ...makeDcodeSandboxEntry(), + ...makeDcodeSandboxEntry(false), provider: "nvidia-prod", credentialEnv: "NVIDIA_INFERENCE_API_KEY", }, hydrateCredentialEnv: () => "invalid-rebuild-test-credential", }); configureDcodeSession(harness); - vi.mocked(rebuildProviderPreflight.validateRebuildHostInferenceCredential).mockRestore(); + vi.mocked( + rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential, + ).mockRestore(); await expect( harness.rebuildSandbox("alpha", force ? ["--yes", "--force"] : ["--yes"], { @@ -108,7 +112,7 @@ describe("rebuildSandbox DCode flow: preflight", () => { hydrateCredentialEnv: () => "host-test-key", beforeBackup: () => { vi.mocked( - rebuildProviderPreflight.validateRebuildHostInferenceCredential, + rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential, ).mockResolvedValue(false); }, }); diff --git a/src/lib/actions/sandbox/rebuild-dcode-recovered-provider.test.ts b/src/lib/actions/sandbox/rebuild-dcode-recovered-provider.test.ts index ea5d3255ae6..96b6df66500 100644 --- a/src/lib/actions/sandbox/rebuild-dcode-recovered-provider.test.ts +++ b/src/lib/actions/sandbox/rebuild-dcode-recovered-provider.test.ts @@ -1,8 +1,9 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { cliOpenShellInstalledVersionObserver } from "../../../../test/helpers/rebuild-flow-harness"; import { configureDcodeSession, expectNoDcodeMutation, @@ -25,7 +26,7 @@ describe("rebuildSandbox DCode recovered provider", () => { const harness = createRebuildFlowHarness({ agentName: "langchain-deepagents-code", sandboxEntry: { - ...makeDcodeSandboxEntry(), + ...makeDcodeSandboxEntry(false), provider: "nvidia-prod", nativeNvidiaProviderAttachment, }, @@ -42,77 +43,86 @@ describe("rebuildSandbox DCode recovered provider", () => { }, ); - it("rejects incompatible keyless provider reuse after the live DCode route proof", async () => { - const restoreEnv = snapshotEnv(["COMPATIBLE_API_KEY"]); - delete process.env.COMPATIBLE_API_KEY; - - try { - const harness = createRebuildFlowHarness({ - agentName: "langchain-deepagents-code", - sandboxEntry: makeDcodeSandboxEntry(), - dcodeRouteResults: [{ ok: true }, { ok: true }], + describe("shared-provider recovery on OpenShell 0.0.116", () => { + beforeEach(() => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue({ + ok: true, + version: "0.0.116", }); - configureDcodeSession(harness); - setGatewayProviderMetadata( - harness, - "Name: compatible-endpoint\nType: anthropic\nCredential keys: COMPATIBLE_API_KEY\nConfig keys: OPENAI_BASE_URL\n", - ); + }); - await expect( - harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }), - ).rejects.toThrow("Unsafe gateway credential reuse"); + it("rejects incompatible keyless provider reuse after the live DCode route proof", async () => { + const restoreEnv = snapshotEnv(["COMPATIBLE_API_KEY"]); + delete process.env.COMPATIBLE_API_KEY; - expect(harness.preflightDcodeRouteSpy).toHaveBeenCalledTimes(2); - expect(harness.prepareManagedDcodeRebuildImageSpy).toHaveBeenCalledOnce(); - expect(harness.disposePreparedDcodeRebuildImageSpy).toHaveBeenCalledWith( - harness.preparedDcodeBuildContext, - ); - expectNoDcodeMutation(harness); - } finally { - restoreEnv(); - } - }); + try { + const harness = createRebuildFlowHarness({ + agentName: "langchain-deepagents-code", + sandboxEntry: makeDcodeSandboxEntry(false), + dcodeRouteResults: [{ ok: true }, { ok: true }], + }); + configureDcodeSession(harness); + setGatewayProviderMetadata( + harness, + "Name: compatible-endpoint\nType: anthropic\nCredential keys: COMPATIBLE_API_KEY\nConfig keys: OPENAI_BASE_URL\n", + ); - it("rejects incomplete keyless provider reuse for backup recovery before deletion", async () => { - const restoreEnv = snapshotEnv(["COMPATIBLE_API_KEY"]); - delete process.env.COMPATIBLE_API_KEY; - const recoveryManifest = { - ...makePreparedRecoveryManifest(), - agentType: "langchain-deepagents-code", - agentVersion: "0.1.12", - dir: "/sandbox/.deepagents", - }; + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }), + ).rejects.toThrow("Unsafe gateway credential reuse"); - try { - const harness = createRebuildFlowHarness({ - agentName: "langchain-deepagents-code", - sandboxEntry: makeDcodeSandboxEntry(), - sandboxInventory: { - sandboxes: [{ name: "alpha", phase: "Error", readiness: "terminal" }], - }, - preDeleteLatestManifest: recoveryManifest, - }); - configureDcodeSession(harness); - setGatewayProviderMetadata( - harness, - "Name: compatible-endpoint\nType: openai\nCredential keys: COMPATIBLE_API_KEY\nConfig keys: \n", - ); + expect(harness.preflightDcodeRouteSpy).toHaveBeenCalledTimes(2); + expect(harness.prepareManagedDcodeRebuildImageSpy).toHaveBeenCalledOnce(); + expect(harness.disposePreparedDcodeRebuildImageSpy).toHaveBeenCalledWith( + harness.preparedDcodeBuildContext, + ); + expectNoDcodeMutation(harness); + } finally { + restoreEnv(); + } + }); - await expect( - harness.rebuildSandbox("alpha", ["--yes"], { - throwOnError: true, - recoveryManifest, - }), - ).rejects.toThrow("Unsafe gateway credential reuse"); + it("rejects incomplete keyless provider reuse for backup recovery before deletion", async () => { + const restoreEnv = snapshotEnv(["COMPATIBLE_API_KEY"]); + delete process.env.COMPATIBLE_API_KEY; + const recoveryManifest = { + ...makePreparedRecoveryManifest(), + agentType: "langchain-deepagents-code", + agentVersion: "0.1.12", + dir: "/sandbox/.deepagents", + }; - expect(harness.preflightDcodeRouteSpy).not.toHaveBeenCalled(); - expect(harness.prepareManagedDcodeRebuildImageSpy).toHaveBeenCalledOnce(); - expect(harness.disposePreparedDcodeRebuildImageSpy).toHaveBeenCalledWith( - harness.preparedDcodeBuildContext, - ); - expectNoDcodeMutation(harness); - } finally { - restoreEnv(); - } + try { + const harness = createRebuildFlowHarness({ + agentName: "langchain-deepagents-code", + sandboxEntry: makeDcodeSandboxEntry(false), + sandboxInventory: { + sandboxes: [{ name: "alpha", phase: "Error", readiness: "terminal" }], + }, + preDeleteLatestManifest: recoveryManifest, + }); + configureDcodeSession(harness); + setGatewayProviderMetadata( + harness, + "Name: compatible-endpoint\nType: openai\nCredential keys: COMPATIBLE_API_KEY\nConfig keys: \n", + ); + + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { + throwOnError: true, + recoveryManifest, + }), + ).rejects.toThrow("Unsafe gateway credential reuse"); + + expect(harness.preflightDcodeRouteSpy).not.toHaveBeenCalled(); + expect(harness.prepareManagedDcodeRebuildImageSpy).toHaveBeenCalledOnce(); + expect(harness.disposePreparedDcodeRebuildImageSpy).toHaveBeenCalledWith( + harness.preparedDcodeBuildContext, + ); + expectNoDcodeMutation(harness); + } finally { + restoreEnv(); + } + }); }); }); diff --git a/src/lib/actions/sandbox/rebuild-dcode-target.test.ts b/src/lib/actions/sandbox/rebuild-dcode-target.test.ts index 7fbc950ca7f..24e0961f7c5 100644 --- a/src/lib/actions/sandbox/rebuild-dcode-target.test.ts +++ b/src/lib/actions/sandbox/rebuild-dcode-target.test.ts @@ -82,7 +82,7 @@ describe("DCode native local rebuild authority", () => { }, ); - it("ignores a stale native attachment for an owned Local NIM rebuild", () => { + it("retains the native attachment for an owned Local NIM rebuild", () => { const result = resolveDcodeRebuildTarget( { ...entry, @@ -93,9 +93,9 @@ describe("DCode native local rebuild authority", () => { 8080, ); - expect(result).not.toHaveProperty("nativeLocalProviderAttachment"); - expect(result).not.toHaveProperty("nativeProvider"); - expect(result).not.toHaveProperty("nativeEndpointUrl"); + expect(result.nativeLocalProviderAttachment).toEqual(attachment); + expect(result.nativeProvider).toBe(true); + expect(result.nativeEndpointUrl).toBe(binding.endpointUrl); expect(result.provider).toBe("vllm-local"); }); }); diff --git a/src/lib/actions/sandbox/rebuild-dcode-target.ts b/src/lib/actions/sandbox/rebuild-dcode-target.ts index b2913ea6c9e..85c439f2f8e 100644 --- a/src/lib/actions/sandbox/rebuild-dcode-target.ts +++ b/src/lib/actions/sandbox/rebuild-dcode-target.ts @@ -73,24 +73,16 @@ export function resolveDcodeRebuildTarget( `Re-run with NEMOCLAW_GATEWAY_PORT=${gatewayPort}.`, ); } - const ownedLocalNim = - resumeConfig.provider === "vllm-local" && Boolean(entry.nimContainer?.trim()); - const nativeLocalProviderAttachment = ownedLocalNim - ? null - : normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment); - if ( - !ownedLocalNim && - entry.nativeLocalProviderAttachment !== undefined && - !nativeLocalProviderAttachment - ) { + const nativeLocalProviderAttachment = normalizeNativeLocalProviderAttachment( + entry.nativeLocalProviderAttachment, + ); + if (entry.nativeLocalProviderAttachment !== undefined && !nativeLocalProviderAttachment) { throw new Error("Malformed native local provider attachment; sandbox is untouched"); } - const nativeAttachment = ownedLocalNim - ? null - : recordedNativeProviderAttachment({ - ...entry, - provider: resumeConfig.provider, - }); + const nativeAttachment = recordedNativeProviderAttachment({ + ...entry, + provider: resumeConfig.provider, + }); return { agent: DCODE_AGENT_NAME, gatewayName, diff --git a/src/lib/actions/sandbox/rebuild-destroy-phase.test.ts b/src/lib/actions/sandbox/rebuild-destroy-phase.test.ts index 94e9aeb057b..5d17a821331 100644 --- a/src/lib/actions/sandbox/rebuild-destroy-phase.test.ts +++ b/src/lib/actions/sandbox/rebuild-destroy-phase.test.ts @@ -1198,7 +1198,7 @@ describe("rebuild destroy phase", () => { }), onDeleted: vi.fn(), }), - ).rejects.toThrow("Sandbox host ports did not release after deletion"); + ).rejects.toThrow("Sandbox host port release could not be proved after deletion"); expect(mocks.restoreSandboxLaunchForwards).not.toHaveBeenCalled(); expect(recreateJournal.beginDelete).toHaveBeenCalledOnce(); diff --git a/src/lib/actions/sandbox/rebuild-destroy-phase.ts b/src/lib/actions/sandbox/rebuild-destroy-phase.ts index b51358f8b4c..cc0985b07d3 100644 --- a/src/lib/actions/sandbox/rebuild-destroy-phase.ts +++ b/src/lib/actions/sandbox/rebuild-destroy-phase.ts @@ -569,10 +569,10 @@ export async function runRebuildDestroyPhase( } if (!(await teardownSandboxDashboardForward(sandboxName))) { console.error( - " Sandbox deletion succeeded, but one or more ForwardTcp host ports did not release.", + " Sandbox deletion succeeded, but release of its ForwardTcp host ports could not be proved.", ); input.onDeleteStateAmbiguous?.(); - bail("Sandbox host ports did not release after deletion."); + bail("Sandbox host port release could not be proved after deletion."); return null; } try { diff --git a/src/lib/actions/sandbox/rebuild-flow-credential-preflight.test.ts b/src/lib/actions/sandbox/rebuild-flow-credential-preflight.test.ts index b3c83212ba0..d13eddc85e9 100644 --- a/src/lib/actions/sandbox/rebuild-flow-credential-preflight.test.ts +++ b/src/lib/actions/sandbox/rebuild-flow-credential-preflight.test.ts @@ -1,12 +1,13 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it, vi } from "vitest"; +import { beforeEach, describe, expect, it, vi } from "vitest"; import { nativeLocalIdentity } from "../../inference/native-local/contract"; import { makeMessagingPlan } from "../../../../test/helpers/messaging-plan-fixtures"; import { expectNoSandboxDelete } from "../../../../test/helpers/rebuild-delete-assertions"; import { rebuildBackupPhase, + cliOpenShellInstalledVersionObserver, listHarnessRebuildBackups, } from "../../../../test/helpers/rebuild-flow-harness"; import { @@ -64,15 +65,16 @@ function providerRuntime( return { status: 0, output, stdout: output, stderr: "" }; }; const missingProvider = { status: 1, output: "", stdout: "", stderr: "provider not found" }; - const inventoryOutput = JSON.stringify( - registeredProviders.map((provider) => ({ + const inventoryOutput = JSON.stringify({ + providers: registeredProviders.map((provider) => ({ name: provider, credential_keys: providerCredentialKeys(provider), ...(credentialExpiresAtMs[provider] ? { credential_expires_at_ms: credentialExpiresAtMs[provider] } : {}), })), - ); + next_page_token: "", + }); return (args: string[]) => { const provider = args[0] === "provider" && args[1] === "get" ? args[2] : undefined; return args[0] === "provider" && args[1] === "list" @@ -376,21 +378,43 @@ describe("rebuildSandbox flow: credential preflight", () => { ); }); - it("stops before deletion when native OpenClaw authority is detached during backup", async () => { - const { harness } = createNativeLocalRebuildHarness("openclaw", () => { - vi.mocked(nativeLocalProfile.verifyRecordedNativeLocalProviderAttachment).mockRejectedValue( - new Error("detached during backup"), - ); - }); - await expect( - harness.rebuildSandbox("alpha", ["--yes", "--verbose"], { - throwOnError: true, - }), - ).rejects.toThrow("Native local provider authority changed before sandbox deletion"); - expect(harness.backupSandboxStateSpy).toHaveBeenCalledOnce(); - expectNoSandboxDelete(harness.runOpenshellSpy); - expect(harness.onboardSpy).not.toHaveBeenCalled(); - }); + it.each(["openclaw", "hermes", "langchain-deepagents-code"] as const)( + "retains native %s before backup when its selected credential expires", + async (agent) => { + const { harness, verify } = createNativeLocalRebuildHarness(agent); + verify.mockRejectedValue(new nativeLocalProfile.NativeProviderCredentialExpiredError()); + await expect( + harness.rebuildSandbox("alpha", ["--yes", "--force"], { + throwOnError: true, + }), + ).rejects.toThrow("Native provider credential is expired; sandbox is untouched"); + expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.onboardSpy).not.toHaveBeenCalled(); + }, + ); + + it.each([ + { label: "detaches", error: new Error("detached during backup") }, + { label: "expires", error: new nativeLocalProfile.NativeProviderCredentialExpiredError() }, + ])( + "stops before deletion when native OpenClaw authority $label during backup", + async ({ error }) => { + const { harness } = createNativeLocalRebuildHarness("openclaw", () => { + vi.mocked(nativeLocalProfile.verifyRecordedNativeLocalProviderAttachment).mockRejectedValue( + error, + ); + }); + await expect( + harness.rebuildSandbox("alpha", ["--yes", "--verbose"], { + throwOnError: true, + }), + ).rejects.toThrow("Native local provider authority changed before sandbox deletion"); + expect(harness.backupSandboxStateSpy).toHaveBeenCalledOnce(); + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.onboardSpy).not.toHaveBeenCalled(); + }, + ); it("aborts before backup when the target provider and credential are missing", async () => { const harness = createRebuildFlowHarness({ @@ -604,191 +628,202 @@ describe("rebuildSandbox flow: credential preflight", () => { expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); }); - it("lets validated prepared recovery recreate a missing provider from a host key (#6114)", async () => { - const harness = createRebuildFlowHarness({ - sandboxEntry: { - provider: "compatible-endpoint", - model: MODEL, - credentialEnv: "COMPATIBLE_API_KEY", - endpointUrl: "https://inference.example.test/v1", - preferredInferenceApi: "openai-completions", - }, - hydrateCredentialEnv: () => "host-provider-key", - runOpenshell: providerRuntime([]), - sandboxInventory: { - sandboxes: [{ name: "alpha", phase: "Error", readiness: "terminal" }], - }, - }); - configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { - endpointUrl: "https://inference.example.test/v1", - preferredInferenceApi: "openai-completions", + describe("shared-provider recovery on OpenShell 0.0.116", () => { + beforeEach(() => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue({ + ok: true, + version: "0.0.116", + }); }); - await expect( - harness.rebuildSandbox("alpha", ["--yes"], { - throwOnError: true, - recoveryManifest: makePreparedRecoveryManifest(), - }), - ).resolves.toBeUndefined(); - - expect(harness.runOpenshellSpy).toHaveBeenCalledWith( - ["provider", "get", "compatible-endpoint"], - expect.objectContaining({ ignoreError: true }), - ); - expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); - expect(harness.onboardSpy).toHaveBeenCalledOnce(); - expect(harness.onboardSpy).toHaveBeenCalledWith( - expect.objectContaining({ - rebuildProviderReconfigure: { - sandboxName: "alpha", + it("lets validated prepared recovery recreate a missing provider from a host key (#6114)", async () => { + const harness = createRebuildFlowHarness({ + sandboxEntry: { provider: "compatible-endpoint", model: MODEL, credentialEnv: "COMPATIBLE_API_KEY", endpointUrl: "https://inference.example.test/v1", + preferredInferenceApi: "openai-completions", }, - }), - ); - }); - - it("aborts if the missing provider appears at the delete edge (#6114)", async () => { - const missingProvider = providerRuntime([]); - const registeredProvider = providerRuntime(["compatible-endpoint"], { - "compatible-endpoint": "COMPATIBLE_API_KEY", - }); - const providerLookups = [missingProvider, registeredProvider]; - const harness = createRebuildFlowHarness({ - sandboxEntry: { - provider: "compatible-endpoint", - model: MODEL, - credentialEnv: "COMPATIBLE_API_KEY", + hydrateCredentialEnv: () => "host-provider-key", + runOpenshell: providerRuntime([]), + sandboxInventory: { + sandboxes: [{ name: "alpha", phase: "Error", readiness: "terminal" }], + }, + }); + configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { endpointUrl: "https://inference.example.test/v1", preferredInferenceApi: "openai-completions", - }, - hydrateCredentialEnv: () => "host-provider-key", - runOpenshell: (args) => - args[0] === "provider" ? (providerLookups.shift() ?? registeredProvider)(args) : undefined, - staleRecovery: false, - }); - configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { - endpointUrl: "https://inference.example.test/v1", - preferredInferenceApi: "openai-completions", - }); + }); - await expect( - harness.rebuildSandbox("alpha", ["--yes"], { - throwOnError: true, - recoveryManifest: makePreparedRecoveryManifest(), - }), - ).rejects.toThrow("changed during rebuild preflight"); + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { + throwOnError: true, + recoveryManifest: makePreparedRecoveryManifest(), + }), + ).resolves.toBeUndefined(); - expectNoSandboxDelete(harness.runOpenshellSpy); - expect(harness.onboardSpy).not.toHaveBeenCalled(); - }); + expect(harness.runOpenshellSpy).toHaveBeenCalledWith( + ["provider", "get", "compatible-endpoint"], + expect.objectContaining({ ignoreError: true }), + ); + expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); + expect(harness.onboardSpy).toHaveBeenCalledOnce(); + expect(harness.onboardSpy).toHaveBeenCalledWith( + expect.objectContaining({ + rebuildProviderReconfigure: { + sandboxName: "alpha", + provider: "compatible-endpoint", + model: MODEL, + credentialEnv: "COMPATIBLE_API_KEY", + endpointUrl: "https://inference.example.test/v1", + }, + }), + ); + }); - it("aborts if the provider credential disappears at the delete edge (#6114)", async () => { - let credentialAvailable = true; - const harness = createRebuildFlowHarness({ - sandboxEntry: { - provider: "compatible-endpoint", - model: MODEL, - credentialEnv: "COMPATIBLE_API_KEY", + it("aborts if the missing provider appears at the delete edge (#6114)", async () => { + const missingProvider = providerRuntime([]); + const registeredProvider = providerRuntime(["compatible-endpoint"], { + "compatible-endpoint": "COMPATIBLE_API_KEY", + }); + const providerLookups = [missingProvider, registeredProvider]; + const harness = createRebuildFlowHarness({ + sandboxEntry: { + provider: "compatible-endpoint", + model: MODEL, + credentialEnv: "COMPATIBLE_API_KEY", + endpointUrl: "https://inference.example.test/v1", + preferredInferenceApi: "openai-completions", + }, + hydrateCredentialEnv: () => "host-provider-key", + runOpenshell: (args) => + args[0] === "provider" + ? (providerLookups.shift() ?? registeredProvider)(args) + : undefined, + staleRecovery: false, + }); + configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { endpointUrl: "https://inference.example.test/v1", preferredInferenceApi: "openai-completions", - }, - hydrateCredentialEnv: () => (credentialAvailable ? "host-provider-key" : null), - runOpenshell: providerRuntime([]), - staleRecovery: false, - }); - configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { - endpointUrl: "https://inference.example.test/v1", - preferredInferenceApi: "openai-completions", - }); + }); - const backupPhase = rebuildBackupPhase.runRebuildBackupPhase; - vi.spyOn(rebuildBackupPhase, "runRebuildBackupPhase").mockImplementation(async (args) => { - const result = await backupPhase(args); - credentialAvailable = false; - return result; + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { + throwOnError: true, + recoveryManifest: makePreparedRecoveryManifest(), + }), + ).rejects.toThrow("changed during rebuild preflight"); + + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.onboardSpy).not.toHaveBeenCalled(); }); - await expect( - harness.rebuildSandbox("alpha", ["--yes"], { - throwOnError: true, - recoveryManifest: makePreparedRecoveryManifest(), - }), - ).rejects.toThrow("became unavailable before sandbox deletion"); + it("aborts if the provider credential disappears at the delete edge (#6114)", async () => { + let credentialAvailable = true; + const harness = createRebuildFlowHarness({ + sandboxEntry: { + provider: "compatible-endpoint", + model: MODEL, + credentialEnv: "COMPATIBLE_API_KEY", + endpointUrl: "https://inference.example.test/v1", + preferredInferenceApi: "openai-completions", + }, + hydrateCredentialEnv: () => (credentialAvailable ? "host-provider-key" : null), + runOpenshell: providerRuntime([]), + staleRecovery: false, + }); + configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { + endpointUrl: "https://inference.example.test/v1", + preferredInferenceApi: "openai-completions", + }); - expectNoSandboxDelete(harness.runOpenshellSpy); - expect(harness.onboardSpy).not.toHaveBeenCalled(); - }); + const backupPhase = rebuildBackupPhase.runRebuildBackupPhase; + vi.spyOn(rebuildBackupPhase, "runRebuildBackupPhase").mockImplementation(async (args) => { + const result = await backupPhase(args); + credentialAvailable = false; + return result; + }); + + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { + throwOnError: true, + recoveryManifest: makePreparedRecoveryManifest(), + }), + ).rejects.toThrow("became unavailable before sandbox deletion"); - it("aborts when the delete-edge provider lookup is indeterminate (#6114)", async () => { - const missingProvider = providerRuntime([]); - const indeterminateProvider = () => ({ - status: 7, - output: "", - stdout: "", - stderr: "gateway transport unavailable", + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.onboardSpy).not.toHaveBeenCalled(); }); - const providerLookups = [missingProvider, indeterminateProvider]; - const harness = createRebuildFlowHarness({ - sandboxEntry: { - provider: "compatible-endpoint", - model: MODEL, - credentialEnv: "COMPATIBLE_API_KEY", + + it("aborts when the delete-edge provider lookup is indeterminate (#6114)", async () => { + const missingProvider = providerRuntime([]); + const indeterminateProvider = () => ({ + status: 7, + output: "", + stdout: "", + stderr: "gateway transport unavailable", + }); + const providerLookups = [missingProvider, indeterminateProvider]; + const harness = createRebuildFlowHarness({ + sandboxEntry: { + provider: "compatible-endpoint", + model: MODEL, + credentialEnv: "COMPATIBLE_API_KEY", + endpointUrl: "https://inference.example.test/v1", + preferredInferenceApi: "openai-completions", + }, + hydrateCredentialEnv: () => "host-provider-key", + runOpenshell: (args) => + args[0] === "provider" + ? (providerLookups.shift() ?? indeterminateProvider)(args) + : undefined, + staleRecovery: false, + }); + configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { endpointUrl: "https://inference.example.test/v1", preferredInferenceApi: "openai-completions", - }, - hydrateCredentialEnv: () => "host-provider-key", - runOpenshell: (args) => - args[0] === "provider" - ? (providerLookups.shift() ?? indeterminateProvider)(args) - : undefined, - staleRecovery: false, - }); - configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { - endpointUrl: "https://inference.example.test/v1", - preferredInferenceApi: "openai-completions", - }); + }); - await expect( - harness.rebuildSandbox("alpha", ["--yes"], { - throwOnError: true, - recoveryManifest: makePreparedRecoveryManifest(), - }), - ).rejects.toThrow("could not be verified before sandbox deletion"); + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { + throwOnError: true, + recoveryManifest: makePreparedRecoveryManifest(), + }), + ).rejects.toThrow("could not be verified before sandbox deletion"); - expectNoSandboxDelete(harness.runOpenshellSpy); - expect(harness.onboardSpy).not.toHaveBeenCalled(); - }); + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.onboardSpy).not.toHaveBeenCalled(); + }); - it("keeps prepared recovery fail-closed when the missing provider has no host key (#6114)", async () => { - const harness = createRebuildFlowHarness({ - sandboxEntry: { - provider: "compatible-endpoint", - model: MODEL, - credentialEnv: "COMPATIBLE_API_KEY", + it("keeps prepared recovery fail-closed when the missing provider has no host key (#6114)", async () => { + const harness = createRebuildFlowHarness({ + sandboxEntry: { + provider: "compatible-endpoint", + model: MODEL, + credentialEnv: "COMPATIBLE_API_KEY", + endpointUrl: "https://inference.example.test/v1", + preferredInferenceApi: "openai-completions", + }, + hydrateCredentialEnv: () => null, + runOpenshell: providerRuntime([]), + staleRecovery: true, + }); + configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { endpointUrl: "https://inference.example.test/v1", preferredInferenceApi: "openai-completions", - }, - hydrateCredentialEnv: () => null, - runOpenshell: providerRuntime([]), - staleRecovery: true, - }); - configureSession(harness, "compatible-endpoint", "COMPATIBLE_API_KEY", { - endpointUrl: "https://inference.example.test/v1", - preferredInferenceApi: "openai-completions", - }); + }); - await expect( - harness.rebuildSandbox("alpha", ["--yes"], { - throwOnError: true, - recoveryManifest: makePreparedRecoveryManifest(), - }), - ).rejects.toThrow("Missing gateway provider: compatible-endpoint"); + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { + throwOnError: true, + recoveryManifest: makePreparedRecoveryManifest(), + }), + ).rejects.toThrow("Missing gateway provider: compatible-endpoint"); - expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); - expect(harness.onboardSpy).not.toHaveBeenCalled(); + expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); + expect(harness.onboardSpy).not.toHaveBeenCalled(); + }); }); it("stops before backup when staged Discord would disable an active channel (#13809)", async () => { @@ -851,7 +886,11 @@ describe("rebuildSandbox flow: credential preflight", () => { it("stops before backup when the agent base-image preflight fails", async () => { const harness = createRebuildFlowHarness({ - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, baseImagePreflight: { ok: false, imageRef: null, overrideEnvVar: null }, }); diff --git a/src/lib/actions/sandbox/rebuild-flow-host-credential-target.test.ts b/src/lib/actions/sandbox/rebuild-flow-host-credential-target.test.ts index db3f915bf5b..3cad31be4e6 100644 --- a/src/lib/actions/sandbox/rebuild-flow-host-credential-target.test.ts +++ b/src/lib/actions/sandbox/rebuild-flow-host-credential-target.test.ts @@ -8,14 +8,18 @@ import { installRebuildFlowTestHooks, } from "../../../../test/helpers/rebuild-flow-generic-harness"; import { + cliOpenShellInstalledVersionObserver, rebuildCustomImagePreflight, - rebuildProviderPreflight, + rebuildHostCredentialPreflight, } from "../../../../test/helpers/rebuild-flow-harness"; import type { preflightRebuildImage } from "./rebuild-custom-image-preflight"; -import type { HostCredentialTarget } from "./rebuild-provider-preflight"; -import { nativeProviderRebuildScenario } from "./rebuild-flow-test-fixtures"; +import type { HostCredentialTarget } from "./rebuild/host-credential-preflight"; +import { + customProviderRebuildEntry, + nativeProviderRebuildScenario, +} from "./rebuild-flow-test-fixtures"; -function createRemoteRebuildHarness() { +function createRemoteRebuildHarness(native = true) { const selection = { provider: "compatible-endpoint", model: "test/model", @@ -23,22 +27,32 @@ function createRemoteRebuildHarness() { endpointUrl: "https://inference.example.test/v1", }; const harness = createRebuildFlowHarness({ - sandboxEntry: selection, - hydrateCredentialEnv: (name) => (name === selection.credentialEnv ? "host-provider-key" : null), - runOpenshell: (args) => { - const stdout = - args[1] === "list" - ? JSON.stringify([ - { name: selection.provider, credential_keys: [selection.credentialEnv] }, - ]) - : [ - `Name: ${selection.provider}`, - "Type: openai", - `Credential keys: ${selection.credentialEnv}`, - "Config keys: OPENAI_BASE_URL", - ].join("\n"); - return args[0] === "provider" ? { status: 0, stdout, output: stdout, stderr: "" } : undefined; + sandboxEntry: { + ...selection, + ...(native ? customProviderRebuildEntry(selection.endpointUrl, "openai-responses") : {}), }, + hydrateCredentialEnv: (name) => (name === selection.credentialEnv ? "host-provider-key" : null), + runOpenshell: native + ? undefined + : (args) => { + const stdout = + args[1] === "list" + ? JSON.stringify({ + providers: [ + { name: selection.provider, credential_keys: [selection.credentialEnv] }, + ], + next_page_token: "", + }) + : [ + `Name: ${selection.provider}`, + "Type: openai", + `Credential keys: ${selection.credentialEnv}`, + "Config keys: OPENAI_BASE_URL", + ].join("\n"); + return args[0] === "provider" + ? { status: 0, stdout, output: stdout, stderr: "" } + : undefined; + }, }); Object.assign(harness.session, { sandboxName: "alpha", @@ -57,7 +71,9 @@ describe("rebuildSandbox flow: host credential target", () => { ...overrides, hydrateCredentialEnv: () => "invalid-host-key", }); - const validate = vi.mocked(rebuildProviderPreflight.validateRebuildHostInferenceCredential); + const validate = vi.mocked( + rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential, + ); validate.mockResolvedValue(false); await expect( @@ -78,9 +94,15 @@ describe("rebuildSandbox flow: host credential target", () => { expect(harness.onboardSpy).not.toHaveBeenCalled(); }); - it("rebuilds a legacy row with the inference API recovered from its matching session", async () => { - const harness = createRemoteRebuildHarness(); - const validate = vi.mocked(rebuildProviderPreflight.validateRebuildHostInferenceCredential); + it("rebuilds a legacy row on OpenShell 0.0.116 with the API from its matching session", async () => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue({ + ok: true, + version: "0.0.116", + }); + const harness = createRemoteRebuildHarness(false); + const validate = vi.mocked( + rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential, + ); validate.mockImplementation( async (target: HostCredentialTarget) => target.preferredInferenceApi === "openai-responses", ); @@ -106,9 +128,13 @@ describe("rebuildSandbox flow: host credential target", () => { ...overrides, hydrateCredentialEnv: () => "nvapi-host-test-key", }); - vi.mocked(rebuildProviderPreflight.validateRebuildHostInferenceCredential).mockImplementation( + vi.mocked( + rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential, + ).mockImplementation( async ( - ...args: Parameters + ...args: Parameters< + typeof rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential + > ) => { const onIndeterminate = args[3] as (() => void) | undefined; onIndeterminate?.(); @@ -132,9 +158,9 @@ describe("rebuildSandbox flow: host credential target", () => { async (_mode, flags) => { const harness = createRemoteRebuildHarness(); let credentialAccepted = true; - vi.mocked(rebuildProviderPreflight.validateRebuildHostInferenceCredential).mockImplementation( - async () => credentialAccepted, - ); + vi.mocked( + rebuildHostCredentialPreflight.validateRebuildHostInferenceCredential, + ).mockImplementation(async () => credentialAccepted); const prepareImage = vi.mocked(rebuildCustomImagePreflight.preflightRebuildImage); const originalPrepareImage = prepareImage.getMockImplementation()!; prepareImage.mockImplementation(async (...args: Parameters) => { diff --git a/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts b/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts index 9d40d44fcfc..01467e4a59a 100644 --- a/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts +++ b/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts @@ -16,11 +16,18 @@ import { snapshotEnv, tempFiles, } from "../../../../test/helpers/rebuild-flow-generic-harness"; -import { makePreparedRecoveryManifest } from "./rebuild-flow-test-fixtures"; +import { + customProviderRebuildEntry, + makePreparedRecoveryManifest, +} from "./rebuild-flow-test-fixtures"; import { HERMES_API_PORT_ENV, resolveOnboardHermesApiPort } from "../../onboard/hermes-api-port"; import { enforceRemovedImmutabilityMigrationBoundary } from "../../state/migrations/removed-immutability"; import type { SandboxRuntimeSnapshot } from "../../state/registry/runtime-snapshot"; -import { registry } from "../../../../test/helpers/rebuild-flow-harness"; +import { + registry, + rebuildOnboardDependencies, + openClawLifecycle, +} from "../../../../test/helpers/rebuild-flow-harness"; const enforceRemovedImmutabilityMigrationBoundaryReal = enforceRemovedImmutabilityMigrationBoundary; @@ -43,6 +50,43 @@ function dockerGpuRuntimeSnapshot(device = "nvidia.com/gpu=0"): SandboxRuntimeSn describe("rebuildSandbox flow: lifecycle", () => { installRebuildFlowTestHooks(); + it.each(["verified", "denied"])( + "rebinds outer restored inference before final start: %s", + async (state) => { + const harness = createRebuildFlowHarness(); + const refresh = vi + .spyOn(rebuildOnboardDependencies, "refreshRestoredRegisteredOpenclawInferenceRoute") + .mockImplementation(async () => { + await (state === "denied" + ? Promise.reject(new Error("binding changed")) + : Promise.resolve()); + }); + const result = await harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }).then( + () => "ready", + () => "refused", + ); + expect(result).toBe(state === "verified" ? "ready" : "refused"); + expect(refresh).toHaveBeenCalledOnce(); + expect(refresh).toHaveBeenCalledWith( + expect.objectContaining({ name: "alpha", gatewayName: "nemoclaw" }), + undefined, + registry.getSandbox, + ); + expect(harness.restoreSandboxStateSpy.mock.invocationCallOrder[0]).toBeLessThan( + refresh.mock.invocationCallOrder[0]!, + ); + expect(harness.finishOpenClawMaintenanceWindowSpy).toHaveBeenCalledTimes( + state === "verified" ? 1 : 0, + ); + expect( + vi.mocked(openClawLifecycle.abortUnregisteredOpenClawPostRestoreDoctor), + ).toHaveBeenCalledTimes(state === "denied" ? 1 : 0); + expect(refresh.mock.invocationCallOrder[0]).toBeLessThan( + harness.finishOpenClawMaintenanceWindowSpy.mock.invocationCallOrder[0] ?? Infinity, + ); + }, + ); + it.each([undefined, "/srv/nemoclaw/recreated-gateway"])( "uses the recorded gateway directory during rebuild unless explicitly overridden (%s)", async (explicitStateDir) => { @@ -971,9 +1015,8 @@ describe("rebuildSandbox flow: lifecycle", () => { const harness = createRebuildFlowHarness({ applyPreset: () => true, sandboxEntry: { - provider: "compatible-endpoint", + ...customProviderRebuildEntry("https://compatible.example.test/v1"), model: "reasoning-model", - endpointUrl: "https://compatible.example.test/v1", compatibleEndpointReasoning: "true", compatibleEndpointReasoningEffort: "high", mcp: { bridges: { github: mcpEntry } }, diff --git a/src/lib/actions/sandbox/rebuild-flow-target-credentials.test.ts b/src/lib/actions/sandbox/rebuild-flow-target-credentials.test.ts index e4529408705..192819061cf 100644 --- a/src/lib/actions/sandbox/rebuild-flow-target-credentials.test.ts +++ b/src/lib/actions/sandbox/rebuild-flow-target-credentials.test.ts @@ -34,7 +34,7 @@ describe("rebuildSandbox flow: target credentials", () => { it("rejects recorded web search when the target agent does not support it", async () => { const harness = createRebuildFlowHarness({ - sandboxEntry: { agent: "hermes", provider: "nvidia-router", webSearchEnabled: true }, + sandboxEntry: { agent: "hermes", provider: "nvidia-prod", webSearchEnabled: true }, }); await expect( diff --git a/src/lib/actions/sandbox/rebuild-flow-target-image.test.ts b/src/lib/actions/sandbox/rebuild-flow-target-image.test.ts index fc3ab5ecd6a..689b6a43c35 100644 --- a/src/lib/actions/sandbox/rebuild-flow-target-image.test.ts +++ b/src/lib/actions/sandbox/rebuild-flow-target-image.test.ts @@ -146,7 +146,11 @@ describe("rebuildSandbox flow: target image", () => { it("stops before sandbox mutation when Hermes base-image preflight fails (#11072)", async () => { const harness = createRebuildFlowHarness({ - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, baseImagePreflight: { ok: false, imageRef: null, overrideEnvVar: null }, }); @@ -380,10 +384,10 @@ describe("rebuildSandbox flow: target image", () => { } }); - it("rebuilds a Hermes router with a non-matching session (#5735)", async () => { + it("preserves a legacy Router target and unrelated session before rebuild (#5735)", async () => { const harness = createRebuildFlowHarness({ applyPreset: () => true, - sandboxEntry: { agent: "hermes", provider: "nvidia-router", model: "router-model" }, + sandboxEntry: { provider: "nvidia-router", model: "router-model" }, sessionSandboxName: "some-other-sandbox", }); harness.session.routerPid = 4242; @@ -391,13 +395,11 @@ describe("rebuildSandbox flow: target image", () => { await expect( harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }), - ).resolves.toBeUndefined(); + ).rejects.toThrow("Legacy shared local inference cannot be rebuilt safely"); - expect(harness.runOpenshellSpy).toHaveBeenCalledWith( - ["sandbox", "delete", "-g", "nemoclaw", "alpha"], - expect.objectContaining({ ignoreError: true }), - ); - expect(harness.onboardSpy).toHaveBeenCalled(); + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); + expect(harness.onboardSpy).not.toHaveBeenCalled(); expect(harness.session.routerPid).toBe(4242); expect(harness.session.routerCredentialHash).toBe("router-credential-hash"); }); diff --git a/src/lib/actions/sandbox/rebuild-flow-target-session.test.ts b/src/lib/actions/sandbox/rebuild-flow-target-session.test.ts index db97ffcc478..52c9c107c0b 100644 --- a/src/lib/actions/sandbox/rebuild-flow-target-session.test.ts +++ b/src/lib/actions/sandbox/rebuild-flow-target-session.test.ts @@ -1,13 +1,14 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { expectNoSandboxDelete } from "../../../../test/helpers/rebuild-delete-assertions"; import { createRebuildFlowHarness, installRebuildFlowTestHooks, snapshotEnv, } from "../../../../test/helpers/rebuild-flow-generic-harness"; +import { cliOpenShellInstalledVersionObserver } from "../../../../test/helpers/rebuild-flow-harness"; import type { ServingProfileProvenance } from "../../inference/serving/profile-provenance"; const savedProfileProvenance: ServingProfileProvenance = { @@ -173,6 +174,107 @@ describe("rebuildSandbox flow: target session", () => { } }); + it.each([ + { provider: "ollama-local", version: "0.1.3" }, + { provider: "vllm-local", version: "0.1.3" }, + { provider: "ollama-local", version: null }, + { provider: "vllm-local", version: null }, + ])( + "refuses a retained Pi $provider rebuild before state changes with runtime version=$version", + async ({ provider, version }) => { + const harness = createRebuildFlowHarness({ + sandboxEntry: { + agent: "pi", + provider, + model: "local-model", + createdAt: "2026-09-01T00:00:00Z", + nativeLocalProviderAttachment: undefined, + }, + }); + const installedVersion = vi + .spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion") + .mockReturnValue( + version === null + ? { ok: false, error: { kind: "unavailable", message: "OpenShell binary not found." } } + : { ok: true, version }, + ); + const before = structuredClone(harness.session); + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }), + ).rejects.toThrow("Legacy shared local inference cannot be rebuilt safely"); + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.runOpenshellSpy).not.toHaveBeenCalled(); + expect(harness.registryUpdateSpy).not.toHaveBeenCalled(); + expect(harness.onboardSpy).not.toHaveBeenCalled(); + expect(harness.session).toEqual(before); + expect(installedVersion).toHaveBeenCalledOnce(); + }, + ); + + it.each([ + { provider: "compatible-endpoint", version: "0.1.3" }, + { provider: "compatible-anthropic-endpoint", version: "0.1.3" }, + { provider: "compatible-endpoint", version: null }, + ])( + "refuses retained public $provider rebuild before deletion on OpenShell $version", + async ({ provider, version }) => { + const harness = createRebuildFlowHarness({ + sandboxEntry: { + agent: "openclaw", + provider, + model: "served-model", + endpointUrl: "https://integrate.api.nvidia.com/v1", + createdAt: "2026-09-01T00:00:00Z", + nativeHostedProviderAttachment: undefined, + nativeNvidiaProviderAttachment: undefined, + }, + }); + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue( + version === null + ? { ok: false, error: { kind: "unavailable", message: "OpenShell binary not found." } } + : { ok: true, version }, + ); + const before = structuredClone(harness.session); + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }), + ).rejects.toThrow("Legacy shared custom inference cannot be rebuilt safely"); + expectNoSandboxDelete(harness.runOpenshellSpy); + expect(harness.runOpenshellSpy).not.toHaveBeenCalled(); + expect(harness.registryUpdateSpy).not.toHaveBeenCalled(); + expect(harness.onboardSpy).not.toHaveBeenCalled(); + expect(harness.session).toEqual(before); + }, + ); + + it.each(["0.1.5", "0.1.7", null])( + "refuses retained legacy NVIDIA rebuild before mutation on OpenShell %s", + async (version) => { + const harness = createRebuildFlowHarness({ + sandboxEntry: { + agent: "openclaw", + provider: "nvidia-nim", + model: "nvidia/model", + createdAt: "2026-09-01T00:00:00Z", + nativeNvidiaProviderAttachment: undefined, + }, + }); + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue( + version === null + ? { ok: false, error: { kind: "unavailable", message: "OpenShell binary not found." } } + : { ok: true, version }, + ); + const before = structuredClone(harness.session); + await expect( + harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }), + ).rejects.toThrow("Legacy shared NVIDIA inference cannot be rebuilt safely"); + expect(harness.backupSandboxStateSpy).not.toHaveBeenCalled(); + expect(harness.runOpenshellSpy).not.toHaveBeenCalled(); + expect(harness.registryUpdateSpy).not.toHaveBeenCalled(); + expect(harness.onboardSpy).not.toHaveBeenCalled(); + expect(harness.session).toEqual(before); + }, + ); + it("uses the exact preflighted agent base image only for the recreate", async () => { const overrideEnvVar = "NEMOCLAW_HERMES_SANDBOX_BASE_IMAGE_REF"; const restoreEnv = snapshotEnv([overrideEnvVar]); @@ -181,7 +283,11 @@ describe("rebuildSandbox flow: target session", () => { try { const harness = createRebuildFlowHarness({ - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, baseImagePreflight: { ok: true, imageRef: "nemoclaw-hermes-sandbox-base-local:12345678", diff --git a/src/lib/actions/sandbox/rebuild-flow-test-fixtures.ts b/src/lib/actions/sandbox/rebuild-flow-test-fixtures.ts index f3bff555303..6ba6adeb29a 100644 --- a/src/lib/actions/sandbox/rebuild-flow-test-fixtures.ts +++ b/src/lib/actions/sandbox/rebuild-flow-test-fixtures.ts @@ -5,6 +5,35 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; import { fixedNativeProvider } from "../../inference/native-provider/fixed"; +import { customNativeProvider } from "../../inference/native-provider/custom"; + +export function customProviderRebuildEntry( + endpointUrl = "https://inference-api.nvidia.com/v1", + api: "openai-completions" | "openai-responses" = "openai-completions", +) { + const definition = customNativeProvider({ + provider: "compatible-endpoint", + endpointUrl, + api, + credentialEnv: "COMPATIBLE_API_KEY", + allowedIps: ["8.8.8.8"], + }); + return { + provider: definition.logicalProvider, + endpointUrl, + credentialEnv: definition.custom.credentialEnv, + preferredInferenceApi: api, + nativeHostedProviderAttachment: { + schemaVersion: 1 as const, + providerName: definition.providerName, + profileId: definition.profileId, + providerId: "11111111-2222-4333-8444-555555555555", + endpointUrl: definition.endpointUrl, + allowedIps: definition.allowedIps, + custom: definition.custom, + }, + }; +} export function makeActiveTeamsMessagingPlan() { return { @@ -164,13 +193,16 @@ export function nativeProviderRebuildScenario(replaced: boolean, provider = "nvi "Resource version: 1", ].join("\n"); } else if (args[1] === "list") { - stdout = JSON.stringify([ - { - name: providerName, - credential_keys: [definition.credentialEnv], - credential_expires_at_ms: {}, - }, - ]); + stdout = JSON.stringify({ + providers: [ + { + name: providerName, + credential_keys: [definition.credentialEnv], + credential_expires_at_ms: {}, + }, + ], + next_page_token: "", + }); } else return undefined; return { status: 0, stdout, output: stdout, stderr: "" }; }, diff --git a/src/lib/actions/sandbox/rebuild-hermes-cron-restore.test.ts b/src/lib/actions/sandbox/rebuild-hermes-cron-restore.test.ts index 25e881ef4e9..9703256eb9f 100644 --- a/src/lib/actions/sandbox/rebuild-hermes-cron-restore.test.ts +++ b/src/lib/actions/sandbox/rebuild-hermes-cron-restore.test.ts @@ -9,6 +9,14 @@ import { validateHermesCronRestoreBackup } from "../../state/rebuild/hermes-cron const processMocks = vi.hoisted(() => ({ executePrivilegedSandboxCommand: vi.fn(), + executeProviderCommand: vi.fn(), + resolveTarget: vi.fn(), +})); + +vi.mock("../../sandbox/privileged-exec", async (importOriginal) => ({ + ...(await importOriginal()), + executePrivilegedSandboxCommand: processMocks.executeProviderCommand, + resolvePrivilegedSandboxTarget: processMocks.resolveTarget, })); vi.mock("./process-recovery", async (importOriginal) => ({ @@ -20,6 +28,7 @@ import { beginHermesCronRestore, completeHermesCronRestoreAfterGatewayReplacement, HERMES_CRON_RESTORE_DRAIN_MARKER_ROLLBACK_FAILED_CODE, + HERMES_RECOVERY_GENERATION_OBSERVER, isHermesCronRestoreDrainMarkerRollbackFailure, observeHermesCronReplacement, prepareHermesCronRestoreRecovery, @@ -149,6 +158,8 @@ describe("Hermes cron rebuild restore contract", () => { beforeEach(() => { processMocks.executePrivilegedSandboxCommand.mockReset(); + processMocks.executeProviderCommand.mockReset(); + processMocks.resolveTarget.mockReset().mockReturnValue({ resourceHandle: "a".repeat(64) }); backupPath = mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-cron-")); }); @@ -441,6 +452,59 @@ describe("Hermes cron rebuild restore contract", () => { ).toThrow("receipt failed validation"); }); + it("reconciles an empty owner-assisted publication once against the same pinned container", () => { + processMocks.executePrivilegedSandboxCommand.mockReturnValue({ + status: 1, + stdout: "", + stderr: `${CONTROL_ERROR_PREFIX}{"code":"gateway-generation-owner-read-required","message":"owner read required"}`, + }); + processMocks.executeProviderCommand + .mockReturnValueOnce({ status: 0, stdout: Buffer.from("null"), stderr: Buffer.alloc(0) }) + .mockReturnValueOnce({ status: 1, stdout: Buffer.alloc(0), stderr: Buffer.alloc(0) }) + .mockReturnValueOnce({ + status: 0, + stdout: Buffer.from( + preparationReceipt("not-required", { gateway_recovery_requested: false }), + ), + stderr: Buffer.alloc(0), + }); + expect(prepareHermesCronRestoreRecovery("alpha")).toEqual({ + disposition: "not-required", + gatewayRecoveryRequested: false, + }); + expect(processMocks.executeProviderCommand).toHaveBeenCalledTimes(3); + expect(processMocks.executeProviderCommand.mock.calls[2]).toEqual([ + "alpha", + [ + "/opt/hermes/.venv/bin/python", + "-I", + "/usr/local/lib/nemoclaw/hermes-cron-restore-control.py", + "prepare-recover", + ], + { + expectedResourceHandle: "a".repeat(64), + sanitizeEnvironment: true, + timeout: 25_000, + maxOutputBytes: 2048, + }, + ]); + }); + + it("refuses a repeated empty owner-assisted publication without retrying the owner read", () => { + processMocks.executePrivilegedSandboxCommand.mockReturnValue({ + status: 1, + stdout: "", + stderr: `${CONTROL_ERROR_PREFIX}{"code":"gateway-generation-owner-read-required","message":"owner read required"}`, + }); + processMocks.executeProviderCommand + .mockReturnValueOnce({ status: 0, stdout: Buffer.from("null"), stderr: Buffer.alloc(0) }) + .mockReturnValue({ status: 1, stdout: Buffer.alloc(0), stderr: Buffer.alloc(0) }); + expect(() => prepareHermesCronRestoreRecovery("alpha")).toThrow( + "Hermes cron prepare-recover failed", + ); + expect(processMocks.executeProviderCommand).toHaveBeenCalledTimes(3); + }); + it.each([ ["missing", undefined], ["negative", -1], @@ -599,6 +663,81 @@ describe("Hermes cron rebuild restore contract", () => { ); }); + it("pins the owner observation and root publication to the same container without argv secrets", () => { + processMocks.executePrivilegedSandboxCommand.mockReturnValue({ + status: 1, + stdout: "", + stderr: `${CONTROL_ERROR_PREFIX}{"code":"gateway-generation-owner-read-required","message":"owner read required"}`, + }); + const observation = Buffer.from('{"version":1,"generation":"opaque","identity":[]}'); + processMocks.executeProviderCommand + .mockReturnValueOnce({ status: 0, stdout: observation, stderr: Buffer.alloc(0) }) + .mockReturnValueOnce({ + status: 0, + stdout: Buffer.from( + preparationReceipt("not-required", { gateway_recovery_requested: true }), + ), + stderr: Buffer.alloc(0), + }); + expect(prepareHermesCronRestoreRecovery("alpha")).toEqual({ + disposition: "not-required", + gatewayRecoveryRequested: true, + }); + expect(processMocks.executeProviderCommand.mock.calls).toEqual([ + [ + "alpha", + ["/opt/hermes/.venv/bin/python", "-I", "-c", HERMES_RECOVERY_GENERATION_OBSERVER], + { + expectedResourceHandle: "a".repeat(64), + sanitizeEnvironment: true, + timeout: 25_000, + maxOutputBytes: 2048, + sandboxUser: "998:999", + }, + ], + [ + "alpha", + [ + "/opt/hermes/.venv/bin/python", + "-I", + "/usr/local/lib/nemoclaw/hermes-cron-restore-control.py", + "prepare-recover", + "--generation-observation-stdin", + ], + { + expectedResourceHandle: "a".repeat(64), + sanitizeEnvironment: true, + timeout: 25_000, + maxOutputBytes: 2048, + input: observation, + }, + ], + ]); + }); + + it.each([ + { + status: 2, + stdout: Buffer.alloc(0), + stderr: Buffer.from("owner observer failed"), + }, + { status: 0, stdout: Buffer.alloc(0), stderr: Buffer.alloc(0) }, + { status: 0, stdout: Buffer.alloc(2049), stderr: Buffer.alloc(0) }, + { status: 0, stdout: Buffer.from("null"), stderr: Buffer.from("untrusted diagnostic") }, + { status: 0, stdout: Buffer.from("null"), stderr: Buffer.alloc(0), signal: "SIGTERM" }, + ])("refuses an unsuccessful owner observation before root publication (%j)", (result) => { + processMocks.executePrivilegedSandboxCommand.mockReturnValue({ + status: 1, + stdout: "", + stderr: `${CONTROL_ERROR_PREFIX}{"code":"gateway-generation-owner-read-required","message":"owner read required"}`, + }); + processMocks.executeProviderCommand.mockReturnValue(result); + expect(() => prepareHermesCronRestoreRecovery("alpha")).toThrow( + "could not be observed by its owner", + ); + expect(processMocks.executeProviderCommand).toHaveBeenCalledTimes(1); + }); + it("reconciles one empty prepare-recover transport failure through its idempotent receipt", () => { processMocks.executePrivilegedSandboxCommand .mockReturnValueOnce({ status: 1, stdout: "", stderr: "" }) diff --git a/src/lib/actions/sandbox/rebuild-hermes-inference-restore.test.ts b/src/lib/actions/sandbox/rebuild-hermes-inference-restore.test.ts new file mode 100644 index 00000000000..c4d55037783 --- /dev/null +++ b/src/lib/actions/sandbox/rebuild-hermes-inference-restore.test.ts @@ -0,0 +1,102 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 +import { beforeEach, it, expect, vi } from "vitest"; +import type * as registry from "../../state/registry"; +import * as sandboxConfig from "../../sandbox/config"; +import * as openshellRuntime from "../../adapters/openshell/runtime"; +import { fingerprintOpenShellSandboxId } from "../../adapters/openshell/sandbox-identity"; +import { restoreExternalHermesInference } from "./rebuild-hermes-post-restore"; +vi.mock("../../sandbox/config", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + resolveAgentConfig: vi.fn(actual.resolveAgentConfig), + readSandboxConfig: vi.fn(actual.readSandboxConfig), + writeSandboxConfig: vi.fn(actual.writeSandboxConfig), + }; +}); + +beforeEach(() => { + vi.mocked(sandboxConfig.resolveAgentConfig).mockReset(); + vi.mocked(sandboxConfig.readSandboxConfig).mockReset(); + vi.mocked(sandboxConfig.writeSandboxConfig).mockReset(); +}); +function restoredExternalHermes() { + const entry = { + name: "alpha", + agent: "hermes", + gatewayName: "nemoclaw-19080", + lifecycleGeneration: "replacement-generation", + lifecycleLiveIdentityFingerprint: fingerprintOpenShellSandboxId("replacement-id"), + provider: "compatible-endpoint", + model: "selected-model", + endpointUrl: "https://models.example.test/v1", + workload: { kind: "external-image" }, + nativeHostedProviderAttachment: { profileId: "native-custom" }, + } as registry.SandboxEntry; + const read = vi.fn(() => entry); + const capture = vi.spyOn(openshellRuntime, "captureOpenshell").mockReturnValue({ + status: 0, + output: "Id: replacement-id\n", + }); + const config = { + model: { api_key: "[STRIPPED_BY_MIGRATION]" }, + user_setting: "preserved", + }; + vi.mocked(sandboxConfig.resolveAgentConfig).mockReturnValue({ agentName: "hermes" } as never); + vi.mocked(sandboxConfig.readSandboxConfig).mockReturnValue(config); + const write = vi.mocked(sandboxConfig.writeSandboxConfig).mockImplementation(() => undefined); + const run = () => + restoreExternalHermesInference( + entry, + { gatewayName: "nemoclaw-19080", workspace: "default" }, + read, + ); + return { entry, config, capture, write, run }; +} + +it("reapplies registered Hermes inference after archive sanitation (#12603)", async () => { + const f = restoredExternalHermes(); + await f.run(); + expect(f.config).toMatchObject({ + model: { + default: "selected-model", + api_key: "${COMPATIBLE_API_KEY}", + base_url: "https://models.example.test/v1", + }, + user_setting: "preserved", + }); + expect(f.write).toHaveBeenCalledWith( + "alpha", + expect.objectContaining({ agentName: "hermes" }), + f.config, + "nemoclaw-19080", + ); + expect(f.capture).toHaveBeenCalledWith( + ["sandbox", "get", "-g", "nemoclaw-19080", "alpha"], + expect.objectContaining({ + env: expect.objectContaining({ OPENSHELL_GATEWAY: "nemoclaw-19080" }), + }), + ); +}); + +it("rejects a replaced Hermes sandbox before writing restored inference (#12603)", async () => { + const f = restoredExternalHermes(); + f.capture + .mockReturnValueOnce({ status: 0, output: "Id: replacement-id\n" }) + .mockReturnValue({ status: 0, output: "Id: unrelated-id\n" }); + await expect(f.run()).rejects.toThrow("live identity"); + expect(f.write).not.toHaveBeenCalled(); +}); + +it("rejects a changed Hermes inference selection before configuration write (#12603)", async () => { + const f = restoredExternalHermes(); + f.capture + .mockImplementationOnce(() => ({ status: 0, output: "Id: replacement-id\n" })) + .mockImplementation(() => { + f.entry.model = "unrelated-model"; + return { status: 0, output: "Id: replacement-id\n" }; + }); + await expect(f.run()).rejects.toThrow("inference selection changed"); + expect(f.write).not.toHaveBeenCalled(); +}); diff --git a/src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts b/src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts index f2deb32c788..46d4ab7ac47 100644 --- a/src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts +++ b/src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; import { afterEach, describe, expect, it, vi } from "vitest"; import * as projection from "../../onboard/hermes/initial-inference-route"; import { patchHermesInferenceConfig } from "../inference-set"; @@ -218,7 +219,11 @@ describe("Hermes rebuild post-restore verification", () => { detachedProviderEntries: [mcpEntry], scrubbedAdapterEntries: [mcpEntry], }, - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, }); await expect( @@ -251,7 +256,11 @@ describe("Hermes rebuild post-restore verification", () => { detachedProviderEntries: [mcpEntry], scrubbedAdapterEntries: [mcpEntry], }, - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, }); harness.restoreMcpBridgesAfterRebuildSpy.mockRejectedValueOnce(new Error("reload failed")); @@ -268,7 +277,11 @@ describe("Hermes rebuild post-restore verification", () => { const harness = createRebuildFlowHarness({ agentName: "hermes", sessionAgentName: null, - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, }); await expect( @@ -287,7 +300,11 @@ describe("Hermes rebuild post-restore verification", () => { it("rebinds and verifies ordinary Hermes after restoring its durable state", async () => { const harness = createRebuildFlowHarness({ agentName: "hermes", - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, }); await expect( @@ -306,7 +323,11 @@ describe("Hermes rebuild post-restore verification", () => { const harness = createRebuildFlowHarness({ agentName: "hermes", sessionAgentName: "langchain-deepagents-code", - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, }); await expect( @@ -368,6 +389,9 @@ describe("restored Hermes NVIDIA credential handoff", () => { const io = { withSandboxMutationLock: async (_name: string, action: () => Promise) => action(), readNvidiaCredentialHandle: vi.fn(async () => handle), + readRegisteredCredentialHandle: vi.fn(async () => { + throw new Error("NVIDIA restore must not read registered-provider credentials"); + }), readCredentialHandle: vi.fn(async () => { throw new Error("NVIDIA restore must not read local-provider credentials"); }), @@ -418,6 +442,8 @@ describe("restored Hermes NVIDIA credential handoff", () => { expect(f.io.readCredentialHandle).not.toHaveBeenCalled(); expect(f.io.writeSandboxConfig).toHaveBeenCalledExactlyOnceWith("alpha", f.target, expected, { restoreNativeLocalCredential: true, + gatewayName: f.runtime.gatewayName, + deferReload: true, }); expect(f.io.writeSandboxConfig.mock.calls[0]![2]).toBe(f.config); expect(f.io.restartGateway).not.toHaveBeenCalled(); @@ -455,6 +481,8 @@ describe("restored Hermes NVIDIA credential handoff", () => { expect(f.io.readCredentialHandle).not.toHaveBeenCalled(); expect(f.io.writeSandboxConfig).toHaveBeenCalledExactlyOnceWith("alpha", f.target, expected, { restoreNativeLocalCredential: true, + gatewayName: f.runtime.gatewayName, + deferReload: true, }); expect(f.io.restartGateway).not.toHaveBeenCalled(); }, @@ -678,40 +706,51 @@ describe("restored Hermes route identity", () => { afterEach(() => vi.restoreAllMocks()); - it("pins the replacement live identity and selected runtime before deferred projection", async () => { - const f = fixture(); - await refreshRestoredHermesInferenceRoute( - "alpha", - runtime, - registry.getSandbox, - commandEnvironment, - ); - expect(f.factory).toHaveBeenCalledWith( - expect.objectContaining({ getSandbox: expect.any(Function) }), - { activation: "deferred", workload: "registered", projection: "credential" }, - ); - expect(f.project).toHaveBeenCalledWith( - "alpha", - "selected-model", - "compatible-endpoint", - runtime.gatewayName, - expect.any(Function), - ); - expect(f.observe).toHaveBeenCalledWith( - ["sandbox", "get", "-g", runtime.gatewayName, "alpha"], - expect.objectContaining({ - env: expect.objectContaining({ - OPENSHELL_GATEWAY: runtime.gatewayName, - OPENSHELL_WORKSPACE: runtime.workspace, - OPENSHELL_LOCAL_TLS_DIR: runtime.localTlsDir, + it.each(["compatible-endpoint", "native-extra"])( + "pins %s replacement identity and selected runtime before deferred projection", + async (provider) => { + const f = fixture(); + f.entry.provider = provider; + f.entry.registeredInferenceProviderAttachment = + provider === "native-extra" + ? { + ...registeredInferenceFixture(true).binding, + gatewayName: runtime.gatewayName, + } + : undefined; + await refreshRestoredHermesInferenceRoute( + "alpha", + runtime, + registry.getSandbox, + commandEnvironment, + ); + expect(f.factory).toHaveBeenCalledWith( + expect.objectContaining({ getSandbox: expect.any(Function) }), + { activation: "deferred", workload: "registered", projection: "credential" }, + ); + expect(f.project).toHaveBeenCalledWith( + "alpha", + "selected-model", + provider, + runtime.gatewayName, + expect.any(Function), + ); + expect(f.observe).toHaveBeenCalledWith( + ["sandbox", "get", "-g", runtime.gatewayName, "alpha"], + expect.objectContaining({ + env: expect.objectContaining({ + OPENSHELL_GATEWAY: runtime.gatewayName, + OPENSHELL_WORKSPACE: runtime.workspace, + OPENSHELL_LOCAL_TLS_DIR: runtime.localTlsDir, + }), + replaceEnv: true, + timeout: 30000, }), - replaceEnv: true, - timeout: 30000, - }), - ); - expect(f.observe.mock.calls[0][1]?.env).not.toHaveProperty("ABSENT"); - expect(f.write).toHaveBeenCalledOnce(); - }); + ); + expect(f.observe.mock.calls[0][1]?.env).not.toHaveProperty("ABSENT"); + expect(f.write).toHaveBeenCalledOnce(); + }, + ); it.each([ { diff --git a/src/lib/actions/sandbox/rebuild-hermes-post-restore.ts b/src/lib/actions/sandbox/rebuild-hermes-post-restore.ts index b118df837a5..f9077d9095a 100644 --- a/src/lib/actions/sandbox/rebuild-hermes-post-restore.ts +++ b/src/lib/actions/sandbox/rebuild-hermes-post-restore.ts @@ -1,6 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { isDeepStrictEqual } from "node:util"; +import type { SandboxEntry } from "../../state/registry/types"; +import { observeSandboxOnGateway } from "../../onboard/sandbox-recreate-probe"; +import { createRegisteredSandboxIdentityRevalidation } from "../../onboard/sandbox-recreate-transaction"; +import { initializeExternalHermesInferenceRoute } from "../../onboard/machine/handlers/initial-hermes-inference-route"; import { CLI_NAME } from "../../cli/branding"; import type { OpenShellRuntimeSelection } from "../../adapters/openshell/runtime"; import { fingerprintOpenShellSandboxLiveIdentity } from "../../adapters/openshell/sandbox-identity"; @@ -9,8 +14,12 @@ import { createInitialHermesInferenceRoute, type InitialHermesInferenceRouteDeps, } from "../../onboard/hermes/initial-inference-route"; -import { createRegisteredSandboxIdentityRevalidation } from "../../onboard/sandbox-recreate-transaction"; -import { isDirectSandboxFallbackUnavailableError } from "../../sandbox/privileged-exec"; +import { + isDirectSandboxFallbackUnavailableError, + executePrivilegedSandboxCommand as executeProviderCommand, + resolvePrivilegedSandboxTarget, +} from "../../sandbox/privileged-exec"; +import { managedImageRuntimeIdentity } from "../../onboard/managed-image/agents"; import type { GatewayRestartResult } from "./gateway-restart"; import { checkAndRecoverSandboxProcesses, @@ -21,6 +30,58 @@ import { const HERMES_CRON_CONTROL = "/usr/local/lib/nemoclaw/hermes-cron-restore-control.py"; const HERMES_PYTHON = "/opt/hermes/.venv/bin/python"; +// The owner cannot read the root-only cron controller. This fixed observer +// reads only the supervisor generation; the controller still owns publication. +export const HERMES_RECOVERY_GENERATION_OBSERVER = String.raw` +import json, os, stat, time +from pathlib import Path +home = Path("/sandbox/.hermes") +generation_path = Path("/tmp/nemoclaw-hermes-gateway-recovery-waiting") +owner = home.lstat().st_uid +if os.geteuid() != owner: + raise RuntimeError("Hermes recovery observation requires the home owner") +deadline = time.monotonic() + 2.0 +observation = None +while True: + try: + descriptor = os.open(generation_path, os.O_RDONLY | os.O_CLOEXEC | os.O_NONBLOCK | os.O_NOFOLLOW) + except FileNotFoundError: + if time.monotonic() >= deadline: + break + time.sleep(0.1) + continue + try: + metadata = os.fstat(descriptor) + if ( + not stat.S_ISREG(metadata.st_mode) + or metadata.st_uid != owner + or stat.S_IMODE(metadata.st_mode) != 0o600 + or metadata.st_nlink != 1 + or metadata.st_size != 68 + ): + raise RuntimeError("Hermes recovery generation metadata is unsafe") + raw = os.read(descriptor, 69) + finally: + os.close(descriptor) + payload = raw.decode("ascii") + if ( + len(payload) != 68 + or not payload.startswith("v1 ") + or not payload.endswith("\n") + or any(c not in "0123456789abcdef" for c in payload[3:-1]) + ): + raise RuntimeError("Hermes recovery generation is invalid") + observation = { + "version": 1, + "generation": payload[3:-1], + "identity": [ + metadata.st_dev, metadata.st_ino, metadata.st_uid, metadata.st_mode, + metadata.st_nlink, metadata.st_size, metadata.st_mtime_ns, metadata.st_ctime_ns, + ], + } + break +print(json.dumps(observation, separators=(",", ":"))) +`; const RECEIPT_PREFIX = "NEMOCLAW_HERMES_CRON_RESTORE_V1:"; const CONTROL_ERROR_PREFIX = "NEMOCLAW_HERMES_CRON_RESTORE_ERROR_V1:"; export const HERMES_CRON_RESTORE_DRAIN_MARKER_ROLLBACK_FAILED_CODE = "drain-marker-rollback-failed"; @@ -697,6 +758,65 @@ function isAmbiguousPrepareRecoveryTransportFailure(error: unknown): boolean { ); } +function prepareHermesRecoveryWithOwnerObservation(sandboxName: string): string { + const expectedResourceHandle = resolvePrivilegedSandboxTarget(sandboxName).resourceHandle; + const owner = managedImageRuntimeIdentity("hermes"); + const command = [HERMES_PYTHON, "-I", HERMES_CRON_CONTROL]; + const options = { + expectedResourceHandle, + sanitizeEnvironment: true, + timeout: CONTROL_TIMEOUT_MS, + maxOutputBytes: 2048, + } as const; + // Container root has no DAC override in the native workload. Keep the read + // with the home owner and the drain/request authority with the root controller. + const observed = executeProviderCommand( + sandboxName, + [HERMES_PYTHON, "-I", "-c", HERMES_RECOVERY_GENERATION_OBSERVER], + { ...options, sandboxUser: `${owner.uid}:${owner.gid}` }, + ); + if ( + observed.status !== 0 || + observed.error || + observed.signal || + observed.stdout.length === 0 || + observed.stdout.length > 2048 || + observed.stderr.length > 0 + ) { + throw new Error("Hermes gateway recovery generation could not be observed by its owner"); + } + // The opaque observation stays off argv. The controller rechecks the inode + // metadata; the supervisor still requires its current nonce and requester PID. + let prepared = executeProviderCommand( + sandboxName, + [...command, "prepare-recover", "--generation-observation-stdin"], + { ...options, input: observed.stdout }, + ); + if ( + prepared.status === 1 && + !prepared.error && + !prepared.signal && + prepared.stdout.length === 0 && + prepared.stderr.length === 0 + ) { + // Publication can finish before Docker delivers its receipt. Reconcile once + // through the idempotent root action, without replaying an old observation. + prepared = executeProviderCommand(sandboxName, [...command, "prepare-recover"], options); + } + if (prepared.error || prepared.signal) { + throw new Error("Hermes cron prepare-recover transport was unavailable"); + } + if (prepared.status !== 0) { + throw new HermesCronRestoreControlFailure( + "prepare-recover", + prepared.status ?? 1, + prepared.stdout.toString("utf8"), + prepared.stderr.toString("utf8"), + ); + } + return prepared.stdout.toString("utf8"); +} + export function prepareHermesCronRestoreRecovery( sandboxName: string, ): HermesCronRestorePreparationOutcome { @@ -705,6 +825,14 @@ export function prepareHermesCronRestoreRecovery( stdout = executeCronRestoreControl(sandboxName, "prepare-recover"); } catch (error) { if (isLegacyCronRestoreControl(error, "prepare-recover")) return "unsupported"; + if ( + error instanceof HermesCronRestoreControlFailure && + error.controlCode === "gateway-generation-owner-read-required" + ) { + return parseCronRestorePreparationReceipt( + prepareHermesRecoveryWithOwnerObservation(sandboxName), + ); + } if (!isAmbiguousPrepareRecoveryTransportFailure(error)) throw error; // The controller publishes its recovery request before it exits. Under @@ -750,6 +878,39 @@ export async function runHermesCronRestoreTransaction SandboxEntry | null, +): Promise { + const revalidate = createRegisteredSandboxIdentityRevalidation(entry, { + readRegistration: readRegistration, + observe: (sandboxName, gatewayName) => + observeSandboxOnGateway({ sandboxName, gatewayName }, undefined, runtimeSelection, 15_000), + }); + if (!revalidate || !entry.gatewayName || !entry.provider || !entry.model) { + throw new Error( + "Restored Hermes inference requires its registered sandbox and route identity.", + ); + } + const registered = structuredClone(entry); + await initializeExternalHermesInferenceRoute({ + sandboxName: entry.name, + gatewayName: entry.gatewayName, + getSandbox: readRegistration, + model: entry.model, + provider: entry.provider, + preferredInferenceApi: entry.preferredInferenceApi ?? null, + revalidateSandboxIdentity: (operation) => { + revalidate(operation); + const current = readRegistration(entry.name); + if (!isDeepStrictEqual(current, registered)) { + throw new Error("Restored Hermes inference selection changed before configuration write."); + } + }, + }); +} + /** Rebind restored native references; the post-restore phase owns the restart. */ export async function refreshRestoredHermesInferenceRoute( sandboxName: string, diff --git a/src/lib/actions/sandbox/rebuild-host-credential-preflight.test.ts b/src/lib/actions/sandbox/rebuild-host-credential-preflight.test.ts index 0221884976d..213b345bd6e 100644 --- a/src/lib/actions/sandbox/rebuild-host-credential-preflight.test.ts +++ b/src/lib/actions/sandbox/rebuild-host-credential-preflight.test.ts @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; -import { validateRebuildHostInferenceCredential } from "./rebuild-provider-preflight"; +import { validateRebuildHostInferenceCredential } from "./rebuild/host-credential-preflight"; const target = { provider: "compatible-endpoint", diff --git a/src/lib/actions/sandbox/rebuild-local-provider-recreate.test.ts b/src/lib/actions/sandbox/rebuild-local-provider-recreate.test.ts index f6f82524437..e5f4c8a87bb 100644 --- a/src/lib/actions/sandbox/rebuild-local-provider-recreate.test.ts +++ b/src/lib/actions/sandbox/rebuild-local-provider-recreate.test.ts @@ -16,6 +16,9 @@ import { import { createLocalInferenceRouteApplier } from "../../onboard/local-inference-route"; const requireDist = createRequire(import.meta.url); +const { cliOpenShellInstalledVersionObserver } = requireDist( + "../../adapters/openshell/installed-version-cli.js", +) as typeof import("../../adapters/openshell/installed-version-cli"); const openshellRuntime = requireDist("../../adapters/openshell/runtime.js") as { captureOpenshell( args: string[], @@ -144,14 +147,20 @@ installRebuildFlowTestHooks({ acceptThirdPartySoftware: true }); describe("rebuild local-provider recreation", () => { it.each(localProviderScenarios)( - "recreates a missing $provider gateway provider through the resumed local setup path", + "recreates a missing $provider gateway provider through Pi shared setup on OpenShell 0.0.116", async ({ provider, model, baseUrl, credentialEnv, setup }) => { + const installedVersion = vi + .spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion") + .mockReturnValue({ + ok: true, + version: "0.0.116", + }); let sourceDeleted = false; let harness!: RebuildFlowHarness; let setupResult: SetupResult | undefined; const routeApplier = makeRouteApplier(); harness = createRebuildFlowHarness({ - // Pi retains the shared-local route exercised here; OpenClaw and Hermes require native attachments. + // Retained Pi shared routes remain rebuildable on the legacy runtime. sandboxEntry: { agent: "pi", provider, @@ -203,6 +212,7 @@ describe("rebuild local-provider recreation", () => { harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }), ).resolves.toBeUndefined(); + expect(installedVersion).toHaveBeenCalledOnce(); const calls = harness.runOpenshellSpy.mock.calls.map((call) => call[0] as string[]); const deleteCall = calls.findIndex( (args) => args.join(" ") === "sandbox delete -g nemoclaw alpha", diff --git a/src/lib/actions/sandbox/rebuild-managed-workload-mutation-guard.test.ts b/src/lib/actions/sandbox/rebuild-managed-workload-mutation-guard.test.ts index 0a4f04e19c4..f7584566c66 100644 --- a/src/lib/actions/sandbox/rebuild-managed-workload-mutation-guard.test.ts +++ b/src/lib/actions/sandbox/rebuild-managed-workload-mutation-guard.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; import { managedStartupE2eProfile } from "../../../../scripts/checks/generate-managed-startup-profile-fixture.mts"; import { mapManagedStartupProfileToAgentEnvironment } from "../../onboard/managed-startup/agent-environment"; @@ -374,3 +375,112 @@ describe("managed workload rebuild mutation guard", () => { }); }); }); + +it.each(["configured", "defaults", "wrong-endpoint"])( + "captures registered model settings before deletion: %s", + (scenario) => { + const binding = registeredInferenceFixture(true).binding; + const model = + scenario === "configured" + ? { + id: "selected", + contextWindow: 131072, + maxTokens: 8192, + input: ["text", "image"], + reasoning: true, + compat: { supportsStore: false }, + } + : { id: "selected" }; + const previousProfile = managedStartupE2eProfile("openclaw"); + const catalog = { + agent: "openclaw", + previousProfile, + previousReceipt: { credentialProxyReplayRequired: false }, + corporateCa: null, + } as unknown as managedWorkload.ManagedWorkloadRebuildCatalogHandoff; + const targetConfig = { + agentDefinition: null, + resumeConfig: { + provider: binding.providerName, + model: "selected", + preferredInferenceApi: binding.inferenceApi, + registeredInferenceProviderAttachment: binding, + }, + durableConfig: { webSearchConfig: null }, + hermesToolGateways: [], + } as unknown as RebuildTargetConfig; + const read = vi.spyOn(managedRebuildProfileDependencies, "readSandboxConfig").mockReturnValue({ + models: { + providers: { + [binding.providerName]: { + baseUrl: + scenario === "wrong-endpoint" ? "https://other.example/v1" : binding.endpointUrl, + api: binding.inferenceApi, + models: [model], + }, + }, + }, + }); + vi.spyOn(managedRebuildProfileDependencies, "resolveAgentConfig").mockReturnValue({ + agentName: "openclaw", + } as never); + const stage = vi + .spyOn(managedWorkload, "stageManagedWorkloadRebuildProfile") + .mockReturnValue(handoff); + const result = () => + prepareManagedRebuildProfileHandoff({ + catalogHandoff: catalog, + targetConfig, + gatewayPort: 18080, + recreateOptions: { toolDisclosure: "progressive" } as RebuildRecreateOnboardOpts, + messagingPlan: null, + environment: {}, + }); + const outcome = (() => { + try { + result(); + return "prepared"; + } catch { + return "refused"; + } + })(); + expect(outcome).toBe(scenario === "wrong-endpoint" ? "refused" : "prepared"); + expect(read).toHaveBeenCalledWith( + binding.sandboxName, + { agentName: "openclaw" }, + binding.gatewayName, + ); + expect(stage.mock.calls).toEqual( + scenario === "wrong-endpoint" + ? [] + : [ + [ + catalog, + expect.objectContaining({ + inference: expect.objectContaining({ + routeProvider: binding.providerName, + routedBaseUrl: binding.endpointUrl, + primaryModelRef: `${binding.providerName}/selected`, + }), + }), + {}, + expect.objectContaining({ openClawSelectedModel: model }), + ], + ], + ); + const environment = managedWorkload.managedWorkloadRebuildProfileEnvironment( + catalog, + {}, + { openClawSelectedModel: model }, + ); + expect(environment.NEMOCLAW_CONTEXT_WINDOW).toBe( + scenario === "configured" ? "131072" : undefined, + ); + expect(environment.NEMOCLAW_MAX_TOKENS).toBe(scenario === "configured" ? "8192" : undefined); + expect(environment.NEMOCLAW_INFERENCE_INPUTS).toBe( + scenario === "configured" ? "text,image" : "text", + ); + expect(environment.NEMOCLAW_REASONING).toBe(scenario === "configured" ? "true" : "false"); + expect(environment.NEMOCLAW_SERVING_PRESET).toBeUndefined(); + }, +); diff --git a/src/lib/actions/sandbox/rebuild-onboard-dependencies.ts b/src/lib/actions/sandbox/rebuild-onboard-dependencies.ts index ec4ff8fc9e9..87cc004422a 100644 --- a/src/lib/actions/sandbox/rebuild-onboard-dependencies.ts +++ b/src/lib/actions/sandbox/rebuild-onboard-dependencies.ts @@ -10,6 +10,7 @@ import type { NativeLocalProviderAttachment } from "../../inference/native-local type RebuildAuthoritativePreflightOptions = RebuildRecreateOnboardOpts & { deferInferenceRouteUntilOnboard?: true; agent?: string | null; + registeredInferenceProviderAttachment?: import("../../inference/config").RegisteredInferenceProviderAttachment; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; model: string; provider: string; @@ -70,6 +71,15 @@ function loadOnboardModule(): RebuildOnboardModule { * the onboarding APIs are side-effect-free named imports. */ export const rebuildOnboardDependencies = { + refreshRestoredRegisteredOpenclawInferenceRoute( + ...args: Parameters< + typeof import("../../onboard/openclaw/initial-inference-route").refreshRestoredRegisteredOpenclawInferenceRoute + > + ): Promise { + return ( + require("../../onboard/openclaw/initial-inference-route") as typeof import("../../onboard/openclaw/initial-inference-route") + ).refreshRestoredRegisteredOpenclawInferenceRoute(...args); + }, verifyRebuiltOpenClawCompatibleEndpoint( options: RebuildCompatibleEndpointSmokeOptions, ): Promise { diff --git a/src/lib/actions/sandbox/rebuild-pipeline.ts b/src/lib/actions/sandbox/rebuild-pipeline.ts index dac1c0b2a64..33ce52dbc05 100644 --- a/src/lib/actions/sandbox/rebuild-pipeline.ts +++ b/src/lib/actions/sandbox/rebuild-pipeline.ts @@ -68,6 +68,7 @@ import { assertSandboxRebuildCommandAvailable, revalidateManagedWorkloadRebuildBeforeDelete, revalidateRebuildRouteBeforeDelete, + revalidateRebuildHostCredentialBeforeDelete, } from "./rebuild-preflight-guards"; import { runHermesCronRestoreBackupPreflight, @@ -83,8 +84,8 @@ import { } from "./rebuild-prepared-recovery"; import { inspectRebuildGatewayProviderRegistration, - validateRebuildHostInferenceCredential, verifyRebuildNativeLocalProviderBeforeDelete, + verifyRebuildRegisteredProviderBeforeDelete, shouldVerifyRebuildGatewayProvider, } from "./rebuild-provider-preflight"; import { @@ -913,34 +914,22 @@ async function rebuildSandboxUnlocked( }, validateAtDeleteEdge: async (runtimeSelection) => { stoppedSource?.assertCurrent(); - const hostCredential = resumeConfig.credentialEnv - ? hydrateCredentialEnv(resumeConfig.credentialEnv) - : null; - let indeterminate = false; - const valid = - !hostCredential || - (await validateRebuildHostInferenceCredential( - resumeConfig, - hostCredential, - undefined, - () => { - indeterminate = true; - }, - )); - if (!valid) { - return { - ok: false, - message: indeterminate - ? "NVIDIA upstream inference was temporarily unavailable before sandbox deletion. The sandbox is untouched; retry rebuild when inference recovers." - : "Host inference credential could not be validated before sandbox deletion. Check the credential and recorded endpoint, then retry rebuild.", - }; - } + const hostCredential = + !resumeConfig.registeredInferenceProviderAttachment && resumeConfig.credentialEnv + ? hydrateCredentialEnv(resumeConfig.credentialEnv) + : null; + const hostCredentialCheck = await revalidateRebuildHostCredentialBeforeDelete( + resumeConfig, + hostCredential, + ); + if (!hostCredentialCheck.ok) return hostCredentialCheck; const nativeAttachment = resumeConfig.nativeNvidiaProviderAttachment ?? resumeConfig.nativeHostedProviderAttachment; if ( !recreateOptions.rebuildProviderReconfigure && resumeConfig.nativeLocalProviderAttachment === undefined && + resumeConfig.registeredInferenceProviderAttachment === undefined && resumeConfig.provider && (nativeAttachment || shouldVerifyRebuildGatewayProvider(resumeConfig.provider)) ) { @@ -1006,6 +995,21 @@ async function rebuildSandboxUnlocked( }; } })(); + if ( + policyValidation.ok && + resumeConfig.registeredInferenceProviderAttachment !== undefined && + !(await verifyRebuildRegisteredProviderBeforeDelete( + sandboxName, + recreateOptions.targetGatewayName, + resumeConfig, + runtimeSelection, + )) + ) { + return { + ok: false, + message: "Registered provider binding changed before sandbox deletion.", + }; + } if (!policyValidation.ok || resumeConfig.nativeLocalProviderAttachment === undefined) return policyValidation; if ( diff --git a/src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts b/src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts index 3f87e1a8dbe..25ba4e3fc7d 100644 --- a/src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts +++ b/src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts @@ -11,10 +11,11 @@ import * as openshellRuntime from "../../adapters/openshell/runtime"; import YAML from "yaml"; import { messagingSandboxFiles } from "../../../../test/support/messaging-sandbox-files"; import { - createBuiltInChannelManifestRegistry, - createBuiltInRenderTemplateResolver, -} from "../../messaging/channels"; -import { MessagingWorkflowPlanner } from "../../messaging/compiler"; + hermesTeamsPlan, + hermesRebuildVerificationCases, + rebuildPostRestoreInput, + runtimeKindByAgent, +} from "../../../../test/support/rebuild-post-restore-fixtures"; import * as agentDefs from "../../agent/defs"; import * as agentRuntime from "../../agent/runtime"; import * as mutableConfigPerms from "../../sandbox/mutable-config-perms"; @@ -32,15 +33,7 @@ import * as rebuildMessaging from "./rebuild-messaging-phase"; import { runRebuildPostRestorePhase } from "./rebuild-post-restore-phase"; import { createRebuildCommandContext } from "./rebuild-preflight-confirmation"; -const processRecovery = restoreWindow; - describe("rebuild post-restore phase", () => { - const runtimeKindByAgent = { - openclaw: "gateway", - hermes: "gateway", - "langchain-deepagents-code": "terminal", - pi: "terminal", - } as const; let agentName: keyof typeof runtimeKindByAgent; let order: string[]; @@ -168,44 +161,52 @@ describe("rebuild post-restore phase", () => { vi.unstubAllEnvs(); }); - function input() { - return { - sandboxName: "alpha", - targetAgentName: agentName, - messagingPlan: null, - backupManifest: null, - mcpEntries: [], - restoreSucceeded: true, - failedPresets: [], - finalBuiltinPresets: [], - failedPresetRemovals: [], - policyPresetReconciliationVerified: true, - preparedBackupRecovery: false, - versionCheck: { expectedVersion: null } as never, - log: vi.fn(), - bail: vi.fn() as never, - }; - } - - async function hermesTeamsPlan(disabled = false) { - vi.stubEnv("MSTEAMS_APP_ID", "test-teams-app"); - vi.stubEnv("MSTEAMS_TENANT_ID", "test-teams-tenant"); - vi.stubEnv("TEAMS_ALLOWED_USERS", "00000000-0000-0000-0000-000000000001"); - vi.stubEnv("MSTEAMS_PORT", "3979"); - return new MessagingWorkflowPlanner( - createBuiltInChannelManifestRegistry(), - undefined, - createBuiltInRenderTemplateResolver(), - ).buildPlan({ - sandboxName: "alpha", + const input = () => rebuildPostRestoreInput(agentName); + + it("reapplies external Hermes inference after verified restart and before MCP restore (#12603)", async () => { + agentName = "hermes"; + vi.mocked(registry.getSandbox).mockReturnValue({ agent: "hermes", - workflow: "rebuild", - isInteractive: false, - configuredChannels: ["teams"], - disabledChannels: disabled ? ["teams"] : [], - credentialAvailability: { MSTEAMS_APP_PASSWORD: true }, + workload: { kind: "external-image" }, + } as never); + const restore = vi + .spyOn(rebuildHermesPostRestore, "restoreExternalHermesInference") + .mockImplementation(async () => { + expect( + rebuildHermesPostRestore.restartHermesGatewayAfterStateRestore, + ).toHaveBeenCalledOnce(); + expect(rebuildMcp.restoreMcpAfterRebuild).not.toHaveBeenCalled(); + }); + await runRebuildPostRestorePhase({ + ...input(), + backupManifest: { backupPath: "/backup" } as never, }); - } + expect(restore).toHaveBeenCalledOnce(); + }); + it.each(["restart-failed", "restart-health-timeout"] as const)( + "does not restore inference or MCP after Hermes %s (#12603)", + async (failure) => { + agentName = "hermes"; + vi.mocked(registry.getSandbox).mockReturnValue({ + agent: "hermes", + workload: { kind: "external-image" }, + } as never); + vi.mocked(rebuildHermesPostRestore.restartHermesGatewayAfterStateRestore).mockResolvedValue( + failure, + ); + const restore = vi + .spyOn(rebuildHermesPostRestore, "restoreExternalHermesInference") + .mockResolvedValue(); + await expect( + runRebuildPostRestorePhase({ + ...input(), + backupManifest: { backupPath: "/backup" } as never, + }), + ).rejects.toThrow("requires a verified gateway restart"); + expect(restore).not.toHaveBeenCalled(); + expect(rebuildMcp.restoreMcpAfterRebuild).not.toHaveBeenCalled(); + }, + ); it.each([false, true])( "restores the manifest-owned Hermes config before restart without enabling a disabled channel: disabled=%s", @@ -343,12 +344,12 @@ describe("rebuild post-restore phase", () => { "mcp", "host-forward", ]); - expect(processRecovery.beginUnregisteredOpenClawBackupQuiesce).toHaveBeenCalledExactlyOnceWith( + expect(restoreWindow.beginUnregisteredOpenClawBackupQuiesce).toHaveBeenCalledExactlyOnceWith( "alpha", undefined, ); - expect(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledOnce(); - expect(processRecovery.abortUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); + expect(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledOnce(); + expect(restoreWindow.abortUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); expect(pairingSettlement.settleOrdinaryOpenClawPairing).toHaveBeenCalledExactlyOnceWith( "alpha", ); @@ -461,6 +462,8 @@ describe("rebuild post-restore phase", () => { it.each([ [false, "scope-upgrade-not-approved", undefined], + [true, "scope-upgrade-not-approved", undefined], + [false, "pairing-incompatible", "paired.clientMode"], [true, "pairing-incompatible", "paired.clientMode"], ] as const)( "rejects rebuild with recovery=%s when pairing is incomplete", @@ -475,6 +478,9 @@ describe("rebuild post-restore phase", () => { expect(args.bail).toHaveBeenCalledWith("OpenClaw pairing remained incomplete after rebuild."); const errors = vi.mocked(console.error).mock.calls.flat().join("\n"); expect(errors).toContain(pairingField ? `${reason} (${pairingField})` : reason); + expect(vi.mocked(console.log).mock.calls.flat().join("\n")).not.toContain( + "rebuild completed", + ); }, ); @@ -554,7 +560,7 @@ describe("rebuild post-restore phase", () => { await runRebuildPostRestorePhase(input()); expect(readFileSync(sessionsPath, "utf8")).toBe(original); - expect(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledOnce(); + expect(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledOnce(); } finally { rmSync(directory, { recursive: true, force: true }); } @@ -562,14 +568,12 @@ describe("rebuild post-restore phase", () => { it("reuses the maintenance window established before filesystem restore", async () => { const window = { sandboxName: "alpha", kind: "backup" } as const; - vi.mocked(processRecovery.beginUnregisteredOpenClawBackupQuiesce).mockClear(); + vi.mocked(restoreWindow.beginUnregisteredOpenClawBackupQuiesce).mockClear(); await runRebuildPostRestorePhase({ ...input(), openClawDoctorWindow: window }); - expect(processRecovery.beginUnregisteredOpenClawBackupQuiesce).not.toHaveBeenCalled(); - expect(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledWith( - window, - ); + expect(restoreWindow.beginUnregisteredOpenClawBackupQuiesce).not.toHaveBeenCalled(); + expect(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledWith(window); expect(order).toEqual(["messaging", "native-start", "mcp", "host-forward"]); }); @@ -587,7 +591,7 @@ describe("rebuild post-restore phase", () => { await runRebuildPostRestorePhase(args); - expect(processRecovery.beginUnregisteredOpenClawBackupQuiesce).toHaveBeenCalledWith( + expect(restoreWindow.beginUnregisteredOpenClawBackupQuiesce).toHaveBeenCalledWith( "alpha", runtimeSelection, ); @@ -626,7 +630,7 @@ describe("rebuild post-restore phase", () => { }); it("finishes offline writes but stops online finalization when native start fails (#9946)", async () => { - vi.mocked(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).mockResolvedValue({ + vi.mocked(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).mockResolvedValue({ ok: false, stage: "restart", detail: "sensitive doctor output", @@ -642,7 +646,7 @@ describe("rebuild post-restore phase", () => { expect(messagingHostForward.ensureMessagingHostForwardAfterRebuild).not.toHaveBeenCalled(); expect(args.bail).toHaveBeenCalledWith("OpenClaw native final start failed during rebuild."); expect( - processRecovery.abortUnregisteredOpenClawPostRestoreDoctor, + restoreWindow.abortUnregisteredOpenClawPostRestoreDoctor, ).toHaveBeenCalledExactlyOnceWith({ sandboxName: "alpha", kind: "backup", @@ -704,7 +708,7 @@ describe("rebuild post-restore phase", () => { rebuildHermesPostRestore.completeHermesCronRestoreAfterGatewayReplacement, ).not.toHaveBeenCalled(); expect( - vi.mocked(processRecovery.abortUnregisteredOpenClawPostRestoreDoctor).mock.calls, + vi.mocked(restoreWindow.abortUnregisteredOpenClawPostRestoreDoctor).mock.calls, ).toEqual(agent === "openclaw" ? [[{ sandboxName: "alpha", kind: "backup" }]] : []); const output = [ ...vi.mocked(console.error).mock.calls, @@ -736,12 +740,12 @@ describe("rebuild post-restore phase", () => { "Could not retire pending messaging removals after rebuild.", ); expect( - processRecovery.abortUnregisteredOpenClawPostRestoreDoctor, + restoreWindow.abortUnregisteredOpenClawPostRestoreDoctor, ).toHaveBeenCalledExactlyOnceWith({ sandboxName: "alpha", kind: "backup", }); - expect(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); + expect(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); }); it("releases the maintenance gate before MCP restoration", async () => { @@ -751,15 +755,15 @@ describe("rebuild post-restore phase", () => { await expect(runRebuildPostRestorePhase(input())).rejects.toThrow("MCP restoration failed"); - expect(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledOnce(); - expect(processRecovery.abortUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); + expect(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).toHaveBeenCalledOnce(); + expect(restoreWindow.abortUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); }); it("does not mask a native start failure when the maintenance abort itself throws", async () => { - vi.mocked(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).mockRejectedValue( + vi.mocked(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).mockRejectedValue( new Error("original restoration failure"), ); - vi.mocked(processRecovery.abortUnregisteredOpenClawPostRestoreDoctor).mockRejectedValue( + vi.mocked(restoreWindow.abortUnregisteredOpenClawPostRestoreDoctor).mockRejectedValue( new Error("sensitive abort failure"), ); const args = input(); @@ -792,9 +796,12 @@ describe("rebuild post-restore phase", () => { ].map((operation) => vi.mocked(operation).mock.invocationCallOrder); expect(calls.map((invocations) => invocations.length)).toEqual([1, 1, 1, 1, 1, 1]); expect(calls.flat()).toEqual([...calls.flat()].sort((a, b) => a - b)); - expect(processRecovery.beginUnregisteredOpenClawBackupQuiesce).not.toHaveBeenCalled(); - expect(processRecovery.finishUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); - expect(mutableConfigPerms.inspectMutableHermesConfigPerms).toHaveBeenCalledWith("alpha"); + expect(restoreWindow.beginUnregisteredOpenClawBackupQuiesce).not.toHaveBeenCalled(); + expect(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).not.toHaveBeenCalled(); + expect(mutableConfigPerms.inspectMutableHermesConfigPerms).toHaveBeenCalledWith( + "alpha", + registry.getSandbox, + ); expect(verification).toEqual({ complete: true, mutableConfigPermissionsVerified: true }); expect( rebuildHermesPostRestore.verifyHermesGatewayAfterStateRestoreForCronGate, @@ -874,29 +881,7 @@ describe("rebuild post-restore phase", () => { expect(vi.mocked(console.log).mock.calls.flat().join("\n")).not.toContain("rebuild completed"); }); - it.each([ - { - cronGate: false, - preparedBackupRecovery: false, - restoreSucceeded: true, - gatewayDetail: "nemoclaw alpha gateway restart", - recoveryDetail: "If gateway health is still unverified", - }, - { - cronGate: true, - preparedBackupRecovery: false, - restoreSucceeded: true, - gatewayDetail: "Hermes gateway health was not verified after state restore.", - recoveryDetail: "Hermes cron dispatch remains drained", - }, - { - cronGate: false, - preparedBackupRecovery: true, - restoreSucceeded: false, - gatewayDetail: "Correct the reported restore problem", - recoveryDetail: "nemoclaw alpha recover", - }, - ])( + it.each(hermesRebuildVerificationCases)( "reports unavailable verification with cron gate=$cronGate, prepared recovery=$preparedBackupRecovery, and restore=$restoreSucceeded (#12004)", async ({ cronGate, @@ -1040,22 +1025,22 @@ describe("rebuild post-restore phase", () => { }); it.each([false, true])( - "reports the failed Hermes write check and retry command with prepared recovery %s", + "rejects incomplete Hermes config verification with prepared recovery %s", async (preparedBackupRecovery) => { agentName = "hermes"; vi.mocked(mutableConfigPerms.inspectMutableHermesConfigPerms).mockReturnValue({ verified: false, errors: ["config.yaml remains read-only"], }); - const args = { ...input(), preparedBackupRecovery }; - - const verification = await runRebuildPostRestorePhase(args); + const failure = new Error("Hermes verification failed"); + const bail = vi.fn((): never => { + throw failure; + }); + const args = { ...input(), preparedBackupRecovery, bail }; - expect(args.bail).toHaveBeenCalledTimes(Number(preparedBackupRecovery)); - expect(verification).toEqual( - preparedBackupRecovery - ? undefined - : { complete: false, mutableConfigPermissionsVerified: false }, + await expect(runRebuildPostRestorePhase(args)).rejects.toBe(failure); + expect(args.bail).toHaveBeenCalledExactlyOnceWith( + "Hermes post-restore verification failed for 'alpha'.", ); expect(args.log).toHaveBeenCalledWith( "Hermes mutable config posture was not verified: config.yaml remains read-only", diff --git a/src/lib/actions/sandbox/rebuild-post-restore-phase.ts b/src/lib/actions/sandbox/rebuild-post-restore-phase.ts index 603b08dacbb..f0ba7c297fd 100644 --- a/src/lib/actions/sandbox/rebuild-post-restore-phase.ts +++ b/src/lib/actions/sandbox/rebuild-post-restore-phase.ts @@ -30,6 +30,7 @@ import { } from "./rebuild/cli-exit"; import { completeHermesCronRestoreAfterGatewayReplacement, + restoreExternalHermesInference, type HermesCronRestoreIdentity, type HermesPostRestoreGatewayRestartState, isHermesCronRestoreDrainMarkerRollbackFailure, @@ -171,6 +172,15 @@ function printRebuildVersionFailureRecovery( return failureMessage; } +function printRebuildPairingFailure(pairing: { + readonly reason: string; + readonly pairingField?: string; +}): void { + console.error( + ` OpenClaw pairing remains incomplete after rebuild: ${pairing.reason}${formatOpenClawPairingMismatchField(pairing.pairingField)}`, + ); +} + function printHermesApiTokenChangeNotice(sandboxName: string, targetAgentName: string): void { if (targetAgentName !== "hermes") { return; @@ -405,11 +415,35 @@ export async function runRebuildPostRestorePhase( hermesPostRestoreGatewayDeps, ) : "not-applicable"; + if ( + restoreSucceeded && + backupManifest && + targetAgentName === "hermes" && + recreatedEntry.workload?.kind === "external-image" + ) { + // Native startup validates restored state and refreshes its integrity + // anchor. Only then can the guarded writer replace stripped references. + if (hermesGatewayRestartState !== "restarted") { + throw new Error( + "Restored Hermes inference requires a verified gateway restart. The rebuild backup is preserved.", + ); + } + await restoreExternalHermesInference( + recreatedEntry, + mcpRuntimeSelection, + registry.getSandbox, + ); + } if (targetAgentName === "openclaw") { if (!openClawDoctorWindow) { bail("OpenClaw gateway-down maintenance authority was lost during rebuild."); return; } + await rebuildOnboardDependencies.refreshRestoredRegisteredOpenclawInferenceRoute( + recreatedEntry, + mcpRuntimeSelection, + registry.getSandbox, + ); log("Releasing OpenClaw for one final start after all offline post-restore writes"); const startResult = await finishUnregisteredOpenClawPostRestoreDoctor(openClawDoctorWindow); log(`OpenClaw native final start: ${startResult.ok ? "verified" : startResult.stage}`); @@ -505,7 +539,10 @@ export async function runRebuildPostRestorePhase( verifiedAgentVersion = rebuiltVersion.sandboxVersion; } if (targetAgentName === "hermes") { - const mutableConfigVerification = inspectMutableHermesConfigPerms(sandboxName); + const mutableConfigVerification = inspectMutableHermesConfigPerms( + sandboxName, + registry.getSandbox, + ); mutableConfigPermissionsVerified = mutableConfigVerification.verified; if (mutableConfigPermissionsVerified) { log("Verified the rebuilt Hermes mutable config posture"); @@ -631,10 +668,7 @@ export async function runRebuildPostRestorePhase( ? await settleOrdinaryOpenClawPairing(sandboxName) : portablePairing; if (pairing.kind !== "settled") { - const pairingField = "pairingField" in pairing ? pairing.pairingField : null; - console.error( - ` OpenClaw pairing remains incomplete after rebuild: ${pairing.reason}${formatOpenClawPairingMismatchField(pairingField)}`, - ); + printRebuildPairingFailure(pairing); if (backupManifest) console.error(` Backup is preserved at: ${backupManifest.backupPath}`); console.error( ` Resolve the pairing failure, then rerun \`${CLI_NAME} ${sandboxName} rebuild --yes\`.`, @@ -699,7 +733,9 @@ export async function runRebuildPostRestorePhase( } if ( targetAgentName === "hermes" && - (hermesGatewayRestoreUnverified || mcpBridgeRestoreUnverified) + (hermesGatewayRestoreUnverified || + mcpBridgeRestoreUnverified || + !mutableConfigPermissionsVerified) ) { bail(`Hermes post-restore verification failed for '${sandboxName}'.`); return; diff --git a/src/lib/actions/sandbox/rebuild-preflight-guards.ts b/src/lib/actions/sandbox/rebuild-preflight-guards.ts index 222c14a1f42..a640f55a838 100644 --- a/src/lib/actions/sandbox/rebuild-preflight-guards.ts +++ b/src/lib/actions/sandbox/rebuild-preflight-guards.ts @@ -15,7 +15,10 @@ import { preflightGatewayRouteDiscovery, } from "../../inference/gateway-route-compatibility"; import { normalizeNativeLocalProviderAttachment } from "../../inference/native-local/contract"; -import { normalizeInferenceSelection } from "../../inference/selection"; +import { + normalizeInferenceSelection, + recordedRegisteredInferenceProviderAttachment, +} from "../../inference/selection"; import { resolveSandboxGatewayName } from "../../onboard/gateway-binding"; import { requireRuntimeProviderBundleForSandbox } from "../../onboard/runtime-provider/access"; import { assertHermesPortableCommandUnavailable } from "../../onboard/experimental/portable-agent-lifecycle"; @@ -29,6 +32,10 @@ import * as registry from "../../state/registry"; import { withLock } from "../../state/registry/lock"; import { load, save } from "../../state/registry/persistence"; import type { SandboxEntry, SandboxRegistry } from "../../state/registry/types"; +import { + validateRebuildHostInferenceCredential, + type HostCredentialTarget, +} from "./rebuild/host-credential-preflight"; import type { RebuildBail } from "./rebuild-credential-preflight"; import type { RebuildSandboxEntry } from "./rebuild-flow-helpers"; import type { RebuildVersionCheck } from "./rebuild-preflight-confirmation"; @@ -98,6 +105,28 @@ function hardRouteConflict( return "Prepared rebuild inference route is missing its provider or model."; } const target = sandboxes.find((entry) => entry.name === sandboxName); + if (target?.registeredInferenceProviderAttachment !== undefined) { + try { + const binding = recordedRegisteredInferenceProviderAttachment(target); + if ( + !binding || + binding.gatewayName !== gatewayName || + binding.sandboxName !== sandboxName || + binding.providerName !== route.provider + ) + return "Registered provider binding does not match the rebuild target."; + const discovery = preflightGatewayRouteDiscovery({ + gatewayName, + sandboxName, + route: { ...route, model: route.model }, + sandboxes, + nativeLocalSelection: true, + }); + return discovery.ok ? null : formatGatewayRouteConflict(discovery.result); + } catch { + return "Registered provider binding does not match the rebuild target."; + } + } if (target?.nativeLocalProviderAttachment !== undefined) { const local = normalizeNativeLocalProviderAttachment(target.nativeLocalProviderAttachment); if ( @@ -107,8 +136,7 @@ function hardRouteConflict( : target.agent === "hermes" ? local.agent === "hermes" : (target.agent === "openclaw" || target.agent === "pi") && - local.agent === target.agent && - !target.nimContainer) || + local.agent === target.agent) || local.sandboxName !== sandboxName || local.gatewayName !== gatewayName || local.provider !== route.provider @@ -177,6 +205,8 @@ export function commitRebuildRoutePreflight( const migratedSandboxNames: string[] = []; for (const peer of Object.values(sandboxRegistry.sandboxes)) { if ( + target.registeredInferenceProviderAttachment !== undefined || + peer.registeredInferenceProviderAttachment !== undefined || target.nativeLocalProviderAttachment !== undefined || peer.nativeLocalProviderAttachment !== undefined || peer.name === input.sandboxName || @@ -484,3 +514,22 @@ export function assertRebuildEntryUnchanged( bail, ); } + +export async function revalidateRebuildHostCredentialBeforeDelete( + target: HostCredentialTarget, + hostCredential: string | null, +): Promise<{ ok: true } | { ok: false; message: string }> { + let indeterminate = false; + const valid = + !hostCredential || + (await validateRebuildHostInferenceCredential(target, hostCredential, undefined, () => { + indeterminate = true; + })); + if (valid) return { ok: true }; + return { + ok: false, + message: indeterminate + ? "NVIDIA upstream inference was temporarily unavailable before sandbox deletion. The sandbox is untouched; retry rebuild when inference recovers." + : "Host inference credential could not be validated before sandbox deletion. Check the credential and recorded endpoint, then retry rebuild.", + }; +} diff --git a/src/lib/actions/sandbox/rebuild-preflight-phase.ts b/src/lib/actions/sandbox/rebuild-preflight-phase.ts index 4ca28b11201..92dc35b54ea 100644 --- a/src/lib/actions/sandbox/rebuild-preflight-phase.ts +++ b/src/lib/actions/sandbox/rebuild-preflight-phase.ts @@ -6,10 +6,12 @@ import path from "node:path"; import type { RebuildSandboxOptions } from "../../domain/lifecycle/options"; import { hasLegacySharedLocalInferenceRoute, + hasLegacySharedCustomInferenceRoute, isPersistedPiSharedLocalInferenceRoute, } from "../../inference/selection"; import type { SandboxMessagingPlan } from "../../messaging"; import { hydrateCredentialEnv } from "../../onboard/credential-env"; +import { getInstalledOpenshellVersion } from "../../onboard/openshell-version"; import { DCODE_AUTO_APPROVAL_FEATURE } from "../../onboard/dcode-auto-approval"; import { managedSandboxFeatureIssue } from "../../onboard/managed-sandbox-feature"; import { @@ -137,21 +139,26 @@ export function runHermesCronRestoreBackupPreflight({ } } -/** Stop before backup or deletion when an old local route cannot be recreated. */ -export function blockRebuildOnLegacySharedLocalRoute( +/** Stop before backup or deletion when a shared route cannot be recreated. */ +export function blockRebuildOnLegacySharedRoute( sandboxName: string, sandboxEntry: RebuildSandboxEntry, bail: RebuildBail, ): boolean { + const local = hasLegacySharedLocalInferenceRoute(sandboxEntry); + const custom = hasLegacySharedCustomInferenceRoute(sandboxEntry); + const legacyNvidia = sandboxEntry.provider === "nvidia-nim"; + if (!local && !custom && !legacyNvidia) return false; if ( - !hasLegacySharedLocalInferenceRoute(sandboxEntry) || - isPersistedPiSharedLocalInferenceRoute(sandboxEntry) + (local ? isPersistedPiSharedLocalInferenceRoute(sandboxEntry) : custom || legacyNvidia) && + /^0\.0\.\d+$/.test(getInstalledOpenshellVersion() ?? "") ) return false; + const route = local ? "local" : legacyNvidia ? "NVIDIA" : "custom"; printRebuildPreflightFailure( - `sandbox '${sandboxName}' uses a legacy shared local inference route without a native provider attachment.`, - "Back up the sandbox, then re-onboard it with --fresh --name --recreate-sandbox before upgrading OpenShell.", - "Legacy shared local inference cannot be rebuilt safely", + `sandbox '${sandboxName}' uses a legacy shared ${route} inference route without a native provider attachment.`, + "Back up the sandbox, then re-onboard it with --fresh --name --recreate-sandbox.", + `Legacy shared ${route} inference cannot be rebuilt safely`, bail, ); return true; @@ -179,7 +186,8 @@ export async function runRebuildPreflightPhase( } = createRebuildCommandContext(options, opts); const sandboxEntry = getRebuildSandboxEntryOrBail(sandboxName, bail); if (!sandboxEntry) return null; - if (blockRebuildOnLegacySharedLocalRoute(sandboxName, sandboxEntry, bail)) return null; + if (blockRebuildOnLegacySharedRoute(sandboxName, sandboxEntry, bail)) return null; + if (blockRebuildOnRetainedSandboxRecovery(sandboxEntry, bail)) return null; restoreRecordedRebuildGatewayStateDir(sandboxEntry); if (blockRebuildOnRetainedSandboxRecovery(sandboxEntry, bail)) return null; const confirmedEntrySnapshot = JSON.stringify(sandboxEntry); diff --git a/src/lib/actions/sandbox/rebuild-preflight-target-phase.test.ts b/src/lib/actions/sandbox/rebuild-preflight-target-phase.test.ts index e74aeba39e9..7bb4582baf8 100644 --- a/src/lib/actions/sandbox/rebuild-preflight-target-phase.test.ts +++ b/src/lib/actions/sandbox/rebuild-preflight-target-phase.test.ts @@ -3,8 +3,11 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import { cliOpenShellInstalledVersionObserver } from "../../adapters/openshell/installed-version-cli"; + import type { RebuildSandboxEntry } from "./rebuild-flow-helpers"; -import { blockRebuildOnLegacySharedLocalRoute } from "./rebuild-preflight-phase"; +import { blockRebuildOnLegacySharedRoute } from "./rebuild-preflight-phase"; +import { customNativeProvider } from "../../inference/native-provider/custom"; import { restoreEnvBulk } from "../../../../test/helpers/env-test-helpers"; import type { @@ -268,6 +271,12 @@ describe("rebuild legacy local route preflight", () => { { agent: "openclaw", provider: "ollama-local", createdAt: undefined }, { agent: "openclaw", provider: "vllm-local", createdAt: undefined }, { agent: "openclaw", provider: "llama-cpp-local", createdAt: undefined }, + { + agent: "openclaw", + provider: "compatible-endpoint", + endpointUrl: "http://127.0.0.1:8000/v1", + createdAt: "2026-09-01T00:00:00Z", + }, { agent: "langchain-deepagents-code", provider: "ollama-local", @@ -290,7 +299,7 @@ describe("rebuild legacy local route preflight", () => { ...state, } as RebuildSandboxEntry; - expect(() => blockRebuildOnLegacySharedLocalRoute("alpha", entry, bail)).toThrow( + expect(() => blockRebuildOnLegacySharedRoute("alpha", entry, bail)).toThrow( "Legacy shared local inference cannot be rebuilt safely", ); expect(error.mock.calls.flat().join("\n")).toContain("Back up the sandbox"); @@ -298,7 +307,7 @@ describe("rebuild legacy local route preflight", () => { }, ); - it("leaves Local NIM on its existing shared rebuild path", () => { + it("blocks a legacy Local NIM rebuild before mutation", () => { const bail = vi.fn((message: string): never => { throw new Error(message); }); @@ -309,8 +318,10 @@ describe("rebuild legacy local route preflight", () => { model: "nim-model", nimContainer: "nim-alpha", } as RebuildSandboxEntry; - expect(blockRebuildOnLegacySharedLocalRoute("alpha", entry, bail)).toBe(false); - expect(bail).not.toHaveBeenCalled(); + expect(() => blockRebuildOnLegacySharedRoute("alpha", entry, bail)).toThrow( + "Legacy shared local inference cannot be rebuilt safely", + ); + expect(bail).toHaveBeenCalledOnce(); }); it("leaves attached selections on their existing preflight path", () => { @@ -325,25 +336,129 @@ describe("rebuild legacy local route preflight", () => { model: "local-model", nativeLocalProviderAttachment: {}, } as RebuildSandboxEntry; - expect(blockRebuildOnLegacySharedLocalRoute("alpha", entry, bail)).toBe(false); + expect(blockRebuildOnLegacySharedRoute("alpha", entry, bail)).toBe(false); expect( - blockRebuildOnLegacySharedLocalRoute( + blockRebuildOnLegacySharedRoute("alpha", { ...entry, pendingRouteReservation: true }, bail), + ).toBe(false); + expect(bail).not.toHaveBeenCalled(); + }); +}); + +describe("retained shared rebuild compatibility", () => { + it("keeps a verified older runtime eligible for shared NVIDIA rebuild", () => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue({ + ok: true, + version: "0.0.116", + }); + const bail = vi.fn((message: string): never => { + throw new Error(message); + }); + expect( + blockRebuildOnLegacySharedRoute( "alpha", - { ...entry, pendingRouteReservation: true }, + { name: "alpha", agent: "openclaw", provider: "nvidia-nim" } as RebuildSandboxEntry, + bail, + ), + ).toBe(false); + expect(bail).not.toHaveBeenCalled(); + }); + + it("leaves native NVIDIA selections on their existing preflight path", () => { + const installedVersion = vi.spyOn( + cliOpenShellInstalledVersionObserver, + "observeInstalledVersion", + ); + const bail = vi.fn((message: string): never => { + throw new Error(message); + }); + expect( + blockRebuildOnLegacySharedRoute( + "alpha", + { name: "alpha", agent: "openclaw", provider: "nvidia-prod" } as RebuildSandboxEntry, + bail, + ), + ).toBe(false); + expect(installedVersion).not.toHaveBeenCalled(); + expect(bail).not.toHaveBeenCalled(); + }); + + it("keeps a verified older runtime eligible for shared custom rebuild", () => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue({ + ok: true, + version: "0.0.116", + }); + const bail = vi.fn((message: string): never => { + throw new Error(message); + }); + expect( + blockRebuildOnLegacySharedRoute( + "alpha", + { + name: "alpha", + agent: "openclaw", + provider: "compatible-endpoint", + endpointUrl: "https://custom.example/v1", + } as RebuildSandboxEntry, bail, ), ).toBe(false); expect(bail).not.toHaveBeenCalled(); }); -}); -describe("Pi shared local rebuild compatibility", () => { - it("keeps a published Pi shared route eligible for rebuild (#12975)", () => { + it("verifies the recorded custom attachment before admitting a native rebuild", () => { + const custom = { + provider: "compatible-endpoint", + endpointUrl: "https://custom.example/v1", + api: "openai-completions", + credentialEnv: "TEAM_KEY", + allowedIps: ["8.8.8.8"], + } as const; + const definition = customNativeProvider(custom); + const attachment = { + schemaVersion: 1, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "provider-id", + endpointUrl: definition.endpointUrl, + allowedIps: definition.allowedIps, + custom, + } as const; + const entry = { + name: "alpha", + agent: "openclaw", + provider: custom.provider, + endpointUrl: custom.endpointUrl, + credentialEnv: custom.credentialEnv, + preferredInferenceApi: custom.api, + nativeHostedProviderAttachment: attachment, + } as RebuildSandboxEntry; + const bail = vi.fn((message: string): never => { + throw new Error(message); + }); + expect(blockRebuildOnLegacySharedRoute("alpha", entry, bail)).toBe(false); + expect(() => + blockRebuildOnLegacySharedRoute( + "alpha", + { + ...entry, + endpointUrl: "https://different.example/v1", + }, + bail, + ), + ).toThrow("does not match"); + expect(bail).not.toHaveBeenCalled(); + }); + + it("keeps a published Pi shared route eligible for rebuild on OpenShell 0.0.116 (#12975)", () => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue({ + ok: true, + version: "0.0.116", + }); const bail = vi.fn((message: string): never => { throw new Error(message); }); expect( - blockRebuildOnLegacySharedLocalRoute( + blockRebuildOnLegacySharedRoute( "alpha", { name: "alpha", diff --git a/src/lib/actions/sandbox/rebuild-provider-preflight.test.ts b/src/lib/actions/sandbox/rebuild-provider-preflight.test.ts index 8596554d383..3cae6c395bc 100644 --- a/src/lib/actions/sandbox/rebuild-provider-preflight.test.ts +++ b/src/lib/actions/sandbox/rebuild-provider-preflight.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; import { HOSTED_NATIVE_PROVIDERS, hostedNativeProvider, @@ -20,6 +21,7 @@ import { inspectRebuildGatewayProviderRegistration, shouldVerifyRebuildGatewayProvider, verifyRebuildNativeLocalProviderBeforeDelete, + verifyRebuildRegisteredProviderBeforeDelete, } from "./rebuild-provider-preflight"; import type { RebuildResumeConfig } from "./rebuild-resume-config"; import * as managedProviderAdapter from "../../adapters/openshell/managed-provider-adapter"; @@ -27,7 +29,7 @@ import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider import { nativeLocalIdentity } from "../../inference/native-local/contract"; import { preflightRebuildCredentials } from "./rebuild-credential-preflight"; import { rebuildOnboardDependencies } from "./rebuild-onboard-dependencies"; -import * as providerPreflight from "./rebuild-provider-preflight"; +import * as hostCredentialPreflight from "./rebuild/host-credential-preflight"; import type { RebuildSandboxEntry } from "./rebuild-flow-helpers"; vi.mock("../../adapters/openshell/provider-policy", () => ({ @@ -108,6 +110,7 @@ describe.each(["langchain-deepagents-code", "hermes", "openclaw"] as const)( type: attachment.profileId, credentialKeys: [binding.credentialEnv], configKeys: [], + credentialExpiresAtMs: {} as Record | undefined, revision: { id: "owned", resourceVersion: 1 }, }, })), @@ -159,6 +162,20 @@ describe.each(["langchain-deepagents-code", "hermes", "openclaw"] as const)( { target: { kind: "named", gatewayName: "nemoclaw" }, providerName: f.attachment.providerName, + includeCredentialExpirations: true, + }, + ], + [ + { + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: f.attachment.providerName, + }, + ], + [ + { + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: f.attachment.providerName, + includeCredentialExpirations: true, }, ], [ @@ -189,16 +206,97 @@ describe.each(["langchain-deepagents-code", "hermes", "openclaw"] as const)( ).resolves.toBe(true); expect(getSandbox.mock.calls).toEqual([["alpha"], ["alpha"]]); expect(f.adapter.inspectProviderProfile).toHaveBeenCalledOnce(); - expect(f.adapter.getProvider).toHaveBeenCalledExactlyOnceWith({ - target: { kind: "named", gatewayName: "nemoclaw" }, - providerName: f.attachment.providerName, - }); + expect(f.adapter.getProvider.mock.calls).toEqual([ + [ + { + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: f.attachment.providerName, + includeCredentialExpirations: true, + }, + ], + [ + { + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: f.attachment.providerName, + }, + ], + ]); expect(f.adapter.listProviderAttachments).toHaveBeenCalledExactlyOnceWith({ target: { kind: "named", gatewayName: "nemoclaw" }, sandboxName: "alpha", }); }); + it.each([ + { label: "expired", metadata: { NEMOCLAW_LOCAL_INFERENCE_TOKEN: 1 }, accepted: false }, + { label: "missing expiry inventory", metadata: undefined, accepted: false }, + { + label: "negative expiry", + metadata: { NEMOCLAW_LOCAL_INFERENCE_TOKEN: -1 }, + accepted: false, + }, + { + label: "invalid expiry", + metadata: { NEMOCLAW_LOCAL_INFERENCE_TOKEN: NaN }, + accepted: false, + }, + { + label: "fractional expiry", + metadata: { NEMOCLAW_LOCAL_INFERENCE_TOKEN: 1.5 }, + accepted: false, + }, + { + label: "explicit non-expiring", + metadata: { NEMOCLAW_LOCAL_INFERENCE_TOKEN: 0 }, + accepted: true, + }, + { + label: "future expiry", + metadata: { NEMOCLAW_LOCAL_INFERENCE_TOKEN: Number.MAX_SAFE_INTEGER }, + accepted: true, + }, + ])("checks $label credentials at the delete edge", async ({ metadata, accepted }) => { + const f = nativeFixture(); + const observed = await f.adapter.getProvider(); + f.adapter.getProvider.mockClear().mockResolvedValue({ + ...observed, + value: { ...observed.value, credentialExpiresAtMs: metadata }, + }); + await expect( + verifyRebuildNativeLocalProviderBeforeDelete( + "alpha", + "nemoclaw", + f.selection, + undefined, + () => f.entry, + ), + ).resolves.toBe(accepted); + }); + + it.each([ + { label: "provider replacement", revision: { id: "replacement", resourceVersion: 1 } }, + { label: "credential update", revision: { id: "owned", resourceVersion: 2 } }, + ])("rejects $label during expiry observation", async ({ revision }) => { + const f = nativeFixture(); + const observed = await f.adapter.getProvider(); + f.adapter.getProvider + .mockClear() + .mockResolvedValueOnce(observed) + .mockResolvedValue({ + ...observed, + value: { ...observed.value, revision }, + }); + await expect( + verifyRebuildNativeLocalProviderBeforeDelete( + "alpha", + "nemoclaw", + f.selection, + undefined, + () => f.entry, + ), + ).resolves.toBe(false); + }); + it.each([ { label: "profile drift", @@ -217,6 +315,7 @@ describe.each(["langchain-deepagents-code", "hermes", "openclaw"] as const)( type: f.attachment.profileId, credentialKeys: [f.attachment.credentialEnv], configKeys: [], + credentialExpiresAtMs: {}, revision: { id: "replacement", resourceVersion: 2 }, }, }), @@ -298,7 +397,7 @@ describe.each(["langchain-deepagents-code", "hermes", "openclaw"] as const)( const f = nativeFixture(); f.hydrate.mockReturnValue("test-only-rejected-key"); const validate = vi - .spyOn(providerPreflight, "validateRebuildHostInferenceCredential") + .spyOn(hostCredentialPreflight, "validateRebuildHostInferenceCredential") .mockResolvedValue(false); await expect(preflightRebuildCredentials(f.entry, vi.fn(), throwingBail)).rejects.toThrow( "Host inference credential validation failed", @@ -358,6 +457,31 @@ describe("shouldVerifyRebuildGatewayProvider", () => { expect(log).not.toHaveBeenCalled(); expect(bail).not.toHaveBeenCalled(); }); + + it("does not print malformed provider or credential identifiers", async () => { + const adapter = createCliOpenShellProviderAdapter(); + vi.spyOn(adapter, "getProvider").mockResolvedValue({ + ok: false, + error: { kind: "command", reason: "not_found", message: "missing" }, + }); + vi.spyOn(managedProviderAdapter, "createManagedProviderAdapter").mockReturnValue(adapter); + const output: string[] = []; + vi.spyOn(console, "error").mockImplementation((message = "") => output.push(String(message))); + const log = vi.fn((message: string) => output.push(message)); + const secret = "nvapi-test-secret"; + + await expect( + checkRebuildGatewayProviderOrBail( + `provider\n${secret}`, + `CREDENTIAL\n${secret}`, + log, + throwingBail, + ), + ).rejects.toThrow("Missing gateway provider: unavailable"); + + expect(output.join("\n")).not.toContain(secret); + expect(output.join("\n")).toContain("provider 'unavailable'"); + }); }); describe("canRecreateMissingRebuildGatewayProvider", () => { @@ -731,3 +855,39 @@ describe("hosted native rebuild provider preflight", () => { }, ); }); + +it.each(["valid", "replaced", "detached", "selection-changed"] as const)( + "checks the external binding at the destructive rebuild boundary: %s", + async (state) => { + const f = registeredInferenceFixture(state !== "detached"); + f.state.id = state === "replaced" ? "replacement" : f.state.id; + vi.spyOn(managedProviderAdapter, "createManagedProviderAdapter").mockReturnValue(f.adapter); + const resume = config({ + agent: "openclaw", + provider: f.binding.providerName, + model: "model-a", + endpointUrl: f.binding.endpointUrl, + credentialEnv: f.binding.credentialEnv, + preferredInferenceApi: f.binding.inferenceApi, + registeredInferenceProviderAttachment: f.binding, + }); + const entry = { + ...resume, + name: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + model: state === "selection-changed" ? "other-model" : resume.model, + }; + expect( + await verifyRebuildRegisteredProviderBeforeDelete( + f.binding.sandboxName, + f.binding.gatewayName, + resume, + undefined, + () => entry, + ), + ).toBe(state === "valid"); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + }, +); diff --git a/src/lib/actions/sandbox/rebuild-provider-preflight.ts b/src/lib/actions/sandbox/rebuild-provider-preflight.ts index 5da698e37e2..11d94fde230 100644 --- a/src/lib/actions/sandbox/rebuild-provider-preflight.ts +++ b/src/lib/actions/sandbox/rebuild-provider-preflight.ts @@ -1,15 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { validateNvidiaApiKeyValue } from "../../credentials/api-key-validation"; -import { getCompatibleAnthropicOpenAiSurfaceBaseUrl } from "../../inference/anthropic/openai-surface"; import { - assertEndpointResolvesPublic, - parseTrustedPrivateInferenceHostsFromEnv, -} from "../../inference/endpoint-ssrf-preflight"; -import { usesNvidiaEndpointProbePayload } from "../../inference/openai-probe-models"; -import { RETRIABLE_HTTP_PROBE_STATUSES } from "../../inference/probe/transient-http-policy"; -import { getRemoteProviderConfigForName } from "../../onboard/inference-providers/provider-selection-keys"; + recordedRegisteredInferenceProviderAttachment, + verifyRegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; import type { RebuildSandboxEntry } from "./rebuild-flow-helpers"; import { @@ -58,110 +53,6 @@ const { REMOTE_PROVIDER_CONFIG } = require("../../onboard/providers") as { >; }; -type ProbeResult = { ok: boolean; validated?: boolean; failures?: unknown[] }; - -function transientNvidiaValidationFailure(result: ProbeResult): boolean { - return ( - Array.isArray(result.failures) && - result.failures.length > 0 && - result.failures.every((failure) => { - if (typeof failure !== "object" || failure === null) return false; - const { httpStatus, curlStatus } = failure as Record; - return ( - (curlStatus === 0 && - typeof httpStatus === "number" && - RETRIABLE_HTTP_PROBE_STATUSES.has(httpStatus)) || - ((curlStatus === 28 || curlStatus === -110) && - (httpStatus === 0 || httpStatus === null || httpStatus === undefined)) - ); - }) - ); -} -type Probe = ( - endpointUrl: string, - model: string, - apiKey: string, - options: Record, -) => ProbeResult | Promise; - -const probes = require("../../inference/onboard-probes") as { - probeAnthropicEndpoint: Probe; - probeOpenAiLikeEndpointOptimized: Probe; - getProbeExtraHeaders(provider: string): string[] | undefined; -}; - -export type HostCredentialTarget = Pick< - RebuildSandboxEntry, - "provider" | "model" | "endpointUrl" | "credentialEnv" | "preferredInferenceApi" ->; - -/** Validate the host credential that recreate can submit, without changing the gateway. */ -export async function validateRebuildHostInferenceCredential( - target: HostCredentialTarget, - credentialValue: string, - deps = { - assertEndpointResolvesPublic, - probeAnthropicEndpoint: probes.probeAnthropicEndpoint, - probeOpenAiLikeEndpointOptimized: probes.probeOpenAiLikeEndpointOptimized, - }, - onIndeterminate?: () => void, -): Promise { - const config = getRemoteProviderConfigForName(target.provider, REMOTE_PROVIDER_CONFIG); - // Local, routed, and Hermes authentication retain their existing preflights. - if (!config || config.providerName === "llama-cpp-local" || target.provider === "hermes-provider") - return true; - if ( - target.provider === "compatible-anthropic-endpoint" && - isBedrockRuntimeEndpoint(target.endpointUrl) - ) { - return true; - } - if (validateNvidiaApiKeyValue(credentialValue, target.credentialEnv ?? "")) return false; - let endpointUrl = config.endpointUrl || target.endpointUrl; - if (!endpointUrl || !target.model) return false; - - try { - const pins = !config.endpointUrl - ? await deps.assertEndpointResolvesPublic(endpointUrl, undefined, { - trustedPrivateHosts: parseTrustedPrivateInferenceHostsFromEnv(process.env), - }) - : null; - if (pins && !pins.ok) return false; - const useAnthropic = - config.providerType === "anthropic" && target.preferredInferenceApi !== "openai-completions"; - if (config.providerType === "anthropic" && !useAnthropic) { - endpointUrl = getCompatibleAnthropicOpenAiSurfaceBaseUrl(endpointUrl); - } - const probe = useAnthropic - ? deps.probeAnthropicEndpoint - : deps.probeOpenAiLikeEndpointOptimized; - const result = await probe(endpointUrl, target.model, credentialValue, { - skipResponsesProbe: target.preferredInferenceApi !== "openai-responses", - probeStreaming: false, - provider: target.provider, - extraHeaders: probes.getProbeExtraHeaders(target.provider ?? ""), - useNvidiaEndpointProbePayload: usesNvidiaEndpointProbePayload(target.provider ?? ""), - ...(pins?.ok - ? { - pinnedAddresses: pins.addresses, - trustedPrivateCapability: pins.trustedPrivateCapability, - } - : {}), - }); - if ( - !result.ok && - usesNvidiaEndpointProbePayload(target.provider ?? "") && - transientNvidiaValidationFailure(result) - ) { - onIndeterminate?.(); - } - return result.ok && result.validated !== false; - } catch { - // Probe errors and response bodies can contain credentials. - return false; - } -} - export type RebuildGatewayProviderRegistration = | "registered" | "expired" @@ -183,6 +74,45 @@ function rebuildProviderAdapter( ); } +/** Recheck the external provider without acquiring ownership of it. */ +export async function verifyRebuildRegisteredProviderBeforeDelete( + sandboxName: string, + gatewayName: string, + config: RebuildResumeConfig, + runtimeSelection?: OpenShellRuntimeSelection, + getSandbox: (name: string) => RebuildSandboxEntry | null = registry.getSandbox, +): Promise { + try { + const expected = recordedRegisteredInferenceProviderAttachment({ + ...config, + name: sandboxName, + gatewayName, + }); + const currentMatches = () => { + const current = getSandbox(sandboxName); + return ( + current?.agent === config.agent && + current.model === config.model && + JSON.stringify(recordedRegisteredInferenceProviderAttachment(current)) === + JSON.stringify(expected) + ); + }; + if ( + !expected || + (runtimeSelection && runtimeSelection.gatewayName !== gatewayName) || + !currentMatches() + ) + return false; + await verifyRegisteredInferenceProviderAttachment({ + adapter: rebuildProviderAdapter(runtimeSelection), + expected, + }); + return currentMatches(); + } catch { + return false; + } +} + /** Revalidate the frozen scoped authority immediately before destructive rebuild work. */ export async function verifyRebuildNativeLocalProviderBeforeDelete( sandboxName: string, @@ -234,6 +164,8 @@ export async function inspectRebuildGatewayProviderRegistration( credentialKey?: string | null, nativeAttachment?: NativeProviderAttachment, ): Promise { + const safeProvider = providerDiagnosticName(provider); + const safeCredential = credentialKey ? credentialDiagnosticName(credentialKey) : null; let nativeLifecycle: ReturnType | undefined; if (nativeAttachment) { try { @@ -279,13 +211,13 @@ export async function inspectRebuildGatewayProviderRegistration( ? "missing" : "indeterminate"; log( - `${phase} gateway provider check: provider '${provider}' ${ + `${phase} gateway provider check: provider '${safeProvider}' ${ registration === "registered" ? "is registered" : registration === "expired" - ? `has expired credential ${credentialKey}` + ? `has expired credential ${safeCredential}` : registration === "credential_missing" - ? `does not expose credential ${credentialKey}` + ? `does not expose credential ${safeCredential}` : registration === "missing" ? "is explicitly missing" : "could not be verified" @@ -319,15 +251,30 @@ function rebuildCredentialReuseCondition(condition: RecoveredProviderReuseReject } } +const PROVIDER_DIAGNOSTIC_NAME = /^[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}$/u; +const CREDENTIAL_DIAGNOSTIC_NAME = /^[A-Z][A-Z0-9_]{0,127}$/u; + +function providerDiagnosticName(value: string): string { + return PROVIDER_DIAGNOSTIC_NAME.test(value) ? value : "unavailable"; +} + +function credentialDiagnosticName(value: string): string { + return CREDENTIAL_DIAGNOSTIC_NAME.test(value) ? value : "unavailable"; +} + function printMissingRebuildGatewayProvider(provider: string, credentialEnv: string | null): void { + const safeProvider = providerDiagnosticName(provider); + const safeCredential = credentialEnv ? credentialDiagnosticName(credentialEnv) : null; console.error(""); console.error( - ` ${_RD}Rebuild preflight failed:${R} provider '${provider}' is not registered in OpenShell.`, + ` ${_RD}Rebuild preflight failed:${R} provider '${safeProvider}' is not registered in OpenShell.`, // lgtm[js/clear-text-logging] Strictly allow-listed provider identifier. ); console.error(" The sandbox registry still points at this upstream provider,"); console.error(" so rebuild will not recreate it before destroying the sandbox."); - if (credentialEnv) { - console.error(` Rebuild cannot rely on ${credentialEnv} while that provider is missing.`); + if (safeCredential) { + console.error( + ` Rebuild cannot rely on ${safeCredential} while that provider is missing.`, // lgtm[js/clear-text-logging] Strictly allow-listed environment-variable name. + ); } console.error(""); console.error(" Re-register the provider in OpenShell or rerun onboard, then retry rebuild."); @@ -335,9 +282,10 @@ function printMissingRebuildGatewayProvider(provider: string, credentialEnv: str } function printIndeterminateRebuildGatewayProvider(provider: string): void { + const safeProvider = providerDiagnosticName(provider); console.error(""); console.error( - ` ${_RD}Rebuild preflight failed:${R} could not verify provider '${provider}' in OpenShell.`, + ` ${_RD}Rebuild preflight failed:${R} could not verify provider '${safeProvider}' in OpenShell.`, // lgtm[js/clear-text-logging] Strictly allow-listed provider identifier. ); console.error(" The provider lookup did not return an explicit not-found response."); console.error(" Check gateway connectivity and authentication, then retry rebuild."); @@ -348,14 +296,18 @@ function printExpiredRebuildGatewayProviderCredential( provider: string, credentialKey: string, ): void { + const safeProvider = providerDiagnosticName(provider); + const safeCredential = credentialDiagnosticName(credentialKey); console.error(""); console.error( - ` ${_RD}Rebuild preflight failed:${R} provider '${provider}' credential ${credentialKey} is expired in OpenShell.`, + ` ${_RD}Rebuild preflight failed:${R} provider '${safeProvider}' credential ${safeCredential} is expired in OpenShell.`, // lgtm[js/clear-text-logging] Strictly allow-listed identifiers. ); console.error( " Rebuild will not destroy a sandbox that it cannot restore to working inference.", ); - console.error(` Refresh ${credentialKey} in OpenShell or rerun onboard, then retry rebuild.`); + console.error( + ` Refresh ${safeCredential} in OpenShell or rerun onboard, then retry rebuild.`, // lgtm[js/clear-text-logging] Strictly allow-listed environment-variable name. + ); console.error(" Sandbox is untouched — no data was lost."); } @@ -363,15 +315,17 @@ function printMissingRebuildGatewayProviderCredential( provider: string, credentialKey: string, ): void { + const safeProvider = providerDiagnosticName(provider); + const safeCredential = credentialDiagnosticName(credentialKey); console.error(""); console.error( - ` ${_RD}Rebuild preflight failed:${R} provider '${provider}' no longer exposes credential ${credentialKey}.`, + ` ${_RD}Rebuild preflight failed:${R} provider '${safeProvider}' no longer exposes credential ${safeCredential}.`, // lgtm[js/clear-text-logging] Strictly allow-listed identifiers. ); console.error( " Rebuild will not destroy a sandbox that it cannot restore to working inference.", ); console.error( - ` Re-register '${provider}' with ${credentialKey} in OpenShell or rerun onboard, then retry rebuild.`, + ` Re-register '${safeProvider}' with ${safeCredential} in OpenShell or rerun onboard, then retry rebuild.`, // lgtm[js/clear-text-logging] Strictly allow-listed identifiers. ); console.error(" Sandbox is untouched — no data was lost."); } @@ -417,6 +371,8 @@ export async function checkRebuildGatewayProviderOrBail( ): Promise { if (!options.nativeAttachment && !shouldVerifyRebuildGatewayProvider(provider)) return true; if (!provider) return false; + const safeProvider = providerDiagnosticName(provider); + const safeCredential = credentialEnv ? credentialDiagnosticName(credentialEnv) : null; const registration = await inspectRebuildGatewayProviderRegistration( provider, @@ -430,12 +386,12 @@ export async function checkRebuildGatewayProviderOrBail( if (registration === "registered") return true; if (registration === "expired" && credentialEnv) { printExpiredRebuildGatewayProviderCredential(provider, credentialEnv); - bail(`Expired gateway provider credential: ${provider}/${credentialEnv}`); + bail(`Expired gateway provider credential: ${safeProvider}/${safeCredential}`); return false; } if (registration === "credential_missing" && credentialEnv) { printMissingRebuildGatewayProviderCredential(provider, credentialEnv); - bail(`Missing gateway provider credential: ${provider}/${credentialEnv}`); + bail(`Missing gateway provider credential: ${safeProvider}/${safeCredential}`); return false; } if ( @@ -448,19 +404,19 @@ export async function checkRebuildGatewayProviderOrBail( ) { options.onProviderReconfigureRequired?.(provider, credentialEnv); log( - `Preflight gateway provider check: validated prepared recovery will recreate missing provider '${provider}' from ${credentialEnv}`, + `Preflight gateway provider check: validated prepared recovery will recreate missing provider '${safeProvider}' from ${safeCredential}`, ); return true; } if (registration === "indeterminate") { printIndeterminateRebuildGatewayProvider(provider); - bail(`Could not verify gateway provider: ${provider}`); + bail(`Could not verify gateway provider: ${safeProvider}`); return false; } printMissingRebuildGatewayProvider(provider, credentialEnv); - bail(`Missing gateway provider: ${provider}`); + bail(`Missing gateway provider: ${safeProvider}`); return false; } @@ -498,6 +454,15 @@ export async function checkRebuildGatewayCredentialReuseOrBail( bail: (msg: string, code?: number) => never, deps: GatewayCredentialReusePreflightDeps = defaultGatewayCredentialReusePreflightDeps(), ): Promise { + if (config.registeredInferenceProviderAttachment !== undefined) { + const expected = config.registeredInferenceProviderAttachment; + if ( + await verifyRebuildRegisteredProviderBeforeDelete(sandboxName, expected.gatewayName, config) + ) + return true; + bail("Could not verify registered provider credential reuse; sandbox is untouched"); + return false; + } if (config.nativeLocalProviderAttachment !== undefined) { try { await verifyRecordedNativeLocalProviderAttachment({ diff --git a/src/lib/actions/sandbox/rebuild-resume-config.test.ts b/src/lib/actions/sandbox/rebuild-resume-config.test.ts index d616dbeb5ab..2c86e2fe3b0 100644 --- a/src/lib/actions/sandbox/rebuild-resume-config.test.ts +++ b/src/lib/actions/sandbox/rebuild-resume-config.test.ts @@ -1,11 +1,13 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; import { HOSTED_NATIVE_PROVIDERS, hostedNativeProvider, } from "../../inference/native-provider/hosted"; import { createRequire } from "node:module"; +import { nativeLocalIdentity } from "../../inference/native-local/contract"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -1003,3 +1005,89 @@ describe("fixed hosted rebuild authority", () => { ).toThrow(/native|Native/); }); }); + +it.each([ + { agent: "openclaw" as const, api: "openai-completions" as const }, + { agent: "openclaw" as const, api: "openai-responses" as const }, + { agent: "hermes" as const, api: "openai-completions" as const }, + { agent: "hermes" as const, api: "openai-responses" as const }, +])( + "preserves the compatible $agent $api attachment and proxy owner across rebuild handoff", + ({ agent, api }) => { + vi.spyOn(onboardSession, "loadSession").mockReturnValue(null); + const binding = { + agent, + provider: "compatible-endpoint" as const, + endpointUrl: "http://host.openshell.internal:11435/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated" as const, + gatewayName: "nemoclaw", + sandboxName: "alpha", + ...(api === "openai-responses" ? { api } : {}), + }; + const attachment = { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1 as const, + providerId: "owned-provider", + }; + const result = prepareRebuildResumeConfig( + "alpha", + entry({ + agent, + provider: "compatible-endpoint", + model: "served-model", + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN", + preferredInferenceApi: api, + nativeLocalProviderAttachment: attachment, + }), + agent, + noopLog, + throwingBail, + ); + expect(result).toMatchObject({ + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN", + preferredInferenceApi: api, + nativeLocalProviderAttachment: attachment, + }); + }, +); + +it("retains the external inference binding without borrowing an unrelated resume session", () => { + const f = registeredInferenceFixture(); + vi.spyOn(onboardSession, "loadSession").mockReturnValue({ + sandboxName: "other", + provider: "openai-api", + model: "other", + endpointUrl: "https://other.example/v1", + }); + const selected = entry({ + gatewayName: f.binding.gatewayName, + provider: f.binding.providerName, + model: "model-a", + endpointUrl: f.binding.endpointUrl, + credentialEnv: f.binding.credentialEnv, + preferredInferenceApi: f.binding.inferenceApi, + registeredInferenceProviderAttachment: f.binding, + }); + expect( + prepareRebuildResumeConfig("alpha", selected, "openclaw", noopLog, throwingBail), + ).toMatchObject({ + provider: f.binding.providerName, + endpointUrl: f.binding.endpointUrl, + credentialEnv: f.binding.credentialEnv, + preferredInferenceApi: f.binding.inferenceApi, + registeredInferenceProviderAttachment: f.binding, + }); + expect(() => + prepareRebuildResumeConfig( + "alpha", + { ...selected, endpointUrl: "https://other.example/v1" }, + "openclaw", + noopLog, + throwingBail, + ), + ).toThrow("sandbox is untouched"); +}); diff --git a/src/lib/actions/sandbox/rebuild-resume-config.ts b/src/lib/actions/sandbox/rebuild-resume-config.ts index aeeaadbe161..ada2b22ca29 100644 --- a/src/lib/actions/sandbox/rebuild-resume-config.ts +++ b/src/lib/actions/sandbox/rebuild-resume-config.ts @@ -13,6 +13,10 @@ import { normalizeNativeLocalProviderAttachment, type NativeLocalProviderAttachment, } from "../../inference/native-local/contract"; +import { + recordedRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; import { CLI_NAME } from "../../cli/branding"; import { RD as _RD, D, R } from "../../cli/terminal-style"; import { @@ -53,6 +57,7 @@ const hermesProviderAuth = require("../../hermes-provider-auth") as { * from ambient selection env. */ export interface RebuildResumeConfig { + readonly registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; readonly nativeNvidiaProviderAttachment?: NativeNvidiaProviderAttachment; readonly nativeHostedProviderAttachment?: NativeProviderAttachment; readonly agent: string | null; @@ -107,7 +112,9 @@ export function prepareRebuildResumeConfig( bail: (msg: string, code?: number) => never, ): RebuildResumeConfig | null { let nativeAttachment; + let registeredInferenceProviderAttachment; try { + registeredInferenceProviderAttachment = recordedRegisteredInferenceProviderAttachment(sb); nativeAttachment = recordedNativeProviderAttachment({ ...sb, provider: sb.provider }); } catch { bail("Malformed native provider attachment; sandbox is untouched"); @@ -212,6 +219,7 @@ export function prepareRebuildResumeConfig( !isLocalInferenceProvider(trustedSelection.provider) && trustedSelection.provider !== hermesProviderAuth.HERMES_PROVIDER_NAME && !rebuildEndpoint.known && + !registeredInferenceProviderAttachment && !explicitTargetEndpoint ) { console.error(""); @@ -238,7 +246,9 @@ export function prepareRebuildResumeConfig( // not this sandbox, after exact sandbox/provider/model checks and URL // canonicalization. // 3. The target sandbox's own matching session endpoint, validated below. - let endpointUrl = rebuildEndpoint.known ? rebuildEndpoint.endpointUrl : explicitTargetEndpoint; + let endpointUrl = + registeredInferenceProviderAttachment?.endpointUrl ?? + (rebuildEndpoint.known ? rebuildEndpoint.endpointUrl : explicitTargetEndpoint); if ( !endpointUrl && !rebuildEndpoint.known && @@ -285,6 +295,7 @@ export function prepareRebuildResumeConfig( return null; } const credentialEnv = + registeredInferenceProviderAttachment?.credentialEnv ?? hostedDefinition?.custom?.credentialEnv ?? hostedDefinition?.credentialEnv ?? getRebuildCredentialEnvFromRegistry( @@ -294,6 +305,7 @@ export function prepareRebuildResumeConfig( ); return { + ...(registeredInferenceProviderAttachment ? { registeredInferenceProviderAttachment } : {}), ...(nativeNvidiaProviderAttachment ? { nativeNvidiaProviderAttachment } : {}), ...(nativeHostedProviderAttachment ? { nativeHostedProviderAttachment } : {}), agent: rebuildAgent, diff --git a/src/lib/actions/sandbox/rebuild-route-preflight.test.ts b/src/lib/actions/sandbox/rebuild-route-preflight.test.ts index 8a6cd60b12e..8dfa7c46bd8 100644 --- a/src/lib/actions/sandbox/rebuild-route-preflight.test.ts +++ b/src/lib/actions/sandbox/rebuild-route-preflight.test.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; + import { describe, expect, it, vi } from "vitest"; import type { SandboxEntry, SandboxRegistry } from "../../state/registry/types"; @@ -469,3 +471,33 @@ describe("revalidateRebuildRouteBeforeDelete", () => { }); }); }); + +it("keeps external rebuild independent while retaining damaged shared-peer denial", () => { + const f = registeredInferenceFixture(); + const binding = { ...f.binding, gatewayName: "nemoclaw", sandboxName: "target" }; + const target = sandbox("target", binding.providerName, { + endpointUrl: binding.endpointUrl, + credentialEnv: binding.credentialEnv, + preferredInferenceApi: binding.inferenceApi, + registeredInferenceProviderAttachment: binding, + }); + const peer = sandbox("peer", "compatible-endpoint", { credentialEnv: "COMPATIBLE_API_KEY" }); + const valid = transactionDependencies(registry(target, peer)); + expect( + commitRebuildRoutePreflight( + { sandboxName: "target", gatewayName: "nemoclaw", targetUpdate: targetUpdate(target) }, + valid.dependencies, + ), + ).toMatchObject({ ok: true, receipt: { migratedSandboxNames: [] } }); + expect(valid.persisted().sandboxes.peer).toEqual(peer); + const damaged = transactionDependencies( + registry(target, { ...peer, endpointUrl: null, credentialEnv: null }), + ); + expect( + commitRebuildRoutePreflight( + { sandboxName: "target", gatewayName: "nemoclaw", targetUpdate: targetUpdate(target) }, + damaged.dependencies, + ), + ).toMatchObject({ ok: false }); + expect(damaged.save).not.toHaveBeenCalled(); +}); diff --git a/src/lib/actions/sandbox/rebuild-target-runtime.test.ts b/src/lib/actions/sandbox/rebuild-target-runtime.test.ts index 594f9a38ec5..a5ad6bbaa30 100644 --- a/src/lib/actions/sandbox/rebuild-target-runtime.test.ts +++ b/src/lib/actions/sandbox/rebuild-target-runtime.test.ts @@ -305,6 +305,7 @@ describe("authoritative rebuild readiness", () => { type: receipt.profileId, credentialKeys: [binding.credentialEnv], configKeys: [], + credentialExpiresAtMs: {}, revision: { id: "owned", resourceVersion: 1 }, }, })), @@ -353,6 +354,11 @@ describe("authoritative rebuild readiness", () => { expect(recreateOptions.rebuildGatewayAuthority).toBe(authority); expect(readiness).toHaveBeenCalledOnce(); expect(route).not.toHaveBeenCalled(); + expect(providerAdapter.getProvider).toHaveBeenCalledWith({ + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: receipt.providerName, + includeCredentialExpirations: true, + }); expect(providerAdapter.getProvider).toHaveBeenCalledWith({ target: { kind: "named", gatewayName: "nemoclaw" }, providerName: receipt.providerName, diff --git a/src/lib/actions/sandbox/rebuild-target-runtime.ts b/src/lib/actions/sandbox/rebuild-target-runtime.ts index 69dd719eef5..73de37b5dc9 100644 --- a/src/lib/actions/sandbox/rebuild-target-runtime.ts +++ b/src/lib/actions/sandbox/rebuild-target-runtime.ts @@ -123,6 +123,7 @@ export async function preflightRebuildTargetHostCredential( target: RebuildTargetConfig, bail: RebuildBail, ): Promise { + if (target.resumeConfig.registeredInferenceProviderAttachment) return true; const credentialEnv = target.credentialEnv; return preflightRebuildHostCredential( { ...target.resumeConfig, credentialEnv }, @@ -304,6 +305,7 @@ export async function preflightAuthoritativeOnboardRuntime( ...recreateOptions, ...options, agent: resumeConfig.agent, + registeredInferenceProviderAttachment: resumeConfig.registeredInferenceProviderAttachment, nativeLocalProviderAttachment: resumeConfig.nativeLocalProviderAttachment, model: resumeConfig.model, provider: resumeConfig.provider, diff --git a/src/lib/actions/sandbox/rebuild/host-credential-preflight.ts b/src/lib/actions/sandbox/rebuild/host-credential-preflight.ts new file mode 100644 index 00000000000..879b0c0d547 --- /dev/null +++ b/src/lib/actions/sandbox/rebuild/host-credential-preflight.ts @@ -0,0 +1,130 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { validateNvidiaApiKeyValue } from "../../../credentials/api-key-validation"; +import { getCompatibleAnthropicOpenAiSurfaceBaseUrl } from "../../../inference/anthropic/openai-surface"; +import { + assertEndpointResolvesPublic, + parseTrustedPrivateInferenceHostsFromEnv, +} from "../../../inference/endpoint-ssrf-preflight"; +import { usesNvidiaEndpointProbePayload } from "../../../inference/openai-probe-models"; +import { RETRIABLE_HTTP_PROBE_STATUSES } from "../../../inference/probe/transient-http-policy"; +import { getRemoteProviderConfigForName } from "../../../onboard/inference-providers/provider-selection-keys"; +import { isBedrockRuntimeEndpoint } from "../../../inference/bedrock-runtime"; +import type { RebuildSandboxEntry } from "../rebuild-flow-helpers"; + +const { REMOTE_PROVIDER_CONFIG } = require("../../../onboard/providers") as { + REMOTE_PROVIDER_CONFIG: Record< + string, + { + providerName: string; + providerType: string; + endpointUrl: string; + credentialEnv: string | null; + } + >; +}; + +type ProbeResult = { ok: boolean; validated?: boolean; failures?: unknown[] }; + +function transientNvidiaValidationFailure(result: ProbeResult): boolean { + return ( + Array.isArray(result.failures) && + result.failures.length > 0 && + result.failures.every((failure) => { + if (typeof failure !== "object" || failure === null) return false; + const { httpStatus, curlStatus } = failure as Record; + return ( + (curlStatus === 0 && + typeof httpStatus === "number" && + RETRIABLE_HTTP_PROBE_STATUSES.has(httpStatus)) || + ((curlStatus === 28 || curlStatus === -110) && + (httpStatus === 0 || httpStatus === null || httpStatus === undefined)) + ); + }) + ); +} +type Probe = ( + endpointUrl: string, + model: string, + apiKey: string, + options: Record, +) => ProbeResult | Promise; + +const probes = require("../../../inference/onboard-probes") as { + probeAnthropicEndpoint: Probe; + probeOpenAiLikeEndpointOptimized: Probe; + getProbeExtraHeaders(provider: string): string[] | undefined; +}; + +export type HostCredentialTarget = Pick< + RebuildSandboxEntry, + "provider" | "model" | "endpointUrl" | "credentialEnv" | "preferredInferenceApi" +>; + +/** Validate the host credential that recreate can submit, without changing the gateway. */ +export async function validateRebuildHostInferenceCredential( + target: HostCredentialTarget, + credentialValue: string, + deps = { + assertEndpointResolvesPublic, + probeAnthropicEndpoint: probes.probeAnthropicEndpoint, + probeOpenAiLikeEndpointOptimized: probes.probeOpenAiLikeEndpointOptimized, + }, + onIndeterminate?: () => void, +): Promise { + const config = getRemoteProviderConfigForName(target.provider, REMOTE_PROVIDER_CONFIG); + // Local, routed, and Hermes authentication retain their existing preflights. + if (!config || config.providerName === "llama-cpp-local" || target.provider === "hermes-provider") + return true; + if ( + target.provider === "compatible-anthropic-endpoint" && + isBedrockRuntimeEndpoint(target.endpointUrl) + ) { + return true; + } + if (validateNvidiaApiKeyValue(credentialValue, target.credentialEnv ?? "")) return false; + let endpointUrl = config.endpointUrl || target.endpointUrl; + if (!endpointUrl || !target.model) return false; + + try { + const pins = !config.endpointUrl + ? await deps.assertEndpointResolvesPublic(endpointUrl, undefined, { + trustedPrivateHosts: parseTrustedPrivateInferenceHostsFromEnv(process.env), + }) + : null; + if (pins && !pins.ok) return false; + const useAnthropic = + config.providerType === "anthropic" && target.preferredInferenceApi !== "openai-completions"; + if (config.providerType === "anthropic" && !useAnthropic) { + endpointUrl = getCompatibleAnthropicOpenAiSurfaceBaseUrl(endpointUrl); + } + const probe = useAnthropic + ? deps.probeAnthropicEndpoint + : deps.probeOpenAiLikeEndpointOptimized; + const result = await probe(endpointUrl, target.model, credentialValue, { + skipResponsesProbe: target.preferredInferenceApi !== "openai-responses", + probeStreaming: false, + provider: target.provider, + extraHeaders: probes.getProbeExtraHeaders(target.provider ?? ""), + useNvidiaEndpointProbePayload: usesNvidiaEndpointProbePayload(target.provider ?? ""), + ...(pins?.ok + ? { + pinnedAddresses: pins.addresses, + trustedPrivateCapability: pins.trustedPrivateCapability, + } + : {}), + }); + if ( + !result.ok && + usesNvidiaEndpointProbePayload(target.provider ?? "") && + transientNvidiaValidationFailure(result) + ) { + onIndeterminate?.(); + } + return result.ok && result.validated !== false; + } catch { + // Probe errors and response bodies can contain credentials. + return false; + } +} diff --git a/src/lib/actions/sandbox/start.test.ts b/src/lib/actions/sandbox/start.test.ts index 68cb2518bc4..0e94a9ee490 100644 --- a/src/lib/actions/sandbox/start.test.ts +++ b/src/lib/actions/sandbox/start.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -1347,3 +1348,46 @@ describe("startSandbox persisted native inference selection", () => { }, ); }); + +it.each(["valid", "replaced"] as const)( + "starts a recorded external provider only after identity verification: %s", + async (state) => { + const f = registeredInferenceFixture(true); + const binding = { ...f.binding, sandboxName: "my-sandbox" }; + const verifyRegisteredAttachment = vi.fn( + state === "replaced" ? async () => Promise.reject(new Error("replaced")) : async () => {}, + ); + const probeInferenceInvocation = vi.fn< + NonNullable + >(async () => ({ ok: true })); + const h = harness({ verifyRegisteredAttachment, probeInferenceInvocation }); + h.getSandbox.mockReturnValue( + sandbox({ + stopped: true, + gatewayName: binding.gatewayName, + agent: "openclaw", + provider: binding.providerName, + model: "model-a", + endpointUrl: binding.endpointUrl, + credentialEnv: binding.credentialEnv, + preferredInferenceApi: binding.inferenceApi, + registeredInferenceProviderAttachment: binding, + }), + ); + expect((await startSandbox("my-sandbox", h.deps)).exitCode).toBe(state === "valid" ? 0 : 1); + expect(verifyRegisteredAttachment).toHaveBeenCalledWith({ + sandboxName: "my-sandbox", + gatewayName: binding.gatewayName, + expected: binding, + }); + expect(probeInferenceInvocation).toHaveBeenCalledTimes(state === "valid" ? 1 : 0); + expect(probeInferenceInvocation.mock.calls[0]?.[0]).toEqual( + state === "valid" + ? expect.objectContaining({ + registeredInferenceProviderAttachment: binding, + nativeEndpointUrl: binding.endpointUrl, + }) + : undefined, + ); + }, +); diff --git a/src/lib/actions/sandbox/start.ts b/src/lib/actions/sandbox/start.ts index 8f4192d6124..339475757eb 100644 --- a/src/lib/actions/sandbox/start.ts +++ b/src/lib/actions/sandbox/start.ts @@ -2,6 +2,10 @@ // SPDX-License-Identifier: Apache-2.0 import { setTimeout as sleep } from "node:timers/promises"; +import { + recordedRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; import type { OpenShellSandboxObserver } from "../../adapters/openshell/sandbox-observer"; import { retryUntilAsync } from "../../core/retry"; @@ -30,7 +34,10 @@ import { READINESS_INFERENCE_INVOCATION_TIMEOUT_MS, type SandboxInferenceInvocationResult, } from "./inference-invocation-probe"; -import { isTransientInferenceInvocationFailure } from "./inference-route-health"; +import { + isTransientInferenceInvocationFailure, + verifyRegisteredStatusAttachment, +} from "./inference-route-health"; import { hasRecordedNativeNvidiaAttachment } from "./launch-readiness/health"; import { hermesPortableLifecycleLockOptions, withSandboxLifecycleLock } from "./gateway-state"; import { getPersistedSandboxTargetGatewayName } from "./gateway-target"; @@ -92,6 +99,7 @@ export interface SandboxStartDeps extends StandardSandboxLifecycleDeps { delayGatewayProcessProbe?: (delayMs: number) => Promise; now?: () => number; probeInferenceInvocation?: typeof probeSandboxInferenceInvocation; + verifyRegisteredAttachment?: typeof verifyRegisteredStatusAttachment; qualifyLegacyPortableProfile?: typeof qualifyLegacyHermesPortableLifecycleProfile; recoverPortableSandbox?: typeof recoverPortableAgentSandboxLifecycle; requalifyPortableSandbox?: typeof requalifyPortableAgentSandboxAuthority; @@ -152,12 +160,26 @@ async function checkStartedSandboxInference( const model = (sandbox.model ?? "").trim(); const provider = (sandbox.provider ?? "").trim(); if (!model || !provider) return null; + let registered: RegisteredInferenceProviderAttachment | undefined; let nativeProvider: boolean; let nativeEndpointUrl: string | undefined; try { const receipt = recordedNativeProviderAttachment(sandbox); - nativeProvider = Boolean(receipt); - nativeEndpointUrl = receipt?.endpointUrl; + registered = recordedRegisteredInferenceProviderAttachment(sandbox); + if (registered) { + if ( + registered.sandboxName !== sandboxName || + registered.gatewayName !== getPersistedSandboxTargetGatewayName(sandbox) + ) + throw new Error("Registered inference target changed"); + await (deps.verifyRegisteredAttachment ?? verifyRegisteredStatusAttachment)({ + sandboxName, + gatewayName: registered.gatewayName, + expected: registered, + }); + } + nativeProvider = Boolean(receipt || registered); + nativeEndpointUrl = registered?.endpointUrl ?? receipt?.endpointUrl; } catch { return { ok: false, @@ -179,6 +201,7 @@ async function checkStartedSandboxInference( ...(sandbox.nativeLocalProviderAttachment ? { nativeLocalProviderAttachment: sandbox.nativeLocalProviderAttachment } : {}), + ...(registered ? { registeredInferenceProviderAttachment: registered } : {}), ...(nativeProvider ? { nativeProvider: true } : {}), ...(nativeEndpointUrl ? { nativeEndpointUrl } : {}), }; diff --git a/src/lib/actions/sandbox/status-flow.test.ts b/src/lib/actions/sandbox/status-flow.test.ts index 5bdb7d1d023..d9d1c0265aa 100644 --- a/src/lib/actions/sandbox/status-flow.test.ts +++ b/src/lib/actions/sandbox/status-flow.test.ts @@ -149,6 +149,8 @@ describe("showSandboxStatus flow", () => { const output = harness.logSpy.mock.calls.flat().join("\n"); expect(output).toContain("Inference: not verified (gateway/sandbox state not verified)"); + expect(output).toContain("Failure layer: gateway_unreachable"); + expect(harness.classifyGatewayFailureSpy).toHaveBeenCalledExactlyOnceWith("alpha"); expect(output).not.toContain( "Inference: not probed (another lifecycle operation is in progress)", ); diff --git a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts index 41564c127c3..268a0adcd96 100644 --- a/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts +++ b/src/lib/actions/sandbox/status-snapshot-inference-health.test.ts @@ -3,7 +3,6 @@ import { describe, expect, it, vi } from "vitest"; -import { nativeLocalIdentity } from "../../inference/native-local/contract"; import type { ProviderHealthStatus } from "../../inference/health"; import type { SandboxEntry } from "../../state/registry"; import type { SandboxInferenceInvocationResult } from "./inference-invocation-probe"; @@ -62,27 +61,20 @@ describe("collectSandboxStatusSnapshot inference route health", () => { it.each([true, false])( "reports native local attachment health without querying the shared route: %s (#12558)", async (attached) => { - const binding = { - provider: "ollama-local", - endpointUrl: "http://host.openshell.internal:11434/v1", - credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", - authMode: "sentinel", - gatewayName: "nemoclaw", - sandboxName: "alpha", - } as const; - const receipt = { - ...binding, - ...nativeLocalIdentity(binding), - schemaVersion: 1 as const, - providerId: "owned-local-provider", - }; + const receipt = nativeLocalTestReceipt( + "ollama-local", + "http://host.openshell.internal:11434/v1", + { + authMode: "sentinel", + }, + ); const options = snapshotDeps( null, null, { ok: true }, { - provider: binding.provider, - gatewayName: binding.gatewayName, + provider: receipt.provider, + gatewayName: receipt.gatewayName, nativeLocalProviderAttachment: receipt, }, ); @@ -107,7 +99,7 @@ describe("collectSandboxStatusSnapshot inference route health", () => { }); expect(snapshot.inferenceHealth).toMatchObject({ ok: attached, - endpoint: expect.stringContaining(binding.endpointUrl), + endpoint: expect.stringContaining(receipt.endpointUrl), }); expect(observeInferenceRoute).not.toHaveBeenCalled(); expect(verifyNativeLocalProviderAttachmentImpl).toHaveBeenCalledWith({ @@ -120,28 +112,21 @@ describe("collectSandboxStatusSnapshot inference route health", () => { ); it("reports bounded redacted native local verification details and recovery (#12558)", async () => { - const binding = { - provider: "ollama-local", - endpointUrl: "http://host.openshell.internal:11434/v1", - credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", - authMode: "sentinel", - gatewayName: "nemoclaw", - sandboxName: "alpha", - } as const; - const receipt = { - ...binding, - ...nativeLocalIdentity(binding), - schemaVersion: 1 as const, - providerId: "owned-local-provider", - }; + const receipt = nativeLocalTestReceipt( + "ollama-local", + "http://host.openshell.internal:11434/v1", + { + authMode: "sentinel", + }, + ); const secret = `nvapi-${"a".repeat(40)}`; const options = snapshotDeps( null, null, { ok: true }, { - provider: binding.provider, - gatewayName: binding.gatewayName, + provider: receipt.provider, + gatewayName: receipt.gatewayName, nativeLocalProviderAttachment: receipt, }, ); @@ -444,41 +429,23 @@ describe("collectSandboxStatusSnapshot inference route health", () => { ]); }); - it("does not set okLabel for a 5xx route failure, and classifies it unhealthy (#6846)", async () => { - const gateway: SandboxInferenceRouteHealth = { - ok: false, - endpoint: "https://inference.local/v1/models", - httpStatus: 502, - detail: - "Inference gateway returned HTTP 502 on https://inference.local/v1/models; the route is reachable but unhealthy.", - }; - - const snapshot = await collectSandboxStatusSnapshot("alpha", snapshotDeps(gateway)); - - expect(snapshot.inferenceHealth).toMatchObject({ - ok: false, - failureLabel: "unhealthy", - }); - expect(snapshot.inferenceHealth?.okLabel).toBeUndefined(); - }); - - it("does not set okLabel when the route is unreachable (#6846)", async () => { - const gateway: SandboxInferenceRouteHealth = { - ok: false, - endpoint: "https://inference.local/v1/models", - httpStatus: 0, - detail: - "Inference gateway unreachable on https://inference.local/v1/models from inside the sandbox.", - }; - - const snapshot = await collectSandboxStatusSnapshot("alpha", snapshotDeps(gateway)); - - expect(snapshot.inferenceHealth).toMatchObject({ - ok: false, - failureLabel: "unreachable", - }); - expect(snapshot.inferenceHealth?.okLabel).toBeUndefined(); - }); + it.each([ + [502, "unhealthy", "Inference gateway returned HTTP 502."], + [0, "unreachable", "Inference gateway unreachable."], + ] as const)( + "reports a failed route with no okLabel: HTTP %s (#6846)", + async (httpStatus, failureLabel, detail) => { + const gateway: SandboxInferenceRouteHealth = { + ok: false, + endpoint: "https://inference.local/v1/models", + httpStatus, + detail, + }; + const snapshot = await collectSandboxStatusSnapshot("alpha", snapshotDeps(gateway)); + expect(snapshot.inferenceHealth).toMatchObject({ ok: false, failureLabel }); + expect(snapshot.inferenceHealth?.okLabel).toBeUndefined(); + }, + ); it("stays unprobed with no okLabel when the gateway probe is unavailable (#6846)", async () => { const snapshot = await collectSandboxStatusSnapshot("alpha", snapshotDeps(null)); @@ -587,34 +554,6 @@ describe("collectSandboxStatusSnapshot inference route health", () => { expect(upstream?.detail).toContain("does not attribute this result to the sandbox route"); }); - it("keeps an unreachable upstream check after the route served a request (#9595)", async () => { - const gateway: SandboxInferenceRouteHealth = { - ok: true, - endpoint: "https://inference.local/v1/models", - httpStatus: 200, - detail: - "Inference gateway responded HTTP 200 on https://inference.local/v1/models (full chain reachable).", - }; - const providerHealth: ProviderHealthStatus = { - ok: false, - probed: true, - providerLabel: "NVIDIA Endpoints", - endpoint: "https://integrate.api.nvidia.com/v1/chat/completions", - detail: "model invocation probe could not reach the endpoint", - failureLabel: "unreachable", - }; - - const snapshot = await collectSandboxStatusSnapshot( - "alpha", - snapshotDeps(gateway, providerHealth), - ); - - expect(snapshot.inferenceHealth?.subprobes).toContainEqual({ - ...providerHealth, - probeLabel: "upstream", - }); - }); - it("keeps an unauthorized auth proxy subprobe after the route served a request (#9595)", async () => { const gateway: SandboxInferenceRouteHealth = { ok: true, @@ -1206,6 +1145,94 @@ describe("status native local request identity (#13338)", () => { ); }); +it.each([ + { state: "ready", agent: "openclaw" }, + { state: "ready", agent: "hermes" }, + { state: "detached", agent: "openclaw" }, + { state: "changed-binding", agent: "openclaw" }, + { state: "request-failed", agent: "openclaw" }, +])("verifies registered provider readiness: $state for $agent", async ({ state, agent }) => { + const binding = { + schemaVersion: 1 as const, + gatewayName: "nemoclaw", + sandboxName: "alpha", + providerName: "operator-provider", + providerId: "operator-id", + profileId: "operator-profile", + profileDigest: "a".repeat(64), + endpointUrl: "https://api.example.test/v1", + inferenceApi: "openai-completions" as const, + credentialEnv: "OPERATOR_API_KEY", + attachedBySelection: false, + }; + const sandbox = { + name: "alpha", + agent, + gatewayName: binding.gatewayName, + provider: binding.providerName, + model: "model-a", + endpointUrl: binding.endpointUrl, + preferredInferenceApi: binding.inferenceApi, + credentialEnv: binding.credentialEnv, + registeredInferenceProviderAttachment: binding, + }; + sandbox.credentialEnv = state === "changed-binding" ? "OTHER_KEY" : binding.credentialEnv; + const options = snapshotDeps(null, null, { ok: true }, sandbox); + const observeInferenceRoute = vi.fn(); + const verifyRegisteredProviderAttachmentImpl = vi.fn( + state === "detached" + ? async () => { + throw new Error("attachment missing"); + } + : async () => undefined, + ); + const probeSandboxInferenceInvocationImpl = vi.fn(async () => ({ + ok: state !== "request-failed", + httpStatus: state === "request-failed" ? 403 : 200, + detail: "request result", + })); + const snapshot = await collectSandboxStatusSnapshot("alpha", { + ...options, + deps: { + ...options.deps, + inferenceRouteObserver: { observeInferenceRoute }, + verifyRegisteredProviderAttachmentImpl, + probeSandboxInferenceInvocationImpl, + }, + }); + expect(snapshot.inferenceHealth?.ok).toBe(state === "ready"); + expect(snapshot.routeDrift).toBeNull(); + expect(observeInferenceRoute).not.toHaveBeenCalled(); + expect(verifyRegisteredProviderAttachmentImpl).toHaveBeenCalledTimes( + state === "changed-binding" ? 0 : 1, + ); + expect(probeSandboxInferenceInvocationImpl).toHaveBeenCalledTimes( + state === "ready" || state === "request-failed" ? 1 : 0, + ); + expect(probeSandboxInferenceInvocationImpl.mock.calls).toEqual( + state === "ready" || state === "request-failed" + ? [ + [ + expect.objectContaining({ + sandboxName: "alpha", + gatewayName: "nemoclaw", + agentName: agent, + provider: binding.providerName, + registeredInferenceProviderAttachment: binding, + }), + {}, + expect.any(Number), + ], + ] + : [], + ); + expect(snapshot.inferenceHealth?.endpoint).toBe( + state === "changed-binding" + ? "" + : binding.endpointUrl + (state === "detached" ? "" : "/chat/completions"), + ); +}); + describe("Hermes NVIDIA agent credential health", () => { const options = () => snapshotDeps( diff --git a/src/lib/actions/sandbox/status-snapshot.ts b/src/lib/actions/sandbox/status-snapshot.ts index d07b94e6c7f..e6c06eced72 100644 --- a/src/lib/actions/sandbox/status-snapshot.ts +++ b/src/lib/actions/sandbox/status-snapshot.ts @@ -63,6 +63,8 @@ import { HermesNvidiaCredentialVerifierMissingError, verifyHermesNvidiaCredential, readNativeStatusSelection, + verifyRegisteredStatusAttachment, + type VerifyRegisteredStatusAttachment, buildSandboxInferenceRouteHealth, isTransientInferenceInvocationFailure, type ProbeSandboxInferenceInvocation, @@ -371,6 +373,7 @@ interface CollectSandboxStatusSnapshotDeps { getSandboxStatusPreflightImpl?: typeof getSandboxStatusPreflight; getGatewayPresets?: GetGatewayPresets; inspectManagedLlamaCppOwnership?: typeof inspectManagedLlamaCppOwnership; + verifyRegisteredProviderAttachmentImpl?: VerifyRegisteredStatusAttachment; verifyNativeNvidiaProviderAttachmentImpl?: VerifyNativeNvidiaStatusAttachment; verifyNativeLocalProviderAttachmentImpl?: VerifyNativeLocalStatusAttachment; verifyNativeHostedProviderAttachmentImpl?: VerifyNativeHostedStatusAttachment; @@ -605,9 +608,15 @@ export async function collectSandboxStatusSnapshot( ((postRecoveryPreflight ?? initialPreflight)?.suppressInferenceProbe ?? opts.suppressInferenceProbe === true); const nativeSelection = readNativeStatusSelection(sb); - const { attachment: nativeAttachment, label: nativeLabel } = nativeSelection; + const { + attachment: nativeAttachment, + registeredAttachment, + label: nativeLabel, + } = nativeSelection; let nativeAttachmentFailure = nativeSelection.failure; - const nativeProvider = Boolean(nativeAttachment || nativeAttachmentFailure); + const nativeProvider = Boolean( + nativeAttachment || registeredAttachment || nativeAttachmentFailure, + ); let liveResult: OpenShellInferenceRouteResult | null = null; let gatewayName: string | null = null; if (lookup.state === "present") { @@ -667,8 +676,8 @@ export async function collectSandboxStatusSnapshot( sb ) { const expected = nativeAttachment; - if (!gatewayName || !expected) { - nativeAttachmentFailure = + if (!gatewayName || (!expected && !registeredAttachment)) { + nativeAttachmentFailure ??= `Native ${nativeLabel} provider attachment is unavailable for sandbox '${sandboxName}'. ` + `Recreate the sandbox to restore native ${nativeLabel} inference.`; } else { @@ -678,7 +687,15 @@ export async function collectSandboxStatusSnapshot( sb.nativeLocalProviderAttachment === undefined ? undefined : normalizeNativeLocalProviderAttachment(sb.nativeLocalProviderAttachment); - if (local) { + if (registeredAttachment) { + await ( + opts.deps?.verifyRegisteredProviderAttachmentImpl ?? verifyRegisteredStatusAttachment + )({ + gatewayName, + sandboxName, + expected: registeredAttachment, + }); + } else if (local) { await ( opts.deps?.verifyNativeLocalProviderAttachmentImpl ?? verifyNativeLocalStatusAttachment )({ @@ -695,7 +712,7 @@ export async function collectSandboxStatusSnapshot( ? { verify: opts.deps.verifyNativeNvidiaProviderAttachmentImpl } : {}), }); - } else { + } else if (expected) { await verifyNativeHostedStatusAttachment({ gatewayName, sandboxName, @@ -708,8 +725,11 @@ export async function collectSandboxStatusSnapshot( } catch (error) { const detail = sanitizedStatusDetail(error); nativeAttachmentFailure = - `Native ${nativeLabel} provider '${nativeAttachment.providerName}' is unavailable for sandbox '${sandboxName}'` + - `${detail ? `: ${detail}` : "."} Recreate the sandbox to restore native ${nativeLabel} inference.`; + `Native ${nativeLabel} provider '${(registeredAttachment ?? nativeAttachment)?.providerName}' is unavailable for sandbox '${sandboxName}'` + + `${detail ? `: ${detail}` : "."} ` + + (registeredAttachment + ? "Restore the operator provider and select it again for this sandbox." + : `Recreate the sandbox to restore native ${nativeLabel} inference.`); } } } @@ -812,11 +832,16 @@ export async function collectSandboxStatusSnapshot( { sandboxName, gatewayName: gatewayName ?? undefined, - ...(sb?.agent === "langchain-deepagents-code" ? { agentName: sb.agent } : {}), + ...(sb?.agent === "langchain-deepagents-code" || registeredAttachment + ? { agentName: sb?.agent } + : {}), provider: invocationProvider, model: invocationModel, preferredInferenceApi: invocationRoute.preferredInferenceApi, ...(nativeProvider ? { nativeProvider: true } : {}), + ...(registeredAttachment + ? { registeredInferenceProviderAttachment: registeredAttachment } + : {}), ...(sb?.nativeLocalProviderAttachment ? { nativeLocalProviderAttachment: sb.nativeLocalProviderAttachment } : {}), @@ -883,7 +908,8 @@ export async function collectSandboxStatusSnapshot( : buildSandboxInferenceRouteHealth(gatewayChain, providerHealth, invocation, { provider: invocationRoute.provider ?? null, nativeProvider, - nativeEndpointUrl: nativeAttachment?.endpointUrl, + nativeEndpointUrl: registeredAttachment?.endpointUrl ?? nativeAttachment?.endpointUrl, + registeredInferenceProviderAttachment: registeredAttachment, nativeLocalEndpoint: sb?.nativeLocalProviderAttachment === undefined ? undefined diff --git a/src/lib/actions/sandbox/stop.test.ts b/src/lib/actions/sandbox/stop.test.ts index ee9b381287b..f6a76b7c96f 100644 --- a/src/lib/actions/sandbox/stop.test.ts +++ b/src/lib/actions/sandbox/stop.test.ts @@ -468,7 +468,7 @@ describe("stopSandbox", () => { expect(result.message).toContain("release of its host forward ports could not be proved"); expect(h.getSandbox("my-sandbox")?.stopped).toBe(true); expect(h.warn).toHaveBeenCalledWith( - " Warning: a ForwardTcp port for 'my-sandbox' did not release. Retry 'nemoclaw my-sandbox stop'.", + " Warning: ForwardTcp port release for 'my-sandbox' could not be proved. Retry 'nemoclaw my-sandbox stop'.", ); }); diff --git a/src/lib/actions/sandbox/stop.ts b/src/lib/actions/sandbox/stop.ts index eae1b4a318f..063bd0489eb 100644 --- a/src/lib/actions/sandbox/stop.ts +++ b/src/lib/actions/sandbox/stop.ts @@ -51,7 +51,7 @@ async function teardownDashboardForwardBestEffort( try { if ((await teardown(sandboxName)) === false) { warn( - ` Warning: a ForwardTcp port for '${sandboxName}' did not release. Retry '${CLI_NAME} ${sandboxName} stop'.`, + ` Warning: ForwardTcp port release for '${sandboxName}' could not be proved. Retry '${CLI_NAME} ${sandboxName} stop'.`, ); return false; } diff --git a/src/lib/actions/uninstall/agent-state-volume-cleanup.ts b/src/lib/actions/uninstall/agent-state-volume-cleanup.ts index 5d66bcc1c44..8eabdc1db32 100644 --- a/src/lib/actions/uninstall/agent-state-volume-cleanup.ts +++ b/src/lib/actions/uninstall/agent-state-volume-cleanup.ts @@ -6,6 +6,7 @@ import { normalizeRuntimeProviderIdentity } from "../../onboard/runtime-provider import { CURRENT_RUNTIME_PROVIDER_BUNDLES } from "../../onboard/runtime-provider/current"; import type { RuntimeProviderBundleRegistry } from "../../onboard/runtime-provider/contract"; import { removeManagedAgentStateVolumes } from "../../onboard/sandbox-provider-cleanup"; +import { OPENSHELL_DEFAULT_WORKSPACE } from "../../adapters/openshell/sandbox-ssh-host"; export type { ManagedAgentStateVolumeContext }; @@ -58,6 +59,7 @@ export function removeManagedAgentStateVolumesForUninstall( ): boolean { for (const context of contexts) { const results = removeManagedAgentStateVolumes(context, { + workspace: runtime.env.OPENSHELL_WORKSPACE ?? OPENSHELL_DEFAULT_WORKSPACE, runtimeProviders: runtime.runtimeProviders ?? CURRENT_RUNTIME_PROVIDER_BUNDLES, runDocker: (args, options) => runtime.runDocker(["volume", ...args], { diff --git a/src/lib/actions/uninstall/portable-runtime-cleanup.test.ts b/src/lib/actions/uninstall/portable-runtime-cleanup.test.ts index 06dc675e371..4a8fd336b9d 100644 --- a/src/lib/actions/uninstall/portable-runtime-cleanup.test.ts +++ b/src/lib/actions/uninstall/portable-runtime-cleanup.test.ts @@ -901,8 +901,22 @@ describe("portable runtime uninstall cleanup", () => { }, ); - it("removes only exact receipt-owned containers and exact current selector projections (#9189)", async () => { - const test = fixture(); + it.each([ + { name: "legacy", setup: fixture }, + { + name: "paired", + setup: () => { + const test = fixture(); + test.containers.get(ALPHA_ID)!.labels["openshell.ai/isolation-role"] = "sandbox"; + test.addSandbox("alpha", "sandbox-alpha", BETA_ID, { + "openshell.ai/isolation-role": "supervisor", + }); + test.containers.get(BETA_ID)!.name = "openshell-supervisor-sandbox-alpha"; + return test; + }, + }, + ])("cleans receipt-owned runtime ($name)", async ({ setup }) => { + const test = setup(); expect(hasPortableRuntimeCleanup(test.stateDir)).toBe(true); const cleanup = await completeCleanup(test.input, test.deps); @@ -943,7 +957,9 @@ describe("portable runtime uninstall cleanup", () => { sandboxContainersRemoved: 1, selectorsRemoved: ["CONTAINERS_CONF", "NETAVARK_FW"], }); - expect(observations()).toBe(4); + // Initial read plus one retry, two pre-removal identity checks, and + // absence checks in both the removal helper and the transaction. + expect(observations()).toBe(6); expect(test.containers.has(ALPHA_ID)).toBe(false); }); diff --git a/src/lib/actions/uninstall/run-plan-agent-state-volume.test.ts b/src/lib/actions/uninstall/run-plan-agent-state-volume.test.ts index 4d972cb6ecc..08dbf82376b 100644 --- a/src/lib/actions/uninstall/run-plan-agent-state-volume.test.ts +++ b/src/lib/actions/uninstall/run-plan-agent-state-volume.test.ts @@ -2,247 +2,45 @@ // SPDX-License-Identifier: Apache-2.0 import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; - import { describe, expect, it, vi } from "vitest"; import type { PodmanBoundContainerEngine } from "../../adapters/podman"; import { createHermesStateVolumeDockerHarness } from "../../onboard/__test-helpers__/hermes-state-volume"; -import { createDockerRuntimeProviderBundle } from "../../onboard/runtime-provider/docker"; import { createPodmanRuntimeProviderBundle } from "../../onboard/runtime-provider/podman"; import { createRuntimeProviderBundleRegistry } from "../../onboard/runtime-provider/registry"; import { removeManagedAgentStateVolumesForUninstall } from "./agent-state-volume-cleanup"; -import { withSuccessfulPreUninstallBackup } from "../../../../test/support/uninstall-managed-gateway-test-support"; - -import { - type RunResult, - runUninstallPlanProduction as runUninstallPlanBase, - type UninstallRunDeps, - type UninstallRunOptions, -} from "./run-plan"; - -const MANAGED_HERMES_VOLUME_LABELS = { - "io.nvidia.nemoclaw.hermes-state.managed": "true", - "io.nvidia.nemoclaw.hermes-state.sandbox": "hermes", - "io.nvidia.nemoclaw.hermes-state.schema": "1", - "io.nvidia.nemoclaw.hermes-state.target": "/sandbox/.hermes", -}; - -const MANAGED_OPENCLAW_VOLUME_LABELS = { - "io.nvidia.nemoclaw.openclaw-state.managed": "true", - "io.nvidia.nemoclaw.openclaw-state.sandbox": "alpha", - "io.nvidia.nemoclaw.openclaw-state.schema": "1", - "io.nvidia.nemoclaw.openclaw-state.target": "/sandbox/.openclaw", -}; - -function ok(stdout = ""): RunResult { - return { status: 0, stdout, stderr: "" }; -} - -async function runUninstallPlan(options: UninstallRunOptions, deps: UninstallRunDeps) { - return await runUninstallPlanBase( - options, - withSuccessfulPreUninstallBackup({ - resolveGatewayTeardownAuthority: ({ gatewayName, gatewayPort }) => ({ - gatewayName, - gatewayPort, - mode: "nemoclaw-managed", - source: gatewayPort === 8080 ? "packaged-service" : "standalone", - endpoint: null, - stateDir: null, - supervisor: null, - requiredCapabilities: [], - }), - ...deps, - }), - ); -} - -async function runManagedStateVolumeUninstall( - mode: "foreign" | "owned" | "remove-fails" | "remove-fails-once", - destroyUserData: boolean, - containerMode: "absent" | "foreign" | "owned" = "absent", - agent: "hermes" | "openclaw" = "hermes", - registryAgent: string | null = agent, -) { - const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-uninstall-hermes-volume-")); - const stateDir = path.join(home, ".nemoclaw"); - const registryFile = path.join(stateDir, "sandboxes.json"); - const sandboxName = agent === "openclaw" ? "alpha" : "hermes"; - const volumeName = `nemoclaw-${agent}-state-v1-${sandboxName}`; - fs.mkdirSync(stateDir, { recursive: true }); - fs.writeFileSync( - registryFile, - JSON.stringify({ - defaultSandbox: sandboxName, - sandboxes: { - [sandboxName]: { - agent: registryAgent, - gatewayName: "nemoclaw", - gatewayPort: 8080, - name: sandboxName, - openshellDriver: "docker", - workload: { kind: "managed-image" }, - }, - }, - }), - ); - - const dockerCalls: string[][] = []; - const errors: string[] = []; - const events: string[] = []; - const logs: string[] = []; - const containerId = "hermes-sandbox-container"; - let containerPresent = containerMode !== "absent"; - let gatewayRegistered = true; - let volumeRemovalAttempts = 0; - let volumePresent = true; - const run = vi.fn((command: string, args: string[]) => { - events.push(`${command} ${args.join(" ")}`); - switch (`${command} ${args.join(" ")}`) { - case "openshell gateway list": - case "openshell gateway list -o json": - return ok(JSON.stringify(gatewayRegistered ? [{ name: "nemoclaw", active: false }] : [])); - case "openshell gateway remove nemoclaw": - gatewayRegistered = false; - return ok(); - case "openshell sandbox delete --all": - return gatewayRegistered - ? ok() - : { status: 1, stdout: "", stderr: "Unknown gateway 'nemoclaw'" }; - default: - return ok(); - } - }); - const runDocker = vi.fn((args: string[]) => { - dockerCalls.push(args); - events.push(`docker ${args.join(" ")}`); - switch (args.join(" ")) { - case "ps -a --format {{.ID}} {{.Image}} {{.Names}}": - switch (containerPresent ? containerMode : "absent") { - case "owned": - return ok( - `${containerId} ghcr.io/nvidia/nemoclaw/${agent}-sandbox:latest openshell-default--${sandboxName}-runtime-id\n`, - ); - case "foreign": - return ok(`${containerId} redis:7 foreign-service\n`); - default: - return ok(); - } - case `rm -f ${containerId}`: - containerPresent = false; - return ok(`${containerId}\n`); - case `volume inspect --format {{json .}} ${volumeName}`: - return volumePresent - ? ok( - `${JSON.stringify({ - Labels: - mode === "foreign" - ? { "com.example.owner": "foreign" } - : agent === "openclaw" - ? MANAGED_OPENCLAW_VOLUME_LABELS - : MANAGED_HERMES_VOLUME_LABELS, - Name: volumeName, - })}\n`, - ) - : { status: 1, stdout: "", stderr: `Error: no such volume: ${volumeName}` }; - case `volume rm ${volumeName}`: - switch (containerPresent ? "attached" : mode) { - case "attached": - case "remove-fails": - return { status: 1, stdout: "", stderr: "volume is still in use" }; - case "remove-fails-once": - volumeRemovalAttempts += 1; - switch (volumeRemovalAttempts) { - case 1: - return { status: 1, stdout: "", stderr: "volume is still in use" }; - default: - volumePresent = false; - return ok(`${volumeName}\n`); - } - default: - volumePresent = false; - return ok(`${volumeName}\n`); - } - case "volume inspect openshell-cluster-nemoclaw": - return { status: 1, stdout: "", stderr: "Error: no such volume" }; - default: - return ok(); - } - }); +import { runManagedHermesVolumeUninstall } from "../../../../test/support/uninstall-state-volume"; - const execute = () => - runUninstallPlan( - { assumeYes: true, deleteModels: false, destroyUserData, keepOpenShell: false }, - { - commandExists: () => true, - env: { HOME: home } as NodeJS.ProcessEnv, - error: (line) => errors.push(line), - existsSync: (target) => target.startsWith(home) && fs.existsSync(target), - hasPortableRuntimeCleanup: () => false, - isTty: false, - kill: () => true, - log: (line) => logs.push(line), - rmSync: fs.rmSync, - run, - runDocker, - runtimeProviders: createRuntimeProviderBundleRegistry([ - [ - "docker", - createDockerRuntimeProviderBundle({ - captureHostCommand: (_command, args) => { - const result = runDocker(args); - return { - status: result.status ?? 1, - stdout: result.stdout, - stderr: result.stderr, - }; - }, - }), - ], - ]), - }, - ); - const result = await execute(); - - return { - cleanup: () => fs.rmSync(home, { force: true, recursive: true }), - containerId, - containerPresent: () => containerPresent, - dockerCalls, - errors, - events, - gatewayRegistered: () => gatewayRegistered, - logs, - registryFile, - result, - retry: execute, - volumeName, - volumePresent: () => volumePresent, - }; -} - -describe("managed Hermes state volume uninstall", () => { +describe("managed agent state volume uninstall", () => { it.each([ - ["local uninstall", false], - ["destructive uninstall", true], - ])("removes an owned volume during %s", async (_label, destroyUserData) => { - const harness = await runManagedStateVolumeUninstall("owned", destroyUserData); - try { - expect(harness.result.exitCode, harness.errors.join("\n")).toBe(0); - expect(harness.volumePresent()).toBe(false); - expect(harness.dockerCalls).toContainEqual(["volume", "rm", harness.volumeName]); - expect(harness.events.indexOf("openshell sandbox delete --all")).toBeLessThan( - harness.events.indexOf(`docker volume rm ${harness.volumeName}`), + ["hermes", false], + ["hermes", true], + ["openclaw", false], + ["openclaw", true], + ] as const)( + "removes an owned %s volume with destroy-user-data=%s", + async (agent, destroyUserData) => { + const harness = await runManagedHermesVolumeUninstall( + "owned", + destroyUserData, + "absent", + agent, ); - expect(harness.logs).toContain("Removed managed agent state volume for 'hermes'."); - } finally { - harness.cleanup(); - } - }); + try { + expect(harness.result.exitCode, harness.errors.join("\n")).toBe(0); + expect(harness.volumePresent()).toBe(false); + expect(harness.dockerCalls).toContainEqual(["volume", "rm", harness.volumeName]); + expect(harness.events.indexOf("openshell sandbox delete --all")).toBeLessThan( + harness.events.indexOf(`docker volume rm ${harness.volumeName}`), + ); + expect(harness.logs).toContain("Removed managed agent state volume for 'hermes'."); + } finally { + harness.cleanup(); + } + }, + ); it("keeps a same-name foreign volume", async () => { - const harness = await runManagedStateVolumeUninstall("foreign", true); + const harness = await runManagedHermesVolumeUninstall("foreign", true); try { expect(harness.result.exitCode, harness.errors.join("\n")).toBe(0); expect(harness.volumePresent()).toBe(true); @@ -256,7 +54,7 @@ describe("managed Hermes state volume uninstall", () => { }); it("preserves uninstall state when volume removal fails", async () => { - const harness = await runManagedStateVolumeUninstall("remove-fails", true); + const harness = await runManagedHermesVolumeUninstall("remove-fails", true); try { expect(harness.result.exitCode).toBe(1); expect(harness.volumePresent()).toBe(true); @@ -272,7 +70,7 @@ describe("managed Hermes state volume uninstall", () => { }); it("retries volume cleanup before removing the gateway registration", async () => { - const harness = await runManagedStateVolumeUninstall("remove-fails-once", true); + const harness = await runManagedHermesVolumeUninstall("remove-fails-once", true); try { expect(harness.result.exitCode).toBe(1); expect(harness.gatewayRegistered()).toBe(true); @@ -291,7 +89,7 @@ describe("managed Hermes state volume uninstall", () => { }); it("preserves a stopped sandbox and its state when the owning runtime leaves it behind", async () => { - const harness = await runManagedStateVolumeUninstall("owned", true, "owned"); + const harness = await runManagedHermesVolumeUninstall("owned", true, "owned"); try { expect(harness.result.exitCode).toBe(1); expect(harness.containerPresent()).toBe(true); @@ -306,7 +104,7 @@ describe("managed Hermes state volume uninstall", () => { }); it("keeps a foreign container that prevents Hermes state volume removal", async () => { - const harness = await runManagedStateVolumeUninstall("owned", true, "foreign"); + const harness = await runManagedHermesVolumeUninstall("owned", true, "foreign"); try { expect(harness.result.exitCode).toBe(1); expect(harness.containerPresent()).toBe(true); @@ -319,6 +117,20 @@ describe("managed Hermes state volume uninstall", () => { }); }); +const MANAGED_HERMES_VOLUME_LABELS = { + "io.nvidia.nemoclaw.hermes-state.managed": "true", + "io.nvidia.nemoclaw.hermes-state.sandbox": "hermes", + "io.nvidia.nemoclaw.hermes-state.schema": "1", + "io.nvidia.nemoclaw.hermes-state.target": "/sandbox/.hermes", +}; + +const MANAGED_OPENCLAW_VOLUME_LABELS = { + "io.nvidia.nemoclaw.openclaw-state.managed": "true", + "io.nvidia.nemoclaw.openclaw-state.sandbox": "alpha", + "io.nvidia.nemoclaw.openclaw-state.schema": "1", + "io.nvidia.nemoclaw.openclaw-state.target": "/sandbox/.openclaw", +}; + const MANAGED_VOLUME_LABELS = { hermes: MANAGED_HERMES_VOLUME_LABELS, openclaw: MANAGED_OPENCLAW_VOLUME_LABELS, @@ -416,33 +228,15 @@ describe("managed agent state volume provider cleanup authority (#13595)", () => ); }); -it.each([ - ["hermes", false], - ["hermes", true], - ["openclaw", false], - ["openclaw", true], -] as const)( - "removes the declared %s state volume through the uninstall plan (destroyUserData=%s) (#13595)", - async (agent, destroyUserData) => { - const harness = await runManagedStateVolumeUninstall("owned", destroyUserData, "absent", agent); - try { - expect(harness.result.exitCode, harness.errors.join("\n")).toBe(0); - expect(harness.volumePresent()).toBe(false); - expect(harness.dockerCalls).toContainEqual(["volume", "rm", harness.volumeName]); - expect(harness.events.indexOf("openshell sandbox delete --all")).toBeLessThan( - harness.events.indexOf(`docker volume rm ${harness.volumeName}`), - ); - expect(harness.logs).toContain( - `Removed managed agent state volume for '${agent === "openclaw" ? "alpha" : "hermes"}'.`, - ); - } finally { - harness.cleanup(); - } - }, -); - it("removes the OpenClaw state volume of a legacy null-agent registry row (#13595)", async () => { - const harness = await runManagedStateVolumeUninstall("owned", false, "absent", "openclaw", null); + const harness = await runManagedHermesVolumeUninstall( + "owned", + false, + "absent", + "openclaw", + "default", + null, + ); try { expect(harness.result.exitCode, harness.errors.join("\n")).toBe(0); expect(harness.volumePresent()).toBe(false); @@ -458,11 +252,12 @@ it.each([ ] as const)( "preserves uninstall state when %s-agent OpenClaw volume removal fails (#13595)", async (_label, registryAgent) => { - const harness = await runManagedStateVolumeUninstall( + const harness = await runManagedHermesVolumeUninstall( "remove-fails", true, "absent", "openclaw", + "default", registryAgent, ); try { @@ -470,7 +265,7 @@ it.each([ expect(harness.volumePresent()).toBe(true); expect(fs.existsSync(harness.registryFile)).toBe(true); expect(harness.errors.join("\n")).toContain( - "Managed agent state volume 'nemoclaw-openclaw-state-v1-alpha' could not be removed.", + `Managed agent state volume '${harness.volumeName}' could not be removed.`, ); expect(harness.errors).toContain("Preserved NemoClaw state so exact cleanup can be retried."); } finally { diff --git a/src/lib/actions/uninstall/run-plan-local-model-profile.test.ts b/src/lib/actions/uninstall/run-plan-local-model-profile.test.ts index 9cf829b3000..c292beee023 100644 --- a/src/lib/actions/uninstall/run-plan-local-model-profile.test.ts +++ b/src/lib/actions/uninstall/run-plan-local-model-profile.test.ts @@ -5,7 +5,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { describe, expect, it, vi } from "vitest"; +import { beforeEach, afterEach, describe, expect, it, vi } from "vitest"; import { withProvenManagedGatewayProcess, writeManagedGatewayRuntimeProof, @@ -29,6 +29,14 @@ import { type UninstallRunOptions, } from "./run-plan"; +let fixtureHome: string; +beforeEach(() => { + fixtureHome = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-uninstall-profile-")); +}); +afterEach(() => { + fs.rmSync(fixtureHome, { recursive: true, force: true }); +}); + function ok(stdout = ""): RunResult { return { status: 0, stdout, stderr: "" }; } @@ -141,7 +149,7 @@ describe("uninstall local model profile cleanup", () => { ["operator-server", "foreign"], ]); const env = { - HOME: "/tmp/nemoclaw-uninstall-vllm-daemons", + HOME: path.join(fixtureHome, "nemoclaw-uninstall-vllm-daemons"), DOCKER_CONTEXT: "other-local", DOCKER_HOST: "unix:///other/docker.sock", DOCKER_CONFIG: "/other/config", @@ -239,7 +247,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-orphaned-vllm" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-orphaned-vllm"), + } as NodeJS.ProcessEnv, existsSync: () => false, isTty: false, log: () => {}, @@ -268,7 +278,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-unlabeled-vllm" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-unlabeled-vllm"), + } as NodeJS.ProcessEnv, existsSync: () => false, error: (message) => errors.push(message), isTty: false, @@ -296,7 +308,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-malformed-vllm" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-malformed-vllm"), + } as NodeJS.ProcessEnv, existsSync: () => false, error: (message) => errors.push(message), isTty: false, @@ -325,7 +339,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-vllm-removal-failure" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-vllm-removal-failure"), + } as NodeJS.ProcessEnv, existsSync: () => false, error: (message) => errors.push(message), isTty: false, @@ -381,7 +397,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-runtime-name-guard" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-runtime-name-guard"), + } as NodeJS.ProcessEnv, existsSync: () => false, error: (message) => errors.push(message), isTty: false, @@ -405,7 +423,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-runtime-inventory-guard" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-runtime-inventory-guard"), + } as NodeJS.ProcessEnv, existsSync: () => false, error: (message) => errors.push(message), isTty: false, @@ -427,7 +447,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: false, deleteModels: true, keepOpenShell: true }, { commandExists: (command) => command === "openshell", - env: { HOME: "/tmp/nemoclaw-uninstall-model-confirmation" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-model-confirmation"), + } as NodeJS.ProcessEnv, existsSync: () => false, isTty: true, log: (line) => logs.push(line), @@ -494,7 +516,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: true, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "ollama", - env: { HOME: "/tmp/nemoclaw-uninstall-all-ollama-models" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-all-ollama-models"), + } as NodeJS.ProcessEnv, existsSync: () => false, isTty: false, log: () => {}, @@ -531,7 +555,7 @@ describe("uninstall local model profile cleanup", () => { { commandExists: (command) => command === "openshell" || command === "ollama", env: { - HOME: "/tmp/nemoclaw-uninstall-local-ollama-only", + HOME: path.join(fixtureHome, "nemoclaw-uninstall-local-ollama-only"), OLLAMA_HOST: "https://remote.example.test:11434", } as NodeJS.ProcessEnv, existsSync: () => false, @@ -560,7 +584,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: true, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "ollama", - env: { HOME: "/tmp/nemoclaw-uninstall-malformed-ollama" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-malformed-ollama"), + } as NodeJS.ProcessEnv, error: (message) => errors.push(message), existsSync: () => false, isTty: false, @@ -587,7 +613,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: true, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "ollama", - env: { HOME: "/tmp/nemoclaw-uninstall-failed-ollama-inventory" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-failed-ollama-inventory"), + } as NodeJS.ProcessEnv, existsSync: () => false, isTty: false, log: () => {}, @@ -611,7 +639,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: true, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "ollama", - env: { HOME: "/tmp/nemoclaw-uninstall-partial-ollama" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-partial-ollama"), + } as NodeJS.ProcessEnv, existsSync: () => false, isTty: false, log: () => {}, @@ -634,7 +664,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "ollama", - env: { HOME: "/tmp/nemoclaw-uninstall-keep-models" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-keep-models"), + } as NodeJS.ProcessEnv, existsSync: () => false, isTty: false, log: () => {}, @@ -795,7 +827,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: true, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-local-cleanup-failure" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-local-cleanup-failure"), + } as NodeJS.ProcessEnv, existsSync: (target) => String(target).endsWith("/.nemoclaw/managed-llama-cpp"), error: (message) => errors.push(message), isTty: false, @@ -822,7 +856,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-llama-and-bearerless-vllm" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-llama-and-bearerless-vllm"), + } as NodeJS.ProcessEnv, existsSync: (target) => String(target).endsWith("/.nemoclaw/managed-llama-cpp"), isTty: false, log: () => {}, @@ -853,7 +889,9 @@ describe("uninstall local model profile cleanup", () => { { assumeYes: true, deleteModels: false, keepOpenShell: true }, { commandExists: (command) => command === "openshell" || command === "docker", - env: { HOME: "/tmp/nemoclaw-uninstall-vllm-receipt-only" } as NodeJS.ProcessEnv, + env: { + HOME: path.join(fixtureHome, "nemoclaw-uninstall-vllm-receipt-only"), + } as NodeJS.ProcessEnv, existsSync: (target) => String(target).endsWith("/.nemoclaw/host-local-vllm-runtime.json"), isTty: false, log: () => {}, diff --git a/src/lib/actions/uninstall/run-plan.ts b/src/lib/actions/uninstall/run-plan.ts index 93e45bcfd04..548f65dbec1 100644 --- a/src/lib/actions/uninstall/run-plan.ts +++ b/src/lib/actions/uninstall/run-plan.ts @@ -492,12 +492,16 @@ function removePathExcept( const children = fs.readdirSync(target); for (const entry of children) { if (preserveSet.has(entry)) continue; - deps.rmSync(path.join(target, entry), { force: true, recursive: true }); + const nested = preserve + .filter((name) => name.startsWith(`${entry}${path.sep}`)) + .map((name) => name.slice(entry.length + 1)); + if (nested.length === 0) + deps.rmSync(path.join(target, entry), { force: true, recursive: true }); + else if (!removePathExcept(path.join(target, entry), nested, deps)) return false; } // Track preserved order against the declared allowlist so the log line is // stable across filesystems with non-deterministic readdir ordering. - const childSet = new Set(children); - const preserved = preserve.filter((name) => childSet.has(name)); + const preserved = preserve.filter((name) => pathEntryExists(path.join(target, name), deps)); if (preserved.length === 0) { deps.rmSync(target, { force: true, recursive: true }); deps.log(`Removed ${target}`); @@ -4310,6 +4314,9 @@ function recoverAbandonedInterruptedUninstallState( return false; } try { + assertGatewayStatePathSafe(runtime.env.HOME || os.homedir(), path.dirname(source)); + assertGatewayStatePathSafe(runtime.env.HOME || os.homedir(), path.dirname(destination)); + fs.mkdirSync(path.dirname(destination), { mode: 0o700, recursive: true }); fs.renameSync(source, destination); } catch (error) { runtime.warn( @@ -4487,6 +4494,9 @@ async function removeInterruptedStateRootAfterFinalCleanup( return false; } try { + assertGatewayStatePathSafe(runtime.env.HOME || os.homedir(), path.dirname(source)); + assertGatewayStatePathSafe(runtime.env.HOME || os.homedir(), path.dirname(destination)); + fs.mkdirSync(path.dirname(destination), { mode: 0o700, recursive: true }); fs.renameSync(source, destination); } catch (error) { runtime.warn( diff --git a/src/lib/adapters/config/live-export-inference-contract.ts b/src/lib/adapters/config/live-export-inference-contract.ts index 78496f8c4b2..88a2526a29e 100644 --- a/src/lib/adapters/config/live-export-inference-contract.ts +++ b/src/lib/adapters/config/live-export-inference-contract.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { recordedRegisteredInferenceProviderAttachment } from "../../inference/native-provider/registered"; import { recordedNativeProviderAttachment } from "../../inference/native-provider/recorded-selection"; import { exportNativeInferenceReceipt } from "../../domain/config/export-evidence"; import { fixedNativeProviderForAttachment } from "../../inference/native-provider/fixed"; @@ -107,6 +108,13 @@ export function resolveExportInferenceSelection(entry: Readonly) { throw new Error("The native local export binding is invalid or unsupported."); return { normalized: normalizeInferenceSelection(entry), + registeredReceipt: recordedRegisteredInferenceProviderAttachment(entry), nativeReceipt: local ?? recordedNativeProviderAttachment(entry), }; } + +export { + verifyRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; +export { createManagedProviderAdapter } from "../openshell/managed-provider-adapter"; diff --git a/src/lib/adapters/config/live-export-registered.test.ts b/src/lib/adapters/config/live-export-registered.test.ts new file mode 100644 index 00000000000..76a2530efc3 --- /dev/null +++ b/src/lib/adapters/config/live-export-registered.test.ts @@ -0,0 +1,150 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { + raw, + mockSupportedLiveSource, + exportLiveSource, +} from "../../../../test/support/config-export-harness"; +import { observeStableExportSource } from "../../actions/config/observe-export-source"; +import { buildManagedStartupProfile } from "../../onboard/managed-startup/profile-builder"; +import { assert, describe, expect, it, vi } from "vitest"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; +import { createManagedProviderAdapter } from "../openshell/managed-provider-adapter"; +import { captureSanitizedResolvedOpenshell } from "../openshell/sanitized-capture"; +import { getSandboxEntryInference } from "../../state/registry-entry-view"; +import { entry, inventory, provider, startupInput } from "./live-export-source-test-fixture"; +import { createLiveExportSnapshotReader } from "./live-export-source"; + +vi.mock("../openshell/managed-provider-adapter", () => ({ + createManagedProviderAdapter: vi.fn(), +})); + +function registeredSource() { + const f = registeredInferenceFixture(true); + const binding = { ...f.binding, gatewayName: entry.gatewayName }; + const startup = buildManagedStartupProfile({ + ...startupInput, + inference: { + ...startupInput.inference, + routeProvider: binding.providerName, + upstreamProvider: binding.providerName, + routedBaseUrl: binding.endpointUrl, + upstreamEndpointUrl: null, + primaryModelRef: binding.providerName + "/model-a", + compatibility: {}, + }, + }); + mockSupportedLiveSource(3, 3, { + ...entry, + workload: { + ...entry.workload, + encodedProfile: startup.encodedProfile, + startupProfileSha256: startup.startupProfileSha256, + }, + provider: binding.providerName, + endpointUrl: binding.endpointUrl, + credentialEnv: binding.credentialEnv, + preferredInferenceApi: binding.inferenceApi, + registeredInferenceProviderAttachment: binding, + }); + vi.mocked(getSandboxEntryInference).mockReturnValue({ + kind: "configured", + provider: binding.providerName, + model: "model-a", + }); + vi.mocked(createManagedProviderAdapter).mockReturnValue(f.adapter); + const liveProvider = provider().provider; + raw.getProvider.mockResolvedValue({ + provider: { + ...liveProvider, + metadata: { ...liveProvider.metadata, name: binding.providerName, id: binding.providerId }, + type: binding.profileId, + credentials: { VENDOR_API_KEY: { value: "redacted" } }, + config: {}, + }, + }); + const sandbox = inventory().sandbox; + raw.getSandbox.mockResolvedValue({ + sandbox: { ...sandbox, spec: { ...sandbox.spec, providers: [binding.providerName] } }, + }); + return { ...f, binding }; +} + +describe("registered provider live export evidence", () => { + it("exports the verified operator route without claiming provider ownership", async () => { + const f = registeredSource(); + const exported = await exportLiveSource(); + expect(exported.result).toEqual({ ok: true, completion: { kind: "stdout" } }); + expect(exported.writeStdout).toHaveBeenCalledWith( + expect.stringContaining(f.binding.endpointUrl), + ); + expect(exported.writeStdout).toHaveBeenCalledWith( + expect.stringContaining(f.binding.credentialEnv), + ); + expect(exported.writeStdout).toHaveBeenCalledWith(expect.not.stringContaining(f.handle)); + }); + + it("verifies external identity and profile without shared routing or owned profile attestation", async () => { + const f = registeredSource(); + const result = await createLiveExportSnapshotReader().read("alpha"); + expect(result).toMatchObject({ + kind: "observed", + inference: { + provider: f.binding.providerName, + endpoint: f.binding.endpointUrl, + endpointEvidence: { + source: { + kind: "registered-provider", + profileId: f.binding.profileId, + profileDigest: f.binding.profileDigest, + }, + provider: { id: f.binding.providerId }, + }, + }, + }); + expect(raw.getProviderProfile).not.toHaveBeenCalled(); + expect(captureSanitizedResolvedOpenshell).not.toHaveBeenCalled(); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + }); + + it("refuses an externally replaced provider before export", async () => { + const f = registeredSource(); + f.state.id = "replacement"; + await expect(createLiveExportSnapshotReader().read("alpha")).resolves.toMatchObject({ + kind: "read-failed", + }); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it("refuses missing attachment membership", async () => { + const f = registeredSource(); + f.state.attached = false; + await expect(createLiveExportSnapshotReader().read("alpha")).resolves.toMatchObject({ + kind: "read-failed", + }); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); +}); + +it("rejects an export whose external profile evidence disagrees with its recorded digest", async () => { + registeredSource(); + const observed = await createLiveExportSnapshotReader().read("alpha"); + assert(observed.kind === "observed" && observed.inference.endpointEvidence); + const evidence = observed.inference.endpointEvidence; + assert(evidence.source.kind === "registered-provider"); + const tampered = { + ...observed, + inference: { + ...observed.inference, + endpointEvidence: { + ...evidence, + source: { ...evidence.source, profileDigest: "b".repeat(64) }, + }, + }, + }; + await expect( + observeStableExportSource("alpha", { read: async () => tampered }), + ).resolves.toMatchObject({ ok: false }); +}); diff --git a/src/lib/adapters/config/live-export-source-test-fixture.ts b/src/lib/adapters/config/live-export-source-test-fixture.ts index b9d6623f09c..08986d83b7c 100644 --- a/src/lib/adapters/config/live-export-source-test-fixture.ts +++ b/src/lib/adapters/config/live-export-source-test-fixture.ts @@ -314,9 +314,9 @@ function nativeNvidiaEndpoint(): Record { port: 443, ports: [], protocol: "rest", - tls: "", - enforcement: "enforce", - access: "", + tls: 0, + enforcement: 1, + access: 0, rules: [ { allow: { ...emptyNativeAllow, method: "GET", path: "/v1/models" } }, { @@ -368,6 +368,7 @@ export function nativeNvidiaProfile(): Record { "/usr/bin/node", "/opt/hermes/.venv/bin/python", "/opt/hermes/.venv/bin/python3", + "/usr/bin/python3.13", "/opt/venv/bin/python3", "/usr/local/bin/curl", "/usr/bin/curl", diff --git a/src/lib/adapters/config/live-export-source-vllm.test.ts b/src/lib/adapters/config/live-export-source-vllm.test.ts index ed268228930..af5d4bf5204 100644 --- a/src/lib/adapters/config/live-export-source-vllm.test.ts +++ b/src/lib/adapters/config/live-export-source-vllm.test.ts @@ -102,7 +102,6 @@ function mockManagedVllmSource( const observed: ObservedManagedVllmRuntime = { containerId: "a".repeat(64), imageId: `sha256:${"b".repeat(64)}`, - networkId: "c".repeat(64), startedAt: "2026-09-10T12:00:00Z", serving: { backend: "vllm", diff --git a/src/lib/adapters/config/live-export-source.test.ts b/src/lib/adapters/config/live-export-source.test.ts index e9d10fcc8d8..885e9a86f86 100644 --- a/src/lib/adapters/config/live-export-source.test.ts +++ b/src/lib/adapters/config/live-export-source.test.ts @@ -238,7 +238,6 @@ function mockNativeHostedSource(definition: HostedProviderDefinition) { host: string; port: number; allowed_ips?: string[]; - tls?: string; rules: { allow: { method: string; path: string } }[]; }; const base = managedBraveProfile(); @@ -261,8 +260,8 @@ function mockNativeHostedSource(definition: HostedProviderDefinition) { host: endpoint.host, port: endpoint.port, allowedIps: endpoint.allowed_ips ?? [], - tls: endpoint.tls ?? "", - access: "", + tls: 0, + access: 0, rules: endpoint.rules.map((rule) => ({ allow: { ...rule.allow, @@ -314,7 +313,10 @@ describe("live export snapshot reader", () => { "alpha-tavily-search", ]); expect(raw.getProviderProfile).toHaveBeenCalledWith( - { id: agent === "hermes" ? "tavily-hermes-v1" : "tavily", workspace: "default" }, + { + id: agent === "hermes" ? "tavily-hermes-v1" : "tavily", + workspaceScope: { selection: { case: "workspace", value: "default" } }, + }, { signal: expect.any(AbortSignal) }, ); expect(search.readCredential).not.toHaveBeenCalled(); @@ -429,7 +431,10 @@ describe("live export snapshot reader", () => { ]); expect(raw.getProviderProfile).toHaveBeenCalledTimes(2); expect(raw.getProviderProfile).toHaveBeenCalledWith( - { id: "brave", workspace: "default" }, + { + id: "brave", + workspaceScope: { selection: { case: "workspace", value: "default" } }, + }, { signal: expect.any(AbortSignal) }, ); expect(publish).not.toHaveBeenCalled(); @@ -634,7 +639,7 @@ describe("live export snapshot reader", () => { }); const result = await observeStableExportSource("alpha", createLiveExportSnapshotReader()); expect(result).toMatchObject({ ok: true, source: { tools: { disclosure: "direct" } } }); - expect(raw.getSandbox).toHaveBeenCalledTimes(2); + expect(raw.getSandbox).toHaveBeenCalledTimes(4); expect(JSON.stringify(result)).not.toContain(readFailureCanary); }); @@ -814,7 +819,12 @@ describe("live export snapshot reader", () => { expect(yaml).not.toContain(readFailureCanary); expect(raw.getProviderProfile).toHaveBeenCalledTimes(2); expect(raw.getProviderProfile).toHaveBeenCalledWith( - { id: "nemoclaw-nvidia-inference-v1", workspace: profileWorkspace }, + { + id: "nemoclaw-nvidia-inference-v1", + ...(profileWorkspace + ? { workspaceScope: { selection: { case: "workspace", value: profileWorkspace } } } + : {}), + }, { signal: expect.any(AbortSignal) }, ); expect(captureSanitizedResolvedOpenshell).not.toHaveBeenCalled(); @@ -1395,18 +1405,6 @@ describe("live export snapshot reader", () => { }); }, ); - - it("does not expose concrete reader exceptions", async () => { - const canary = "credential-canary-value"; - vi.mocked(loadRegistry).mockImplementation(() => { - throw new Error(canary); - }); - - const result = await createLiveExportSnapshotReader().read("alpha"); - - expect(result).toEqual({ kind: "read-failed", stage: "registry" }); - expect(JSON.stringify(result)).not.toContain(canary); - }); }); describe("dashboard export observation", () => { it("projects registered dashboard and direct tools through complete live observation (#10904)", async () => { diff --git a/src/lib/adapters/config/live-export-source.ts b/src/lib/adapters/config/live-export-source.ts index ed53519a299..6ac763c9d9a 100644 --- a/src/lib/adapters/config/live-export-source.ts +++ b/src/lib/adapters/config/live-export-source.ts @@ -1,6 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + verifyRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "./live-export-inference-contract"; +import { createManagedProviderAdapter } from "./live-export-inference-contract"; import os from "node:os"; import { normalizeNativeLocalProviderAttachment } from "../../inference/native-local/contract"; import { @@ -182,6 +187,42 @@ function endpointSourceFor( return { kind: "provider-config", key: configKey }; } +async function readRegisteredProviderEvidence( + provider: Provider, + gatewayName: string, + registeredReceipt: RegisteredInferenceProviderAttachment, +): Promise { + await verifyRegisteredInferenceProviderAttachment({ + adapter: createManagedProviderAdapter(), + expected: registeredReceipt, + }); + if ( + provider.id !== registeredReceipt.providerId || + provider.name !== registeredReceipt.providerName || + provider.type !== registeredReceipt.profileId || + !provider.credentialKeys.includes(registeredReceipt.credentialEnv) + ) + throw new Error("The live registered provider identity does not match its selection."); + return { + provider: providerIdentity(provider, gatewayName, false), + endpoint: registeredReceipt.endpointUrl, + source: { + kind: "registered-provider", + profileId: registeredReceipt.profileId, + profileDigest: registeredReceipt.profileDigest, + }, + }; +} + +function nativeHostedExportRequest(receipt?: NativeInferenceReceipt) { + if (!receipt?.endpointUrl || normalizeNativeLocalProviderAttachment(receipt)) return {}; + return { + nativeHostedEndpoint: receipt.endpointUrl, + nativeHostedAllowedIps: receipt.allowedIps, + nativeHostedCustom: receipt.custom, + }; +} + async function readProviderEvidence( normalized: ReturnType, routeProvider: string, @@ -189,6 +230,7 @@ async function readProviderEvidence( signal: AbortSignal, managedServing?: ObservedManagedVllmRuntime, nativeReceipt?: NativeInferenceReceipt, + registeredReceipt?: RegisteredInferenceProviderAttachment, ): Promise { const { configKey } = providerContract(normalized.preferredInferenceApi); const profileContract = profileContractFor(routeProvider, nativeReceipt); @@ -197,17 +239,14 @@ async function readProviderEvidence( workspace: "default", name: routeProvider, ...(profileContract ? { profileContract } : {}), - ...(nativeReceipt?.endpointUrl && !normalizeNativeLocalProviderAttachment(nativeReceipt) - ? { - nativeHostedEndpoint: nativeReceipt.endpointUrl, - nativeHostedAllowedIps: nativeReceipt.allowedIps, - nativeHostedCustom: nativeReceipt.custom, - } - : {}), - configKeys: [configKey], + ...nativeHostedExportRequest(nativeReceipt), + configKeys: registeredReceipt ? [] : [configKey], signal, }); if (!provider) throw new Error("The live inference provider is missing."); + if (registeredReceipt) { + return readRegisteredProviderEvidence(provider, gatewayName, registeredReceipt); + } const builtin = provider.builtinInferenceEndpoint !== undefined; if ( !matchesProviderMetadata(provider, normalized, routeProvider, !!managedServing, nativeReceipt) @@ -228,7 +267,7 @@ async function readProviderEvidence( async function resolveLiveInference( entry: Readonly, gatewayName: string, - nativeReceipt: NativeInferenceReceipt | undefined, + nativeReceipt: NativeInferenceReceipt | RegisteredInferenceProviderAttachment | undefined, ): Promise> { if (!nativeReceipt) { const live = await readInferenceRoute(entry, gatewayName); @@ -252,12 +291,13 @@ async function inferenceFor( sandbox: ObservedExportSandboxIdentity, managedServing?: ObservedManagedVllmRuntime, ): Promise { - const { normalized, nativeReceipt } = resolveExportInferenceSelection(entry); + const { normalized, nativeReceipt, registeredReceipt } = resolveExportInferenceSelection(entry); + const selectedReceipt = registeredReceipt ?? nativeReceipt; const gateway = resolveGatewayBinding(entry); - if (nativeReceipt && !sandbox.providerNames.includes(nativeReceipt.providerName)) { + if (selectedReceipt && !sandbox.providerNames.includes(selectedReceipt.providerName)) { throw new Error("The native provider is not attached to the sandbox."); } - const live = await resolveLiveInference(entry, gateway.name, nativeReceipt); + const live = await resolveLiveInference(entry, gateway.name, selectedReceipt); beforeRead("provider-metadata"); const endpointEvidence = await readProviderEvidence( normalized, @@ -266,6 +306,7 @@ async function inferenceFor( signal, managedServing, nativeReceipt, + registeredReceipt, ); let ollamaServing: ObservedExportInference["ollamaServing"]; if (entry.provider === "ollama-local") { @@ -322,6 +363,7 @@ async function effectivePolicy(gateway: ObservedExportGateway, row: Sandbox, sig target: namedOpenShellGateway(gateway.name), workspace: row.workspace, sandboxId: row.id, + name: row.name, signal, }); if (policy.appliedRevision === null) { diff --git a/src/lib/adapters/docker/inspect-identity.test.ts b/src/lib/adapters/docker/inspect-identity.test.ts index 7eff8c7afb8..fed7aceb97d 100644 --- a/src/lib/adapters/docker/inspect-identity.test.ts +++ b/src/lib/adapters/docker/inspect-identity.test.ts @@ -9,13 +9,15 @@ const LABELS = { managedBy: "openshell.ai/managed-by", workspace: "openshell.ai/sandbox-workspace", sandboxId: "openshell.ai/sandbox-id", + namespace: "openshell.ai/sandbox-namespace", + isolationRole: "openshell.ai/isolation-role", }; describe("inspectDockerSandboxIdentities", () => { it("queries only the sandbox-name label and parses one exact row", () => { const inspect = vi.fn((_args: readonly string[]) => ({ status: 0, - stdout: "aaaa000000000000\topenshell\tdefault\tsb-real", + stdout: "aaaa000000000000\topenshell\tdefault\tsb-real\t\t", })); expect( @@ -44,7 +46,7 @@ describe("inspectDockerSandboxIdentities", () => { expect( inspectDockerSandboxIdentities("openshell.ai/sandbox-name=alpha", LABELS, () => ({ status: 0, - stdout: `aaaa000000000000\t${managedBy}\tdefault\tsb-real`, + stdout: `aaaa000000000000\t${managedBy}\tdefault\tsb-real\t\t`, })), ).toEqual({ status: "observed", rows: [], malformedRows: 1 }); }); diff --git a/src/lib/adapters/docker/inspect.ts b/src/lib/adapters/docker/inspect.ts index 437af7ce009..4cb1c6d8c65 100644 --- a/src/lib/adapters/docker/inspect.ts +++ b/src/lib/adapters/docker/inspect.ts @@ -8,6 +8,8 @@ export type DockerSandboxIdentityRow = { managedBy: string; workspace: string; sandboxId: string; + namespace?: string; + isolationRole?: string; }; export type DockerSandboxIdentityObservation = @@ -37,7 +39,13 @@ function isExactBoundedIdentityText(value: string): boolean { /** Inspect and parse the exact Docker rows used by sandbox destroy. */ export function inspectDockerSandboxIdentities( sandboxNameLabel: string, - labelKeys: { managedBy: string; workspace: string; sandboxId: string }, + labelKeys: { + managedBy: string; + workspace: string; + sandboxId: string; + namespace: string; + isolationRole: string; + }, inspect: DockerSandboxIdentityInspect = dockerRun, ): DockerSandboxIdentityObservation { const format = [ @@ -45,6 +53,8 @@ export function inspectDockerSandboxIdentities( `{{.Label "${labelKeys.managedBy}"}}`, `{{.Label "${labelKeys.workspace}"}}`, `{{.Label "${labelKeys.sandboxId}"}}`, + `{{.Label "${labelKeys.namespace}"}}`, + `{{.Label "${labelKeys.isolationRole}"}}`, ].join("\t"); const result = inspect( ["ps", "-a", "--no-trunc", "--filter", `label=${sandboxNameLabel}`, "--format", format], @@ -70,15 +80,29 @@ export function inspectDockerSandboxIdentities( if (rawLine.length === 0) continue; const fields = rawLine.split("\t"); if ( - fields.length !== 4 || + fields.length !== 6 || !DOCKER_CONTAINER_ID_PATTERN.test(fields[0] ?? "") || !fields.every(isExactBoundedIdentityText) ) { malformedRows += 1; continue; } - const [id, managedBy, workspace, sandboxId] = fields as [string, string, string, string]; - rows.push({ id, managedBy, workspace, sandboxId }); + const [id, managedBy, workspace, sandboxId, namespace, isolationRole] = fields as [ + string, + string, + string, + string, + string, + string, + ]; + rows.push({ + id, + managedBy, + workspace, + sandboxId, + ...(namespace ? { namespace } : {}), + ...(isolationRole ? { isolationRole } : {}), + }); } return { status: "observed", rows, malformedRows }; } diff --git a/src/lib/adapters/openshell/README.md b/src/lib/adapters/openshell/README.md index 4955b1f77da..c6fc012eef5 100644 --- a/src/lib/adapters/openshell/README.md +++ b/src/lib/adapters/openshell/README.md @@ -87,7 +87,7 @@ detachment, deletion, replacement cleanup, and other lifecycle operations keep t adapters until the remaining #9806 migration slices land. This does not claim SDK qualification for those operations. -Policy export rejects SDK messages and serialized YAML above 1 MiB. It checks cancellation before conversion and after SDK loading. OpenShell SDK 0.0.116 does not expose a transport receive-size option. +Policy export rejects SDK messages and serialized YAML above 1 MiB. It checks cancellation before conversion and after SDK loading. OpenShell SDK 0.1.7 does not expose a transport receive-size option. ## User file transfers @@ -108,13 +108,12 @@ success, since interruption can arrive during post-transfer verification or lock retry an interrupted or indeterminate transfer. Session exports, onboarding downloads, and plugin copy remain assigned to later #9810 deliveries. -Native NVIDIA setup uses `provider-adapter-cli.ts` to read the named gateway's -`providers_v2_enabled` setting before publishing credentials. OpenShell 0.0.116 -leaves provider-derived policy disabled when that setting is unset. Setup enables -that default and confirms the result, preserves an explicit `false`, and stops on -unknown or failed responses without retrying a settings write. A successfully -enabled setting remains enabled if later provider setup fails; it is gateway state, -not a sandbox-owned resource. +Native NVIDIA and local inference setup verify the named gateway's global policy +history before publishing credentials. OpenShell 0.1.7 composes attached provider +policies automatically when no global policy override is active. Setup accepts +verified empty history or a superseded current global revision. It rejects an +active override, an unreadable response, or malformed policy data without changing +gateway settings. Native NVIDIA policy preparation and invocation probes are shared across agent integrations. When NVIDIA is selected, policy preparation adds the mandatory diff --git a/src/lib/adapters/openshell/forward-cli-validation.test.ts b/src/lib/adapters/openshell/forward-cli-validation.test.ts index bfcdf3a0202..50fa832521c 100644 --- a/src/lib/adapters/openshell/forward-cli-validation.test.ts +++ b/src/lib/adapters/openshell/forward-cli-validation.test.ts @@ -69,17 +69,21 @@ describe("CLI OpenShell forward command builders", () => { }); describe("CLI OpenShell forward list parser", () => { - it("parses an ANSI-formatted forward table", () => { + it.each([ + ["legacy", "", "", "default"], + ["workspace", "WORKSPACE ", "team-a ", "team-a"], + ])("parses an ANSI-formatted %s forward table", (_case, header, row, workspace) => { expect( parseCliOpenShellForwardList( - "\u001b[1mSANDBOX\u001b[0m BIND PORT PID STATUS\n" + - "\u001b[32mdemo\u001b[0m 127.0.0.1 18789 4312 running", + `${header}\u001b[1mSANDBOX\u001b[0m BIND PORT PID STATUS\n` + + `${row}\u001b[32mdemo\u001b[0m 127.0.0.1 18789 4312 running`, "", ), ).toEqual({ ok: true, rows: [ { + workspace, sandboxName: "demo", bind: "127.0.0.1", port: 18_789, @@ -99,6 +103,12 @@ describe("CLI OpenShell forward list parser", () => { ["missing header", "demo 127.0.0.1 18789 4312 running"], ["missing column", `${listHeader}\ndemo 127.0.0.1 18789 running`], ["extra column", `${listHeader}\ndemo 127.0.0.1 18789 4312 running extra`], + ["missing workspace", `WORKSPACE ${listHeader}\ndemo 127.0.0.1 18789 4312 running`], + ["invalid workspace", `WORKSPACE ${listHeader}\n../other demo 127.0.0.1 18789 4312 running`], + [ + "duplicate port across workspaces", + `WORKSPACE ${listHeader}\ndefault demo 127.0.0.1 18789 4312 running\nother demo 127.0.0.1 18789 9876 running`, + ], ["invalid bind", `${listHeader}\ndemo localhost 18789 4312 running`], ["invalid port", `${listHeader}\ndemo 127.0.0.1 65536 4312 running`], ["invalid PID", `${listHeader}\ndemo 127.0.0.1 18789 0 running`], diff --git a/src/lib/adapters/openshell/forward-cli.test.ts b/src/lib/adapters/openshell/forward-cli.test.ts index 310232fb48a..74e778e831c 100644 --- a/src/lib/adapters/openshell/forward-cli.test.ts +++ b/src/lib/adapters/openshell/forward-cli.test.ts @@ -102,9 +102,12 @@ describe("CLI OpenShell forward observations", () => { }, ); - it("returns stale only when the legacy inspector owns the listed PID", async () => { + it.each([ + ["legacy", legacyForwardList], + ["workspace", "WORKSPACE " + legacyForwardList.replaceAll("\n", "\ndefault ")], + ])("returns stale only when the inspector owns the %s table PID", async (_case, output) => { const { adapter, inspect, inspectLegacy } = createHarness({ - run: async () => captured(0, legacyForwardList), + run: async () => captured(0, output), }); await expect(adapter.observeForwards({ forwards: [forward] })).resolves.toEqual([ @@ -1186,6 +1189,29 @@ describe("CLI OpenShell direct forward start", () => { }); describe("CLI OpenShell legacy forward retirement", () => { + it.each([ + ["workspace table", "WORKSPACE " + legacyForwardList.replaceAll("\n", "\nother "), forward], + ["legacy table outside default", legacyForwardList, { ...forward, workspace: "other" }], + ])( + "refuses another workspace in %s without requesting authority or stopping", + async (_case, output, requested) => { + const authorize = vi.fn(async () => {}); + const { adapter, run } = createHarness({ + runtimeSelection: { ...runtimeSelection, workspace: requested.workspace }, + run: async () => captured(0, output), + }); + + await expect(adapter.retireLegacyForward({ forward: requested, authorize })).resolves.toEqual( + { + state: "refused", + observation: { state: "foreign", forward: requested }, + }, + ); + expect(authorize).not.toHaveBeenCalled(); + expect(run).toHaveBeenCalledOnce(); + }, + ); + it("omits the workspace flag for legacy CLIs that predate workspace selection", async () => { const run = vi.fn(async (_executable, args) => args.includes("list") ? captured(0, legacyForwardList) : captured(0), @@ -1203,57 +1229,63 @@ describe("CLI OpenShell legacy forward retirement", () => { expect(run.mock.calls.filter(([, args]) => args.includes("stop"))).toHaveLength(1); }); - it("checks authority again, stops once, and verifies release", async () => { - const operations: string[] = []; - const run: RunCommand = async (_executable, args) => { - const operation = args.includes("list") ? "list" : "stop"; - operations.push(operation); - return operation === "list" ? captured(0, legacyForwardList) : captured(0); - }; - const inspectLegacy: InspectLegacyListener = async (_identity, expectedPid) => { - operations.push(`legacy:${String(expectedPid)}`); - return { state: "owned", pid: expectedPid }; - }; - const probePort: ProbePort = async () => { - operations.push("probe"); - return { state: "unbound" }; - }; - const authorize = vi.fn(async () => { - operations.push("authorize"); - }); - const { adapter, run: runMock } = createHarness({ inspectLegacy, probePort, run }); + it.each([ + ["legacy", legacyForwardList], + ["workspace", "WORKSPACE " + legacyForwardList.replaceAll("\n", "\ndefault ")], + ])( + "checks authority, stops once, and verifies release for the %s table", + async (_case, output) => { + const operations: string[] = []; + const run: RunCommand = async (_executable, args) => { + const operation = args.includes("list") ? "list" : "stop"; + operations.push(operation); + return operation === "list" ? captured(0, output) : captured(0); + }; + const inspectLegacy: InspectLegacyListener = async (_identity, expectedPid) => { + operations.push(`legacy:${String(expectedPid)}`); + return { state: "owned", pid: expectedPid }; + }; + const probePort: ProbePort = async () => { + operations.push("probe"); + return { state: "unbound" }; + }; + const authorize = vi.fn(async () => { + operations.push("authorize"); + }); + const { adapter, run: runMock } = createHarness({ inspectLegacy, probePort, run }); - await expect(adapter.retireLegacyForward({ forward, authorize })).resolves.toEqual({ - state: "retired", - forward, - }); - expect(authorize).toHaveBeenCalledTimes(2); - expect(authorize).toHaveBeenNthCalledWith(1, forward); - expect(authorize).toHaveBeenNthCalledWith(2, forward); - expect(operations).toEqual([ - "list", - "legacy:4312", - "authorize", - "list", - "legacy:4312", - "authorize", - "stop", - "probe", - ]); - expect(runMock.mock.calls.filter(([, args]) => args.includes("stop"))).toEqual([ - [ - executable, - buildCliOpenShellLegacyForwardStopArgs(forward), - expect.objectContaining({ - environment: { - OPENSHELL_GATEWAY: "nemoclaw", - OPENSHELL_WORKSPACE: "default", - }, - outputLimitBytes: 64 * 1_024, - }), - ], - ]); - }); + await expect(adapter.retireLegacyForward({ forward, authorize })).resolves.toEqual({ + state: "retired", + forward, + }); + expect(authorize).toHaveBeenCalledTimes(2); + expect(authorize).toHaveBeenNthCalledWith(1, forward); + expect(authorize).toHaveBeenNthCalledWith(2, forward); + expect(operations).toEqual([ + "list", + "legacy:4312", + "authorize", + "list", + "legacy:4312", + "authorize", + "stop", + "probe", + ]); + expect(runMock.mock.calls.filter(([, args]) => args.includes("stop"))).toEqual([ + [ + executable, + buildCliOpenShellLegacyForwardStopArgs(forward), + expect.objectContaining({ + environment: { + OPENSHELL_GATEWAY: "nemoclaw", + OPENSHELL_WORKSPACE: "default", + }, + outputLimitBytes: 64 * 1_024, + }), + ], + ]); + }, + ); it.each([ ["owned", "not_needed", noActiveForwards, { state: "owned", pid: 4_321 } as const], diff --git a/src/lib/adapters/openshell/forward-cli.ts b/src/lib/adapters/openshell/forward-cli.ts index ee1b6989c62..57002b227d4 100644 --- a/src/lib/adapters/openshell/forward-cli.ts +++ b/src/lib/adapters/openshell/forward-cli.ts @@ -244,6 +244,7 @@ export type CliOpenShellForwardAdapterDeps = Readonly<{ }>; export type CliOpenShellLegacyForwardRow = Readonly<{ + workspace: string; sandboxName: string; bind: "127.0.0.1" | "0.0.0.0"; port: number; @@ -260,7 +261,7 @@ function stripTerminalFormatting(value: string): string | null { return TERMINAL_CONTROL_RE.test(stripped) ? null : stripped; } -/** Parse the complete OpenShell 0.0.116 legacy forward table. */ +/** Parse tracked forwards from OpenShell 0.1.3 and the legacy cleanup CLI. */ export function parseCliOpenShellForwardList( stdout: string, stderr: string, @@ -278,7 +279,9 @@ export function parseCliOpenShellForwardList( const lines = output.split(/\r?\n/u).map((line) => line.trim()); if (lines.some((line) => line.length === 0)) return { ok: false, error: SCHEMA_ERROR }; - if (lines.shift()?.split(/\s+/u).join(" ") !== "SANDBOX BIND PORT PID STATUS") { + const header = lines.shift()?.split(/\s+/u).join(" "); + const workspaceColumn = header === "WORKSPACE SANDBOX BIND PORT PID STATUS"; + if (!workspaceColumn && header !== "SANDBOX BIND PORT PID STATUS") { return { ok: false, error: SCHEMA_ERROR }; } if (lines.length === 0) return { ok: false, error: SCHEMA_ERROR }; @@ -286,11 +289,15 @@ export function parseCliOpenShellForwardList( const occupiedPorts = new Set(); for (const line of lines) { const columns = line.split(/\s+/u); - if (columns.length !== 5) return { ok: false, error: SCHEMA_ERROR }; + if (columns.length !== (workspaceColumn ? 6 : 5)) return { ok: false, error: SCHEMA_ERROR }; + // The pre-workspace CLI can only own forwards in the default workspace. + const workspace = workspaceColumn ? columns.shift() : "default"; const [sandboxName, bind, portText, pidText, status] = columns; const port = Number(portText); const pid = Number(pidText); if ( + !workspace || + !isValidName(workspace) || !sandboxName || !isValidName(sandboxName) || (bind !== "127.0.0.1" && bind !== "0.0.0.0") || @@ -305,7 +312,7 @@ export function parseCliOpenShellForwardList( return { ok: false, error: SCHEMA_ERROR }; } occupiedPorts.add(port); - rows.push({ sandboxName, bind, port, pid, status }); + rows.push({ workspace, sandboxName, bind, port, pid, status }); } return { ok: true, rows }; } @@ -1154,7 +1161,10 @@ function classifyRegistry( ): CliOpenShellForwardRegistryMatch { const samePort = rows.filter((row) => row.port === forward.port); const sameTarget = samePort.filter( - (row) => row.sandboxName === forward.sandboxName && row.bind === forward.localHost, + (row) => + row.workspace === forward.workspace && + row.sandboxName === forward.sandboxName && + row.bind === forward.localHost, ); if (samePort.length > 1 || sameTarget.length > 1) return { state: "indeterminate" }; const legacy = samePort[0]; diff --git a/src/lib/adapters/openshell/gateway-scope.ts b/src/lib/adapters/openshell/gateway-scope.ts index 84882e4004f..120fa990d3c 100644 --- a/src/lib/adapters/openshell/gateway-scope.ts +++ b/src/lib/adapters/openshell/gateway-scope.ts @@ -17,7 +17,7 @@ export { export type OpenShellGatewayTarget = { kind: "named"; gatewayName: string } | { kind: "selected" }; function inferredGatewayFlagIndex(args: readonly string[]): number | null { - if (args[0] === "inference" || args[0] === "provider") return 2; + if (args[0] === "inference" || args[0] === "provider" || args[0] === "policy") return 2; if (args[0] !== "sandbox" || typeof args[1] !== "string") return null; return args[1] === "provider" ? 3 : 2; } diff --git a/src/lib/adapters/openshell/hermes-acp-ssh-cli.test.ts b/src/lib/adapters/openshell/hermes-acp-ssh-cli.test.ts index 98ea61ec8a6..a5360a60f82 100644 --- a/src/lib/adapters/openshell/hermes-acp-ssh-cli.test.ts +++ b/src/lib/adapters/openshell/hermes-acp-ssh-cli.test.ts @@ -142,7 +142,7 @@ function harness( captureOpenShell, createTempConfig, createSessionStatusNonce: () => SESSION_STATUS_NONCE, - openshellVersion: vi.fn(() => "0.0.116"), + openshellVersion: vi.fn(() => "0.1.7"), platform: "linux", resolveOpenshell: () => "/usr/bin/openshell", spawnSsh, @@ -597,6 +597,7 @@ describe("CLI Hermes ACP SSH transport", () => { it.each([ ["native Windows", { platform: "win32" as const }], ["a different OpenShell version", { openshellVersion: vi.fn(() => "0.0.105") }], + ["the frozen headless API version", { openshellVersion: vi.fn(() => "0.1.5") }], ])("rejects %s before preparing SSH", async (_label, overrides) => { const fixture = harness([], overrides); const io = streams(); diff --git a/src/lib/adapters/openshell/hermes-acp-ssh-cli.ts b/src/lib/adapters/openshell/hermes-acp-ssh-cli.ts index 16bf0e296ae..8877515bc75 100644 --- a/src/lib/adapters/openshell/hermes-acp-ssh-cli.ts +++ b/src/lib/adapters/openshell/hermes-acp-ssh-cli.ts @@ -13,7 +13,10 @@ import { accessSync, constants } from "node:fs"; import path from "node:path"; import { pipeline } from "node:stream/promises"; -import { HERMES_LIFECYCLE_DEFINITION } from "../../domain/lifecycle/hermes-definition"; +import { + HERMES_LIFECYCLE_DEFINITION, + HERMES_CLI_OPENSHELL_VERSION, +} from "../../domain/lifecycle/hermes-definition"; import { assertNoOpenShellGatewayEndpointOverride } from "../../openshell-gateway-endpoint-guard"; import { createTempSshConfig, @@ -36,7 +39,7 @@ import { } from "./hermes-acp-ssh"; const ACP_SDK_VERSION = "0.9.0"; -const ACP_SETUP_TIMEOUT_MS = 30_000; +export const ACP_SETUP_TIMEOUT_MS = 30_000; const SSH_CONFIG_MAX_BYTES = 1024 * 1024; const PROBE_MAX_BYTES = 4 * 1024; const SSH_KILL_GRACE_MS = 1_000; @@ -608,7 +611,7 @@ function resolveBinaries( versionOptions, deps.captureOpenShell ?? captureOpenShell, ); - return result === HERMES_LIFECYCLE_DEFINITION.openshellVersion ? { openshell, ssh } : null; + return result === HERMES_CLI_OPENSHELL_VERSION ? { openshell, ssh } : null; } /** Create the OpenShell CLI implementation of the Hermes ACP SSH boundary. */ @@ -623,7 +626,7 @@ export function createCliHermesAcpSshTransport( if (!binaries) { return failure( "unavailable", - `Hermes ACP requires OpenShell ${HERMES_LIFECYCLE_DEFINITION.openshellVersion} and the system SSH client.`, + `Hermes ACP requires OpenShell ${HERMES_CLI_OPENSHELL_VERSION} and the system SSH client.`, 69, ); } diff --git a/src/lib/adapters/openshell/native-hosted-profile-response.test.ts b/src/lib/adapters/openshell/native-hosted-profile-response.test.ts index bbdd6439e94..34ea83515c1 100644 --- a/src/lib/adapters/openshell/native-hosted-profile-response.test.ts +++ b/src/lib/adapters/openshell/native-hosted-profile-response.test.ts @@ -2,7 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 import fs from "node:fs"; -import { describe, expect, it } from "vitest"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it, vi, onTestFinished } from "vitest"; import { managedBraveProfile } from "../../../../test/fixtures/openshell-provider-profile"; import { hostedNativeProvider, @@ -10,6 +12,7 @@ import { type HostedProviderDefinition, } from "../../inference/native-provider/hosted"; import { nativeProviderLifecycle } from "../../inference/native-provider"; +import { boundHermesNativeProfile } from "../../inference/native-provider/hermes-profile"; import { customNativeProvider, boundCustomNativeProfile, @@ -22,14 +25,15 @@ function response(definition: HostedProviderDefinition) { const contract = parseCheckedInProviderProfileContract( definition.custom ? boundCustomNativeProfile(definition, profilePath) - : fs.readFileSync(profilePath, "utf8"), + : definition.endpointUrl + ? boundHermesNativeProfile(definition) + : fs.readFileSync(profilePath, "utf8"), )!; const base = managedBraveProfile(); const credential = contract.boundary.credentials[0]; const endpoint = contract.boundary.endpoints[0] as { host: string; port: number; - tls?: string; allowed_ips?: string[]; rules: { allow: { method: string; path: string } }[]; }; @@ -51,9 +55,9 @@ function response(definition: HostedProviderDefinition) { ...base.endpoints[0], host: endpoint.host, port: endpoint.port, - tls: endpoint.tls ?? "", + tls: 0, allowedIps: endpoint.allowed_ips ?? [], - access: "", + access: 0, rules: endpoint.rules.map((rule) => ({ allow: { ...rule.allow, @@ -77,6 +81,33 @@ describe.each(HOSTED_NATIVE_PROVIDERS)("native $label profile read boundary", (d const value = response(definition); expect(isManagedNativeHostedProfileResponse(value, definition.profileId)).toBe(value); }); + it.each([ + ["disabled TLS inspection", { tls: 1 }], + ["removed explicit TLS termination", { tls: 2 }], + ["removed TLS passthrough", { tls: 3 }], + ["audit-only enforcement", { enforcement: 2 }], + ["unspecified enforcement", { enforcement: 0 }], + ["unrestricted read-write access", { access: 2 }], + ["an unknown access preset", { access: 999 }], + ["legacy string enums", { tls: "", enforcement: "enforce", access: "" }], + ])("rejects %s", (_label, mutation) => { + const value = response(definition); + const profile = { + ...value.profile, + endpoints: [{ ...value.profile.endpoints[0], ...mutation }], + }; + expect(isManagedNativeHostedProfileResponse({ profile }, definition.profileId)).toBeUndefined(); + }); + it.each(["/usr/bin/python3*", "/usr/bin/python3.12", "/unapproved"])( + "rejects replacing the Hermes interpreter with %s", + (path) => { + const value = response(definition); + value.profile.binaries = value.profile.binaries.map((binary) => + binary.path === "/usr/bin/python3.13" ? { path } : binary, + ); + expect(isManagedNativeHostedProfileResponse(value, definition.profileId)).toBeUndefined(); + }, + ); it("rejects extra egress or executable access", () => { const value = response(definition); expect( @@ -165,7 +196,7 @@ it.each([ endpoints: [ { ...value.profile.endpoints[0], - tls: endpointUrl.startsWith("https:") ? "" : "terminate", + tls: 1, }, ], }, @@ -233,3 +264,54 @@ it("requires the exact public IP restriction on a Hermes bound profile", () => { isManagedNativeHostedProfileResponse(value, definition.profileId, endpointUrl, ["10.0.0.1"]), ).toBeUndefined(); }); + +it.each([ + { endpointUrl: "http://8.8.8.8/team/v1", allowedIps: ["8.8.8.8"] }, + { endpointUrl: "https://[2001:4860:4860::8888]/team/v1", allowedIps: ["2001:4860:4860::8888"] }, + { endpointUrl: "http://host.openshell.internal:42103/team/v1", allowedIps: undefined }, +])( + "validates the runtime response for retained Hermes $endpointUrl", + ({ endpointUrl, allowedIps }) => { + const definition = { ...hostedNativeProvider("hermes-provider", endpointUrl)!, allowedIps }; + const value = response(definition); + expect( + isManagedNativeHostedProfileResponse(value, definition.profileId, endpointUrl, allowedIps), + ).toBe(value); + expect( + isManagedNativeHostedProfileResponse(value, definition.profileId, endpointUrl, ["1.1.1.1"]), + ).toBeUndefined(); + value.profile.endpoints[0].port += 1; + expect( + isManagedNativeHostedProfileResponse(value, definition.profileId, endpointUrl, allowedIps), + ).toBeUndefined(); + }, +); + +it.each([ + { service: "gateway", directory: ".nemoclaw/gateways/42103", routerPort: null }, + { service: "Model Router", directory: ".nemoclaw", routerPort: 42103 }, +])( + "rejects a retained Hermes response after its port is reserved for $service", + ({ directory, routerPort }) => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-response-port-")); + onTestFinished(() => fs.rmSync(home, { recursive: true, force: true })); + vi.stubEnv("HOME", home); + vi.stubEnv("NEMOCLAW_GATEWAY_PORT", ""); + vi.stubEnv("NEMOCLAW_MODEL_ROUTER_PORT", ""); + const endpointUrl = "http://host.openshell.internal:42103/team/v1"; + const definition = hostedNativeProvider("hermes-provider", endpointUrl)!; + const value = response(definition); + expect(isManagedNativeHostedProfileResponse(value, definition.profileId, endpointUrl)).toBe( + value, + ); + const root = path.join(home, directory); + fs.mkdirSync(root, { recursive: true }); + fs.writeFileSync( + path.join(root, "onboard-session.json"), + JSON.stringify({ routerPort, routerPid: null, routerCredentialHash: null }), + ); + expect( + isManagedNativeHostedProfileResponse(value, definition.profileId, endpointUrl), + ).toBeUndefined(); + }, +); diff --git a/src/lib/adapters/openshell/native-hosted-profile-response.ts b/src/lib/adapters/openshell/native-hosted-profile-response.ts index e95bfe1fc84..52aae4e3b41 100644 --- a/src/lib/adapters/openshell/native-hosted-profile-response.ts +++ b/src/lib/adapters/openshell/native-hosted-profile-response.ts @@ -1,7 +1,9 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { requireHermesPublicPins } from "../../inference/native-provider/hermes-pins"; +import { isAllowedOpenShellSandboxBridgeUrl } from "../../private-networks"; +import { isProtectedLocalInferencePort } from "../../inference/gateway/local-endpoint-ports"; +import { requireHermesEndpointPins } from "../../inference/native-provider/hermes-pins"; import { customNativeProvider, type CustomNativeBinding, @@ -46,6 +48,12 @@ export function hostedNativeProfileBoundary( const definition = hostedNativeProfile(profileId, endpointUrl, custom); if (!definition) return undefined; const endpoint = new URL(definition.endpoint); + if ( + definition.logicalProvider === "hermes-provider" && + isAllowedOpenShellSandboxBridgeUrl(endpoint) && + isProtectedLocalInferencePort(Number(endpoint.port)) + ) + return undefined; const base = endpoint.pathname.replace(/\/+$/, ""); const prefix = definition.api === "anthropic-messages" ? `${custom ? base : ""}/v1` : base; let paths = ["chat/completions"]; @@ -54,11 +62,11 @@ export function hostedNativeProfileBoundary( else if (definition.logicalProvider === "openai-api") paths.push("responses"); let pins: readonly string[] | undefined; if (custom) pins = definition.allowedIps ?? []; - else if (definition.endpointUrl) pins = requireHermesPublicPins(allowedIps); + else if (definition.endpointUrl) + pins = requireHermesEndpointPins(definition.endpointUrl, allowedIps); return { profileId: definition.profileId as HostedNativeProfileId, ...(pins ? { allowedIps: pins } : {}), - ...(custom && endpoint.protocol === "https:" ? { tls: "terminate" as const } : {}), credentialEnv: definition.credentialEnv, authStyle: definition.api === "anthropic-messages" ? "header" : "bearer", headerName: definition.api === "anthropic-messages" ? "x-api-key" : "authorization", diff --git a/src/lib/adapters/openshell/native-inference-profile-response.ts b/src/lib/adapters/openshell/native-inference-profile-response.ts index 7a0d18f1db4..16f09ffbe7b 100644 --- a/src/lib/adapters/openshell/native-inference-profile-response.ts +++ b/src/lib/adapters/openshell/native-inference-profile-response.ts @@ -21,7 +21,6 @@ export type NativeProfileBoundary = Readonly<{ headerName: string; host: string; port?: number; - tls?: "terminate"; allowedIps?: readonly string[]; binaries?: readonly string[]; rules: readonly Readonly<{ method: "GET" | "POST"; path: string }>[]; @@ -104,9 +103,9 @@ function nativeEndpoint(value: unknown, expected: NativeProfileBoundary): boolea endpoint.port === (expected.port ?? 443) && emptyArray(endpoint.ports) && endpoint.protocol === "rest" && - endpoint.tls === (expected.tls ?? "") && - endpoint.enforcement === "enforce" && - endpoint.access === "" && + endpoint.tls === 0 && + endpoint.enforcement === 1 && + endpoint.access === 0 && Array.isArray(rules) && rules.length === expected.rules.length && expected.rules.every((rule, index) => nativeRule(rules[index], rule.method, rule.path)) && @@ -139,6 +138,7 @@ function nativeBinaries(value: unknown, expected?: readonly string[]): boolean { "/usr/bin/node", "/opt/hermes/.venv/bin/python", "/opt/hermes/.venv/bin/python3", + "/usr/bin/python3.13", "/opt/venv/bin/python3", "/usr/local/bin/curl", "/usr/bin/curl", diff --git a/src/lib/adapters/openshell/native-nvidia-profile-response.ts b/src/lib/adapters/openshell/native-nvidia-profile-response.ts index 7823b9b34f3..4e857fc6939 100644 --- a/src/lib/adapters/openshell/native-nvidia-profile-response.ts +++ b/src/lib/adapters/openshell/native-nvidia-profile-response.ts @@ -17,7 +17,6 @@ export type NativeProfileBoundary = Readonly<{ headerName: string; host: string; port?: number; - tls?: "terminate"; allowedIps?: readonly string[]; rules: readonly Readonly<{ method: "GET" | "POST"; path: string }>[]; }>; @@ -99,9 +98,9 @@ function nativeEndpoint(value: unknown, expected: NativeProfileBoundary): boolea endpoint.port === (expected.port ?? 443) && emptyArray(endpoint.ports) && endpoint.protocol === "rest" && - endpoint.tls === (expected.tls ?? "") && - endpoint.enforcement === "enforce" && - endpoint.access === "" && + endpoint.tls === 0 && + endpoint.enforcement === 1 && + endpoint.access === 0 && Array.isArray(rules) && rules.length === expected.rules.length && expected.rules.every((rule, index) => nativeRule(rules[index], rule.method, rule.path)) && diff --git a/src/lib/adapters/openshell/policy-boundary.ts b/src/lib/adapters/openshell/policy-boundary.ts index 2945f226008..cccd5a31a40 100644 --- a/src/lib/adapters/openshell/policy-boundary.ts +++ b/src/lib/adapters/openshell/policy-boundary.ts @@ -4,6 +4,7 @@ import { isDeepStrictEqual } from "node:util"; import { + openShellHostMatches as canonicalOpenShellHostMatches, assertOpenShellPolicyEndpointPins as assertCanonicalOpenShellPolicyEndpointPins, assertPolicyRequirementContainment as assertCanonicalPolicyRequirementContainment, buildOpenShellSandboxPolicyInspectionArgs as buildCanonicalOpenShellSandboxPolicyInspectionArgs, @@ -34,6 +35,7 @@ import { stripCredentials } from "../../security/credential-filter"; // sourceOfTruth: nemoclaw/src/shared/openshell-policy-boundary.cts // generatedBoundary: build:cli emits the canonical .cjs/.d.cts before this // adapter boundary is compiled. Keep this file implementation-free. +export const openShellHostMatches = canonicalOpenShellHostMatches; export const assertOpenShellPolicyEndpointPins = assertCanonicalOpenShellPolicyEndpointPins; export const parseOpenShellPolicy = parseCanonicalOpenShellPolicy; export const classifyOpenShellGlobalPolicyHistory = classifyCanonicalOpenShellGlobalPolicyHistory; diff --git a/src/lib/adapters/openshell/provider-adapter-cli-expiration.test.ts b/src/lib/adapters/openshell/provider-adapter-cli-expiration.test.ts index 26b156de7e3..12967619653 100644 --- a/src/lib/adapters/openshell/provider-adapter-cli-expiration.test.ts +++ b/src/lib/adapters/openshell/provider-adapter-cli-expiration.test.ts @@ -6,6 +6,10 @@ import { describe, expect, it, vi } from "vitest"; import { createCliOpenShellProviderAdapter, type RunProviderCommand } from "./provider-adapter-cli"; import { namedOpenShellGateway, selectedOpenShellGateway } from "./sandbox-observer"; +function inventory(providers: unknown[], nextPageToken = ""): string { + return JSON.stringify({ providers, next_page_token: nextPageToken }); +} + function captured(status: number | null, stdout = "", stderr = "") { return { status, stdout, stderr }; } @@ -26,7 +30,7 @@ describe("CLI OpenShell provider credential expiration metadata", () => { .mockReturnValueOnce( captured( 0, - JSON.stringify([ + inventory([ { name: "other-provider", credential_keys: ["OTHER_API_KEY"], @@ -68,10 +72,10 @@ describe("CLI OpenShell provider credential expiration metadata", () => { "list", "-g", "nemoclaw-18080", - "--limit", + "--page-size", "1000", - "--offset", - "0", + "--page-token", + "", "--output", "json", ], @@ -90,7 +94,7 @@ describe("CLI OpenShell provider credential expiration metadata", () => { .mockReturnValueOnce( captured( 1, - JSON.stringify([ + inventory([ { name: "search-prod", credentials: { TAVILY_API_KEY: credentialValue }, @@ -118,15 +122,15 @@ describe("CLI OpenShell provider credential expiration metadata", () => { }); it("finds a requested provider beyond the first inventory page", async () => { - const page = Array.from({ length: 1000 }, (_, index) => ({ name: `other-${index}` })); + const page = [{ name: "other-provider" }]; const run = vi .fn() .mockReturnValueOnce(captured(0, PROVIDER_GET_OUTPUT)) - .mockReturnValueOnce(captured(0, JSON.stringify(page))) + .mockReturnValueOnce(captured(0, inventory(page, "next-page"))) .mockReturnValueOnce( captured( 0, - JSON.stringify([ + inventory([ { name: "search-prod", credential_keys: ["TAVILY_API_KEY"], @@ -147,10 +151,10 @@ describe("CLI OpenShell provider credential expiration metadata", () => { expect(run.mock.calls[2][0]).toEqual([ "provider", "list", - "--limit", - "1000", - "--offset", + "--page-size", "1000", + "--page-token", + "next-page", "--output", "json", ]); @@ -163,7 +167,7 @@ describe("CLI OpenShell provider credential expiration metadata", () => { .mockReturnValueOnce( captured( 0, - JSON.stringify([ + inventory([ { name: "search-prod", credential_keys: ["OTHER_API_KEY"], @@ -189,10 +193,11 @@ describe("CLI OpenShell provider credential expiration metadata", () => { }); it("stops at the bounded inventory limit when the provider cannot be found", async () => { + let pageNumber = 0; const page = Array.from({ length: 1000 }, (_, index) => ({ name: `other-${index}` })); const run = vi .fn() - .mockReturnValue(captured(0, JSON.stringify(page))) + .mockImplementation(() => captured(0, inventory(page, `page-${++pageNumber}`))) .mockReturnValueOnce(captured(0, PROVIDER_GET_OUTPUT)); const result = await createCliOpenShellProviderAdapter({ run }).getProvider({ target: selectedOpenShellGateway(), @@ -208,17 +213,37 @@ describe("CLI OpenShell provider credential expiration metadata", () => { expect(run).toHaveBeenCalledTimes(11); }); + it("rejects a repeated continuation token", async () => { + const run = vi + .fn() + .mockReturnValue(captured(0, inventory([], "same-page"))) + .mockReturnValueOnce(captured(0, PROVIDER_GET_OUTPUT)); + const result = await createCliOpenShellProviderAdapter({ run }).getProvider({ + target: selectedOpenShellGateway(), + providerName: "search-prod", + includeCredentialExpirations: true, + }); + expect(result).toMatchObject({ ok: false, error: { kind: "schema" } }); + expect(run).toHaveBeenCalledTimes(3); + }); + it.each([ + ["legacy unpaginated output", JSON.stringify([{ name: "search-prod", credential_keys: [] }])], + ["missing continuation token", JSON.stringify({ providers: [] })], + ["invalid continuation token", JSON.stringify({ providers: [], next_page_token: 1 })], + ["oversized continuation token", inventory([], "a".repeat(4097))], + ["NUL in continuation token", inventory([], "bad\0token")], + ["missing target on the last page", inventory([{ name: "other-provider" }])], [ "duplicate target", - JSON.stringify([ + inventory([ { name: "search-prod", credential_keys: [], credential_expires_at_ms: {} }, { name: "search-prod", credential_keys: [], credential_expires_at_ms: {} }, ]), ], [ "invalid credential keys", - JSON.stringify([ + inventory([ { name: "search-prod", credential_keys: "TAVILY_API_KEY", @@ -228,7 +253,7 @@ describe("CLI OpenShell provider credential expiration metadata", () => { ], [ "unsafe credential key", - JSON.stringify([ + inventory([ { name: "search-prod", credential_keys: ["TAVILY_API_KEY"], @@ -238,7 +263,7 @@ describe("CLI OpenShell provider credential expiration metadata", () => { ], [ "unsafe expiry timestamp", - JSON.stringify([ + inventory([ { name: "search-prod", credential_keys: ["TAVILY_API_KEY"], @@ -248,7 +273,7 @@ describe("CLI OpenShell provider credential expiration metadata", () => { ], [ "out-of-range expiry timestamp", - JSON.stringify([ + inventory([ { name: "search-prod", credential_keys: ["TAVILY_API_KEY"], diff --git a/src/lib/adapters/openshell/provider-adapter-cli-registered.test.ts b/src/lib/adapters/openshell/provider-adapter-cli-registered.test.ts new file mode 100644 index 00000000000..8092dab5bf3 --- /dev/null +++ b/src/lib/adapters/openshell/provider-adapter-cli-registered.test.ts @@ -0,0 +1,49 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 +import { it, expect, vi } from "vitest"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; +import { createCliOpenShellProviderAdapter } from "./provider-adapter-cli"; +import { namedOpenShellGateway } from "./sandbox-observer"; +function captured(status: number, stdout: string) { + return { status, stdout, stderr: "" }; +} +it("reads operator inference projection only when explicitly requested", async () => { + const profile = registeredInferenceFixture().profile; + const run = vi.fn(() => captured(0, JSON.stringify(profile))); + const adapter = createCliOpenShellProviderAdapter({ run, environment: {} }); + const request = { + target: namedOpenShellGateway("nemoclaw-18080"), + profileType: "operator-inference", + }; + await expect(adapter.inspectProviderProfile(request)).resolves.toEqual({ + ok: true, + value: { credentialKeys: ["VENDOR_API_KEY"] }, + }); + await expect( + adapter.inspectProviderProfile({ ...request, includeInferenceProjection: true }), + ).resolves.toMatchObject({ + ok: true, + value: { + inference: { + digest: expect.stringMatching(/^[a-f0-9]{64}$/), + credentials: [ + { envVars: ["VENDOR_API_KEY"], authStyle: "bearer", headerName: "authorization" }, + ], + endpoints: [{ host: "native.example", port: 443 }], + }, + }, + }); + expect(run).toHaveBeenLastCalledWith( + [ + "provider", + "profile", + "-g", + "nemoclaw-18080", + "export", + "operator-inference", + "--output", + "json", + ], + expect.any(Object), + ); +}); diff --git a/src/lib/adapters/openshell/provider-adapter-cli.ts b/src/lib/adapters/openshell/provider-adapter-cli.ts index 6e5bbf4d743..878b2722572 100644 --- a/src/lib/adapters/openshell/provider-adapter-cli.ts +++ b/src/lib/adapters/openshell/provider-adapter-cli.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { inspectInferenceProviderProfile } from "./provider-profile"; +import { requireSandboxPolicyAuthority } from "./provider-policy"; import { SUPPORTED_OPENSHELL_FALLBACK_VERSION, usesOpenShellNativeProjection, @@ -578,62 +580,17 @@ export function createCliOpenShellProviderAdapter( async (request) => { const targetError = namedGatewayEndpointOverrideError(request.target, environment); if (targetError) return failure(targetError); - const read = () => { - const result = invoke(["settings", "get", "--global", "--json"], request); - const error = commandError(result); - if (error) return failure(error); - try { - const document = JSON.parse(bufferOrStringToText(result.stdout)); - const value = document.settings?.providers_v2_enabled; - if (document.scope === "global" && ["true", "false", ""].includes(value)) { - return success(value); - } - } catch { - /* Fail closed on incompatible settings output. */ - } - return failure({ - kind: "schema", - message: "OpenShell did not return the global provider policy setting.", - }); - }; - const before = read(); - if (!before.ok) return before; - if (before.value === "true") return success(undefined); - if (before.value === "false") { + try { + await requireSandboxPolicyAuthority(async (args) => invoke(args, request)); + return success(undefined); + } catch (error) { return failure({ kind: "command", - reason: "conflict", + reason: "uncertain", message: - "Native NVIDIA inference requires provider-derived policy, but providers_v2_enabled is explicitly disabled on this gateway. Ask the gateway administrator to enable it.", + error instanceof Error ? error.message : "Could not verify sandbox policy authority.", }); } - // OpenShell 0.0.116 defaults this gate off. Preserve explicit administrator - // choices; enable only its unset default before publishing a native provider. - const updated = invoke( - [ - "settings", - "set", - "--global", - "--key", - "providers_v2_enabled", - "--value", - "true", - "--yes", - ], - request, - ); - const error = commandError(updated); - if (error) return failure(error); - const after = read(); - if (!after.ok) return after; - return after.value === "true" - ? success(undefined) - : failure({ - kind: "command", - reason: "uncertain", - message: - "OpenShell did not confirm provider-derived policy activation. Inspect gateway settings before retrying.", - }); }; const listProviders: OpenShellProviderAdapter["listProviders"] = async (request) => { @@ -782,6 +739,7 @@ export function createCliOpenShellProviderAdapter( const credentialState = parseCliOpenShellProviderCredentialState( commandStdout(inventory), request.providerName, + projectedRuntime, ); if (credentialState && "nextPageToken" in credentialState) { if (!projectedRuntime || seenPageTokens.has(credentialState.nextPageToken)) { @@ -905,7 +863,10 @@ export function createCliOpenShellProviderAdapter( isDeepStrictEqual(JSON.parse(exported) as unknown, snapshot), ); } - return success({ credentialKeys }); + const inference = request.includeInferenceProjection + ? inspectInferenceProviderProfile(commandStdout(result)) + : undefined; + return success({ credentialKeys, ...(inference ? { inference } : {}) }); }; const deleteProvider: OpenShellProviderAdapter["deleteProvider"] = async ( diff --git a/src/lib/adapters/openshell/provider-adapter.ts b/src/lib/adapters/openshell/provider-adapter.ts index 5d959050b55..a69320841fa 100644 --- a/src/lib/adapters/openshell/provider-adapter.ts +++ b/src/lib/adapters/openshell/provider-adapter.ts @@ -1,7 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import type { CheckedInProviderProfileContract } from "./provider-profile"; +import type { + CheckedInProviderProfileContract, + InferenceProviderProfileProjection, +} from "./provider-profile"; import type { OpenShellGatewayTarget } from "./sandbox-observer"; @@ -89,6 +92,8 @@ export type OpenShellProviderMetadata = Readonly<{ export type OpenShellProviderProfileInspection = Readonly<{ credentialKeys: readonly string[]; + /** Non-secret declared inference projection; never an ownership grant. */ + inference?: InferenceProviderProfileProjection; }>; export type CreateOpenShellProviderRequest = OpenShellProviderRequest & @@ -121,6 +126,7 @@ export type ImportOpenShellProviderProfileRequest = OpenShellProviderRequest & export type InspectOpenShellProviderProfileRequest = OpenShellProviderRequest & Readonly<{ profileType: string; + includeInferenceProjection?: boolean; expectedProfile?: CheckedInProviderProfileContract; /** Verify operator-owned registration inputs and retain an advisory profile snapshot. */ registrationCredentials?: readonly string[] | "from-existing"; diff --git a/src/lib/adapters/openshell/provider-inventory-projected.test.ts b/src/lib/adapters/openshell/provider-inventory-projected.test.ts index 753d463d32a..5808bd25667 100644 --- a/src/lib/adapters/openshell/provider-inventory-projected.test.ts +++ b/src/lib/adapters/openshell/provider-inventory-projected.test.ts @@ -12,7 +12,7 @@ vi.mock("../../onboard/openshell-version", async (original) => ({ }, })); -describe.each(["0.1.3", "0.1.5"])("OpenShell %s projection", (version) => { +describe.each(["0.1.3", "0.1.5", "0.1.7"])("OpenShell %s projection", (version) => { beforeEach(() => { runtime.version = version; }); diff --git a/src/lib/adapters/openshell/provider-metadata-cli.ts b/src/lib/adapters/openshell/provider-metadata-cli.ts index 0d209c89d27..6b830dcd638 100644 --- a/src/lib/adapters/openshell/provider-metadata-cli.ts +++ b/src/lib/adapters/openshell/provider-metadata-cli.ts @@ -156,6 +156,7 @@ function parseProviderKeyArray(value: unknown): readonly string[] | null { export function parseCliOpenShellProviderCredentialState( output: string, providerName: string, + projectedRuntime: boolean, ): CliOpenShellProviderCredentialState | { nextPageToken: string } | null | undefined { if ( !isValidCliOpenShellProviderIdentifier(providerName) || @@ -170,8 +171,9 @@ export function parseCliOpenShellProviderCredentialState( } catch { return null; } + if (Array.isArray(parsed) === projectedRuntime) return null; let nextPageToken: string | undefined; - if (!Array.isArray(parsed)) { + if (projectedRuntime) { if (!parsed || typeof parsed !== "object") return null; const page = parsed as Record; if ( diff --git a/src/lib/adapters/openshell/provider-policy-cli.test.ts b/src/lib/adapters/openshell/provider-policy-cli.test.ts index 82325f2bdcf..e8b6049e623 100644 --- a/src/lib/adapters/openshell/provider-policy-cli.test.ts +++ b/src/lib/adapters/openshell/provider-policy-cli.test.ts @@ -4,75 +4,43 @@ import { describe, expect, it, vi } from "vitest"; import { createCliOpenShellProviderAdapter, type RunProviderCommand } from "./provider-adapter-cli"; import { namedOpenShellGateway } from "./sandbox-observer"; - -function captured(status: number | null, stdout = "", stderr = "") { - return { status, stdout, stderr }; -} +import { absentPolicy } from "./provider-policy.test-support"; describe("native provider policy prerequisite", () => { const target = namedOpenShellGateway("brev"); - const settings = (value: string) => - captured(0, JSON.stringify({ scope: "global", settings: { providers_v2_enabled: value } })); - - it("enables the pinned runtime's unset default and confirms the named gateway", async () => { - const run = vi - .fn() - .mockReturnValueOnce(settings("")) - .mockReturnValueOnce(captured(0)) - .mockReturnValueOnce(settings("true")); + it("verifies sandbox policy authority without mutating the named gateway", async () => { + const run = vi.fn().mockReturnValue(absentPolicy); await expect( createCliOpenShellProviderAdapter({ run }).ensureProviderPolicyComposition({ target }), ).resolves.toEqual({ ok: true, value: undefined }); expect(run.mock.calls.map(([args]) => args)).toEqual([ - ["settings", "get", "-g", "brev", "--global", "--json"], - [ - "settings", - "set", - "-g", - "brev", - "--global", - "--key", - "providers_v2_enabled", - "--value", - "true", - "--yes", - ], - ["settings", "get", "-g", "brev", "--global", "--json"], + ["policy", "list", "-g", "brev", "--global", "--page-size", "1", "--output", "json"], ]); }); - - it.each(["true", "false", "invalid"])( - "preserves existing %s settings without mutation", - async (value) => { - const run = vi.fn().mockReturnValue(settings(value)); - const result = await createCliOpenShellProviderAdapter({ - run, - }).ensureProviderPolicyComposition({ target }); - expect(result.ok).toBe(value === "true"); - expect(run).toHaveBeenCalledTimes(1); - }, - ); - - it("does not retry an ambiguous settings mutation", async () => { + it.each(["loaded", "superseded"])("checks the current %s global revision", async (status) => { const run = vi .fn() - .mockReturnValueOnce(settings("")) - .mockReturnValueOnce(captured(null, "", "connection lost")); + .mockReturnValueOnce({ + status: 0, + stdout: JSON.stringify({ revisions: [{}], next_page_token: "" }), + }) + .mockReturnValueOnce({ status: 0, stdout: JSON.stringify({ scope: "global", status }) }); + const result = await createCliOpenShellProviderAdapter({ run }).ensureProviderPolicyComposition( + { target }, + ); + expect(result.ok).toBe(status === "superseded"); + expect(run.mock.calls.map(([args]) => args[1])).toEqual(["list", "get"]); + }); + it.each([ + { status: null, stdout: "", stderr: "connection lost" }, + { status: 1, stdout: "", stderr: "permission denied" }, + { status: 0, stdout: "{}", stderr: "" }, + ])("stops on an unverified response without changing gateway state", async (result) => { + const run = vi.fn().mockReturnValue(result); expect( (await createCliOpenShellProviderAdapter({ run }).ensureProviderPolicyComposition({ target })) .ok, ).toBe(false); - expect(run).toHaveBeenCalledTimes(2); - }); - - it("fails when the gateway does not confirm activation", async () => { - const run = vi - .fn() - .mockReturnValueOnce(settings("")) - .mockReturnValueOnce(captured(0)) - .mockReturnValueOnce(settings("")); - expect( - await createCliOpenShellProviderAdapter({ run }).ensureProviderPolicyComposition({ target }), - ).toMatchObject({ ok: false, error: { reason: "uncertain" } }); + expect(run).toHaveBeenCalledTimes(1); }); }); diff --git a/src/lib/adapters/openshell/provider-policy-projected.test.ts b/src/lib/adapters/openshell/provider-policy-projected.test.ts index cc44f3de5be..cf23668cc35 100644 --- a/src/lib/adapters/openshell/provider-policy-projected.test.ts +++ b/src/lib/adapters/openshell/provider-policy-projected.test.ts @@ -12,7 +12,7 @@ vi.mock("../../onboard/openshell-version", async (original) => ({ }, })); -describe.each(["0.1.3", "0.1.5"])("OpenShell %s projection", (version) => { +describe.each(["0.1.3", "0.1.5", "0.1.7"])("OpenShell %s projection", (version) => { beforeEach(() => { runtime.version = version; }); diff --git a/src/lib/adapters/openshell/provider-policy.test-support.ts b/src/lib/adapters/openshell/provider-policy.test-support.ts index 023473cb772..39933e19e60 100644 --- a/src/lib/adapters/openshell/provider-policy.test-support.ts +++ b/src/lib/adapters/openshell/provider-policy.test-support.ts @@ -3,33 +3,23 @@ import { vi } from "vitest"; -export const absentPolicy = { status: 0, stdout: "", stderr: "No global policy history found\n" }; -export function fixture(enabled = "true", policy = absentPolicy) { - const run = vi.fn(async (args: string[]) => { - if (args[0] === "policy") { - if (args[1] === "get" && policy === absentPolicy) - return { - status: 1, - stdout: "", - stderr: 'status: NotFound, message: "no global policy revision found"', - }; - if (args[1] === "list" && policy !== absentPolicy && policy.status === 0) - return { - status: 0, - stdout: "VERSION HASH STATUS CREATED ERROR\n1 fixture Loaded 0\n", - stderr: "", - }; - return policy; +export const absentPolicy = { + status: 0, + stdout: JSON.stringify({ revisions: [], next_page_token: "" }), + stderr: "", +}; +export function fixture(policy = absentPolicy) { + return vi.fn(async (args: string[]) => { + if (args[0] !== "policy") throw new Error("Unexpected provider-policy command"); + if (args[1] === "list") { + return policy === absentPolicy + ? absentPolicy + : { + status: 0, + stdout: JSON.stringify({ revisions: [{ scope: "global" }], next_page_token: "" }), + stderr: "", + }; } - if (args[1] === "set") { - enabled = "true"; - return { status: 0, stdout: "", stderr: "" }; - } - return { - status: 0, - stdout: JSON.stringify({ scope: "global", settings: { providers_v2_enabled: enabled } }), - stderr: "", - }; + return policy; }); - return run; } diff --git a/src/lib/adapters/openshell/provider-policy.test.ts b/src/lib/adapters/openshell/provider-policy.test.ts index ba1280b8c91..97eda092193 100644 --- a/src/lib/adapters/openshell/provider-policy.test.ts +++ b/src/lib/adapters/openshell/provider-policy.test.ts @@ -3,57 +3,38 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import { initializeNativeProviderPolicy, requireNativeProviderPolicy } from "./provider-policy"; - import type { OpenShellProviderAdapter, OpenShellProviderMetadata } from "./provider-adapter"; import { ensureNativeLocalProvider } from "../../inference/native-local/profile"; -import { redactOnboardErrorText } from "../../onboard/diagnostics/redaction"; - -import { fixture } from "./provider-policy.test-support"; +import { absentPolicy, fixture } from "./provider-policy.test-support"; afterEach(() => vi.unstubAllEnvs()); describe("native provider policy prerequisites", () => { - it("reads the named existing gateway without changing its settings (#12558)", async () => { - const run = fixture(); - await requireNativeProviderPolicy("selected", run); - expect(run.mock.calls.map(([args]) => args)).toEqual([ - ["settings", "get", "-g", "selected", "--global", "--json"], - ["policy", "list", "-g", "selected", "--global", "--limit", "1"], - ]); - }); - it.each(["false", ""])( - "refuses %s composition without writing (#12558)", - async (value) => { - const run = fixture(value); + it.each([requireNativeProviderPolicy, initializeNativeProviderPolicy])( + "verifies the named gateway without changing settings", + async (check) => { + const run = fixture(); + await check("selected", run); + expect(run.mock.calls.map(([args]) => args)).toEqual([ + ["policy", "list", "-g", "selected", "--global", "--page-size", "1", "--output", "json"], + ]); + }, + ); + it.each(["loaded", "pending", "failed", "unspecified"])( + "refuses a global override with status %s", + async (status) => { + const run = fixture({ + status: 0, + stdout: JSON.stringify({ scope: "global", status }), + stderr: "", + }); await expect(requireNativeProviderPolicy("selected", run)).rejects.toThrow( - /administrator.*enable composition/, + /global policy override/, ); - expect(run.mock.calls.every(([args]) => args[1] === "get")).toBe(true); + expect(run.mock.calls.map(([args]) => args[1])).toEqual(["list", "get"]); }, ); - it("keeps the full recovery command visible after onboard line bounding (#13540)", async () => { - const gatewayName = "g".repeat(19); - const run = fixture(""); - const error = await requireNativeProviderPolicy(gatewayName, run).catch((e: unknown) => e); - expect(error).toBeInstanceOf(Error); - const displayed = redactOnboardErrorText((error as Error).message); - expect(displayed).toContain( - `openshell settings set -g ${gatewayName} --global --key providers_v2_enabled --value true --yes`, - ); - expect(displayed).toContain("Then retry."); - }); - it("refuses an active global override even when composition is enabled (#12558)", async () => { - const run = fixture("true", { - status: 0, - stdout: JSON.stringify({ scope: "global", status: "loaded" }), - stderr: "", - }); - await expect(requireNativeProviderPolicy("selected", run)).rejects.toThrow( - /global policy override/, - ); - expect(run.mock.calls.map(([args]) => args[1])).toEqual(["get", "list", "get"]); - }); - it("accepts deleted global policy history (#12558)", async () => { - const run = fixture("true", { + it("accepts deleted global policy history", async () => { + const run = fixture({ status: 0, stdout: JSON.stringify({ scope: "global", status: "superseded" }), stderr: "", @@ -69,117 +50,42 @@ describe("native provider policy prerequisites", () => { }, { status: 1, stdout: "", stderr: "permission denied" }, { status: null, stdout: "", stderr: "connection lost" }, - ])( - "stops fresh initialization when policy history cannot be verified (#12558)", - async (response) => { - const run = vi.fn(async () => response); - await expect(initializeNativeProviderPolicy("selected", run)).rejects.toThrow( - /prerequisites/, - ); - expect(run).toHaveBeenCalledExactlyOnceWith([ - "policy", - "list", - "-g", - "selected", - "--global", - "--limit", - "1", - ]); - }, - ); - it("does not accept policy history without a verified current revision (#12558)", async () => { - const run = fixture(); - run.mockResolvedValueOnce({ - status: 0, - stdout: "VERSION HASH STATUS CREATED ERROR\n", - stderr: "", - }); - await expect(initializeNativeProviderPolicy("selected", run)).rejects.toThrow(/prerequisites/); - expect(run.mock.calls.map(([args]) => args[1])).toEqual(["list", "get"]); - }); - it("initializes a fresh gateway once and verifies applied settings (#12558)", async () => { - const run = fixture(""); - await initializeNativeProviderPolicy("selected", run); - expect(run.mock.calls.filter(([args]) => args[1] === "set")).toHaveLength(1); - expect(run.mock.calls.at(-2)?.[0]).toContain("get"); - }); - it("reconciles a lost initialization response without another write (#12558)", async () => { - const original = fixture(); - const run = vi.fn(async (args: string[]) => - args[1] === "set" ? { status: null, error: new Error("connection lost") } : original(args), - ); - await expect(initializeNativeProviderPolicy("selected", run)).resolves.toBeUndefined(); - expect(run.mock.calls.filter(([args]) => args[1] === "set")).toHaveLength(1); - }); - it("observes the setting after an initialization transport exception (#12558)", async () => { - const run = fixture(); - run.mockResolvedValueOnce({ status: 0, stdout: "", stderr: "No global policy history found" }); - run.mockRejectedValueOnce(new Error("connection lost")); - await expect(initializeNativeProviderPolicy("selected", run)).resolves.toBeUndefined(); - expect(run.mock.calls.filter(([args]) => args[1] === "set")).toHaveLength(1); - }); - it.each([ - { status: 1, stdout: "", stderr: "connection lost" }, { status: 0, stdout: "{broken", stderr: "" }, { status: 0, stdout: "[]", stderr: "" }, - ])("rejects an unverified gateway response (#12558)", async (response) => { - await expect(requireNativeProviderPolicy("selected", async () => response)).rejects.toThrow(); - }); - it("continues only after verifying disabled composition and sandbox authority (#12558)", async () => { - const original = fixture("false"); - const run = vi.fn(async (args: string[]) => - args[1] === "set" ? { status: 1, stdout: "", stderr: "unavailable" } : original(args), - ); - await expect(initializeNativeProviderPolicy("selected", run)).resolves.toBe("disabled"); - expect(run.mock.calls.map(([args]) => args[1])).toEqual(["list", "set", "get", "list"]); + { status: 0, stdout: "{}", stderr: "" }, + { status: 0, stdout: JSON.stringify({ revisions: [], next_page_token: "next" }), stderr: "" }, + { status: 0, stdout: " ".repeat(65_537), stderr: "" }, + ])("rejects unverified global policy history", async (response) => { + const run = vi.fn(async () => response); await expect(requireNativeProviderPolicy("selected", run)).rejects.toThrow( - /enable composition/, + /prerequisites|policy history|response exceeded/, ); - expect(run.mock.calls.filter(([args]) => args[1] === "set")).toHaveLength(1); + expect(run).toHaveBeenCalledTimes(1); }); - it.each(["", "", null, false])( - "rejects indeterminate composition value %s after initialization (#12558)", - async (value) => { - const run = fixture(); - run.mockResolvedValueOnce({ - status: 0, - stdout: "", - stderr: "No global policy history found", - }); - run.mockResolvedValueOnce({ status: 1, stdout: "", stderr: "unavailable" }); - run.mockResolvedValueOnce({ - status: 0, - stdout: JSON.stringify({ scope: "global", settings: { providers_v2_enabled: value } }), - stderr: "", - }); - await expect(initializeNativeProviderPolicy("selected", run)).rejects.toThrow( - /Could not verify/, - ); - expect(run.mock.calls.filter(([args]) => args[1] === "set")).toHaveLength(1); - }, - ); - it("rejects a policy override introduced during disabled initialization (#12558)", async () => { - const run = fixture("false", { + it("does not accept history without a verified current revision", async () => { + const run = fixture({ status: 1, stdout: "", stderr: "permission denied" }); + await expect(requireNativeProviderPolicy("selected", run)).rejects.toThrow(/prerequisites/); + expect(run.mock.calls.map(([args]) => args[1])).toEqual(["list", "get"]); + }); + it("rejects a superseded revision from another scope", async () => { + const run = fixture({ status: 0, - stdout: JSON.stringify({ scope: "global", status: "loaded" }), + stdout: JSON.stringify({ scope: "sandbox", status: "superseded" }), stderr: "", }); - run.mockResolvedValueOnce({ status: 0, stdout: "", stderr: "No global policy history found" }); - run.mockResolvedValueOnce({ status: 1, stdout: "", stderr: "unavailable" }); - await expect(initializeNativeProviderPolicy("selected", run)).rejects.toThrow( + await expect(requireNativeProviderPolicy("selected", run)).rejects.toThrow( /global policy override/, ); - expect(run.mock.calls.filter(([args]) => args[1] === "set")).toHaveLength(1); }); - it("keeps an unreadable post-write state fatal (#12558)", async () => { - const run = fixture(); - run.mockResolvedValueOnce({ status: 0, stdout: "", stderr: "No global policy history found" }); - run.mockRejectedValueOnce(new Error("lost response")); - run.mockResolvedValueOnce({ status: 1, stdout: "", stderr: "unreachable" }); - await expect(initializeNativeProviderPolicy("selected", run)).rejects.toThrow(/prerequisites/); - expect(run.mock.calls.filter(([args]) => args[1] === "set")).toHaveLength(1); + it("requires successful transport even when the history payload appears empty", async () => { + await expect( + requireNativeProviderPolicy("selected", async () => ({ + ...absentPolicy, + error: new Error("lost response"), + })), + ).rejects.toThrow(/prerequisites/); }); - it("rejects an ambient endpoint override before any command (#12558)", async () => { + it("rejects an ambient endpoint override before any command", async () => { vi.stubEnv("OPENSHELL_GATEWAY_ENDPOINT", "https://another.example:443"); const run = fixture(); await expect(requireNativeProviderPolicy("selected", run)).rejects.toThrow(); @@ -188,8 +94,14 @@ describe("native provider policy prerequisites", () => { }); // Native local callers verify policy prerequisites without taking ownership of settings. -function nativeLocalPolicyFixture(enabled: "true" | "false") { - const policyCommand = fixture(enabled); +function nativeLocalPolicyFixture(override: boolean) { + const policyCommand = override + ? fixture({ + status: 0, + stdout: JSON.stringify({ scope: "global", status: "loaded" }), + stderr: "", + }) + : fixture(); let provider: OpenShellProviderMetadata | undefined; const providerAdapter = { importProviderProfile: vi.fn(async () => ({ ok: true })), @@ -228,15 +140,15 @@ function nativeLocalPolicyFixture(enabled: "true" | "false") { return { action, providerAdapter, policyCommand }; } -it("keeps enabled native-local composition read-only (#12558)", async () => { - const { action, providerAdapter, policyCommand } = nativeLocalPolicyFixture("true"); +it("keeps native-local composition read-only (#12558)", async () => { + const { action, providerAdapter, policyCommand } = nativeLocalPolicyFixture(false); await expect(action).resolves.toMatchObject({ sandboxName: "alice" }); expect(providerAdapter.ensureProviderPolicyComposition).not.toHaveBeenCalled(); expect(policyCommand.mock.calls.some(([args]) => args.includes("set"))).toBe(false); }); -it("rejects disabled native-local composition without activating it (#12558)", async () => { - const { action, providerAdapter, policyCommand } = nativeLocalPolicyFixture("false"); +it("rejects global policy override before native-local provider mutation (#12558)", async () => { + const { action, providerAdapter, policyCommand } = nativeLocalPolicyFixture(true); await expect(action).rejects.toThrow("administrator"); expect(providerAdapter.ensureProviderPolicyComposition).not.toHaveBeenCalled(); expect(policyCommand.mock.calls.some(([args]) => args.includes("set"))).toBe(false); diff --git a/src/lib/adapters/openshell/provider-policy.ts b/src/lib/adapters/openshell/provider-policy.ts index d40890c829f..16257b136b8 100644 --- a/src/lib/adapters/openshell/provider-policy.ts +++ b/src/lib/adapters/openshell/provider-policy.ts @@ -60,7 +60,7 @@ function document(result: PolicyCommandResult): Record { throw new Error("OpenShell returned invalid provider-policy prerequisites."); } -async function requireSandboxPolicyAuthority(run: PolicyCommand): Promise { +export async function requireSandboxPolicyAuthority(run: PolicyCommand): Promise { if (usesOpenShellNativeProjection(SUPPORTED_OPENSHELL_FALLBACK_VERSION)) { const history = document( await run(["policy", "list", "--global", "--page-size", "1", "--output", "json"]), @@ -91,7 +91,7 @@ async function requireSandboxPolicyAuthority(run: PolicyCommand): Promise const policy = document(await run(["policy", "get", "--global", "--output", "json"])); if (policy.scope === "global" && policy.status === "superseded") return; throw new Error( - "Native local inference requires sandbox-owned policy. This gateway has a global policy override; ask its administrator to prepare the gateway before retrying.", + "Native inference requires sandbox-owned policy. This gateway has a global policy override; ask its administrator to prepare the gateway before retrying.", ); } @@ -127,7 +127,7 @@ export async function requireNativeProviderPolicy( await requireSandboxPolicyAuthority(run); } -/** Called only by the owner immediately after creating a fresh gateway. */ +/** Verify fresh gateway policy authority without changing gateway settings. */ export async function initializeNativeProviderPolicy( gatewayName: string, command?: PolicyCommand, diff --git a/src/lib/adapters/openshell/provider-profile.test.ts b/src/lib/adapters/openshell/provider-profile.test.ts index 778ab7889d3..3aaeba4c290 100644 --- a/src/lib/adapters/openshell/provider-profile.test.ts +++ b/src/lib/adapters/openshell/provider-profile.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import assert from "node:assert/strict"; import path from "node:path"; import { describe, expect, it } from "vitest"; @@ -13,6 +14,35 @@ import { const PROFILE_ID = "langfuse-hermes-v1"; describe("OpenShell endpointless provider profiles", () => { + it.each([ + ["omitted files", undefined, true], + ["empty files", [], true], + [ + "workload file", + [{ path: "client.yaml", content: "unowned", env_var: "CLIENT_CONFIG" }], + false, + ], + ["malformed files", {}, false], + ["null files", null, false], + ] as const)( + "checks provider-managed files before profile reuse: %s", + (_name, files, accepted) => { + const profile = { + id: PROFILE_ID, + credentials: [], + endpoints: [], + binaries: [], + inference_capable: false, + }; + const expected = parseCheckedInProviderProfileContract(JSON.stringify(profile)); + assert.ok(expected, "Valid fixture profile"); + const exported = JSON.stringify({ ...profile, files }); + + expect(exportedProviderProfileMatchesContract(exported, expected)).toBe(accepted); + expect(parseCheckedInProviderProfileContract(exported) !== null).toBe(accepted); + }, + ); + it("resolves a checked-in profile path for the requested profile", () => { expect(endpointlessProviderProfilePath("/repo", PROFILE_ID)).toBe( path.join("/repo", "nemoclaw-blueprint", "provider-profiles", "langfuse-hermes-v1.yaml"), @@ -20,6 +50,7 @@ describe("OpenShell endpointless provider profiles", () => { }); it.each([ + ["stable_placeholder", true], ["path_template", "/v1/{credential}"], ["token_grant", { endpoint: "https://attacker.example/token" }], ])("rejects an unowned exported credential control: %s (#12558)", (key, value) => { diff --git a/src/lib/adapters/openshell/provider-profile.ts b/src/lib/adapters/openshell/provider-profile.ts index a0f3855b7ad..568e31e1132 100644 --- a/src/lib/adapters/openshell/provider-profile.ts +++ b/src/lib/adapters/openshell/provider-profile.ts @@ -2,6 +2,21 @@ // SPDX-License-Identifier: Apache-2.0 import path from "node:path"; +import { createHash } from "node:crypto"; +export type InferenceProviderProfileProjection = Readonly<{ + digest: string; + credentials: readonly Readonly<{ + envVars: readonly string[]; + authStyle: string; + headerName: string; + }>[]; + endpoints: readonly Readonly<{ + host: string; + port: number; + ports: readonly number[]; + enforcesCredentialedRest: boolean; + }>[]; +}>; import { isDeepStrictEqual } from "node:util"; import YAML from "yaml"; @@ -48,6 +63,14 @@ function providerProfileBoundary(value: unknown): ProviderProfileBoundary | null ) { return null; } + // NemoClaw's owned profiles do not materialize workload files or file-path + // environment variables. OpenShell 0.1.3 can, so reject that extra authority. + if ( + profile.files !== undefined && + (!Array.isArray(profile.files) || profile.files.length !== 0) + ) { + return null; + } const credentials = profile.credentials.map((value) => { const credential = recordValue(value); if ( @@ -132,3 +155,69 @@ export function isMissingProviderProfile(output: string, profileId: string): boo export function endpointlessProviderProfilePath(root: string, profileId: string): string { return path.join(root, "nemoclaw-blueprint", "provider-profiles", `${profileId}.yaml`); } + +/** Read declared inference fields without claiming ownership of an operator profile. */ +export function inspectInferenceProviderProfile( + source: string, +): InferenceProviderProfileProjection | undefined { + const profile = recordValue(JSON.parse(source) as unknown); + if ( + !profile || + profile.inference_capable !== true || + !Array.isArray(profile.credentials) || + !Array.isArray(profile.endpoints) + ) + return undefined; + const credentials: { envVars: string[]; authStyle: string; headerName: string }[] = []; + for (const value of profile.credentials) { + const credential = recordValue(value); + if ( + !credential || + !Array.isArray(credential.env_vars) || + credential.env_vars.some( + (key) => typeof key !== "string" || !/^[A-Z][A-Z0-9_]{0,255}$/u.test(key), + ) || + typeof credential.auth_style !== "string" || + typeof credential.header_name !== "string" + ) + return undefined; + credentials.push({ + envVars: credential.env_vars as string[], + authStyle: credential.auth_style, + headerName: credential.header_name.toLowerCase(), + }); + } + const endpoints: InferenceProviderProfileProjection["endpoints"][number][] = []; + for (const value of profile.endpoints) { + const endpoint = recordValue(value); + const ports = endpoint?.ports === undefined ? [] : endpoint.ports; + if ( + !endpoint || + typeof endpoint.host !== "string" || + !Array.isArray(ports) || + ports.some((port) => !Number.isInteger(port) || port < 1 || port > 65535) || + (!ports.length && + (!Number.isInteger(endpoint.port) || + Number(endpoint.port) < 1 || + Number(endpoint.port) > 65535)) + ) + return undefined; + endpoints.push({ + host: endpoint.host.toLowerCase(), + port: Number(endpoint.port ?? 0), + ports: ports.length ? ports : [Number(endpoint.port)], + enforcesCredentialedRest: + typeof endpoint.protocol === "string" && + endpoint.protocol.toLowerCase() === "rest" && + endpoint.enforcement === "enforce" && + (endpoint.tls === undefined || endpoint.tls === "") && + (endpoint.allow_uninspected_credentials === undefined || + endpoint.allow_uninspected_credentials === false), + }); + } + return { + digest: createHash("sha256").update(JSON.stringify(profile)).digest("hex"), + credentials, + endpoints, + }; +} diff --git a/src/lib/adapters/openshell/providers.test.ts b/src/lib/adapters/openshell/providers.test.ts index 43f3ce48679..a38d696fac5 100644 --- a/src/lib/adapters/openshell/providers.test.ts +++ b/src/lib/adapters/openshell/providers.test.ts @@ -144,7 +144,10 @@ describe("OpenShell provider evidence", () => { }, }); expect(raw.getProviderProfile).toHaveBeenCalledExactlyOnceWith( - { id: profileContract, workspace: "default" }, + { + id: profileContract, + workspaceScope: { selection: { case: "workspace", value: "default" } }, + }, { signal: input.signal }, ); expect(readCredential).not.toHaveBeenCalled(); @@ -177,7 +180,11 @@ describe("OpenShell provider evidence", () => { ], ["credential refresh", { credentials: [{ ...credential, refresh: {} }] }], ["a foreign endpoint", { endpoints: [{ ...endpoint, host: "foreign.example" }] }], - ["a read-write access preset", { endpoints: [{ ...endpoint, access: "read-write" }] }], + ["a read-write access preset", { endpoints: [{ ...endpoint, access: 2 }] }], + ["an unknown access preset", { endpoints: [{ ...endpoint, access: 999 }] }], + ["audit-only enforcement", { endpoints: [{ ...endpoint, enforcement: 2 }] }], + ["unspecified enforcement", { endpoints: [{ ...endpoint, enforcement: 0 }] }], + ["legacy string enforcement", { endpoints: [{ ...endpoint, enforcement: "enforce" }] }], [ "uninspected credentials", { endpoints: [{ ...endpoint, allowUninspectedCredentials: true }] }, @@ -241,7 +248,7 @@ describe("OpenShell provider evidence", () => { config: { OPENAI_BASE_URL: "https://api.example/v1" }, }); expect(raw.getProvider).toHaveBeenCalledWith( - { name: "alpha", workspace: "default" }, + { name: "alpha", workspaceScope: { selection: { case: "workspace", value: "default" } } }, { signal: input.signal }, ); expect(connect).toHaveBeenCalledWith(target); @@ -312,7 +319,12 @@ describe("OpenShell provider evidence", () => { }, }); expect(raw.getProviderProfile).toHaveBeenCalledExactlyOnceWith( - { id: "brave", workspace: profileWorkspace }, + { + id: "brave", + ...(profileWorkspace + ? { workspaceScope: { selection: { case: "workspace", value: profileWorkspace } } } + : {}), + }, { signal: input.signal }, ); expect(readSecret).not.toHaveBeenCalled(); @@ -340,7 +352,7 @@ describe("OpenShell provider evidence", () => { { label: "discovery override", change: { discovery: {} } }, { label: "an empty access preset", - change: { endpoints: [{ ...managedBraveProfile().endpoints[0], access: "" }] }, + change: { endpoints: [{ ...managedBraveProfile().endpoints[0], access: 0 }] }, }, ])("rejects managed Brave profiles with $label (#10904)", async ({ change }) => { const { connect, raw } = fixture(); @@ -367,7 +379,12 @@ describe("OpenShell provider evidence", () => { }); expect(result).toMatchObject({ profileWorkspace: workspace, managedProfile: null }); expect(raw.getProviderProfile).toHaveBeenCalledWith( - { id: "openai", workspace }, + { + id: "openai", + ...(workspace + ? { workspaceScope: { selection: { case: "workspace", value: workspace } } } + : {}), + }, { signal: expect.any(AbortSignal) }, ); }, @@ -523,7 +540,12 @@ describe("OpenShell provider evidence", () => { builtinInferenceEndpoint: "https://integrate.api.nvidia.com/v1", }); expect(raw.getProviderProfile).toHaveBeenCalledWith( - { id: "nvidia", workspace: profileWorkspace }, + { + id: "nvidia", + ...(profileWorkspace + ? { workspaceScope: { selection: { case: "workspace", value: profileWorkspace } } } + : {}), + }, { signal: input.signal }, ); expect(connect).toHaveBeenCalledExactlyOnceWith(target); @@ -756,7 +778,7 @@ describe("OpenShell sandbox export evidence", () => { }); expect(JSON.stringify(result)).not.toContain(canary); expect(raw.getSandbox).toHaveBeenCalledWith( - { name: "alpha", workspace: "default" }, + { name: "alpha", workspaceScope: { selection: { case: "workspace", value: "default" } } }, { signal: input.signal }, ); }); @@ -786,10 +808,10 @@ describe("OpenShell sandbox export evidence", () => { }); it.each([1, 2])( - "reads source %i configuration by verified ID without settings values", + "reads source %i configuration by name and verifies its ID without settings values", async (policySource) => { const { connect, raw } = fixture(); - const input = { ...request(), sandboxId: "verified-id" }; + const input = { ...request(), sandboxId: "resource-id" }; const response = await raw.getSandboxConfig(); raw.getSandboxConfig.mockClear().mockResolvedValue({ ...response, @@ -798,7 +820,7 @@ describe("OpenShell sandbox export evidence", () => { }); const result = await createSandboxConfig(connect, async () => "version: 1\n").get(input); expect(result).toMatchObject({ - sandboxId: "verified-id", + sandboxId: "resource-id", workspace: "default", revision: 3, configRevision: "18446744073709551615", @@ -807,13 +829,46 @@ describe("OpenShell sandbox export evidence", () => { policy: { document: "version: 1\n", appliedRevision: policySource === 1 ? 3 : 4 }, }); expect(raw.getSandboxConfig).toHaveBeenCalledWith( - { sandboxId: "verified-id" }, + { + name: "alpha", + workspaceScope: { selection: { case: "workspace", value: "default" } }, + }, { signal: input.signal }, ); expect(JSON.stringify(result)).not.toContain(canary); }, ); + it.each([{ id: "replacement-id" }, { workspace: "other" }])( + "rejects changed sandbox identity %j after reading configuration", + async (change) => { + const { connect, raw } = fixture(); + const response = await raw.getSandbox(); + raw.getSandbox.mockClear().mockResolvedValue({ + ...response, + sandbox: { + ...response.sandbox, + metadata: { ...response.sandbox.metadata, ...change }, + }, + }); + const serialize = vi.fn(async () => "version: 1\n"); + await expect( + createSandboxConfig(connect, serialize).get({ ...request(), sandboxId: "resource-id" }), + ).rejects.toMatchObject({ kind: "schema" }); + expect(serialize).not.toHaveBeenCalled(); + }, + ); + + it("rejects a missing sandbox after reading configuration", async () => { + const { connect, raw } = fixture(); + raw.getSandbox.mockRejectedValue({ code: 5 }); + const serialize = vi.fn(async () => "version: 1\n"); + await expect( + createSandboxConfig(connect, serialize).get({ ...request(), sandboxId: "resource-id" }), + ).rejects.toMatchObject({ kind: "schema" }); + expect(serialize).not.toHaveBeenCalled(); + }); + it("does not serialize a configuration response that arrives after abort", async () => { const { connect, raw } = fixture(); const response = await raw.getSandboxConfig(); @@ -826,7 +881,7 @@ describe("OpenShell sandbox export evidence", () => { await expect( createSandboxConfig(connect, serialize).get({ ...request(), - sandboxId: "verified-id", + sandboxId: "resource-id", signal: controller.signal, }), ).rejects.toMatchObject({ kind: "timeout" }); @@ -845,7 +900,7 @@ describe("OpenShell sandbox export evidence", () => { const response = await raw.getSandboxConfig(); raw.getSandboxConfig.mockResolvedValue({ ...response, ...change }); await expect( - createSandboxConfig(connect).get({ ...request(), sandboxId: "verified-id" }), + createSandboxConfig(connect).get({ ...request(), sandboxId: "resource-id" }), ).rejects.toMatchObject({ kind: "schema" }); }); }); diff --git a/src/lib/adapters/openshell/providers.ts b/src/lib/adapters/openshell/providers.ts index d1481a8a4dc..9592369abb9 100644 --- a/src/lib/adapters/openshell/providers.ts +++ b/src/lib/adapters/openshell/providers.ts @@ -99,7 +99,16 @@ async function readBuiltinNvidiaEndpoint( readValue( BuiltinNvidiaProfileResponseSchema, await client.raw.getProviderProfile( - { id: "nvidia", workspace: profileWorkspace }, + { + id: "nvidia", + ...(profileWorkspace === "" + ? {} + : { + workspaceScope: { + selection: { case: "workspace" as const, value: profileWorkspace }, + }, + }), + }, { signal: request.signal }, ), ); @@ -133,7 +142,16 @@ async function readManagedProfile( try { response = await client.raw.getProviderProfile( // Resolve the actual profile binding; the default-workspace import is managed state. - { id: profileId, workspace: profileWorkspace }, + { + id: profileId, + ...(profileWorkspace === "" + ? {} + : { + workspaceScope: { + selection: { case: "workspace" as const, value: profileWorkspace }, + }, + }), + }, { signal: request.signal }, ); } catch (error) { @@ -333,7 +351,10 @@ export function createProviders( try { // The pinned SDK has no curated gateway provider reader. response = await client.raw.getProvider( - { name, workspace: request.workspace }, + { + name, + workspaceScope: { selection: { case: "workspace", value: request.workspace } }, + }, { signal: request.signal }, ); } catch (error) { diff --git a/src/lib/adapters/openshell/resolve-shared.ts b/src/lib/adapters/openshell/resolve-shared.ts index 050c16aef6d..64d7134224d 100644 --- a/src/lib/adapters/openshell/resolve-shared.ts +++ b/src/lib/adapters/openshell/resolve-shared.ts @@ -3,7 +3,7 @@ import { spawnSync, type SpawnSyncReturns } from "node:child_process"; -import { HERMES_LIFECYCLE_DEFINITION } from "../../domain/lifecycle/hermes-definition"; +import { HERMES_CLI_OPENSHELL_VERSION } from "../../domain/lifecycle/hermes-definition"; import { assertPodmanExecutableAuthority, capturePodmanExecutableAuthority, @@ -13,7 +13,7 @@ import { } from "../podman/executable-authority"; import { resolveOpenshell } from "./resolve"; -export const HERMES_PORTABLE_OPENSHELL_VERSION = HERMES_LIFECYCLE_DEFINITION.openshellVersion; +export const HERMES_PORTABLE_OPENSHELL_VERSION = HERMES_CLI_OPENSHELL_VERSION; const VERSION_TIMEOUT_MS = 5_000; const VERSION_MAX_BUFFER_BYTES = 16 * 1024; const SEMVER_PATTERN = /(?:^|[^0-9.])([0-9]+\.[0-9]+\.[0-9]+)(?![0-9.])/u; diff --git a/src/lib/adapters/openshell/resolve.test.ts b/src/lib/adapters/openshell/resolve.test.ts index 1c4371cf585..45ba8b58951 100644 --- a/src/lib/adapters/openshell/resolve.test.ts +++ b/src/lib/adapters/openshell/resolve.test.ts @@ -48,7 +48,7 @@ function executableAuthorityHarness() { resolve: (env) => env.NEMOCLAW_OPENSHELL_BIN ?? AUTHORITY_BINARY, runVersion: () => ({ status: 0, - stdout: "openshell 0.0.116\n", + stdout: "openshell 0.1.7\n", stderr: "", }), }; @@ -213,7 +213,7 @@ describe("Hermes portable OpenShell executable authority", () => { fs.writeFileSync(executablePath, "openshell-binary", { mode: 0o755 }); const runVersion = vi.fn(() => ({ status: 0, - stdout: "openshell 0.0.116\n", + stdout: "openshell 0.1.7\n", stderr: "", })); const lstat = (filePath: string): PodmanExecutableStat => { @@ -348,7 +348,7 @@ describe("Hermes portable OpenShell executable authority", () => { }, ); - it("captures and reuses the exact canonical OpenShell 0.0.116 generation (#9211)", () => { + it("captures and reuses the canonical OpenShell 0.1.7 generation (#9211)", () => { const harness = executableAuthorityHarness(); const authority = captureHermesPortableOpenShellExecutableAuthority( AUTHORITY_BINARY, @@ -357,7 +357,7 @@ describe("Hermes portable OpenShell executable authority", () => { harness.deps, ); - expect(authority.version).toBe("0.0.116"); + expect(authority.version).toBe("0.1.7"); expect(authority.executable.executablePath).toBe(AUTHORITY_BINARY); expect( assertHermesPortableOpenShellExecutableAuthority(authority, childEnv, childEnv, harness.deps), @@ -434,27 +434,30 @@ describe("Hermes portable OpenShell executable authority", () => { ).toThrow("executable generation changed after reservation"); }); - it("rejects OpenShell 0.0.101 before authority capture or reuse (#9211)", () => { - const harness = executableAuthorityHarness(); - expect(() => - captureHermesPortableOpenShellExecutableAuthority(AUTHORITY_BINARY, childEnv, childEnv, { - ...harness.deps, - runVersion: () => ({ status: 0, stdout: "openshell 0.0.101\n", stderr: "" }), - }), - ).toThrow("requires OpenShell 0.0.116"); - - const reuseHarness = executableAuthorityHarness(); - const authority = captureHermesPortableOpenShellExecutableAuthority( - AUTHORITY_BINARY, - childEnv, - childEnv, - reuseHarness.deps, - ); - expect(() => - assertHermesPortableOpenShellExecutableAuthority(authority, childEnv, childEnv, { - ...reuseHarness.deps, - runVersion: () => ({ status: 0, stdout: "openshell 0.0.101\n", stderr: "" }), - }), - ).toThrow("requires OpenShell 0.0.116"); - }); + it.each(["0.0.101", "0.1.5", "0.1.6", "0.1.8"])( + "rejects OpenShell %s before authority capture or reuse (#9211)", + (version) => { + const harness = executableAuthorityHarness(); + expect(() => + captureHermesPortableOpenShellExecutableAuthority(AUTHORITY_BINARY, childEnv, childEnv, { + ...harness.deps, + runVersion: () => ({ status: 0, stdout: `openshell ${version}\n`, stderr: "" }), + }), + ).toThrow("requires OpenShell 0.1.7"); + + const reuseHarness = executableAuthorityHarness(); + const authority = captureHermesPortableOpenShellExecutableAuthority( + AUTHORITY_BINARY, + childEnv, + childEnv, + reuseHarness.deps, + ); + expect(() => + assertHermesPortableOpenShellExecutableAuthority(authority, childEnv, childEnv, { + ...reuseHarness.deps, + runVersion: () => ({ status: 0, stdout: `openshell ${version}\n`, stderr: "" }), + }), + ).toThrow("requires OpenShell 0.1.7"); + }, + ); }); diff --git a/src/lib/adapters/openshell/sandbox-config.test.ts b/src/lib/adapters/openshell/sandbox-config.test.ts index e5105ea935a..17157550c6b 100644 --- a/src/lib/adapters/openshell/sandbox-config.test.ts +++ b/src/lib/adapters/openshell/sandbox-config.test.ts @@ -66,7 +66,6 @@ describe("SDK policy document conversion", () => { it("preserves untouched policy fields without mutating the input", () => { const endpoint = { protocol: "mcp", - tls: "terminate", allowed_ips: ["192.0.2.1"], json_rpc_max_body_bytes: 4096, mcp: { strict_tool_names: false }, @@ -74,7 +73,6 @@ describe("SDK policy document conversion", () => { const before = structuredClone(endpoint); expect(exportedEndpoint(endpoint)).toEqual({ protocol: "mcp", - tls: "terminate", allowed_ips: ["192.0.2.1"], mcp: { strict_tool_names: false, max_body_bytes: 4096 }, }); @@ -113,7 +111,6 @@ describe("SDK policy document conversion", () => { host: "api.example", port: 443, protocol: "rest", - tls: "terminate", rules: [ { allow: { @@ -129,7 +126,6 @@ describe("SDK policy document conversion", () => { host: "api.example", port: 443, protocol: "rest", - tls: "terminate", rules: [ { allow: { @@ -224,14 +220,25 @@ describe("SDK policy document conversion", () => { }); }); - it("preserves provider-composed rules and ignores the deprecated binary harness flag", () => { + it("preserves provider restrictions without runtime ownership markers", () => { expect( sdkPolicyDocument({ version: 1, network_policies: { "provider:nvidia": { name: "provider:nvidia", - endpoints: [{ host: "api.example", port: 443 }], + endpoints: [ + { + host: "api.example", + port: 443, + advisor_proposed: true, + provider_credentialed: true, + token_grant_owner: "provider-owned-endpoint", + credential_binding: { provider: "nvidia" }, + request_body_credential_rewrite: true, + allow_uninspected_credentials: false, + }, + ], binaries: [{ path: "/usr/bin/curl", harness: true }], }, }, @@ -241,7 +248,15 @@ describe("SDK policy document conversion", () => { network_policies: { "provider:nvidia": { name: "provider:nvidia", - endpoints: [{ host: "api.example", port: 443 }], + endpoints: [ + { + host: "api.example", + port: 443, + credential_binding: { provider: "nvidia" }, + request_body_credential_rewrite: true, + allow_uninspected_credentials: false, + }, + ], binaries: [{ path: "/usr/bin/curl" }], }, }, diff --git a/src/lib/adapters/openshell/sandbox-config.ts b/src/lib/adapters/openshell/sandbox-config.ts index d79a1938db1..5540bd0b1d6 100644 --- a/src/lib/adapters/openshell/sandbox-config.ts +++ b/src/lib/adapters/openshell/sandbox-config.ts @@ -7,6 +7,7 @@ import { sortCanonicalMappings } from "../../config/canonical-mapping"; import { isSandboxPolicyCredentialFree } from "../../policy/sandbox-policy-validation"; import type { SandboxConfiguration } from "../../domain/sandbox/configuration"; import type { OpenShellSandboxPolicyRead } from "./sandbox-policy"; +import { createSandboxes } from "./sandboxes"; import { PolicyJsonSchema, SandboxConfigResponseSchema, @@ -31,21 +32,32 @@ export function createSandboxConfig( ) { return { get: ( - request: ReadRequest & Readonly<{ sandboxId: string }>, + request: ReadRequest & Readonly<{ sandboxId: string; name: string }>, ): Promise => readOpenShell(request, async () => { const sandboxId = text(request.sandboxId); + const name = text(request.name); const client = await connect(request.target); request.signal.throwIfAborted(); - // sandbox.getConfig(name) performs a new name lookup and omits workspace identity. + // OpenShell 0.1.3 selects configuration by workspace and public name. const config = readValue( SandboxConfigResponseSchema, - await client.raw.getSandboxConfig({ sandboxId }, { signal: request.signal }), + await client.raw.getSandboxConfig( + { + name, + workspaceScope: { selection: { case: "workspace", value: request.workspace } }, + }, + { signal: request.signal }, + ), ); if (config.workspace !== request.workspace) { throw new OpenShellReadError("schema"); } request.signal.throwIfAborted(); + // Reject replacement under the same name before associating policy with the observed ID. + const confirmed = await createSandboxes(async () => client).get(request); + if (confirmed?.id !== sandboxId) throw new OpenShellReadError("schema"); + request.signal.throwIfAborted(); const document = await serializePolicy(config.policy, request.signal); request.signal.throwIfAborted(); return owned({ @@ -147,8 +159,44 @@ function compactEndpointPorts(endpoint: PolicyDocument, ports: PolicyEndpointJso } } +const POLICY_ENDPOINT_MODES = { + tls: { + NETWORK_TLS_MODE_UNSPECIFIED: "", + NETWORK_TLS_MODE_SKIP: "skip", + }, + enforcement: { + NETWORK_ENFORCEMENT_MODE_UNSPECIFIED: "", + NETWORK_ENFORCEMENT_MODE_ENFORCE: "enforce", + NETWORK_ENFORCEMENT_MODE_AUDIT: "audit", + }, + access: { + NETWORK_ACCESS_PRESET_UNSPECIFIED: "", + NETWORK_ACCESS_PRESET_READ_ONLY: "read-only", + NETWORK_ACCESS_PRESET_READ_WRITE: "read-write", + NETWORK_ACCESS_PRESET_FULL: "full", + }, +} as const; + +function convertEndpointModes(endpoint: PolicyDocument): void { + for (const [field, values] of Object.entries(POLICY_ENDPOINT_MODES)) { + const value = endpoint[field]; + if (value === undefined) continue; + if (typeof value !== "string" || !Object.hasOwn(values, value)) { + throw new OpenShellReadError("schema"); + } + const authored = (values as Readonly>)[value]; + if (authored === "") delete endpoint[field]; + else endpoint[field] = authored; + } +} + function convertEndpoint(value: PolicyEndpointJson): PolicyDocument { const endpoint: PolicyDocument = { ...value }; + // Match OpenShell's authored policy projection; the gateway owns these markers. + delete endpoint.advisor_proposed; + delete endpoint.provider_credentialed; + delete endpoint.token_grant_owner; + convertEndpointModes(endpoint); const mcp = value.protocol?.toLowerCase() === "mcp"; const options: NonNullable & { max_body_bytes?: number } = { ...value.mcp, diff --git a/src/lib/adapters/openshell/sandbox-identity.test.ts b/src/lib/adapters/openshell/sandbox-identity.test.ts index e3ed9ddf6f0..17adb7ae8b8 100644 --- a/src/lib/adapters/openshell/sandbox-identity.test.ts +++ b/src/lib/adapters/openshell/sandbox-identity.test.ts @@ -120,7 +120,28 @@ describe("OpenShell sandbox identity parsing", () => { }); describe("OpenShell sandbox identity reading", () => { - it.each(["0.0.116", "0.1.3", "0.1.5"])( + function readIdentityPage(nextPageToken: string) { + return resolveCreatedOpenShellSandboxId({ + sandboxName: "alpha", + gatewayName: "nemoclaw", + createAttemptNonce: CREATE_ATTEMPT_NONCE, + runCaptureOpenshell: () => + JSON.stringify({ + sandboxes: JSON.parse(sandboxListJson()), + next_page_token: nextPageToken, + }), + }); + } + + it("accepts a complete 0.1.2 identity page", () => { + expect(readIdentityPage("")).toBe("sandbox-alpha"); + }); + + it("rejects an incomplete 0.1.2 identity page", () => { + expect(() => readIdentityPage("more")).toThrow("selector-output-malformed"); + }); + + it.each(["0.0.116", "0.1.3", "0.1.5", "0.1.7"])( "binds the first ID to an exact create-attempt label on %s (#9833)", (version) => { runtime.version = version; diff --git a/src/lib/adapters/openshell/sandbox-lifecycle-sdk.test.ts b/src/lib/adapters/openshell/sandbox-lifecycle-sdk.test.ts index 3e9f5bbd422..e33f02363b6 100644 --- a/src/lib/adapters/openshell/sandbox-lifecycle-sdk.test.ts +++ b/src/lib/adapters/openshell/sandbox-lifecycle-sdk.test.ts @@ -22,8 +22,22 @@ function harness() { sandbox: { get, waitReady }, })); const waitForStartPoll = vi.fn(async () => undefined); - const lifecycle = createSdkOpenShellSandboxStateLifecycle({ connect, waitForStartPoll }); - return { connect, get, lifecycle, startSandbox, stopSandbox, waitForStartPoll, waitReady }; + const waitForStopPoll = vi.fn(async () => undefined); + const lifecycle = createSdkOpenShellSandboxStateLifecycle({ + connect, + waitForStartPoll, + waitForStopPoll, + }); + return { + connect, + get, + lifecycle, + startSandbox, + stopSandbox, + waitForStartPoll, + waitForStopPoll, + waitReady, + }; } describe("OpenShell SDK sandbox lifecycle", () => { @@ -41,11 +55,17 @@ describe("OpenShell SDK sandbox lifecycle", () => { expect(connect).toHaveBeenCalledTimes(2); expect(get).toHaveBeenCalledTimes(4); expect(startSandbox).toHaveBeenCalledWith( - { name: "alpha", workspace: "default" }, + { + name: "alpha", + workspaceScope: { selection: { case: "workspace", value: "default" } }, + }, { signal: expect.any(AbortSignal) }, ); expect(stopSandbox).toHaveBeenCalledWith( - { name: "alpha", workspace: "default" }, + { + name: "alpha", + workspaceScope: { selection: { case: "workspace", value: "default" } }, + }, { signal: expect.any(AbortSignal) }, ); expect(get).toHaveBeenCalledWith("alpha", { signal: expect.any(AbortSignal) }); @@ -53,7 +73,7 @@ describe("OpenShell SDK sandbox lifecycle", () => { it.each([ { action: "start" as const, expectedTimeoutMs: 135_000, settledPhase: "ready" }, - { action: "stop" as const, expectedTimeoutMs: 75_000, settledPhase: "stopped" }, + { action: "stop" as const, expectedTimeoutMs: 150_000, settledPhase: "stopped" }, ])("uses the $action default timeout", async ({ action, expectedTimeoutMs, settledPhase }) => { const test = harness(); test.get.mockResolvedValue({ id: sandboxId, phase: settledPhase }); @@ -358,48 +378,94 @@ describe("OpenShell SDK sandbox lifecycle", () => { expect(get).toHaveBeenCalledOnce(); }); - it("waits for the same sandbox to report Stopped after the stop RPC", async () => { - const get = vi - .fn() - .mockResolvedValueOnce({ id: sandboxId, phase: "ready" }) - .mockResolvedValueOnce({ id: sandboxId, phase: "stopping" }) - .mockResolvedValueOnce({ id: sandboxId, phase: "Stopped" }); - const waitForStopPoll = vi.fn(async () => undefined); - const lifecycle = createSdkOpenShellSandboxStateLifecycle({ - connect: async () => ({ - raw: { - startSandbox: async () => ({ sandbox: { metadata: { id: sandboxId } } }), - stopSandbox: async () => ({ sandbox: { metadata: { id: sandboxId } } }), - }, - sandbox: { - get, - }, - }), - waitForStopPoll, - }); - - await expect(lifecycle.stopSandbox(request)).resolves.toEqual({ kind: "accepted" }); - expect(get).toHaveBeenCalledTimes(3); - expect(waitForStopPoll).toHaveBeenCalledOnce(); - }); + it.each(["Completed", "Error"])( + "reports %s after a successful stop RPC without waiting for the deadline", + async (phase) => { + const { lifecycle, get, stopSandbox, waitForStopPoll } = harness(); + get.mockResolvedValueOnce({ id: sandboxId, phase: "ready" }); + get.mockResolvedValue({ id: sandboxId, phase }); - it("fails at the shared deadline when the exact sandbox never reaches Stopped", async () => { - const lifecycle = createSdkOpenShellSandboxStateLifecycle({ - connect: async () => ({ - raw: { - startSandbox: async () => ({ sandbox: { metadata: { id: sandboxId } } }), - stopSandbox: async () => ({ sandbox: { metadata: { id: sandboxId } } }), - }, - sandbox: { - get: async () => ({ id: sandboxId, phase: "stopping" }), + await expect(lifecycle.stopSandbox(request)).resolves.toEqual({ + kind: "failed", + error: { + kind: "command", + reason: "failed", + message: `OpenShell sandbox is ${phase} after the stop request; Stopped was not confirmed.`, }, - }), - waitForStopPoll: async () => await new Promise(() => undefined), - }); + }); + expect(stopSandbox).toHaveBeenCalledOnce(); + expect(get).toHaveBeenCalledTimes(2); + expect(waitForStopPoll).not.toHaveBeenCalled(); + }, + ); - await expect(lifecycle.stopSandbox({ ...request, timeoutMs: 5 })).resolves.toEqual({ - kind: "failed", - error: { kind: "timeout", message: "OpenShell timed out." }, - }); + it("waits through both container stop periods and confirms the same sandbox is Stopped", async () => { + vi.useFakeTimers(); + try { + const get = vi + .fn() + .mockResolvedValueOnce({ id: sandboxId, phase: "ready" }) + .mockResolvedValueOnce({ id: sandboxId, phase: "stopping" }) + .mockResolvedValueOnce({ id: sandboxId, phase: "Stopped" }); + const waitForStopPoll = vi.fn(async () => { + await new Promise((resolve) => setTimeout(resolve, 45_000)); + }); + const lifecycle = createSdkOpenShellSandboxStateLifecycle({ + connect: async () => ({ + raw: { + startSandbox: async () => ({ sandbox: { metadata: { id: sandboxId } } }), + stopSandbox: async () => { + await new Promise((resolve) => setTimeout(resolve, 90_000)); + return { sandbox: { metadata: { id: sandboxId } } }; + }, + }, + sandbox: { + get, + }, + }), + waitForStopPoll, + }); + + const stopped = lifecycle.stopSandbox(request); + await vi.advanceTimersByTimeAsync(135_000); + await expect(stopped).resolves.toEqual({ kind: "accepted" }); + expect(get).toHaveBeenCalledTimes(3); + expect(waitForStopPoll).toHaveBeenCalledOnce(); + } finally { + vi.useRealTimers(); + } }); + + it.each([ + { timeoutMs: undefined, elapsedMs: 150_000 }, + { timeoutMs: 5, elapsedMs: 5 }, + ])( + "fails after $elapsedMs ms when the sandbox never reaches Stopped", + async ({ timeoutMs, elapsedMs }) => { + vi.useFakeTimers(); + try { + const lifecycle = createSdkOpenShellSandboxStateLifecycle({ + connect: async () => ({ + raw: { + startSandbox: async () => ({ sandbox: { metadata: { id: sandboxId } } }), + stopSandbox: async () => ({ sandbox: { metadata: { id: sandboxId } } }), + }, + sandbox: { + get: async () => ({ id: sandboxId, phase: "stopping" }), + }, + }), + waitForStopPoll: async () => await new Promise(() => undefined), + }); + + const stopped = lifecycle.stopSandbox({ ...request, timeoutMs }); + await vi.advanceTimersByTimeAsync(elapsedMs); + await expect(stopped).resolves.toEqual({ + kind: "failed", + error: { kind: "timeout", message: "OpenShell timed out." }, + }); + } finally { + vi.useRealTimers(); + } + }, + ); }); diff --git a/src/lib/adapters/openshell/sandbox-lifecycle-sdk.ts b/src/lib/adapters/openshell/sandbox-lifecycle-sdk.ts index ea1a47a6432..8c99494db61 100644 --- a/src/lib/adapters/openshell/sandbox-lifecycle-sdk.ts +++ b/src/lib/adapters/openshell/sandbox-lifecycle-sdk.ts @@ -30,17 +30,23 @@ type SdkSandboxRef = Readonly<{ id: string; phase: string }>; type SdkSandboxMutationResponse = Readonly<{ sandbox?: Readonly<{ metadata?: Readonly<{ id?: string }> }>; }>; +type SdkSandboxMutationRequest = Readonly<{ + name: string; + workspaceScope: Readonly<{ + selection: Readonly<{ case: "workspace"; value: "default" }>; + }>; +}>; type SdkClient = Readonly<{ sandbox: Readonly<{ get(name: string, options: CallOptions): Promise; }>; raw: Readonly<{ startSandbox( - request: Readonly<{ name: string; workspace: string }>, + request: SdkSandboxMutationRequest, options: CallOptions, ): Promise; stopSandbox( - request: Readonly<{ name: string; workspace: string }>, + request: SdkSandboxMutationRequest, options: CallOptions, ): Promise; }>; @@ -60,8 +66,9 @@ export type SdkOpenShellSandboxStateLifecycleDeps = Readonly<{ // minute while the immutable workload container was already running. The // lifecycle still requires the same sandbox identity to publish Ready. const SDK_START_INITIAL_ERROR_GRACE_POLLS = 40; -const DEFAULT_MUTATION_TIMEOUT_MS = 75_000; const DEFAULT_START_TIMEOUT_MS = 135_000; +// Native Podman stops the supervisor and workload sequentially before observing exit. +const DEFAULT_STOP_TIMEOUT_MS = 150_000; function lifecycleError( error: unknown, @@ -139,8 +146,7 @@ async function mutate( const controller = new AbortController(); const timeout = setTimeout( () => controller.abort(), - request.timeoutMs ?? - (action === "start" ? DEFAULT_START_TIMEOUT_MS : DEFAULT_MUTATION_TIMEOUT_MS), + request.timeoutMs ?? (action === "stop" ? DEFAULT_STOP_TIMEOUT_MS : DEFAULT_START_TIMEOUT_MS), ); const aborted = new Promise((_resolve, reject) => { controller.signal.addEventListener( @@ -172,7 +178,13 @@ async function mutate( observedPhase = observed.phase; const operation = action === "start" ? client.raw.startSandbox : client.raw.stopSandbox; const mutation = await Promise.race([ - operation({ name: request.sandboxName, workspace: "default" }, { signal: controller.signal }), + operation( + { + name: request.sandboxName, + workspaceScope: { selection: { case: "workspace", value: "default" } }, + }, + { signal: controller.signal }, + ), aborted, ]); if ( @@ -212,6 +224,19 @@ async function mutate( } const phase = current.phase.toLowerCase(); if (phase === (action === "start" ? "ready" : "stopped")) break; + // A successful stop RPC can preserve the terminal process result when + // compute is missing. That result does not establish the Stopped state + // required before NemoClaw records an intentional stop. + if (action === "stop" && (phase === "completed" || phase === "error")) { + return { + kind: "failed", + error: { + kind: "command", + reason: "failed", + message: `OpenShell sandbox is ${current.phase} after the stop request; Stopped was not confirmed.`, + }, + }; + } if (action === "start" && !allowInitialError(phase) && isTerminalSandboxPhase(phase)) { return { kind: "failed", diff --git a/src/lib/adapters/openshell/sandbox-policy-cli.test.ts b/src/lib/adapters/openshell/sandbox-policy-cli.test.ts index bbd05dbec68..fc49e52dc19 100644 --- a/src/lib/adapters/openshell/sandbox-policy-cli.test.ts +++ b/src/lib/adapters/openshell/sandbox-policy-cli.test.ts @@ -295,6 +295,69 @@ describe("CLI OpenShell sandbox policy writer", () => { ]); }); + it.each([ + { code: "ETIMEDOUT", status: null, category: "timeout", exit: "unknown" }, + { code: "ENOENT", status: null, category: "missing-executable", exit: "unknown" }, + { code: "EACCES", status: 1, category: "permission", exit: "nonzero" }, + { code: "EPERM", status: 1, category: "permission", exit: "nonzero" }, + { code: "ENOBUFS", status: null, category: "output-limit", exit: "unknown" }, + { code: "credential-value", status: 0, category: "other", exit: "zero" }, + { code: "credential-value", status: Number.NaN, category: "other", exit: "unknown" }, + { code: "credential-value", status: -1, category: "other", exit: "unknown" }, + { code: "credential-value", status: 256, category: "other", exit: "unknown" }, + ])( + "reports only fixed policy capture metadata for $code / $status", + async ({ code, status, category, exit }) => { + const capture = vi.fn(() => + captured({ + status, + output: "credential-value", + stdout: "credential-value", + stderr: "credential-value", + error: Object.assign(new Error("credential-value"), { code }), + signal: "SIGTERM", + }), + ); + const result = await createCliOpenShellSandboxPolicyWriter({ capture }).setSandboxPolicy({ + target: selectedOpenShellGateway(), + sandboxName: "alpha", + document: POLICY, + }); + expect(result.outcome).toEqual({ + kind: "ambiguous", + detail: `OpenShell did not confirm the policy submission (status=${exit}; signal=present; capture=${category}; stdout=present; stderr=present)`, + }); + expect(JSON.stringify(result)).not.toContain("credential-value"); + expect(capture).toHaveBeenCalledOnce(); + }, + ); + + it.each([ + { streams: {}, expected: "stdout=unavailable; stderr=unavailable" }, + { streams: { stdout: "", stderr: "" }, expected: "stdout=empty; stderr=empty" }, + { + streams: { stdout: "credential-value", stderr: "denied credential-value" }, + expected: "stdout=present; stderr=present", + }, + ])( + "retains ambiguity without printing unrecognized output: $expected", + async ({ streams, expected }) => { + const capture = vi.fn(() => captured({ status: 1, output: "credential-value", ...streams })); + const result = await createCliOpenShellSandboxPolicyWriter({ capture }).setSandboxPolicy({ + target: selectedOpenShellGateway(), + sandboxName: "alpha", + document: POLICY, + }); + expect(result).toEqual({ + status: 1, + outcome: { + kind: "ambiguous", + detail: `OpenShell did not confirm the policy submission (status=nonzero; signal=none; capture=none; ${expected})`, + }, + }); + expect(JSON.stringify(result)).not.toContain("credential-value"); + }, + ); it("reports an allowlisted reason for an unconfirmed policy submission", async () => { const stderr = "Error: × code: 'The system is not in a state required for the operation's execution', " + @@ -336,7 +399,8 @@ describe("CLI OpenShell sandbox policy writer", () => { expect(outcome).toEqual({ kind: "ambiguous", - detail: "OpenShell did not confirm the policy submission.", + detail: + "OpenShell did not confirm the policy submission (status=nonzero; signal=none; capture=none; stdout=unavailable; stderr=present)", }); expect(JSON.stringify(outcome)).not.toContain(opaqueValue); expect(JSON.stringify(outcome)).not.toMatch(/[\u0000-\u001f\u007f]/u); diff --git a/src/lib/adapters/openshell/sandbox-policy-cli.ts b/src/lib/adapters/openshell/sandbox-policy-cli.ts index dbc22bd9c46..af430b019fe 100644 --- a/src/lib/adapters/openshell/sandbox-policy-cli.ts +++ b/src/lib/adapters/openshell/sandbox-policy-cli.ts @@ -185,14 +185,46 @@ function capturedOutput(captured: CapturedOpenShellCommandResult): string { export const classifyCliOpenShellSandboxPolicySetResult = classifyOpenShellSandboxPolicySetResult; -function safeAmbiguousPolicyDetail(detail: string): string { +const POLICY_CAPTURE_ERROR_CLASSES = new Map([ + ["ETIMEDOUT", "timeout"], + ["ENOENT", "missing-executable"], + ["EACCES", "permission"], + ["EPERM", "permission"], + ["ENOBUFS", "output-limit"], +]); + +function policySubmissionStatus(captured: CapturedOpenShellCommandResult): string { + const status = captured.status; + let exit = "unknown"; + if (status !== null && Number.isInteger(status) && status >= 0 && status <= 255) { + exit = status === 0 ? "zero" : "nonzero"; + } + const code = (captured.error as NodeJS.ErrnoException | undefined)?.code; + const error = captured.error ? (POLICY_CAPTURE_ERROR_CLASSES.get(code ?? "") ?? "other") : "none"; + const presence = (stream: unknown) => { + if (typeof stream === "string" || Buffer.isBuffer(stream)) { + return stream.length > 0 ? "present" : "empty"; + } + return "unavailable"; + }; + // Fixed categories only: policy commands can echo credentials and submitted documents. + return ( + `status=${exit}; signal=${captured.signal ? "present" : "none"}; capture=${error}; ` + + `stdout=${presence(captured.stdout)}; stderr=${presence(captured.stderr)}` + ); +} + +function safeAmbiguousPolicyDetail( + detail: string, + captured: CapturedOpenShellCommandResult, +): string { const diagnostic = stripAnsi(detail).replace( /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/gu, " ", ); return /network endpoint ambiguity validation failed/iu.test(diagnostic) ? "OpenShell did not confirm the policy submission: network endpoint ambiguity validation failed." - : "OpenShell did not confirm the policy submission."; + : `OpenShell did not confirm the policy submission (${policySubmissionStatus(captured)})`; } function parsePolicySet( @@ -202,7 +234,7 @@ function parsePolicySet( if (outcome.kind === "rejected") { outcome = { ...outcome, message: redactCredentialText(outcome.message) }; } else if (outcome.kind === "ambiguous") { - outcome = { kind: "ambiguous", detail: safeAmbiguousPolicyDetail(outcome.detail) }; + outcome = { kind: "ambiguous", detail: safeAmbiguousPolicyDetail(outcome.detail, captured) }; } return { status: captured.status, outcome }; } diff --git a/src/lib/adapters/openshell/sandbox-presence.test.ts b/src/lib/adapters/openshell/sandbox-presence.test.ts index 231eac713e4..8f6a427eb66 100644 --- a/src/lib/adapters/openshell/sandbox-presence.test.ts +++ b/src/lib/adapters/openshell/sandbox-presence.test.ts @@ -69,3 +69,53 @@ describe("structured OpenShell sandbox identity", () => { ).toEqual({ kind: "unknown" }); }); }); + +describe("complete OpenShell 0.1.2 sandbox inventory", () => { + const page = (sandboxes: unknown[], next_page_token: unknown = "") => ({ + status: 0, + stdout: JSON.stringify({ sandboxes, next_page_token }), + stderr: "", + }); + + it("reads one durable identity from a complete page", () => { + expect(observeOpenShellSandboxIdentity("alpha", page([row()]))).toEqual({ + kind: "present", + id: "sandbox-alpha", + phase: "Ready", + }); + }); + + it("proves absence only from a complete valid inventory", () => { + expect(classifyOpenShellSandboxPresence("alpha", page([]))).toBe("absent"); + expect(classifyOpenShellSandboxPresence("alpha", page([row({ name: "beta" })]))).toBe("absent"); + }); + + it.each(["next", " ", null, 0, undefined])( + "rejects a nonempty or invalid continuation token %s", + (token) => { + const response = { + status: 0, + stdout: JSON.stringify({ sandboxes: [], next_page_token: token }), + stderr: "", + }; + expect(classifyOpenShellSandboxPresence("alpha", response)).toBe("unknown"); + }, + ); + + it("rejects partial presence rather than assuming unique identity", () => { + expect(observeOpenShellSandboxIdentity("alpha", page([row()], "next"))).toEqual({ + kind: "unknown", + }); + }); + + it("retains row, duplicate and diagnostic rejection for complete pages", () => { + expect(classifyOpenShellSandboxPresence("alpha", page([row(), row()]))).toBe("unknown"); + expect(classifyOpenShellSandboxPresence("alpha", page([row({ labels: null })]))).toBe( + "unknown", + ); + expect(classifyOpenShellSandboxPresence("alpha", { ...page([]), stderr: "warning" })).toBe( + "unknown", + ); + expect(classifyOpenShellSandboxPresence("alpha", { ...page([]), status: 1 })).toBe("unknown"); + }); +}); diff --git a/src/lib/adapters/openshell/sandboxes.ts b/src/lib/adapters/openshell/sandboxes.ts index a5f0ad24a4d..4b6366a5c81 100644 --- a/src/lib/adapters/openshell/sandboxes.ts +++ b/src/lib/adapters/openshell/sandboxes.ts @@ -36,7 +36,10 @@ export function createSandboxes(connect: ConnectOpenShellReader = connectOpenShe try { // sandbox.get() omits workspace, template image, and the active policy version. response = await client.raw.getSandbox( - { name, workspace: request.workspace }, + { + name, + workspaceScope: { selection: { case: "workspace", value: request.workspace } }, + }, { signal: request.signal }, ); } catch (error) { diff --git a/src/lib/adapters/openshell/sdk-read-schema.ts b/src/lib/adapters/openshell/sdk-read-schema.ts index bbefb3ced36..e17b180991f 100644 --- a/src/lib/adapters/openshell/sdk-read-schema.ts +++ b/src/lib/adapters/openshell/sdk-read-schema.ts @@ -68,8 +68,9 @@ const ManagedRestEndpointFields = { port: Type.Literal(443), ports: Type.Union([Type.Tuple([]), Type.Tuple([Type.Literal(443)])]), protocol: Type.Literal("rest"), - tls: Type.Literal(""), - enforcement: Type.Literal("enforce"), + // OpenShell 0.1.2 raw protobuf responses use numeric network enums. + tls: Type.Literal(0), // NetworkTlsMode.UNSPECIFIED + enforcement: Type.Literal(1), // NetworkEnforcementMode.ENFORCE allowedIps: Type.Tuple([]), denyRules: Type.Tuple([]), allowEncodedSlash: Type.Literal(false), @@ -118,7 +119,7 @@ export const ManagedBraveProfileResponseSchema = Type.Object({ Type.Object({ ...ManagedRestEndpointFields, host: Type.Literal("api.search.brave.com"), - access: Type.Literal("read-write"), + access: Type.Literal(2), // NetworkAccessPreset.READ_WRITE rules: Type.Tuple([]), requestBodyCredentialRewrite: Type.Literal(false), }), @@ -161,7 +162,7 @@ function tavilyProfile(id: "tavily" | "tavily-hermes-v1", binaries: readonly str Type.Object({ ...ManagedRestEndpointFields, host: Type.Literal("api.tavily.com"), - access: Type.Literal(""), + access: Type.Literal(0), // NetworkAccessPreset.UNSPECIFIED (explicit rules) rules: Type.Tuple([tavilyRule("/search"), tavilyRule("/extract")]), requestBodyCredentialRewrite: Type.Literal(true), }), diff --git a/src/lib/adapters/openshell/sdk-read.ts b/src/lib/adapters/openshell/sdk-read.ts index 38943b28e4f..8e376c3ebc4 100644 --- a/src/lib/adapters/openshell/sdk-read.ts +++ b/src/lib/adapters/openshell/sdk-read.ts @@ -18,16 +18,28 @@ export type ReadRequest = Readonly<{ }>; type Options = Readonly<{ signal: AbortSignal }>; +type WorkspaceScope = Readonly<{ + selection: Readonly<{ case: "workspace"; value: string }>; +}>; /** Only SDK reads used by configuration export. Integration tests check the SDK boundary. */ export interface OpenShellReadClient { readonly raw: { - getProvider(request: { name: string; workspace: string }, options: Options): Promise; + getProvider( + request: { name: string; workspaceScope: WorkspaceScope }, + options: Options, + ): Promise; getProviderProfile( - request: { id: string; workspace: string }, + request: { id: string; workspaceScope?: WorkspaceScope }, + options: Options, + ): Promise; + getSandbox( + request: { name: string; workspaceScope: WorkspaceScope }, + options: Options, + ): Promise; + getSandboxConfig( + request: { name: string; workspaceScope: WorkspaceScope }, options: Options, ): Promise; - getSandbox(request: { name: string; workspace: string }, options: Options): Promise; - getSandboxConfig(request: { sandboxId: string }, options: Options): Promise; }; } export type ConnectOpenShellReader = ( diff --git a/src/lib/adapters/sandbox/command-transport.ts b/src/lib/adapters/sandbox/command-transport.ts index 2eb137377f3..0689d0a1a24 100644 --- a/src/lib/adapters/sandbox/command-transport.ts +++ b/src/lib/adapters/sandbox/command-transport.ts @@ -1,6 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { spawnSync } from "node:child_process"; +import { resolveOpenshellSandboxSshHost } from "../openshell/sandbox-ssh-host"; +import { captureSandboxSshConfigCommand } from "../openshell/command-execution"; +import { resolveOpenshellBinaryOrNull } from "../openshell/resolve-shared"; +import { createTempSshConfig, runWithTempSshConfigCleanup } from "../../sandbox/temp-ssh-config"; import type { OpenShellSandboxBufferedCommandExecutor, OpenShellSandboxCommandError, @@ -138,3 +143,63 @@ export async function executeSandboxExecCommand( }, ); } + +// Maintenance deliberately holds the gateway down. OpenShell exec requires +// Ready, while its SSH transport remains available to the sandbox owner. +export async function executeSandboxMaintenanceCommand( + sandboxName: string, + command: string, + timeout: number, + options: SandboxExecCommandExecutionOptions = {}, +): Promise { + const deadline = Date.now() + timeout; + const binary = resolveOpenshellBinaryOrNull(); + if (!binary) return null; + const environment = buildSandboxCommandEnvironment(options.runtimeSelection, options.runtimeEnv); + const config = captureSandboxSshConfigCommand(binary, sandboxName, { + env: environment, + replaceEnv: true, + gatewayName: options.runtimeSelection?.gatewayName, + ignoreError: true, + timeout: Math.max(1, Math.floor(timeout / 3)), + }); + if (config.status !== 0 || config.error) return null; + const host = resolveOpenshellSandboxSshHost(sandboxName, config.output); + if (!host) return null; + const remaining = deadline - Date.now(); + if (remaining <= 0) return null; + const temporary = createTempSshConfig(config.output, "nemoclaw-openclaw-maintenance-"); + const completed = runWithTempSshConfigCleanup(temporary, () => { + const result = spawnSync( + "ssh", + [ + "-F", + temporary.file, + "-o", + "StrictHostKeyChecking=no", + "-o", + "UserKnownHostsFile=/dev/null", + "-o", + "ConnectTimeout=10", + "-o", + "LogLevel=ERROR", + host, + "/bin/bash --noprofile --norc -p -s", + ], + { + encoding: "utf8", + input: `builtin unset OPENCLAW_GATEWAY_TOKEN\n${command}`, + env: environment, + timeout: remaining, + maxBuffer: 1024 * 1024, + stdio: ["pipe", "pipe", "pipe"], + killSignal: "SIGKILL", + }, + ); + if (result.error || result.signal || result.status === null || result.status === 255) + return null; + return { status: result.status, stdout: result.stdout.trim(), stderr: result.stderr.trim() }; + }); + if (completed.cleanupError) throw completed.cleanupError; + return completed.result; +} diff --git a/src/lib/adapters/sandbox/maintenance-command.test.ts b/src/lib/adapters/sandbox/maintenance-command.test.ts new file mode 100644 index 00000000000..c131267a7d6 --- /dev/null +++ b/src/lib/adapters/sandbox/maintenance-command.test.ts @@ -0,0 +1,88 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import { beforeEach, expect, it, vi } from "vitest"; +const { spawnSync, captureSandboxSshConfigCommand } = vi.hoisted(() => ({ + spawnSync: vi.fn(), + captureSandboxSshConfigCommand: vi.fn(), +})); +vi.mock("node:child_process", async (original) => ({ + ...(await original()), + spawnSync, +})); +vi.mock("../openshell/command-execution", async (original) => ({ + ...(await original()), + captureSandboxSshConfigCommand, +})); +vi.mock("../openshell/resolve-shared", async (original) => ({ + ...(await original()), + resolveOpenshellBinaryOrNull: () => "/pinned/openshell", +})); +import { executeSandboxMaintenanceCommand } from "./command-transport"; +const config = "Host openshell-alpha\n HostName localhost\n User sandbox\n"; +const runtimeSelection = { + gatewayName: "selected", + localTlsDir: "/selected/tls", + workspace: "default", +}; + +beforeEach(() => { + captureSandboxSshConfigCommand.mockReset().mockReturnValue({ status: 0, output: config }); + spawnSync.mockReset().mockReturnValue({ status: 0, stdout: "ready\n", stderr: "", signal: null }); +}); + +it("uses selected OpenShell SSH authority during maintenance and removes its private config (#12603)", async () => { + vi.stubEnv("OPENSHELL_GATEWAY", "ambient"); + vi.stubEnv("OPENSHELL_TOKEN", "ambient-token"); + let configPath = ""; + spawnSync.mockImplementation((binary, argv, options) => { + configPath = argv[1]; + expect(binary).toBe("ssh"); + expect(fs.readFileSync(configPath, "utf8")).toBe(config); + expect(fs.statSync(configPath).mode & 0o777).toBe(0o600); + expect(argv.at(-2)).toBe("openshell-alpha"); + expect(argv.at(-1)).toContain("--noprofile"); + expect(options.input).toBe("builtin unset OPENCLAW_GATEWAY_TOKEN\necho ready"); + expect(options.env).toMatchObject({ + OPENSHELL_GATEWAY: "selected", + OPENSHELL_LOCAL_TLS_DIR: "/selected/tls", + }); + expect(options.env).not.toHaveProperty("OPENSHELL_TOKEN"); + expect(options.timeout).toBeGreaterThan(0); + expect(options.timeout).toBeLessThanOrEqual(2000); + return { status: 0, stdout: "ready\n", stderr: "", signal: null }; + }); + await expect( + executeSandboxMaintenanceCommand("alpha", "echo ready", 2000, { runtimeSelection }), + ).resolves.toEqual({ status: 0, stdout: "ready", stderr: "" }); + expect(captureSandboxSshConfigCommand).toHaveBeenCalledWith( + "/pinned/openshell", + "alpha", + expect.objectContaining({ + env: expect.objectContaining({ OPENSHELL_GATEWAY: "selected" }), + replaceEnv: true, + }), + ); + expect(fs.existsSync(configPath)).toBe(false); +}); + +it.each([ + { status: 1, output: "unavailable" }, + { status: 0, output: "Host openshell-other\n HostName localhost\n" }, +])("does not execute with unavailable or mismatched SSH authority: %j", async (result) => { + captureSandboxSshConfigCommand.mockReturnValue(result); + await expect(executeSandboxMaintenanceCommand("alpha", "mutate", 1000)).resolves.toBeNull(); + expect(spawnSync).not.toHaveBeenCalled(); +}); + +it.each([ + { status: 255 }, + { status: null, signal: "SIGKILL" }, + { status: 0, error: new Error("timeout") }, +])("does not accept or retry an unconfirmed SSH command: %j", async (failure) => { + spawnSync.mockReturnValue({ stdout: "partial", stderr: "", ...failure }); + await expect(executeSandboxMaintenanceCommand("alpha", "mutate", 1000)).resolves.toBeNull(); + expect(spawnSync).toHaveBeenCalledOnce(); + expect(fs.existsSync(spawnSync.mock.calls[0][1][1])).toBe(false); +}); diff --git a/src/lib/agent/candidate-authority.ts b/src/lib/agent/candidate-authority.ts index 508f58bd130..d02c8b92acd 100644 --- a/src/lib/agent/candidate-authority.ts +++ b/src/lib/agent/candidate-authority.ts @@ -13,8 +13,8 @@ export const CANDIDATE_QUALIFICATION_RECEIPT_DIGESTS: Readonly< Record > = Object.freeze({ pi: Object.freeze([ - "d5531c1cfd9591395560b8c1767f834695e92d8361e358106f613c24a39aebf4", - "8326a66000d169dc05416e6218c17514780ae49863e5b7699de549f3e9ad0c39", + "9e51cbdb42187939b574181d52d38edcd71624c68169a691627344304cca6f41", + "dddf4d84e15eb64e14be8c3be735e2dc7e27433bf586eec1e73c3b9674f37174", ]), }); diff --git a/src/lib/agent/deep-agents-code-base-image.test.ts b/src/lib/agent/deep-agents-code-base-image.test.ts index 240c4d5734a..ca88bf8fea8 100644 --- a/src/lib/agent/deep-agents-code-base-image.test.ts +++ b/src/lib/agent/deep-agents-code-base-image.test.ts @@ -99,12 +99,12 @@ describe("Deep Agents Code base image compatibility", () => { makeAgent({ name: "langchain-deepagents-code", displayName: "LangChain Deep Agents Code", - expectedVersion: "0.1.34", + expectedVersion: "0.1.54", }), "/test/root/agents/langchain-deepagents-code/Dockerfile.base", ); mocks.dockerCapture - .mockReturnValueOnce("0.1.34") + .mockReturnValueOnce("0.1.54") .mockReturnValueOnce("nemoclaw-dcode-sandbox-identity-ok") .mockReturnValueOnce(""); diff --git a/src/lib/agent/gateway-readiness.ts b/src/lib/agent/gateway-readiness.ts index 6c70aee0d61..82ae2cdaa25 100644 --- a/src/lib/agent/gateway-readiness.ts +++ b/src/lib/agent/gateway-readiness.ts @@ -1,8 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { waitUntilAsync } from "../core/wait"; -import { createReadinessWaitOptions } from "../onboard/readiness-wait"; +import { createReadinessWaitOptions, waitUntilAsync } from "../onboard/readiness-wait"; export async function waitForAgentGatewayReady(options: { timeoutSeconds: number; diff --git a/src/lib/blueprint-runner.ts b/src/lib/blueprint-runner.ts index 2c00ad9a8e8..5f80aa72b79 100755 --- a/src/lib/blueprint-runner.ts +++ b/src/lib/blueprint-runner.ts @@ -6,6 +6,85 @@ import { redactForLog } from "./security/redact.js"; import { sdkOpenShellGatewayHealthObserver } from "../../nemoclaw/dist/shared/openshell-gateway-health-sdk.js"; import type { OpenShellGatewayHealthObserver } from "../../nemoclaw/dist/shared/openshell-observation-boundary.cjs"; +import type { resumeBlueprintNativeInferenceRun } from "./inference/native-provider/blueprint-resume"; + +import type { createBlueprintNativeInference } from "./inference/native-provider/blueprint"; + +/** Observe existing native authority without creating, repairing, or attaching providers. */ +export async function observeBlueprintNativeInference( + requested: Readonly<{ + gatewayName: string; + sandboxName: string; + provider: string; + model: string; + api: string; + endpointUrl?: string; + requiresCredential?: boolean; + }>, +): Promise { + const { findSandboxAcrossGatewayRoots } = await import("./state/registry/cross-port"); + const { verifyBlueprintNativeInferenceSelection } = + await import("./actions/sandbox/inference-route-health"); + const { createRegisteredSandboxIdentityRevalidation } = + await import("./onboard/sandbox-recreate-transaction"); + const { observeSandboxOnGateway } = await import("./onboard/sandbox-recreate-probe"); + const { findBlueprintNativeReceipt, observeBlueprintCreatedInference } = + await import("./inference/native-provider/blueprint"); + const readBlueprintSandboxRegistration = (sandboxName: string) => + findSandboxAcrossGatewayRoots(sandboxName)?.entry ?? null; + const entry = readBlueprintSandboxRegistration(requested.sandboxName); + if (!entry) { + const receipt = findBlueprintNativeReceipt(requested); + if (receipt) return observeBlueprintCreatedInference(requested, receipt); + } + const revalidate = createRegisteredSandboxIdentityRevalidation(entry, { + readRegistration: readBlueprintSandboxRegistration, + observe: (sandboxName, gatewayName) => + observeSandboxOnGateway({ sandboxName, gatewayName }, undefined, undefined, 15_000), + }); + if (!entry || !revalidate) + throw new Error("Runtime identity apply requires recorded sandbox lifecycle authority."); + revalidate("inspect native inference for runtime identity apply"); + // Each registered native family verifies its declared credential against live provider metadata. + const identity = await verifyBlueprintNativeInferenceSelection(entry, requested); + revalidate("complete native inference inspection for runtime identity apply"); + const current = readBlueprintSandboxRegistration(requested.sandboxName); + if (!current || identity !== (await verifyBlueprintNativeInferenceSelection(current, requested))) + throw new Error("Native inference selection changed during runtime identity inspection."); + revalidate("complete native inference selection verification"); + return identity; +} + +export async function prepareBlueprintNativeInference( + requested: Parameters[0] & + Readonly<{ timeoutSeconds: number }>, +): Promise { + const { reconcileBlueprintNativeTimeout, findBlueprintNativeReceipt, requireCreatedSandbox } = + await import("./inference/native-provider/blueprint"); + const { findSandboxAcrossGatewayRoots } = await import("./state/registry/cross-port"); + const { createRegisteredSandboxIdentityRevalidation } = + await import("./onboard/sandbox-recreate-transaction"); + const { observeSandboxOnGateway } = await import("./onboard/sandbox-recreate-probe"); + const readRegistration = (sandboxName: string) => + findSandboxAcrossGatewayRoots(sandboxName)?.entry ?? null; + const entry = readRegistration(requested.sandboxName); + const receipt = entry ? null : findBlueprintNativeReceipt(requested); + const revalidate = receipt + ? () => requireCreatedSandbox(receipt) + : createRegisteredSandboxIdentityRevalidation(entry, { + readRegistration, + observe: (sandboxName, gatewayName) => + observeSandboxOnGateway({ sandboxName, gatewayName }, undefined, undefined, 15_000), + }); + if (!revalidate) + throw new Error("Runtime identity apply requires recorded sandbox lifecycle authority."); + await reconcileBlueprintNativeTimeout( + requested, + () => observeBlueprintNativeInference(requested), + () => revalidate("update native inference timeout"), + ); +} + const RUNNER_MODULE = "../nemoclaw/blueprint/runner.js"; const MAX_DIAGNOSTIC_CHARACTERS = 1_024; const UNSAFE_DIAGNOSTIC_CHARACTERS = @@ -30,14 +109,30 @@ async function run(): Promise { const runner = (await import(RUNNER_MODULE)) as Readonly<{ main( argv: string[], - options: Readonly<{ gatewayHealthObserver: OpenShellGatewayHealthObserver }>, + options: Readonly<{ + gatewayHealthObserver: OpenShellGatewayHealthObserver; + nativeInferenceObserver: typeof observeBlueprintNativeInference; + nativeInferenceCreator: typeof createBlueprintNativeInference; + nativeInferenceResumer: typeof resumeBlueprintNativeInferenceRun; + nativeInferencePreparer: typeof prepareBlueprintNativeInference; + }>, ): Promise; }>; await runner.main(process.argv.slice(2), { gatewayHealthObserver: sdkOpenShellGatewayHealthObserver, + nativeInferenceObserver: observeBlueprintNativeInference, + nativeInferenceCreator: async (input) => { + const { createBlueprintNativeInference } = + await import("./inference/native-provider/blueprint"); + return createBlueprintNativeInference(input); + }, + nativeInferencePreparer: prepareBlueprintNativeInference, + nativeInferenceResumer: async (input) => { + const { resumeBlueprintNativeInferenceRun } = + await import("./inference/native-provider/blueprint-resume"); + return resumeBlueprintNativeInferenceRun(input); + }, }); } -void run().catch(reportFailure); - -export {}; +if (require.main === module) void run().catch(reportFailure); diff --git a/src/lib/domain/config/export-document.test.ts b/src/lib/domain/config/export-document.test.ts index 24006508ff6..9b3150d8318 100644 --- a/src/lib/domain/config/export-document.test.ts +++ b/src/lib/domain/config/export-document.test.ts @@ -117,6 +117,7 @@ describe("export config builder", () => { it.each([ { compatibility: "strict", expected: "hard_requirement" }, + { compatibility: "hard_requirement", expected: "hard_requirement" }, { compatibility: "best_effort", expected: "best_effort" }, ])( "exports Landlock $compatibility as $expected without changing source policy", diff --git a/src/lib/domain/config/export-evidence.ts b/src/lib/domain/config/export-evidence.ts index 617580ec2ce..65f67e7388b 100644 --- a/src/lib/domain/config/export-evidence.ts +++ b/src/lib/domain/config/export-evidence.ts @@ -79,6 +79,7 @@ export const EXPORT_REGISTRY_EVIDENCE_KEYS = [ "nativeNvidiaProviderAttachment", "nativeLocalProviderAttachment", "nativeHostedProviderAttachment", + "registeredInferenceProviderAttachment", "nimContainer", "observabilityEnabled", "openshellDriver", @@ -184,6 +185,11 @@ export interface ObservedExportEndpointEvidence { readonly kind: "provider-config"; readonly key: "OPENAI_BASE_URL" | "ANTHROPIC_BASE_URL"; } + | { + readonly kind: "registered-provider"; + readonly profileId: string; + readonly profileDigest: string; + } | { readonly kind: "builtin-profile"; readonly profileId: "nvidia" } | { readonly kind: "managed-profile"; @@ -213,7 +219,6 @@ export interface ObservedManagedVllmRuntime { readonly serving: import("../../config/model").NemoClawManagedVllmServing; readonly containerId: string; readonly imageId: string; - readonly networkId: string; readonly startedAt: string; } diff --git a/src/lib/domain/config/verify-export-defaults.test.ts b/src/lib/domain/config/verify-export-defaults.test.ts index 18800a24cb5..f1899ffaca7 100644 --- a/src/lib/domain/config/verify-export-defaults.test.ts +++ b/src/lib/domain/config/verify-export-defaults.test.ts @@ -181,7 +181,7 @@ describe("effective v1alpha1 export defaults (#12132)", () => { }, ); it.runIf(process.env.NEMOCLAW_RUN_V1_CONFIG_COMPATIBILITY === "1")( - "compiles strict Deep Agents policy with the pinned v1 consumer", + "compiles fail-closed Deep Agents policy with the pinned v1 consumer", testTimeoutOptions(12 * 60_000), async () => { const dcode = dcodeSnapshot(); @@ -189,7 +189,7 @@ describe("effective v1alpha1 export defaults (#12132)", () => { ...dcode, policy: { ...dcode.policy, - document: dcode.policy.document + "landlock:\n compatibility: strict\n", + document: dcode.policy.document + "landlock:\n compatibility: hard_requirement\n", }, }; const result = await exportSnapshots([strictDcode]); diff --git a/src/lib/domain/config/verify-export-source.ts b/src/lib/domain/config/verify-export-source.ts index 871823f2e77..94edf5fef51 100644 --- a/src/lib/domain/config/verify-export-source.ts +++ b/src/lib/domain/config/verify-export-source.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { recordedRegisteredInferenceProviderAttachment } from "../../inference/native-provider/registered"; import type * as TypeBoxValueModule from "typebox/value" with { "resolution-mode": "import" }; import { isDeepStrictEqual } from "node:util"; import { cloneAndDeepFreeze } from "../../core/immutable"; @@ -612,6 +613,7 @@ function expectedManagedStartupProfile(entry: ObservedExportRegistry): ManagedSt selected.preferredInferenceApi, exportUsesNativeInference(entry) || entry.nativeLocalProviderAttachment !== undefined, exportRouteEndpoint(entry), + recordedRegisteredInferenceProviderAttachment(entry), ); const projection = EXPORT_AGENT_PROFILE_PROJECTIONS[agent](inference); const search = exportWebSearchBinding(entry); @@ -1000,10 +1002,74 @@ function managedEndpointEvidenceMatchesRoute( ); } +function registeredEndpointEvidenceMatchesRoute( + snapshot: QualifiedExportSnapshot, + evidence: NonNullable & { + source: { kind: "registered-provider"; profileId: string; profileDigest: string }; + }, +): boolean { + const { inference } = snapshot; + try { + const binding = recordedRegisteredInferenceProviderAttachment(snapshot.registry); + return ( + !!binding && + isDeepStrictEqual( + [ + evidence.source.profileId, + evidence.source.profileDigest, + evidence.provider.id, + evidence.provider.name, + evidence.provider.gatewayName, + evidence.endpoint, + inference.provider, + inference.endpoint, + inference.api, + inference.credentialEnv, + ], + [ + binding.profileId, + binding.profileDigest, + binding.providerId, + binding.providerName, + binding.gatewayName, + binding.endpointUrl, + binding.providerName, + binding.endpointUrl, + binding.inferenceApi, + binding.credentialEnv, + ], + ) + ); + } catch { + return false; + } +} + +function providerConfigEndpointEvidenceMatches( + inference: QualifiedExportSnapshot["inference"], + evidence: NonNullable, +): boolean { + let expectedConfigKey: string | null = null; + if (inference.api === "anthropic-messages") expectedConfigKey = "ANTHROPIC_BASE_URL"; + else if (["openai-completions", "openai-responses"].includes(inference.api)) + expectedConfigKey = "OPENAI_BASE_URL"; + return ( + evidence.source.kind === "provider-config" && + expectedConfigKey !== null && + evidence.source.key === expectedConfigKey + ); +} + function endpointEvidenceMatchesRoute(snapshot: QualifiedExportSnapshot): boolean { const { inference } = snapshot; const evidence = inference.endpointEvidence; if (!evidence) return false; + if (evidence.source.kind === "registered-provider") { + return registeredEndpointEvidenceMatchesRoute(snapshot, { + ...evidence, + source: evidence.source, + }); + } if (snapshot.registry.nativeLocalProviderAttachment !== undefined) return nativeLocalEndpointEvidenceMatches(snapshot); if (evidence.source.kind === "managed-profile") @@ -1017,15 +1083,7 @@ function endpointEvidenceMatchesRoute(snapshot: QualifiedExportSnapshot): boolea ) ); } - let expectedConfigKey: string | null = null; - if (inference.api === "anthropic-messages") expectedConfigKey = "ANTHROPIC_BASE_URL"; - else if (["openai-completions", "openai-responses"].includes(inference.api)) - expectedConfigKey = "OPENAI_BASE_URL"; - return ( - evidence.source.kind === "provider-config" && - expectedConfigKey !== null && - evidence.source.key === expectedConfigKey - ); + return providerConfigEndpointEvidenceMatches(inference, evidence); } function validateSandboxIdentity( diff --git a/src/lib/domain/lifecycle/hermes-definition.test.ts b/src/lib/domain/lifecycle/hermes-definition.test.ts index de31f5cd4d7..eb93065bbed 100644 --- a/src/lib/domain/lifecycle/hermes-definition.test.ts +++ b/src/lib/domain/lifecycle/hermes-definition.test.ts @@ -4,16 +4,20 @@ import { describe, expect, it } from "vitest"; import { loadAgent } from "../../agent/defs"; -import { HERMES_LIFECYCLE_DEFINITION } from "./hermes-definition"; +import { HERMES_LIFECYCLE_DEFINITION, HERMES_CLI_OPENSHELL_VERSION } from "./hermes-definition"; describe("Hermes lifecycle definition", () => { + it("keeps the managed CLI runtime separate from the frozen first API", () => { + expect(HERMES_CLI_OPENSHELL_VERSION).toBe("0.1.7"); + expect(HERMES_LIFECYCLE_DEFINITION.openshellVersion).toBe("0.1.5"); + }); it("matches the packaged Hermes manifest version (#10613)", () => { const hermes = loadAgent("hermes"); expect(HERMES_LIFECYCLE_DEFINITION).toEqual({ agent: hermes.name, agentVersion: hermes.expectedVersion, - openshellVersion: "0.0.116", + openshellVersion: "0.1.5", }); }); }); diff --git a/src/lib/domain/lifecycle/hermes-definition.ts b/src/lib/domain/lifecycle/hermes-definition.ts index 848f245ac74..8b1c479f09b 100644 --- a/src/lib/domain/lifecycle/hermes-definition.ts +++ b/src/lib/domain/lifecycle/hermes-definition.ts @@ -5,5 +5,8 @@ export const HERMES_LIFECYCLE_DEFINITION = Object.freeze({ agent: "hermes" as const, agentVersion: "0.21.3" as const, - openshellVersion: "0.0.116" as const, + openshellVersion: "0.1.5" as const, }); + +/** OpenShell runtime selected by the Hermes CLI; independent of the first lifecycle API. */ +export const HERMES_CLI_OPENSHELL_VERSION = "0.1.7" as const; diff --git a/src/lib/hermes-managed-route.ts b/src/lib/hermes-managed-route.ts index d3b92cefe44..ac301f25f1e 100644 --- a/src/lib/hermes-managed-route.ts +++ b/src/lib/hermes-managed-route.ts @@ -53,6 +53,7 @@ export type HermesManagedRoute = { inferenceApi: string; contextWindow?: number | null; nativeCredentialHandle?: string; + registeredCredentialEnv?: string; }; function isObjectRecord(value: unknown): value is Record { @@ -88,16 +89,34 @@ export function hermesProviderKey(provider: string): string { export function applyHermesManagedRoute( config: Record, route: HermesManagedRoute, + runtimeLocalCredentialHandle?: string, ): asserts config is Record & HermesManagedRouting { const nativeHosted = nativeHostedAgentConfig(route.upstreamProvider, route.baseUrl); - const nativeLocal = nativeLocalCredentialReference(route.upstreamProvider, route.baseUrl); + const nativeLocal = nativeLocalCredentialReference( + route.upstreamProvider, + route.baseUrl, + undefined, + route.nativeCredentialHandle ?? runtimeLocalCredentialHandle, + ); let apiKey = nativeLocal ? `\${${NATIVE_LOCAL_CREDENTIAL_ENV}}` : nativeHosted ? `\${${nativeHosted.credentialEnv}}` : managedInferenceApiKey(route.baseUrl, HERMES_PROXY_REWRITE_SENTINEL); if (route.nativeCredentialHandle !== undefined) { - if (nativeLocal) { + if (route.registeredCredentialEnv) { + const env = route.registeredCredentialEnv; + if ( + !/^[A-Z][A-Z0-9_]{0,255}$/u.test(env) || + !new RegExp(`^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_${env}$`, "u").test( + route.nativeCredentialHandle, + ) || + /[\r\n]/u.test(route.nativeCredentialHandle) + ) + throw new Error( + "Registered Hermes inference requires its provider-bound credential handle", + ); + } else if (nativeLocal) { const match = new RegExp(NATIVE_LOCAL_RUNTIME_REFERENCE_PATTERN).exec( route.nativeCredentialHandle, ); @@ -156,26 +175,47 @@ export function applyHermesManagedRoute( if (apiMode) customProvider.api_mode = apiMode; const providers = isObjectRecord(config.providers) ? { ...config.providers } : {}; - if (previousProviderKey && previousProviderKey !== providerKey) { + if ( + !route.registeredCredentialEnv && + previousProviderKey && + previousProviderKey !== providerKey + ) { delete providers[previousProviderKey]; } - providers[providerKey] = providerConfig; + providers[providerKey] = route.registeredCredentialEnv + ? { + ...(isObjectRecord(providers[providerKey]) ? providers[providerKey] : {}), + ...providerConfig, + } + : providerConfig; const customProviders = Array.isArray(config.custom_providers) ? config.custom_providers.filter( (entry) => !isObjectRecord(entry) || - (entry.name !== previousUpstream.provider && entry.name !== providerName), + ((route.registeredCredentialEnv || entry.name !== previousUpstream.provider) && + entry.name !== providerName), ) : []; - customProviders.push(customProvider); + const retainedProvider = + route.registeredCredentialEnv && Array.isArray(config.custom_providers) + ? config.custom_providers.find( + (entry) => isObjectRecord(entry) && entry.name === providerName, + ) + : undefined; + customProviders.push({ + ...(isObjectRecord(retainedProvider) ? retainedProvider : {}), + ...customProvider, + }); config._nemoclaw_upstream = { provider: providerName, provider_key: providerKey, model: route.model, }; - config.model = modelConfig; + config.model = route.registeredCredentialEnv + ? { ...(isObjectRecord(config.model) ? config.model : {}), ...modelConfig } + : modelConfig; config.providers = providers; config.custom_providers = customProviders; } @@ -199,19 +239,30 @@ export function refreshHermesNativeLocalCredential( ); const previous = model.api_key; const nvidia = + !route.registeredCredentialEnv && route.upstreamProvider === "nvidia-prod" && managedInferenceApiKey(route.baseUrl, "") === NVIDIA_INFERENCE_PLACEHOLDER; - const pattern = nvidia - ? NATIVE_NVIDIA_SCOPED_HANDLE_PATTERN - : new RegExp(NATIVE_LOCAL_RUNTIME_REFERENCE_PATTERN); - const template = nvidia ? NVIDIA_INFERENCE_PLACEHOLDER : `\${${NATIVE_LOCAL_CREDENTIAL_ENV}}`; + const credentialEnv = route.registeredCredentialEnv ?? NATIVE_LOCAL_CREDENTIAL_ENV; + const pattern = route.registeredCredentialEnv + ? new RegExp(`^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_${credentialEnv}$`) + : nvidia + ? NATIVE_NVIDIA_SCOPED_HANDLE_PATTERN + : new RegExp(NATIVE_LOCAL_RUNTIME_REFERENCE_PATTERN); + const template = nvidia ? NVIDIA_INFERENCE_PLACEHOLDER : `\${${credentialEnv}}`; const reference = pattern.exec(String(previous)); // The sentinel identifies damage, not authority. Only the registered restore // caller may admit it after verifying live identity and provider binding. const recoverStripped = options.recoverStrippedNvidia === true && nvidia && previous === "[STRIPPED_BY_MIGRATION]"; if ( - (!nvidia && !nativeLocalCredentialReference(route.upstreamProvider, route.baseUrl)) || + (!route.registeredCredentialEnv && + !nvidia && + !nativeLocalCredentialReference( + route.upstreamProvider, + route.baseUrl, + undefined, + route.nativeCredentialHandle, + )) || upstream.provider !== route.upstreamProvider || upstream.provider_key !== providerKey || upstream.model !== route.model || @@ -228,7 +279,9 @@ export function refreshHermesNativeLocalCredential( typeof previous !== "string" || managed.api_key !== previous || matches[0].api_key !== previous || - (previous !== template && reference?.[0] !== previous && !recoverStripped) + ((Boolean(route.registeredCredentialEnv) || previous !== template) && + reference?.[0] !== previous && + !recoverStripped) ) { throw new Error("Restored Hermes inference configuration does not match its registered route."); } diff --git a/src/lib/hermes-switchyard-routing.ts b/src/lib/hermes-switchyard-routing.ts index fcbe9f573e6..264543cd743 100644 --- a/src/lib/hermes-switchyard-routing.ts +++ b/src/lib/hermes-switchyard-routing.ts @@ -4,7 +4,7 @@ import { Buffer } from "node:buffer"; import { - hostedNativeProvider, + fixedHostedNativeProvider, HOSTED_NATIVE_PROVIDERS, nativeHostedAgentConfig, } from "./inference/native-provider/agent-config.ts"; @@ -279,7 +279,7 @@ function clientProfile(providerId: HermesSwitchyardProvider): SwitchyardClientPr extraHeaders: {}, }); } - const definition = hostedNativeProvider(providerId); + const definition = fixedHostedNativeProvider(providerId); if (!definition || definition.logicalProvider === "hermes-provider") { return fail("provider profile is not supported by Switchyard onboarding"); } diff --git a/src/lib/inference/config.test.ts b/src/lib/inference/config.test.ts index 116e22e5c07..3dc735a4b0e 100644 --- a/src/lib/inference/config.test.ts +++ b/src/lib/inference/config.test.ts @@ -384,6 +384,23 @@ describe("inference selection config", () => { }); describe("getSandboxInferenceConfig", () => { + it("keeps Pi private compatible endpoints on their native custom provider", () => { + const endpoint = "http://10.0.0.2:11434/v1"; + expect( + getSandboxInferenceConfig( + "selected-model", + "compatible-endpoint", + "openai-completions", + true, + endpoint, + "pi", + ), + ).toMatchObject({ + inferenceBaseUrl: endpoint, + inferenceApi: "openai-completions", + }); + }); + it("enables streaming usage for ollama-local behind the managed inference route", () => { expect(getSandboxInferenceConfig("qwen3.6:35b", "ollama-local")).toEqual({ providerKey: MANAGED_PROVIDER_ID, @@ -667,7 +684,7 @@ describe("formatInferenceRouteDriftForDisplay", () => { }); it.each([ - ["openclaw", true, "https://inference.local/v1"], + ["openclaw", true, "http://host.openshell.internal:8000/v1"], ["pi", true, "http://host.openshell.internal:8000/v1"], ["pi", false, "https://inference.local/v1"], ["langchain-deepagents-code", false, "https://inference.local/v1"], @@ -725,3 +742,26 @@ it.each([ expect(config.inferenceBaseUrl).toBe(endpoint); expect(config.inferenceApi).toBe(expectedApi); }); + +it.each([ + { agent: "openclaw" as const, api: "openai-completions" as const }, + { agent: "openclaw" as const, api: "openai-responses" as const }, + { agent: "hermes" as const, api: "openai-completions" as const }, + { agent: "hermes" as const, api: "openai-responses" as const }, +])("preserves compatible $agent native API $api", ({ agent, api }) => { + expect( + getSandboxInferenceConfig( + "served-alias", + "compatible-endpoint", + api, + true, + "http://host.openshell.internal:8000/v1", + agent, + ), + ).toMatchObject({ + inferenceBaseUrl: "http://host.openshell.internal:8000/v1", + inferenceApi: api, + primaryModelRef: "inference/served-alias", + providerKey: "inference", + }); +}); diff --git a/src/lib/inference/config.ts b/src/lib/inference/config.ts index 98dc91b2a51..bc111318458 100644 --- a/src/lib/inference/config.ts +++ b/src/lib/inference/config.ts @@ -6,7 +6,8 @@ * All functions are pure. */ -import { nativeLocalCredentialReference } from "./native-local/agent-config"; +import type { RegisteredInferenceProviderAttachment } from "./native-provider/registered"; +import { nativeLocalCredentialReference } from "./native-local/model-router"; import { usesNativeLocalInference, usesNativeLocalInferenceForAgent, @@ -43,7 +44,10 @@ export { isPersistedPiSharedLocalInferenceRoute, normalizeNativeLocalProviderAttachment, } from "./native-local/contract"; -export { hasLegacySharedLocalInferenceRoute } from "./selection"; +export { + hasLegacySharedLocalInferenceRoute, + hasLegacySharedCustomInferenceRoute, +} from "./selection"; export type { NativeLocalProviderAttachment } from "./native-local/contract"; export { OLLAMA_LOCAL_CREDENTIAL_ENV }; export { managedInferenceApiKey, NVIDIA_INFERENCE_PLACEHOLDER } from "../inference-credential"; @@ -358,7 +362,19 @@ export function getSandboxInferenceConfig( nativeProvider?: boolean, nativeEndpointUrl?: string | null, agentName?: string | null, + registered?: RegisteredInferenceProviderAttachment, ): SandboxInferenceConfig { + if (registered) { + if (registered.providerName !== provider) + throw new Error("Registered inference provider does not match selection"); + return { + providerKey: registered.providerName, + primaryModelRef: `${registered.providerName}/${model}`, + inferenceBaseUrl: registered.endpointUrl, + inferenceApi: registered.inferenceApi, + inferenceCompat: null, + }; + } const nativeCustom = nativeProvider === true && isCustomHostedProvider(provider) && @@ -474,7 +490,10 @@ export function getSandboxInferenceConfig( nativeLocalCredentialReference(provider, nativeEndpointUrl, agentName) ) { inferenceBaseUrl = nativeEndpointUrl; - inferenceApi = "openai-completions"; + inferenceApi = + (agentName === "openclaw" || agentName === "hermes") && provider === "compatible-endpoint" + ? selectedApi || "openai-completions" + : "openai-completions"; } return { providerKey, primaryModelRef, inferenceBaseUrl, inferenceApi, inferenceCompat }; } @@ -567,3 +586,10 @@ export function formatInferenceRouteDriftForDisplay( warning: `gateway inference route (${liveProvider}/${liveModel}) differs from the recorded route ${owner} (${recordedRoute}).`, }; } + +export { + recordedRegisteredInferenceProviderAttachment, + normalizeRegisteredInferenceProviderAttachment, + verifyRegisteredInferenceProviderAttachment, +} from "./native-provider/registered"; +export type { RegisteredInferenceProviderAttachment } from "./native-provider/registered"; diff --git a/src/lib/inference/gateway/route-contract.ts b/src/lib/inference/gateway/route-contract.ts index 021538112c7..b84789d510a 100644 --- a/src/lib/inference/gateway/route-contract.ts +++ b/src/lib/inference/gateway/route-contract.ts @@ -1,7 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { getSandboxInferenceConfig, resolveAgentInferenceApi } from "../config"; +import { + getSandboxInferenceConfig, + resolveAgentInferenceApi, + type RegisteredInferenceProviderAttachment, +} from "../config"; /** Resolve the provider route recorded in one managed startup profile. */ export function resolveManagedStartupInferenceRoute( @@ -11,6 +15,7 @@ export function resolveManagedStartupInferenceRoute( preferredInferenceApi: string | null, nativeProvider = true, nativeEndpointUrl?: string | null, + registered?: RegisteredInferenceProviderAttachment, ) { const api = agentName === "langchain-deepagents-code" @@ -23,5 +28,6 @@ export function resolveManagedStartupInferenceRoute( nativeProvider, nativeEndpointUrl, agentName, + registered, ); } diff --git a/src/lib/inference/local-vllm-auth.test.ts b/src/lib/inference/local-vllm-auth.test.ts index 396c665461d..dc63064b045 100644 --- a/src/lib/inference/local-vllm-auth.test.ts +++ b/src/lib/inference/local-vllm-auth.test.ts @@ -22,9 +22,6 @@ const hostLocalRecovery = vi.hoisted(() => ({ endpoint: vi.fn(), bridgeHost: vi.fn<() => string | null>(() => null), })); -const managedBridge = vi.hoisted(() => ({ - host: vi.fn(() => "172.18.0.1"), -})); const managedKey = vi.hoisted(() => ({ load: vi.fn(), })); @@ -38,8 +35,7 @@ vi.mock("./serving/managed-cluster-runtime-receipt", () => ({ vi.mock("./serving/vllm-host-local-lifecycle", async (importOriginal) => ({ ...(await importOriginal()), recoverHostLocalManagedVllmEndpoint: hostLocalRecovery.endpoint, - getLegacyHostLocalManagedVllmBridgeHost: hostLocalRecovery.bridgeHost, - resolveManagedVllmBridgeHost: managedBridge.host, + getLegacyHostLocalManagedVllmLoopbackHost: hostLocalRecovery.bridgeHost, })); vi.mock("./vllm-api-key", async (importOriginal) => ({ ...(await importOriginal()), @@ -111,7 +107,7 @@ function hostLocalDiagnosticCapture(argv: readonly string[]): string { return argv[0] === "curl" ? "200" : argv.at(-1) === "/etc/hosts" - ? "172.18.0.1\thost.openshell.internal" + ? "127.0.0.1\thost.openshell.internal" : argv.includes("-o") ? "503" : ""; @@ -125,8 +121,6 @@ beforeEach(() => { hostLocalRecovery.endpoint.mockReturnValue(null); hostLocalRecovery.bridgeHost.mockReset(); hostLocalRecovery.bridgeHost.mockReturnValue(null); - managedBridge.host.mockReset(); - managedBridge.host.mockReturnValue("172.18.0.1"); managedKey.load.mockReset(); managedKey.load.mockReturnValue(API_KEY); lifecycle.baseUrl.mockReset(); @@ -157,13 +151,13 @@ describe("managed vLLM authentication", () => { expect(capture).not.toHaveBeenCalled(); }); - it("probes bearerless managed vLLM on the OpenShell bridge instead of Docker's default gateway (#14083)", () => { + it("probes bearerless managed vLLM on loopback with host networking (#14083)", () => { vi.stubEnv("DOCKER_CONTEXT", "remote-builder"); vi.stubEnv("DOCKER_HOST", "ssh://remote.example.test"); lifecycle.baseUrl.mockReturnValue(null); - hostLocalRecovery.bridgeHost.mockReturnValue("172.18.0.1"); + hostLocalRecovery.bridgeHost.mockReturnValue("127.0.0.1"); const capture = vi.fn((argv: readonly string[]) => - argv[0] === "curl" || argv.includes("host.openshell.internal:172.18.0.1") + argv[0] === "curl" || argv.includes("host.openshell.internal:127.0.0.1") ? `${JSON.stringify({ data: [{ id: "test/model" }] })}\n200` : "", ); @@ -171,16 +165,17 @@ describe("managed vLLM authentication", () => { expect(validateLocalProvider("vllm-local", capture, vi.fn())).toEqual({ ok: true }); const command = capture.mock.calls.find(([argv]) => argv[0] === "docker")?.[0]; expect(command?.slice(0, 4)).toEqual(["docker", "--context", "default", "run"]); - expect(command).toContain("host.openshell.internal:172.18.0.1"); + expect(command).toContain("host.openshell.internal:127.0.0.1"); expect(command).not.toContain("host.openshell.internal:host-gateway"); expect(command).not.toContain("host.openshell.internal:172.17.0.1"); + expect(command).toEqual(expect.arrayContaining(["--network", "host"])); }); it.each([ { state: "unsafe bindings", failure: new UnsafeLegacyVllmBindingsError() }, { - state: "an unavailable bridge", - failure: new Error("Managed host-local vLLM could not inspect the OpenShell bridge"), + state: "an unavailable inspection", + failure: new Error("Managed vLLM container inspection failed"), }, ])("returns a validation failure when a later inspection finds $state", ({ failure }) => { lifecycle.baseUrl.mockReturnValue(null); @@ -390,7 +385,7 @@ describe("managed vLLM authentication", () => { ).toThrow("Managed host-local vLLM returned an invalid loopback endpoint"); }); - it("pins host-local reachability to the exact OpenShell bridge (#8379)", () => { + it("pins host-local reachability to the native supervisor network (#8379)", () => { lifecycle.baseUrl.mockReturnValue(null); hostLocalRecovery.endpoint.mockReturnValue({ baseUrl: "http://127.0.0.1:8000", @@ -399,13 +394,13 @@ describe("managed vLLM authentication", () => { const command = getLocalProviderContainerReachabilityCheck("vllm-local"); - expect(command).toContain("host.openshell.internal:172.18.0.1"); + expect(command).toContain("host.openshell.internal:127.0.0.1"); expect(command).not.toContain("host.openshell.internal:host-gateway"); expect(command?.at(-1)).toBe("http://host.openshell.internal:8000/health"); - expect(managedBridge.host).toHaveBeenCalledOnce(); + expect(command).toEqual(expect.arrayContaining(["--network", "host"])); }); - it("pins host-local diagnostics to the exact OpenShell bridge (#8379)", () => { + it("pins host-local diagnostics to the native supervisor network (#8379)", () => { lifecycle.baseUrl.mockReturnValue(null); hostLocalRecovery.endpoint.mockReturnValue({ baseUrl: "http://127.0.0.1:8000", @@ -420,15 +415,15 @@ describe("managed vLLM authentication", () => { expect(result.ok).toBe(false); expect(result.diagnostic).toContain("Container curl returned HTTP 503"); - expect(result.diagnostic).toContain("host.openshell.internal resolved to: 172.18.0.1"); + expect(result.diagnostic).toContain("host.openshell.internal resolved to: 127.0.0.1"); expect(dockerCommands).toHaveLength(5); - expect( - dockerCommands.every((argv) => argv.includes("host.openshell.internal:172.18.0.1")), - ).toBe(true); + expect(dockerCommands.every((argv) => argv.includes("host.openshell.internal:127.0.0.1"))).toBe( + true, + ); expect( dockerCommands.every((argv) => !argv.includes("host.openshell.internal:host-gateway")), ).toBe(true); - expect(managedBridge.host).toHaveBeenCalledOnce(); + expect(dockerCommands.every((argv) => argv.includes("host"))).toBe(true); }); it("uses /health only for unauthenticated availability checks", () => { diff --git a/src/lib/inference/local-vllm-health.test.ts b/src/lib/inference/local-vllm-health.test.ts index b1b9f22faac..58e7ae3ecd5 100644 --- a/src/lib/inference/local-vllm-health.test.ts +++ b/src/lib/inference/local-vllm-health.test.ts @@ -54,12 +54,7 @@ describe("recorded bearerless vLLM health ports", () => { (owned) => { const recordedPort = VLLM_PORT === 18080 ? 18081 : 18080; inspectBearerlessVllmBindings( - owned - ? [ - { HostIp: "127.0.0.1", HostPort: String(recordedPort) }, - { HostIp: "172.18.0.1", HostPort: String(recordedPort) }, - ] - : null, + owned ? [{ HostIp: "127.0.0.1", HostPort: String(recordedPort) }] : null, ); const probedEndpoints: string[] = []; const result = probeLocalProviderHealth("vllm-local", { @@ -114,10 +109,7 @@ describe("recorded bearerless vLLM health ports", () => { ])( "keeps the configured local vLLM port when the recorded route has $reason", ({ endpoint: recordedEndpointUrl }) => { - inspectBearerlessVllmBindings([ - { HostIp: "127.0.0.1", HostPort: String(VLLM_PORT) }, - { HostIp: "172.18.0.1", HostPort: String(VLLM_PORT) }, - ]); + inspectBearerlessVllmBindings([{ HostIp: "127.0.0.1", HostPort: String(VLLM_PORT) }]); const result = probeLocalProviderHealth("vllm-local", { recordedEndpointUrl, runCurlProbeImpl: () => ({ diff --git a/src/lib/inference/local.ts b/src/lib/inference/local.ts index 18aaf59a7c6..b8eb67e2f43 100644 --- a/src/lib/inference/local.ts +++ b/src/lib/inference/local.ts @@ -86,10 +86,9 @@ import { recoverInstalledManagedClusterVllmEndpoint, } from "./serving/managed-cluster-runtime-receipt"; import { - getLegacyHostLocalManagedVllmBridgeHost, + getLegacyHostLocalManagedVllmLoopbackHost, ManagedVllmInspectionError, recoverHostLocalManagedVllmEndpoint, - resolveManagedVllmBridgeHost, UnsafeLegacyVllmBindingsError, } from "./serving/vllm-host-local-lifecycle"; import { loadManagedVllmApiKey } from "./vllm-api-key"; @@ -1210,7 +1209,7 @@ export function getManagedVllmProviderState( } if (!managedClusterEndpoint) { if (stationState.kind === "absent") { - getLegacyHostLocalManagedVllmBridgeHost(options.legacyServingPort ?? VLLM_PORT); + getLegacyHostLocalManagedVllmLoopbackHost(options.legacyServingPort ?? VLLM_PORT); } return stationState; } @@ -1681,16 +1680,25 @@ export function getLocalProviderContainerReachabilityCheck( const managed = recoveredManagedVllmBinding(); if (managed.kind === "unavailable") return null; const managedBaseUrl = managed.binding?.baseUrl.replace(/\/v1\/?$/, "") ?? null; - const bridgeHost = managed.binding?.validationBaseUrl - ? resolveManagedVllmBridgeHost() + const loopbackHost = managed.binding?.validationBaseUrl + ? "127.0.0.1" : !managed.binding && !configuredLocalInferenceHostUrl() - ? getLegacyHostLocalManagedVllmBridgeHost(VLLM_PORT) + ? getLegacyHostLocalManagedVllmLoopbackHost(VLLM_PORT) : null; - const hostAlias = `host.openshell.internal:${bridgeHost ?? "host-gateway"}`; + const hostAlias = `host.openshell.internal:${loopbackHost ?? "host-gateway"}`; return [ - ...(managedBaseUrl || bridgeHost ? ["docker", "--context", "default"] : ["docker"]), + ...(managedBaseUrl || loopbackHost ? ["docker", "--context", "default"] : ["docker"]), "run", "--rm", + ...(loopbackHost + ? [ + "--network", + "host", + "--cap-drop=ALL", + "--security-opt=no-new-privileges", + "--read-only", + ] + : []), "--add-host", hostAlias, CONTAINER_REACHABILITY_IMAGE, @@ -2044,16 +2052,15 @@ function collectContainerDiagnostic( ); } const dockerCommand = containerCommand.slice(0, dockerRunIndex); + const probeCommand = containerCommand.slice( + 0, + containerCommand.indexOf(CONTAINER_REACHABILITY_IMAGE) + 1, + ); try { - // Reuse the exact Docker context, host mapping, and URL from the failed check. + // Retain the failed probe's Docker context, network, hardening, and host mapping. const httpStatus = capture( [ - ...dockerCommand, - "run", - "--rm", - "--add-host", - hostAlias, - CONTAINER_REACHABILITY_IMAGE, + ...probeCommand, "-s", "-o", "/dev/null", @@ -2069,19 +2076,7 @@ function collectContainerDiagnostic( ); // Confirm that Docker applied the same host mapping used by the failed check. - const hostsOutput = capture( - [ - ...dockerCommand, - "run", - "--rm", - "--add-host", - hostAlias, - CONTAINER_REACHABILITY_IMAGE, - "cat", - "/etc/hosts", - ], - { ignoreError: true }, - ); + const hostsOutput = capture([...probeCommand, "cat", "/etc/hosts"], { ignoreError: true }); if (!httpStatus && !hostsOutput) { return classifyContainerRunFailure(dockerCommand, capture); diff --git a/src/lib/inference/native-local.test.ts b/src/lib/inference/native-local.test.ts index f2c4f7a308f..bcd3b4c6542 100644 --- a/src/lib/inference/native-local.test.ts +++ b/src/lib/inference/native-local.test.ts @@ -11,8 +11,12 @@ import { prepareNativeLocalSelection } from "./native-local/selection"; import * as routerPort from "../core/model-router-port"; import { getSandboxInferenceConfig } from "./config"; import { createSetupInference, type SetupInferenceDeps } from "../onboard/setup-inference"; +import * as routerSession from "../state/onboard-session"; +import * as routerProcess from "../onboard/model-router-process"; +import { hashCredential } from "../security/credential-hash"; +import { prepareNativeModelRouterSelection } from "../onboard/routed-inference"; import { upsertRoutedProvider } from "../onboard/routed-inference"; -import { describe, expect, it, vi } from "vitest"; +import { beforeEach, describe, expect, it, vi } from "vitest"; import { createCliOpenShellProviderAdapter, type RunProviderCommand, @@ -35,6 +39,7 @@ import { verifyNativeLocalProviderAttachment, verifyRecordedNativeLocalProviderAttachment, retireNativeLocalProvider, + detachNativeLocalProvider, } from "./native-local/profile"; vi.mock("../adapters/openshell/provider-policy", () => ({ @@ -83,6 +88,7 @@ function fixture(selectedBinding: NativeLocalBinding = binding) { type: identity.profileId, credentialKeys: [binding.credentialEnv], configKeys: [], + credentialExpiresAtMs: {}, revision: { id: "owned-id", resourceVersion: 1 }, }, } @@ -135,6 +141,56 @@ function fixture(selectedBinding: NativeLocalBinding = binding) { } describe("native local inference", () => { + it.each(["openclaw", "hermes", "pi", "langchain-deepagents-code"] as const)( + "retains %s Model Router profile, rebuild identity, and retirement ownership", + async (agent) => { + const routerBinding = { + ...binding, + ...(agent === "langchain-deepagents-code" ? {} : { agent }), + provider: "nvidia-router" as const, + endpointUrl: "http://host.openshell.internal:4000/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }; + const f = fixture(routerBinding); + const receipt = await ensureNativeLocalProvider(f.input); + await ensureNativeLocalProviderAttached({ + adapter: f.adapter, + expected: receipt, + sandboxName: "alice", + }); + await verifyRecordedNativeLocalProviderAttachment({ + adapter: f.adapter, + expected: receipt, + agent, + sandboxName: "alice", + gatewayName: "nemoclaw", + provider: "nvidia-router", + }); + const profile = prepareNativeLocalProfile(routerBinding); + expect(profile.document.endpoints[0].rules).toEqual([ + { allow: { method: "GET", path: "/v1/models" } }, + { allow: { method: "POST", path: "/v1/chat/completions" } }, + ]); + expect(nativeLocalIdentity({ ...routerBinding, sandboxName: "bob" }).providerName).not.toBe( + receipt.providerName, + ); + await detachNativeLocalProvider({ + adapter: f.adapter, + expected: receipt, + sandboxName: "alice", + }); + const clearAuthority = vi.fn(); + await retireNativeLocalProvider({ + adapter: f.adapter, + expected: receipt, + sandboxName: "alice", + gatewayName: "nemoclaw", + clearAuthority, + }); + expect(f.adapter.deleteProvider).toHaveBeenCalledOnce(); + expect(clearAuthority).toHaveBeenCalledOnce(); + }, + ); const prepared = prepareNativeLocalProfile(binding); it.each([ [ @@ -207,9 +263,9 @@ describe("native local inference", () => { ["pi", "http://169.254.169.254:12345/v1", false], ["pi", "http://host.openshell.internal:80/v1", false], ["pi", "http://user:secret@host.openshell.internal:12345/v1", false], - ["langchain-deepagents-code", "http://host.openshell.internal:12345/v1", false], - ["openclaw", "http://host.openshell.internal:12345/v1", false], - ["hermes", "http://host.openshell.internal:12345/v1", false], + ["langchain-deepagents-code", "http://host.openshell.internal:12345/v1", true], + ["openclaw", "http://host.openshell.internal:12345/v1", true], + ["hermes", "http://host.openshell.internal:12345/v1", true], ] as const)("classifies %s compatible endpoint %s as native=%s", (agent, endpoint, expected) => { expect(usesNativeLocalInferenceForAgent(agent, "compatible-endpoint", endpoint)).toBe(expected); }); @@ -277,6 +333,7 @@ describe("native local inference", () => { credentialKeys: [expected.credentialEnv], configKeys: [], revision: { id: "different-instance", resourceVersion: 2 }, + credentialExpiresAtMs: {}, }, }); }, @@ -402,20 +459,14 @@ describe("native local inference", () => { ); }); - it("keeps an owned Local NIM selection on the shared route while standalone vLLM is native", () => { - expect(usesNativeLocalInferenceForAgent("openclaw", "vllm-local", null)).toBe(true); - expect(usesNativeLocalInferenceForAgent("langchain-deepagents-code", "vllm-local", null)).toBe( - true, - ); - expect(usesNativeLocalInferenceForAgent("openclaw", "vllm-local", null, true)).toBe(false); - expect(usesNativeLocalInferenceForAgent("pi", "vllm-local", null, true)).toBe(false); - expect( - usesNativeLocalInferenceForAgent("langchain-deepagents-code", "vllm-local", null, true), - ).toBe(false); - expect( - usesNativeLocalInferenceForAgent("langchain-deepagents-code", "llama-cpp-local", null, true), - ).toBe(true); - }); + it.each(["openclaw", "hermes", "pi", "langchain-deepagents-code"])( + "selects native Local NIM for %s", + (agent) => { + expect(usesNativeLocalInferenceForAgent(agent, "vllm-local", binding.endpointUrl, true)).toBe( + true, + ); + }, + ); it("isolates Hermes interpreter permissions and preserves recorded agent identities", () => { expect(nativeLocalIdentity(binding, 1)).toEqual({ @@ -445,6 +496,7 @@ describe("native local inference", () => { }); expect(hermes.document.binaries).toEqual([ "/usr/bin/python3", + "/usr/bin/python3.13", "/opt/hermes/.venv/bin/python", "/opt/hermes/.venv/bin/python3", "/opt/venv/bin/python3", @@ -452,17 +504,17 @@ describe("native local inference", () => { "/usr/bin/curl", ]); expect(hermes.document.endpoints).toEqual(dcode.document.endpoints); - expect(usesNativeLocalInferenceForAgent("hermes", "vllm-local", null, true)).toBe(false); + expect(usesNativeLocalInferenceForAgent("hermes", "vllm-local", null, true)).toBe(true); expect( usesNativeLocalInferenceForAgent("openclaw", "compatible-endpoint", binding.endpointUrl), - ).toBe(false); + ).toBe(true); expect( normalizeNativeLocalBinding({ ...binding, provider: "compatible-endpoint", agent: "openclaw", }), - ).toBeUndefined(); + ).toMatchObject({ agent: "openclaw", provider: "compatible-endpoint" }); }); it.each(["langchain-deepagents-code", "unknown"])( @@ -563,11 +615,19 @@ describe("native local inference", () => { const openClawProfile = prepareNativeLocalProfile({ ...binding, agent: "openclaw" }); expect(openClawProfile.document.binaries).toContain("/usr/local/bin/node"); expect(openClawProfile.document.binaries).toContain("/usr/bin/python3"); + expect(openClawProfile.document.binaries).toContain("/usr/bin/python3.13"); expect(openClawProfile.profileId).not.toBe(profile.profileId); expect(profile.document.binaries).not.toContain("/usr/local/bin/openclaw"); const piProfile = prepareNativeLocalProfile({ ...binding, agent: "pi" }); - expect(piProfile.document.binaries).toContain("/usr/local/bin/node"); - expect(piProfile.document.binaries).toContain("/usr/bin/python3"); + expect(piProfile.document.binaries).toEqual([ + "/usr/bin/python3", + "/usr/bin/python3.13", + "/usr/local/bin/node", + "/usr/bin/node", + "/usr/local/bin/curl", + "/usr/bin/curl", + ]); + expect(piProfile.document.endpoints).toEqual(profile.document.endpoints); expect(piProfile.profileId).not.toBe(profile.profileId); expect(piProfile.profileId).not.toBe(openClawProfile.profileId); expect(profile.document.binaries).not.toContain("/opt/hermes/.venv/bin/python"); @@ -803,6 +863,7 @@ describe("native local inference", () => { name: expected.providerName, type: expected.profileId, credentialKeys: [expected.credentialEnv], + credentialExpiresAtMs: { [expected.credentialEnv]: 0 }, configKeys: [], revision: { id: expected.providerId, resourceVersion: 1 }, }, @@ -922,6 +983,14 @@ vi.mock("../state/gateway-registry", () => ({ })); describe("native OpenClaw Model Router (#13560)", () => { + beforeEach(() => { + vi.spyOn(routerSession, "loadSession").mockReturnValue({ + routerPort: 4000, + routerPid: 123, + routerCredentialHash: hashCredential("test-only-secret"), + } as ReturnType); + vi.spyOn(routerProcess, "doesModelRouterProcessOwnPort").mockReturnValue(true); + }); const routerBinding: NativeLocalBinding = { ...binding, provider: "nvidia-router", @@ -968,7 +1037,8 @@ describe("native OpenClaw Model Router (#13560)", () => { isNonInteractive: () => true, hermesProviderAuth: { HERMES_PROVIDER_NAME: "hermes-provider" }, hydrateCredentialEnv: () => "test-only-secret", - localInference: {}, + localInference: { loadPendingOllamaModelCleanup: () => [] }, + withOllamaModelOwnershipLock: (action: () => unknown) => action(), isRoutedInferenceProvider: (provider: string) => provider === "nvidia-router", reconcileModelRouter: async () => { events.push("router"); @@ -1083,9 +1153,19 @@ describe("native OpenClaw Model Router (#13560)", () => { ); it("restricts a nondefault port to the router endpoint and OpenClaw executables", async () => { - vi.spyOn(routerPort, "resolveConfiguredModelRouterPort").mockReturnValue(12346); + vi.spyOn(routerPort, "loadBlueprintProfile").mockReturnValue({ + model: "nvidia-routed", + provider_name: "nvidia-router", + endpoint: "http://localhost:12346/v1", + router: { enabled: true, port: 12346 }, + }); + vi.mocked(routerSession.loadSession).mockReturnValue({ + routerPort: 12346, + routerPid: 123, + routerCredentialHash: hashCredential("test-only-secret"), + } as ReturnType); const f = fixture({ ...routerBinding, endpointUrl: "http://host.openshell.internal:12346/v1" }); - const expected = await prepareNativeLocalSelection({ ...f.input, ownedModelRouterPort: 12346 }); + const expected = await prepareNativeModelRouterSelection(f.input); const prepared = prepareNativeLocalProfile(expected); expect(prepared.document.endpoints).toEqual([ expect.objectContaining({ @@ -1119,38 +1199,44 @@ describe("native OpenClaw Model Router (#13560)", () => { { provider: "nvidia-router", endpoint: "http://host.openshell.internal:4000/v1", - ownedPort: undefined, + generic: true, }, { provider: "compatible-endpoint", endpoint: "http://host.openshell.internal:4000/v1", - ownedPort: 4000, + generic: true, agent: "pi", }, - { provider: "nvidia-router", endpoint: "http://host.docker.internal:4000/v1", ownedPort: 4000 }, - { provider: "nvidia-router", endpoint: "http://10.1.2.3:4000/v1", ownedPort: 4000 }, + { provider: "nvidia-router", endpoint: "http://host.docker.internal:4000/v1" }, + { provider: "nvidia-router", endpoint: "http://10.1.2.3:4000/v1" }, { provider: "nvidia-router", endpoint: "http://host.openshell.internal:12346/v1", - ownedPort: 12346, }, { provider: "nvidia-router", endpoint: "http://host.openshell.internal:8080/v1", - ownedPort: 8080, }, ])( - "rejects unauthorized $provider endpoint $endpoint with port authority $ownedPort", - async ({ provider, endpoint, ownedPort, agent }) => { - const f = fixture({ - ...routerBinding, - provider: provider as NativeLocalBinding["provider"], - agent: (agent ?? "openclaw") as "openclaw" | "pi", - endpointUrl: endpoint, - }); + "rejects unauthorized $provider endpoint $endpoint", + async ({ provider, endpoint, generic, agent }) => { + const f = fixture(routerBinding); + const input = { + ...f.input, + binding: { + ...routerBinding, + provider, + agent: (agent ?? "openclaw") as "openclaw" | "pi", + endpointUrl: endpoint, + }, + }; await expect( - prepareNativeLocalSelection({ ...f.input, ownedModelRouterPort: ownedPort }), - ).rejects.toThrow("protected host port"); + generic ? prepareNativeLocalSelection(input) : prepareNativeModelRouterSelection(input), + ).rejects.toThrow( + generic + ? "protected host port" + : "Model Router selection does not match its configured endpoint and credential", + ); expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); expect(f.adapter.createProvider).not.toHaveBeenCalled(); }, @@ -1210,27 +1296,204 @@ describe("local inference endpoint port reservations", () => { }); }); -it.each(["http://10.1.2.3:8000/v1", "http://host.openshell.internal:42103/v1"])( - "keeps compatible endpoint %s on the shared route (#13197)", - (endpoint) => { - expect( - usesNativeLocalInferenceForAgent( - "langchain-deepagents-code", - "compatible-endpoint", - endpoint, - ), - ).toBe(false); - }, -); +it.each([ + "http://10.1.2.3:8000/v1", + "http://localhost:42103/v1", + "http://host.docker.internal:42103/v1", +])("keeps compatible endpoint %s on the shared route (#13197)", (endpoint) => { + expect( + usesNativeLocalInferenceForAgent("langchain-deepagents-code", "compatible-endpoint", endpoint), + ).toBe(false); +}); -it("keeps Local NIM and generic compatible endpoints on the shared OpenClaw route", () => { +it("selects Local NIM and compatible OpenClaw endpoints natively", () => { expect( usesNativeLocalInferenceForAgent("openclaw", "vllm-local", binding.endpointUrl, true), - ).toBe(false); + ).toBe(true); expect( usesNativeLocalInferenceForAgent("openclaw", "compatible-endpoint", binding.endpointUrl), - ).toBe(false); + ).toBe(true); expect(usesNativeLocalInferenceForAgent("openclaw", "vllm-local", binding.endpointUrl)).toBe( true, ); }); + +it.each([ + { agent: "openclaw" as const, api: "openai-completions" as const }, + { agent: "openclaw" as const, api: "openai-responses" as const }, + { agent: "hermes" as const, api: "openai-completions" as const }, + { agent: "hermes" as const, api: "openai-responses" as const }, +])( + "binds compatible $agent API $api to its provider identity and request path", + async ({ agent, api: selectedApi }) => { + const api = selectedApi === "openai-responses" ? selectedApi : undefined; + const selected = { + ...binding, + provider: "compatible-endpoint" as const, + agent, + ...(api ? { api } : {}), + }; + const profile = prepareNativeLocalProfile(selected); + const defaultProfile = prepareNativeLocalProfile({ ...selected, api: undefined }); + expect(profile.document.endpoints[0]?.rules).toEqual([ + { allow: { method: "GET", path: "/v1/models" } }, + { allow: { method: "POST", path: api ? "/v1/responses" : "/v1/chat/completions" } }, + ]); + expect(profile.profileId === defaultProfile.profileId).toBe(!api); + const f = fixture(selected); + const attached = await ensureNativeLocalProvider(f.input); + await ensureNativeLocalProviderAttached({ + adapter: f.adapter, + expected: attached, + sandboxName: "alice", + }); + await expect( + verifyNativeLocalProviderAttachment({ + adapter: f.adapter, + expected: attached, + sandboxName: "alice", + }), + ).resolves.toEqual(attached); + expect( + normalizeNativeLocalProviderAttachment({ + ...attached, + api: api ? undefined : "openai-responses", + }), + ).toBeUndefined(); + }, +); + +it.each([ + { provider: "ollama-local", agent: "openclaw", api: "openai-responses" }, + { provider: "compatible-endpoint", agent: "pi", api: "openai-responses" }, + { provider: "compatible-endpoint", agent: "openclaw", api: "anthropic-messages" }, + { + provider: "compatible-endpoint", + agent: "openclaw", + endpointUrl: "http://host.docker.internal:42103/v1", + }, + { + provider: "compatible-endpoint", + agent: "openclaw", + endpointUrl: "http://192.168.1.20:8000/v1", + }, +])("rejects unqualified compatible OpenClaw binding %j", (override) => { + expect(normalizeNativeLocalBinding({ ...binding, ...override })).toBeUndefined(); +}); + +it.each([8080, 4000, 18789, 8642, 11435, 11436, 11437, 11438, 12345, 12346])( + "rejects compatible reserved alias port %s before provider creation", + async (port) => { + const selected = { + ...binding, + provider: "compatible-endpoint" as const, + agent: "openclaw" as const, + endpointUrl: `http://host.openshell.internal:${port}/v1`, + }; + const f = fixture(selected); + await expect(prepareNativeLocalSelection(f.input)).rejects.toThrow(/protected host port/); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + }, +); + +it.each([false, true])( + "preserves a retained Hermes compatible binding before profile replacement with owned proxy=%s", + async (ownedProxy) => { + const selected = { + ...binding, + provider: "compatible-endpoint" as const, + agent: "hermes" as const, + authMode: "authenticated" as const, + endpointUrl: ownedProxy + ? "http://host.openshell.internal:11435/v1" + : "http://host.openshell.internal:42103/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + api: "openai-responses" as const, + }; + const f = fixture(selected); + const previous = { + ...selected, + ...nativeLocalIdentity(selected, 1), + schemaVersion: 1 as const, + providerId: "old-instance", + }; + const credentialEnv = ownedProxy ? "NEMOCLAW_OLLAMA_PROXY_TOKEN" : "COMPATIBLE_API_KEY"; + const resolve = vi.spyOn(localRuntime, "resolveHermesLocalSwitchRuntime"); + f.adapter.importProviderProfile.mockResolvedValue({ + ok: false, + error: { kind: "command", reason: "profile_incompatible", message: "collision" }, + }); + await expect( + prepareNativeLocalSwitch({ + adapter: f.adapter, + previous, + provider: selected.provider, + agentName: "hermes", + gatewayName: selected.gatewayName, + sandboxName: selected.sandboxName, + binding: null, + entry: { + provider: selected.provider, + credentialEnv, + endpointUrl: ownedProxy ? "http://127.0.0.1:8000/v1" : selected.endpointUrl, + }, + credentialEnv, + inferenceApi: selected.api, + resolveCredentialValue: () => "fixture-secret", + readAuthority: f.input.readAuthority, + writeAuthority: f.input.writeAuthority, + }), + ).rejects.toThrow("conflicts"); + expect(resolve.mock.calls[0][1]).toEqual({ + baseUrl: selected.endpointUrl, + credentialEnv, + token: "fixture-secret", + providerType: "openai", + }); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + }, +); + +it("does not reuse a compatible endpoint binding when switching to vLLM", async () => { + const f = fixture(); + const previousBinding = { + ...binding, + provider: "compatible-endpoint" as const, + agent: "hermes" as const, + authMode: "authenticated" as const, + }; + const previous = { + ...previousBinding, + ...nativeLocalIdentity(previousBinding, 1), + schemaVersion: 1 as const, + providerId: "old-instance", + }; + const resolve = vi.spyOn(localRuntime, "resolveHermesLocalSwitchRuntime").mockReturnValue({ + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialValue: "dummy", + authMode: "sentinel", + ownedProxy: false, + }); + f.adapter.importProviderProfile.mockResolvedValue({ + ok: false, + error: { kind: "command", reason: "profile_incompatible", message: "collision" }, + }); + await expect( + prepareNativeLocalSwitch({ + adapter: f.adapter, + previous, + provider: "vllm-local", + agentName: "hermes", + gatewayName: binding.gatewayName, + sandboxName: binding.sandboxName, + binding: null, + credentialEnv: "COMPATIBLE_API_KEY", + resolveCredentialValue: () => "fixture-secret", + readAuthority: f.input.readAuthority, + writeAuthority: f.input.writeAuthority, + }), + ).rejects.toThrow("conflicts"); + expect(resolve).toHaveBeenCalledWith("vllm-local", null, expect.any(Function), undefined); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); +}); diff --git a/src/lib/inference/native-local/agent-config.test.ts b/src/lib/inference/native-local/agent-config.test.ts new file mode 100644 index 00000000000..978ee28a9f0 --- /dev/null +++ b/src/lib/inference/native-local/agent-config.test.ts @@ -0,0 +1,69 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { nativeLocalCredentialReference } from "./agent-config"; +import { nativeLocalCredentialReference as hostCredentialReference } from "./model-router"; +import * as blueprint from "../../core/model-router-port"; + +const endpoint = "http://host.openshell.internal:43123/team/v1"; +const reference = "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN"; +const handle = "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; + +describe("native local image credential projection", () => { + it.each([handle, `openshell:resolve:env:s${"a".repeat(64)}_NEMOCLAW_LOCAL_INFERENCE_TOKEN`])( + "projects an issued handle for a configured Router without reading the host blueprint: %s", + (issued) => { + const loader = vi.spyOn(blueprint, "loadBlueprintProfile").mockImplementation(() => { + throw new Error("The host blueprint is unavailable in an image"); + }); + expect(nativeLocalCredentialReference("team-router", endpoint, undefined, issued)).toBe( + reference, + ); + expect(loader).not.toHaveBeenCalled(); + }, + ); + + it.each([ + undefined, + "", + "raw-key", + reference, + "openshell:resolve:env:v7_NVIDIA_INFERENCE_API_KEY", + "openshell:resolve:env:sabc_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + `${handle}\n`, + `raw\n${handle}`, + `${handle}\nraw`, + ])("does not project a Router credential from an invalid handle: %s", (issued) => { + expect(nativeLocalCredentialReference("team-router", endpoint, undefined, issued)).toBeNull(); + }); + + it.each([ + "http://host.docker.internal:43123/team/v1", + "http://10.1.2.3:43123/team/v1", + "http://127.0.0.1:43123/team/v1", + "http://169.254.169.254:43123/team/v1", + "https://host.openshell.internal:43123/team/v1", + "http://host.openshell.internal:80/team/v1", + `${endpoint}?token=hidden`, + `${endpoint}#fragment`, + "http://user:pass@host.openshell.internal:43123/team/v1", + `${endpoint}%2fadmin`, + ])("does not use an issued Router handle at another endpoint shape: %s", (url) => { + expect(nativeLocalCredentialReference("team-router", url, undefined, handle)).toBeNull(); + }); + + it("keeps host Router admission bound to its configured name, port, and path", () => { + vi.stubEnv("NEMOCLAW_LOCAL_INFERENCE_TOKEN", handle); + vi.spyOn(blueprint, "loadBlueprintProfile").mockReturnValue({ + provider_name: "team-router", + endpoint: "http://localhost:43123/team/v1", + model: "selected", + router: { enabled: true, port: 43123 }, + }); + expect(hostCredentialReference("team-router", endpoint)).toBe(reference); + expect(hostCredentialReference("other-router", endpoint)).toBeNull(); + expect(hostCredentialReference("team-router", endpoint.replace("43123", "43124"))).toBeNull(); + expect(hostCredentialReference("team-router", endpoint.replace("team/v1", "admin"))).toBeNull(); + }); +}); diff --git a/src/lib/inference/native-local/agent-config.ts b/src/lib/inference/native-local/agent-config.ts index f97a139443c..7fdddb58166 100644 --- a/src/lib/inference/native-local/agent-config.ts +++ b/src/lib/inference/native-local/agent-config.ts @@ -44,18 +44,27 @@ export function nativeLocalCredentialReference( provider: string | null | undefined, endpointUrl: string, agentName?: string | null, + runtimeCredentialHandle?: string, ): string | null { + const scoped = new RegExp(NATIVE_LOCAL_RUNTIME_REFERENCE_PATTERN).exec( + runtimeCredentialHandle ?? "", + ); + // The host owns Router admission. Images only project its issued credential + // for the canonical managed alias; they cannot read the host blueprint. + const issuedRouterCredential = + typeof provider === "string" && + /^[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}$/.test(provider) && + scoped !== null && + scoped[0] === runtimeCredentialHandle && + qualifiedNativeLocalEndpointUrl(endpointUrl) !== null && + new URL(endpointUrl).hostname === "host.openshell.internal"; if ( - ![ - "ollama-local", - "vllm-local", - "llama-cpp-local", - "compatible-endpoint", - "nvidia-router", - ].includes(provider ?? "") + !issuedRouterCredential && + !["ollama-local", "vllm-local", "llama-cpp-local", "compatible-endpoint"].includes( + provider ?? "", + ) ) return null; - if (provider === "nvidia-router" && agentName !== "openclaw") return null; const qualified = qualifiedNativeLocalEndpointUrl(endpointUrl); // Canonical custom-hosted selection includes private IP endpoints. Dedicated // local providers still use private bridge IPs; compatible local routes use diff --git a/src/lib/inference/native-local/attachment.ts b/src/lib/inference/native-local/attachment.ts index b1988e54807..cea9ee545c0 100644 --- a/src/lib/inference/native-local/attachment.ts +++ b/src/lib/inference/native-local/attachment.ts @@ -1,8 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { nativeModelRouterEndpoint } from "./model-router"; import { createHash } from "node:crypto"; import { isLocalInferenceProvider, qualifiedNativeLocalEndpointUrl } from "./agent-config"; +import { usesNativeLocalInference } from "./eligibility"; import { isValidName } from "../../../../nemoclaw/dist/shared/sandbox-name.cjs"; import type { NativeProviderAttachment } from "../native-provider/lifecycle"; @@ -10,8 +12,8 @@ export type NativeLocalProvider = | "ollama-local" | "vllm-local" | "llama-cpp-local" - | "nvidia-router" - | "compatible-endpoint"; + | "compatible-endpoint" + | (string & {}); export type NativeLocalBinding = Readonly<{ provider: NativeLocalProvider; /** Tagged agents retain distinct immutable profile identities. */ @@ -19,6 +21,8 @@ export type NativeLocalBinding = Readonly<{ endpointUrl: string; credentialEnv: string; authMode: "authenticated" | "sentinel"; + /** Compatible OpenClaw and Hermes endpoints can select the Responses API. */ + api?: "openai-responses"; gatewayName: string; sandboxName: string; /** Existing runtime publication transaction, when that owner requires one. */ @@ -32,16 +36,18 @@ export function normalizeNativeLocalBinding(value: unknown): NativeLocalBinding const input = value as Record; if ( (!isLocalInferenceProvider(input.provider as string) && - input.provider !== "nvidia-router" && - input.provider !== "compatible-endpoint") || + input.provider !== "compatible-endpoint" && + !nativeModelRouterEndpoint(input.provider as string, input.endpointUrl as string)) || (input.agent !== undefined && input.agent !== "openclaw" && input.agent !== "pi" && input.agent !== "hermes") || (input.agent === "openclaw" && - input.provider !== "nvidia-router" && - !isLocalInferenceProvider(input.provider as string)) || - (input.provider === "nvidia-router" && input.agent !== "openclaw") || + !usesNativeLocalInference(input.provider as string, input.endpointUrl as string)) || + (input.api !== undefined && + (input.api !== "openai-responses" || + (input.agent !== "openclaw" && input.agent !== "hermes") || + input.provider !== "compatible-endpoint")) || typeof input.endpointUrl !== "string" || typeof input.credentialEnv !== "string" || !/^[A-Z][A-Z0-9_]*$/.test(input.credentialEnv) || @@ -62,6 +68,7 @@ export function normalizeNativeLocalBinding(value: unknown): NativeLocalBinding endpointUrl, credentialEnv: input.credentialEnv, authMode: input.authMode, + ...(input.api === "openai-responses" ? { api: input.api } : {}), gatewayName: input.gatewayName, sandboxName: input.sandboxName, ...(input.agent === "hermes" ? { agent: "hermes" as const } : {}), @@ -81,7 +88,7 @@ export function nativeLocalIdentity(binding: NativeLocalBinding, policyVersion: normalized.endpointUrl, normalized.credentialEnv, normalized.authMode, - // Literal private IPv4 endpoints retain their existing immutable boundary. + ...(normalized.api ? [normalized.api] : []), ...(policyVersion === 2 && new URL(normalized.endpointUrl).hostname.startsWith("host.") ? ["private-dual-stack-v1"] : []), diff --git a/src/lib/inference/native-local/contract.ts b/src/lib/inference/native-local/contract.ts index 5091913fff8..b3cef69f784 100644 --- a/src/lib/inference/native-local/contract.ts +++ b/src/lib/inference/native-local/contract.ts @@ -7,15 +7,12 @@ import { } from "./attachment"; export { + gatewayReachableCompatibleEndpointUrl, usesNativeLocalInference, usesNativeLocalInferenceForAgent, isPersistedPiSharedLocalInferenceRoute, } from "./eligibility"; -export { - NATIVE_LOCAL_CREDENTIAL_ENV, - isLocalInferenceProvider, - nativeLocalCredentialReference, -} from "./agent-config"; +export { NATIVE_LOCAL_CREDENTIAL_ENV, isLocalInferenceProvider } from "./agent-config"; export { nativeLocalIdentity, normalizeNativeLocalBinding, @@ -24,6 +21,7 @@ export { type NativeLocalBinding, type NativeLocalProviderAttachment, } from "./attachment"; +export { nativeLocalCredentialReference } from "./model-router"; export function normalizeNativeLocalProviderAuthorities( value: unknown, diff --git a/src/lib/inference/native-local/eligibility.ts b/src/lib/inference/native-local/eligibility.ts index cba43086558..3d140836bf9 100644 --- a/src/lib/inference/native-local/eligibility.ts +++ b/src/lib/inference/native-local/eligibility.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { nativeModelRouterEndpoint } from "./model-router"; import { OLLAMA_PROXY_PORT } from "../../core/ollama-proxy-port"; import { VLLM_PORT } from "../../core/vllm-port"; import { LLAMA_CPP_PORT } from "../llama-cpp/contract"; @@ -13,10 +14,16 @@ export function usesNativeLocalInference( provider: string | null | undefined, endpointUrl?: string | null, ): boolean { - if (isLocalInferenceProvider(provider)) return true; + if (isLocalInferenceProvider(provider) || nativeModelRouterEndpoint(provider, endpointUrl)) + return true; if (provider !== "compatible-endpoint" || !endpointUrl) return false; try { const endpoint = new URL(endpointUrl); + if ( + endpoint.hostname === "host.openshell.internal" && + qualifiedNativeLocalEndpointUrl(endpointUrl) + ) + return true; return ( endpoint.protocol === "http:" && [11434, OLLAMA_PROXY_PORT, VLLM_PORT].includes(Number(endpoint.port)) && @@ -38,10 +45,8 @@ export function usesNativeLocalInferenceForAgent( agentName: string | null | undefined, provider: string | null | undefined, endpointUrl?: string | null, - localNimSelected = false, + _localNimSelected = false, ): boolean { - if (provider === "nvidia-router") return agentName === "openclaw"; - if (provider === "vllm-local" && localNimSelected) return false; if ( agentName === "pi" && provider === "compatible-endpoint" && @@ -52,7 +57,7 @@ export function usesNativeLocalInferenceForAgent( return ( ((agentName === "langchain-deepagents-code" || agentName === "pi" || agentName === "hermes") && usesNativeLocalInference(provider, endpointUrl)) || - (agentName === "openclaw" && isLocalInferenceProvider(provider)) + (agentName === "openclaw" && usesNativeLocalInference(provider, endpointUrl)) ); } @@ -77,6 +82,7 @@ export function isPersistedPiSharedLocalInferenceRoute( typeof recorded.createdAt === "string" && recorded.pendingRouteReservation !== true && recorded.nativeLocalProviderAttachment === undefined && + !nativeModelRouterEndpoint(recorded.provider, recorded.endpointUrl) && usesNativeLocalInferenceForAgent("pi", recorded.provider, recorded.endpointUrl) && (!selected || (recorded.provider === selected.provider && @@ -97,6 +103,8 @@ export function gatewayReachableCompatibleEndpointUrl( provider: string, endpointUrl: string | null | undefined, ): string | null | undefined { + const routerEndpoint = nativeModelRouterEndpoint(provider, endpointUrl); + if (routerEndpoint) return routerEndpoint; if (!LOOPBACK_BRIDGE_PROVIDERS.has(provider) || !endpointUrl) { return endpointUrl; } diff --git a/src/lib/inference/native-local/model-router.ts b/src/lib/inference/native-local/model-router.ts new file mode 100644 index 00000000000..cd19a562bf1 --- /dev/null +++ b/src/lib/inference/native-local/model-router.ts @@ -0,0 +1,88 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { loadBlueprintProfile, type BlueprintInferenceProfile } from "../../core/model-router-port"; +import { + nativeLocalCredentialReference as agentCredentialReference, + qualifiedNativeLocalEndpointUrl, + NATIVE_LOCAL_CREDENTIAL_ENV, +} from "./agent-config"; +import { unsafeEndpointUrlViolation } from "../../core/endpoint-url-safety"; + +/** Recognize the existing provider even when its endpoint needs repair. */ +export function isModelRouterProvider(provider: string | null | undefined): boolean { + return ( + provider === "nvidia-router" || + Boolean(provider && provider === loadBlueprintProfile("routed")?.provider_name) + ); +} + +/** The existing routed blueprint owns the provider name, host port, and API path. */ +export function nativeModelRouterEndpoint( + provider: string | null | undefined, + endpointUrl?: string | null, + profile: BlueprintInferenceProfile | null = loadBlueprintProfile("routed"), +): string | null { + if ( + !profile || + profile.router?.enabled !== true || + (provider !== "nvidia-router" && provider !== profile.provider_name) || + typeof provider !== "string" || + !/^[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}$/.test(provider) + ) + return null; + const port = profile.router.port || 4000; + const selected = endpointUrl ?? profile.endpoint; + if ( + typeof selected !== "string" || + !selected || + typeof profile.endpoint !== "string" || + !profile.endpoint || + unsafeEndpointUrlViolation(selected) + ) + return null; + try { + const endpoint = new URL(selected); + const configured = new URL(profile.endpoint); + const localHosts = ["localhost", "127.0.0.1", "[::1]", "host.openshell.internal"]; + if ( + !Number.isInteger(port) || + port < 1024 || + port > 65535 || + endpoint.protocol !== "http:" || + configured.protocol !== "http:" || + !localHosts.includes(endpoint.hostname) || + !localHosts.includes(configured.hostname) || + endpoint.username || + endpoint.password || + endpoint.search || + endpoint.hash || + configured.username || + configured.password || + configured.search || + configured.hash || + Number(endpoint.port) !== port || + Number(configured.port) !== port || + endpoint.pathname.replace(/\/$/, "") !== configured.pathname.replace(/\/$/, "") || + !/^\/(?:[a-zA-Z0-9_-]+\/)*[a-zA-Z0-9_-]+\/?$/.test(endpoint.pathname) + ) + return null; + return `http://host.openshell.internal:${port}${endpoint.pathname.replace(/\/$/, "")}`; + } catch { + return null; + } +} + +/** Host configuration keeps Router credentials bound to the selected blueprint. */ +export function nativeLocalCredentialReference( + provider: string | null | undefined, + endpointUrl: string, + agentName?: string | null, +): string | null { + if ( + nativeModelRouterEndpoint(provider, endpointUrl) && + qualifiedNativeLocalEndpointUrl(endpointUrl) + ) + return `openshell:resolve:env:${NATIVE_LOCAL_CREDENTIAL_ENV}`; + return agentCredentialReference(provider, endpointUrl, agentName); +} diff --git a/src/lib/inference/native-local/profile.ts b/src/lib/inference/native-local/profile.ts index 28020fe3952..6f9cd7d480f 100644 --- a/src/lib/inference/native-local/profile.ts +++ b/src/lib/inference/native-local/profile.ts @@ -30,6 +30,7 @@ import { } from "./contract"; export { normalizeNativeLocalProviderAttachment, nativeLocalIdentity }; +export { NativeProviderCredentialExpiredError } from "../native-provider/lifecycle"; export type { NativeLocalProviderAttachment }; export function prepareNativeLocalProfile(input: NativeLocalBinding) { @@ -75,7 +76,12 @@ export function prepareNativeLocalProfile(input: NativeLocalBinding) { : ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", ...(dualStack ? ["fc00::/7"] : [])], rules: [ { allow: { method: "GET", path: `${endpoint.pathname}/models` } }, - { allow: { method: "POST", path: `${endpoint.pathname}/chat/completions` } }, + { + allow: { + method: "POST", + path: `${endpoint.pathname}/${binding.api === "openai-responses" ? "responses" : "chat/completions"}`, + }, + }, ], }, ], @@ -193,7 +199,11 @@ export async function verifyRecordedNativeLocalProviderAttachment(input: { (input.gatewayName !== undefined && expected.gatewayName !== input.gatewayName) ) throw new Error("Native local provider authority does not match the rebuild target."); - return verifyNativeLocalProviderAttachment({ ...input, expected }); + return verifyNativeLocalProviderAttachment({ + ...input, + expected, + requireUnexpiredCredential: true, + }); } export async function ensureNativeLocalProviderAttached(input: AttachmentInput) { diff --git a/src/lib/inference/native-local/selection.ts b/src/lib/inference/native-local/selection.ts index b248129c804..8924e6d75ca 100644 --- a/src/lib/inference/native-local/selection.ts +++ b/src/lib/inference/native-local/selection.ts @@ -2,7 +2,6 @@ // SPDX-License-Identifier: Apache-2.0 import { OLLAMA_PROXY_PORT } from "../../core/ollama-proxy-port"; -import { resolveConfiguredModelRouterPort } from "../../core/model-router-port"; import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; import { isProtectedLocalInferencePort } from "../gateway/local-endpoint-ports"; import { @@ -26,8 +25,6 @@ export async function prepareNativeLocalSelection(input: { binding: Omit; credentialValue: string | null; ownedProxy?: boolean; - /** Supplied only after the managed router has reconciled under its port lock. */ - ownedModelRouterPort?: number; readAuthority?: typeof getNativeLocalProviderAuthority; writeAuthority?: typeof setNativeLocalProviderAuthority; }) { @@ -38,17 +35,7 @@ export async function prepareNativeLocalSelection(input: { input.ownedProxy === true && port === OLLAMA_PROXY_PORT && new URL(binding.endpointUrl).hostname === "host.openshell.internal"; - const ownedModelRouterPort = - binding.provider === "nvidia-router" && - input.ownedModelRouterPort === port && - port === resolveConfiguredModelRouterPort() && - new URL(binding.endpointUrl).hostname === "host.openshell.internal" - ? port - : undefined; - if ( - (binding.provider === "nvidia-router" && ownedModelRouterPort === undefined) || - isProtectedLocalInferencePort(port, { ownedProxy, ownedModelRouterPort }) - ) { + if (isProtectedLocalInferencePort(port, { ownedProxy })) { throw new Error("The selected local inference endpoint uses a protected host port."); } const readAuthority = input.readAuthority ?? getNativeLocalProviderAuthority; diff --git a/src/lib/inference/native-nvidia.test.ts b/src/lib/inference/native-nvidia.test.ts index e6bb5831ea6..b66e1b3120e 100644 --- a/src/lib/inference/native-nvidia.test.ts +++ b/src/lib/inference/native-nvidia.test.ts @@ -398,7 +398,9 @@ describe("native NVIDIA OpenShell provider", () => { providerId: "recorded-id", }, }), - ).rejects.toThrow(/is missing.*No provider was changed/u); + ).rejects.toThrow( + /is missing.*no sandboxes after a reinstall, complete uninstall without --keep-openshell.*No provider was changed/u, + ); expect(createProvider).not.toHaveBeenCalled(); }); diff --git a/src/lib/inference/native-provider/agent-config.test.ts b/src/lib/inference/native-provider/agent-config.test.ts index f8799da94cc..4b3e9d43872 100644 --- a/src/lib/inference/native-provider/agent-config.test.ts +++ b/src/lib/inference/native-provider/agent-config.test.ts @@ -1,7 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it, vi, onTestFinished } from "vitest"; import { nativeHostedAgentConfig } from "./agent-config"; import { HOSTED_NATIVE_PROVIDERS } from "./hosted"; @@ -43,3 +47,66 @@ it.each(HOSTED_NATIVE_PROVIDERS)( ).toBeUndefined(); }, ); + +it.each([ + "https://user:secret@example.com/v1", + "https://example.com/v1?token=secret", + "https://example.com/v1#fragment", + "https://example.com/v1%2fextra", + "https://example.com/v1%5cextra", + "https://example.com/v1%0Aextra", + "https://example.com/\n/v1", + "file:///etc/passwd", +])("rejects malformed retained Hermes image configuration %s", (endpoint) => { + expect(() => nativeHostedAgentConfig("hermes-provider", endpoint)).toThrow(); +}); + +it.each(["http://host.openshell.internal:42103/v1", "https://retained.example/team/v1"])( + "serializes only a resolver handle for retained Hermes endpoint %s", + (endpoint) => { + vi.stubEnv("OPENAI_API_KEY", "host-secret-must-not-enter-config"); + expect(nativeHostedAgentConfig("hermes-provider", endpoint)).toEqual({ + credentialEnv: "OPENAI_API_KEY", + apiKey: "openshell:resolve:env:OPENAI_API_KEY", + }); + }, +); + +it("loads image provider configuration without host modules or compiled CLI dependencies", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "native-hosted-image-")); + onTestFinished(() => fs.rmSync(root, { recursive: true, force: true })); + const providers = path.join(root, "inference", "native-provider"); + fs.mkdirSync(providers, { recursive: true }); + fs.copyFileSync( + path.join(import.meta.dirname, "agent-config.ts"), + path.join(providers, "agent-config.ts"), + ); + fs.copyFileSync(path.join(import.meta.dirname, "catalog.ts"), path.join(providers, "catalog.ts")); + const core = path.join(root, "core"); + fs.mkdirSync(core); + fs.copyFileSync( + path.resolve(import.meta.dirname, "../../core/endpoint-url-safety.ts"), + path.join(core, "endpoint-url-safety.ts"), + ); + const result = spawnSync( + process.execPath, + [ + "--input-type=module", + "-e", + ` + import { nativeHostedAgentConfig } from ${JSON.stringify(new URL(`file://${providers}/agent-config.ts`).href)}; + console.log(JSON.stringify(nativeHostedAgentConfig("hermes-provider", "http://host.openshell.internal:42103/v1"))); + `, + ], + { + encoding: "utf8", + timeout: 10_000, + env: { PATH: process.env.PATH, OPENAI_API_KEY: "must-stay-private" }, + }, + ); + expect(result.status, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toEqual({ + credentialEnv: "OPENAI_API_KEY", + apiKey: "openshell:resolve:env:OPENAI_API_KEY", + }); +}); diff --git a/src/lib/inference/native-provider/agent-config.ts b/src/lib/inference/native-provider/agent-config.ts index 11cacb2039e..dd2fe3011a6 100644 --- a/src/lib/inference/native-provider/agent-config.ts +++ b/src/lib/inference/native-provider/agent-config.ts @@ -2,15 +2,18 @@ // SPDX-License-Identifier: Apache-2.0 import { - hostedNativeProvider, + fixedHostedNativeProvider, + recordedHermesEndpointUrl, customNativeCredentialEnv, HOSTED_NATIVE_PROVIDERS, isCustomHostedProvider, -} from "./hosted.ts"; +} from "./catalog.ts"; -export { hostedNativeProvider, HOSTED_NATIVE_PROVIDERS }; +export { fixedHostedNativeProvider, HOSTED_NATIVE_PROVIDERS }; -/** Public configuration only. Never read the host's provider credential here. */ +/** Public configuration only; this grants neither egress nor a provider attachment. + * Host admission validates destinations before OpenShell receives the real credential. + */ export function nativeHostedAgentConfig(provider: string | null | undefined, baseUrl: string) { if (isCustomHostedProvider(provider) && new URL(baseUrl).hostname !== "inference.local") { const endpoint = new URL(baseUrl); @@ -26,16 +29,13 @@ export function nativeHostedAgentConfig(provider: string | null | undefined, bas const credentialEnv = customNativeCredentialEnv(provider!); return { credentialEnv, apiKey: `openshell:resolve:env:${credentialEnv}` }; } - if (!hostedNativeProvider(provider)) return undefined; + const definition = fixedHostedNativeProvider(provider); + if (!definition) return undefined; // Existing shared-route selections keep their original configuration until explicitly changed. if (new URL(baseUrl).hostname === "inference.local") return undefined; - const definition = hostedNativeProvider( - provider, - provider === "hermes-provider" ? baseUrl : undefined, - ); - if (!definition) return undefined; - const endpoint = new URL(baseUrl); - const expected = new URL(definition.endpoint); + const endpoint = + provider === "hermes-provider" ? recordedHermesEndpointUrl(baseUrl) : new URL(baseUrl); + const expected = provider === "hermes-provider" ? endpoint : new URL(definition.endpoint); if ( endpoint.username || endpoint.password || diff --git a/src/lib/inference/native-provider/blueprint-resume.ts b/src/lib/inference/native-provider/blueprint-resume.ts new file mode 100644 index 00000000000..a57577b89ae --- /dev/null +++ b/src/lib/inference/native-provider/blueprint-resume.ts @@ -0,0 +1,132 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { randomUUID } from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { isDeepStrictEqual } from "node:util"; +import { + observeBlueprintCreatedInference, + publishBlueprintNativeConfiguration, + readBlueprintRunPlan, + type BlueprintNativeReceipt, +} from "./blueprint"; + +/** Resume a recorded configuration write without acquiring new resource authority. */ +export async function resumeBlueprintNativeInferenceRun( + input: Readonly<{ + runId: string; + gateway: Readonly<{ name: string; host: string; port: number }>; + sandboxName: string; + revalidate: () => Promise; + }>, + deps?: Parameters[2], +): Promise { + if (!/^[a-zA-Z0-9_-]+$/u.test(input.runId)) throw new Error("Invalid blueprint recovery run ID."); + if (typeof fs.constants.O_NOFOLLOW !== "number" || typeof fs.constants.O_DIRECTORY !== "number") + throw new Error("Blueprint recovery requires no-follow directory descriptors."); + const runs = path.join(os.homedir(), ".nemoclaw/state/runs"); + const directory = path.join(runs, input.runId); + const directoryFd = fs.openSync( + directory, + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | fs.constants.O_DIRECTORY, + ); + try { + const metadata = fs.fstatSync(directoryFd); + if (!metadata.isDirectory()) + throw new Error("Blueprint recovery requires its original run directory."); + const file = path.join(directory, "plan.json"); + const read = (): unknown => { + const current = fs.lstatSync(file); + if ( + !current.isFile() || + current.nlink !== 1 || + fs.existsSync(path.join(directory, "rolled_back")) + ) + throw new Error("Blueprint recovery requires an unchanged, unshared run receipt."); + return readBlueprintRunPlan(file, metadata); + }; + const source = read(); + if (!source || typeof source !== "object" || Array.isArray(source)) + throw new Error("Blueprint recovery requires a valid run receipt."); + const plan = source as Record; + const value = plan.native_inference; + if ( + plan.run_id !== input.runId || + plan.sandbox_name !== input.sandboxName || + !isDeepStrictEqual(plan.gateway, input.gateway) || + plan.sandbox_created_by_apply !== true || + plan.inference_provider_created_by_apply !== true || + !value || + typeof value !== "object" || + Array.isArray(value) + ) + throw new Error("Blueprint recovery receipt does not match this run's resource ownership."); + const receipt = value as BlueprintNativeReceipt; + if ( + receipt.gatewayName !== input.gateway.name || + receipt.sandboxName !== input.sandboxName || + !receipt.sandboxId || + !receipt.providerAttachment || + (receipt.configPublished !== undefined && receipt.configPublished !== true) + ) + throw new Error("Blueprint recovery requires recorded sandbox and provider identities."); + const requireSamePlan = (): void => { + if (!isDeepStrictEqual(read(), source)) + throw new Error( + "Blueprint recovery receipt changed; preserve the run for manual recovery.", + ); + }; + const revalidate = async (): Promise => { + requireSamePlan(); + await input.revalidate(); + requireSamePlan(); + }; + await revalidate(); + if (receipt.configPublished === true) { + await observeBlueprintCreatedInference(receipt, receipt, deps); + await revalidate(); + return; + } + await publishBlueprintNativeConfiguration(receipt, revalidate, deps); + await revalidate(); + const pending = path.join(directory, `.resume-${randomUUID()}.json`); + let created = false; + try { + const fd = fs.openSync(pending, "wx", 0o600); + created = true; + try { + fs.writeFileSync( + fd, + JSON.stringify( + { + ...plan, + native_inference: { ...receipt, configPublished: true }, + }, + null, + 2, + ), + ); + fs.fsyncSync(fd); + } finally { + fs.closeSync(fd); + } + requireSamePlan(); + fs.renameSync(pending, file); + created = false; + const currentDirectory = fs.lstatSync(directory); + if ( + !currentDirectory.isDirectory() || + currentDirectory.dev !== metadata.dev || + currentDirectory.ino !== metadata.ino + ) + throw new Error("Blueprint recovery run directory changed before it could be synced."); + fs.fsyncSync(directoryFd); + } finally { + if (created) fs.unlinkSync(pending); + } + } finally { + fs.closeSync(directoryFd); + } +} diff --git a/src/lib/inference/native-provider/blueprint.test.ts b/src/lib/inference/native-provider/blueprint.test.ts new file mode 100644 index 00000000000..18aa6d09900 --- /dev/null +++ b/src/lib/inference/native-provider/blueprint.test.ts @@ -0,0 +1,1269 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { execFileSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import YAML from "yaml"; +import { resumeBlueprintNativeInferenceRun } from "./blueprint-resume"; +import type { + OpenShellProviderAdapter, + OpenShellProviderMetadata, +} from "../../adapters/openshell/provider-adapter"; +import { exportedProviderProfileMatchesContract } from "../../adapters/openshell/provider-profile"; +import { + createBlueprintNativeInference, + findBlueprintNativeReceipt, + reconcileBlueprintNativeTimeout, + requireCreatedSandbox, + observeBlueprintCreatedInference, + type BlueprintNativeReceipt, +} from "./blueprint"; + +function fixture( + overrides: { + credential?: string; + providerName?: string; + expiry?: number; + collision?: boolean; + wrongSandbox?: boolean; + configFailure?: boolean; + } = {}, +) { + const providerName = overrides.providerName ?? "chosen-provider"; + let source = ""; + let provider: OpenShellProviderMetadata | null = null; + let nonce = ""; + let attached = false; + let sandboxId = "sandbox-id-1"; + const events: string[] = []; + const receipts: BlueprintNativeReceipt[] = []; + const adapter = { + importProviderProfile: vi.fn( + ({ profilePath }) => { + source = fs.readFileSync(profilePath, "utf8"); + return { ok: true }; + }, + ), + getProvider: vi.fn(async () => { + return overrides.collision + ? { + ok: true, + value: { + name: providerName, + type: "operator-profile", + credentialKeys: [], + configKeys: [], + revision: { id: "operator-id", resourceVersion: 1 }, + }, + } + : provider + ? { ok: true, value: provider } + : { ok: false, error: { kind: "command", reason: "not_found", message: "not found" } }; + }), + createProvider: vi.fn(async (request) => { + events.push("create-provider"); + provider = { + name: request.name, + type: request.type, + configKeys: [], + credentialKeys: request.credentials.map((credential) => credential.name), + revision: { id: "provider-id-1", resourceVersion: 1 }, + credentialExpiresAtMs: { OPENAI_API_KEY: overrides.expiry ?? 0 }, + }; + return { ok: true }; + }), + inspectProviderProfile: vi.fn( + async ({ expectedProfile }) => + expectedProfile && + exportedProviderProfileMatchesContract(JSON.stringify(YAML.parse(source)), expectedProfile) + ? { ok: true, value: { credentialKeys: [] } } + : { + ok: false, + error: { + kind: "command", + reason: "profile_incompatible", + message: "profile mismatch", + }, + }, + ), + listProviderAttachments: vi.fn(async () => ({ + ok: true, + value: { names: attached ? [providerName] : [] }, + })), + updateProvider: vi.fn(), + deleteProvider: vi.fn(), + detachProvider: vi.fn(), + attachProvider: vi.fn(async () => { + attached = true; + events.push("attach-provider"); + return { ok: true }; + }), + } as unknown as OpenShellProviderAdapter; + const writeConfig = vi.fn< + NonNullable[1]>["writeConfig"] + >( + overrides.configFailure + ? () => { + events.push("write-config"); + throw new Error("configuration failed"); + } + : () => { + events.push("write-config"); + }, + ); + const deps = { + adapter: vi.fn(() => adapter), + capture: vi.fn(() => + JSON.stringify({ + sandboxes: [ + { + id: sandboxId, + name: "chosen-sandbox", + labels: { + "ai.nvidia.nemoclaw.create-attempt": overrides.wrongSandbox ? "b".repeat(62) : nonce, + }, + resource_version: 1, + created_at: "2026-01-01T00:00:00Z", + phase: "Ready", + current_policy_version: 1, + }, + ], + next_page_token: "", + }), + ), + writeConfig, + readHandle: vi.fn(async () => "openshell:resolve:env:v1_OPENAI_API_KEY"), + }; + const input = { + gatewayName: "chosen-gateway", + sandboxName: "chosen-sandbox", + provider: providerName, + model: "custom-model", + api: "openai-completions", + endpointUrl: "https://8.8.8.8:8000/v1", + prepareSandboxPolicy: vi.fn(async () => { + events.push("verify-policy"); + }), + credentialValue: overrides.credential ?? "test-host-secret", + timeoutSeconds: 180, + persist: vi.fn((receipt: BlueprintNativeReceipt) => { + receipts.push(receipt); + events.push( + receipt.sandboxId + ? "record-sandbox" + : receipt.providerAttachment + ? "record-provider" + : "record-intent", + ); + }), + createSandbox: vi.fn(async (_providerName: string, createAttemptNonce: string) => { + nonce = createAttemptNonce; + events.push("create-sandbox"); + }), + }; + return { + input, + deps, + adapter, + receipts, + events, + source: () => source, + replaceProvider: () => { + provider = { ...provider!, revision: { id: "replacement", resourceVersion: 1 } }; + }, + replaceSandbox: () => { + sandboxId = "replacement-id"; + }, + }; +} + +describe("blueprint native creation", () => { + it.each(["Provider_1.prod", `a${"b".repeat(127)}`])( + "creates and verifies the supported provider name %s", + async (providerName) => { + const f = fixture({ providerName }); + const created = await createBlueprintNativeInference(f.input, f.deps); + expect(f.adapter.createProvider).toHaveBeenCalledExactlyOnceWith( + expect.objectContaining({ name: providerName, fromExisting: false }), + ); + expect(f.receipts.at(-1)).toMatchObject({ + provider: providerName, + configPublished: true, + providerAttachment: { providerName, providerId: "provider-id-1" }, + }); + await expect(created.observe()).resolves.toBe(created.identity); + }, + ); + + it.each(["1provider", "--help", "$(id)", "provider\n::error::forged", `a${"b".repeat(128)}`])( + "rejects malformed provider name %j before host operations", + async (providerName) => { + const f = fixture({ providerName }); + const createDirectory = vi.spyOn(fs, "mkdtempSync"); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow( + "Invalid blueprint native inference selection.", + ); + expect(createDirectory).not.toHaveBeenCalled(); + expect(f.deps.adapter).not.toHaveBeenCalled(); + expect(f.deps.capture).not.toHaveBeenCalled(); + expect(f.deps.readHandle).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(f.input.persist).not.toHaveBeenCalled(); + expect(f.input.createSandbox).not.toHaveBeenCalled(); + expect(f.input.prepareSandboxPolicy).not.toHaveBeenCalled(); + }, + ); + + it("records immutable ownership before later effects and preserves the requested provider, model and timeout", async () => { + const f = fixture(); + const created = await createBlueprintNativeInference(f.input, f.deps); + expect(f.events).toEqual([ + "record-intent", + "create-sandbox", + "record-sandbox", + "verify-policy", + "create-provider", + "record-sandbox", + "attach-provider", + "write-config", + "record-sandbox", + ]); + expect(f.input.createSandbox).toHaveBeenCalledWith( + "chosen-provider", + expect.stringMatching(/^[a-f0-9]{62}$/u), + ); + expect(f.adapter.createProvider).toHaveBeenCalledWith( + expect.objectContaining({ + name: "chosen-provider", + credentials: [{ name: "OPENAI_API_KEY", value: "test-host-secret" }], + fromExisting: false, + }), + ); + expect(f.deps.writeConfig).toHaveBeenCalledWith( + "chosen-sandbox", + [ + { + dotpath: "models.providers.inference", + value: { + baseUrl: "https://8.8.8.8:8000/v1", + api: "openai-completions", + apiKey: "openshell:resolve:env:v1_OPENAI_API_KEY", + timeoutSeconds: 180, + models: [{ id: "custom-model", name: "custom-model" }], + }, + }, + { dotpath: "agents.defaults.model.primary", value: "inference/custom-model" }, + ], + "chosen-gateway", + ); + expect( + JSON.stringify(f.receipts) + f.source() + JSON.stringify(f.deps.capture.mock.calls), + ).not.toContain("test-host-secret"); + expect(f.receipts.at(-1)).toMatchObject({ + sandboxId: "sandbox-id-1", + configPublished: true, + providerAttachment: { providerId: "provider-id-1", providerName: "chosen-provider" }, + }); + await expect(created.observe()).resolves.toBe(created.identity); + }); + + it("creates a credential-free HTTP profile without reading or inheriting another credential", async () => { + const f = fixture({ credential: "" }); + f.input.endpointUrl = "http://8.8.8.8:8000/v1"; + const created = await createBlueprintNativeInference(f.input, f.deps); + await expect(created.observe()).resolves.toBe(created.identity); + expect(YAML.parse(f.source()).credentials).toEqual([]); + expect(f.adapter.createProvider).toHaveBeenCalledWith( + expect.objectContaining({ credentials: [], fromExisting: false }), + ); + expect(f.deps.readHandle).not.toHaveBeenCalled(); + expect(f.deps.writeConfig.mock.calls[0]?.[1]).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + value: expect.objectContaining({ apiKey: "unused", authHeader: false }), + }), + ]), + ); + }); + + it.each(["openai-completions", "anthropic-messages"])( + "rejects credential-bearing HTTP for %s before host operations", + async (api) => { + const f = fixture(); + const createDirectory = vi.spyOn(fs, "mkdtempSync"); + await expect( + createBlueprintNativeInference( + { ...f.input, api, endpointUrl: "http://8.8.8.8:8000/v1" }, + f.deps, + ), + ).rejects.toThrow("Blueprint inference endpoints with credentials require HTTPS."); + expect(createDirectory).not.toHaveBeenCalled(); + expect(f.deps.adapter).not.toHaveBeenCalled(); + expect(f.deps.capture).not.toHaveBeenCalled(); + expect(f.deps.readHandle).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(f.input.persist).not.toHaveBeenCalled(); + expect(f.input.createSandbox).not.toHaveBeenCalled(); + expect(f.input.prepareSandboxPolicy).not.toHaveBeenCalled(); + }, + ); + + it("retains the Anthropic credential boundary and paths", async () => { + const f = fixture(); + await createBlueprintNativeInference( + { ...f.input, api: "anthropic-messages", endpointUrl: "https://8.8.8.8/v1" }, + f.deps, + ); + expect(YAML.parse(f.source())).toMatchObject({ + credentials: [ + { env_vars: ["OPENAI_API_KEY"], auth_style: "header", header_name: "x-api-key" }, + ], + endpoints: [ + { + host: "8.8.8.8", + port: 443, + allowed_ips: ["8.8.8.8"], + rules: [ + { allow: { method: "GET", path: "/v1/models" } }, + { allow: { method: "POST", path: "/v1/messages" } }, + { allow: { method: "POST", path: "/v1/messages/count_tokens" } }, + ], + }, + ], + }); + }); + + it.each([ + "http://127.0.0.1:8000/v1", + "http://10.1.2.3/v1", + "https://example.com/v1", + "https://user:secret@8.8.8.8/v1", + "http://8.8.8.8/v1?key=secret", + "http://8.8.8.8/%2e%2e/v1", + ])( + "refuses an endpoint outside the existing admission boundary before mutation: %s", + async (endpointUrl) => { + const f = fixture(); + await expect( + createBlueprintNativeInference({ ...f.input, endpointUrl }, f.deps), + ).rejects.toThrow(); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + expect(f.input.createSandbox).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + }, + ); + + it.each([{ collision: true }, { wrongSandbox: true }, { expiry: 1 }])( + "rejects conflicting or unusable authority before configuring the agent: %j", + async (options) => { + const f = fixture(options); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + }, + ); + + it.each(["", "test-host-secret"])( + "does not adopt an unrecorded same-profile provider: %j", + async (credential) => { + const f = fixture({ credential }); + vi.mocked(f.adapter.getProvider).mockImplementation(async () => ({ + ok: true, + value: { + name: "chosen-provider", + type: YAML.parse(f.source()).id, + configKeys: [], + credentialKeys: credential ? ["OPENAI_API_KEY"] : [], + revision: { id: "operator-id", resourceVersion: 1 }, + }, + })); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow( + "without a matching", + ); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }, + ); + + it("refuses credential-required reuse of an anonymous native provider", async () => { + const f = fixture({ credential: "" }); + await createBlueprintNativeInference(f.input, f.deps); + await expect( + observeBlueprintCreatedInference( + { ...f.input, requiresCredential: true }, + f.receipts.at(-1)!, + f.deps, + ), + ).rejects.toThrow("does not match"); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + }); + + it("rejects unexpected credentials on a recorded anonymous provider", async () => { + const f = fixture({ credential: "" }); + const created = await createBlueprintNativeInference(f.input, f.deps); + vi.mocked(f.adapter.getProvider).mockResolvedValue({ + ok: true, + value: { + name: "chosen-provider", + type: YAML.parse(f.source()).id, + configKeys: [], + credentialKeys: ["OPENAI_API_KEY"], + revision: { id: "provider-id-1", resourceVersion: 1 }, + }, + }); + await expect(created.observe()).rejects.toThrow("does not match"); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + }); + + it("does not expose inference credentials after policy refusal", async () => { + const f = fixture(); + f.input.prepareSandboxPolicy.mockRejectedValueOnce(new Error("policy refused")); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow("policy refused"); + expect(f.receipts.at(-1)?.sandboxId).toBe("sandbox-id-1"); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.deps.readHandle).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + }); + + it("stops before provider creation when the sandbox changes during policy verification", async () => { + const f = fixture(); + f.input.prepareSandboxPolicy.mockImplementationOnce(async () => { + f.replaceSandbox(); + }); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow( + "identity changed", + ); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + }); + + it("stops before attachment when the sandbox changes after provider creation", async () => { + const f = fixture(); + f.input.persist + .mockImplementationOnce(() => {}) + .mockImplementationOnce(() => {}) + .mockImplementationOnce(() => { + f.replaceSandbox(); + }); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow( + "identity changed", + ); + expect(f.adapter.createProvider).toHaveBeenCalledTimes(1); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + }); + + it("refuses configuration when the sandbox changes as credential verification returns", async () => { + const f = fixture(); + const capture = f.deps.capture.getMockImplementation()!; + let replaced = false; + f.deps.readHandle.mockImplementationOnce(async () => { + // Verification observes the sandbox before and after its provider checks. + f.deps.capture.mockImplementationOnce(capture).mockImplementationOnce(() => { + const result = capture(); + queueMicrotask(() => { + f.replaceSandbox(); + replaced = true; + }); + return result; + }); + return "openshell:resolve:env:v1_OPENAI_API_KEY"; + }); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow( + "identity changed", + ); + expect(replaced).toBe(true); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(f.receipts.at(-1)).toMatchObject({ sandboxId: "sandbox-id-1" }); + expect(f.receipts.at(-1)?.configPublished).toBeUndefined(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + }); + + it("retains resources and refuses retry after configuration fails without a timeout", async () => { + const f = fixture({ configFailure: true }); + const input = { ...f.input, timeoutSeconds: undefined }; + await expect(createBlueprintNativeInference(input, f.deps)).rejects.toThrow( + "configuration failed", + ); + const receipt = f.receipts.at(-1)!; + expect(receipt.sandboxId).toBe("sandbox-id-1"); + expect(receipt.configPublished).toBeUndefined(); + const home = fs.mkdtempSync(path.join(os.tmpdir(), "blueprint-recovery-")); + vi.spyOn(os, "homedir").mockReturnValue(home); + const run = path.join(home, ".nemoclaw/state/runs/nc-failed"); + try { + fs.mkdirSync(run, { recursive: true }); + fs.writeFileSync(path.join(run, "plan.json"), JSON.stringify({ native_inference: receipt })); + expect(() => findBlueprintNativeReceipt(input)).toThrow("incomplete"); + await expect(observeBlueprintCreatedInference(input, receipt, f.deps)).rejects.toThrow( + "publication is incomplete", + ); + } finally { + fs.rmSync(home, { recursive: true, force: true }); + } + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).toHaveBeenCalledTimes(1); + }); + + it("refuses later reuse after replacement or a changed requested model", async () => { + const f = fixture(); + const created = await createBlueprintNativeInference(f.input, f.deps); + await expect( + observeBlueprintCreatedInference( + { ...f.input, model: "other-model" }, + f.receipts.at(-1)!, + f.deps, + ), + ).rejects.toThrow("does not match"); + f.replaceSandbox(); + await expect(created.observe()).rejects.toThrow("identity changed"); + expect(f.deps.writeConfig).toHaveBeenCalledTimes(1); + }); +}); + +async function recoveryFixture() { + const f = fixture(); + await createBlueprintNativeInference(f.input, f.deps); + const home = fs.mkdtempSync(path.join(os.tmpdir(), "blueprint-recovery-")); + vi.spyOn(os, "homedir").mockReturnValue(home); + const directory = path.join(home, ".nemoclaw/state/runs/nc-current"); + fs.mkdirSync(directory, { recursive: true }); + const file = path.join(directory, "plan.json"); + const plan = JSON.stringify({ native_inference: f.receipts.at(-1) }); + fs.writeFileSync(file, plan); + return { ...f, home, directory, file, plan }; +} + +describe("blueprint run receipt recovery", () => { + it("rejects a saved HTTP credential receipt before reuse or timeout reconciliation", async () => { + const f = await recoveryFixture(); + try { + const endpointUrl = "http://8.8.8.8:8000/v1"; + const receipt = { ...f.receipts.at(-1)!, endpointUrl }; + const requested = { ...f.input, endpointUrl }; + fs.writeFileSync(f.file, JSON.stringify({ native_inference: receipt })); + vi.clearAllMocks(); + expect(() => findBlueprintNativeReceipt(requested)).toThrow("require HTTPS"); + const observe = () => observeBlueprintCreatedInference(requested, receipt, f.deps); + await expect(observe()).rejects.toThrow("require HTTPS"); + const config = { readConfig: vi.fn(), configTarget: vi.fn(), writeConfig: vi.fn() }; + await expect( + reconcileBlueprintNativeTimeout(requested, observe, vi.fn(), config), + ).rejects.toThrow("require HTTPS"); + expect(f.deps.adapter).not.toHaveBeenCalled(); + expect(f.deps.capture).not.toHaveBeenCalled(); + expect(f.deps.readHandle).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(config.configTarget).not.toHaveBeenCalled(); + expect(config.readConfig).not.toHaveBeenCalled(); + expect(config.writeConfig).not.toHaveBeenCalled(); + expect(f.input.persist).not.toHaveBeenCalled(); + expect(JSON.parse(fs.readFileSync(f.file, "utf8"))).toEqual({ native_inference: receipt }); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("reads a completed receipt at the byte limit and closes its descriptor", async () => { + const f = await recoveryFixture(); + try { + fs.writeFileSync(f.file, f.plan.padEnd(1024 * 1024, " ")); + const open = vi.spyOn(fs, "openSync"); + expect(findBlueprintNativeReceipt(f.input)).toEqual(f.receipts.at(-1)); + expect(() => fs.fstatSync(open.mock.results[0]!.value)).toThrow(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("skips an oversized receipt without reading its content", async () => { + const f = await recoveryFixture(); + try { + fs.writeFileSync(f.file, f.plan.padEnd(1024 * 1024 + 1, " ")); + const read = vi.spyOn(fs, "readSync"); + const open = vi.spyOn(fs, "openSync"); + expect(findBlueprintNativeReceipt(f.input)).toBeNull(); + expect(read).not.toHaveBeenCalled(); + expect(() => fs.fstatSync(open.mock.results[0]!.value)).toThrow(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("skips a symbolic-link receipt without reading its target", async () => { + const f = await recoveryFixture(); + try { + const outside = path.join(f.home, "outside.json"); + fs.renameSync(f.file, outside); + fs.symlinkSync(outside, f.file); + const read = vi.spyOn(fs, "readSync"); + expect(findBlueprintNativeReceipt(f.input)).toBeNull(); + expect(read).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("rejects a symbolic link substituted immediately before opening the receipt", async () => { + const f = await recoveryFixture(); + const open = fs.openSync; + try { + const outside = path.join(f.home, "outside.json"); + fs.writeFileSync(outside, f.plan); + vi.spyOn(fs, "openSync").mockImplementationOnce((file, flags) => { + fs.unlinkSync(f.file); + fs.symlinkSync(outside, f.file); + return open(file, flags); + }); + const read = vi.spyOn(fs, "readSync"); + expect(findBlueprintNativeReceipt(f.input)).toBeNull(); + expect(read).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("refuses a nonblocking FIFO receipt before reading", async () => { + const f = await recoveryFixture(); + try { + fs.unlinkSync(f.file); + execFileSync("mkfifo", [f.file]); + const read = vi.spyOn(fs, "readSync"); + const openFile = fs.openSync; + const open = vi.spyOn(fs, "openSync").mockImplementationOnce((file, flags) => { + expect(Number(flags) & fs.constants.O_NONBLOCK).toBe(fs.constants.O_NONBLOCK); + return openFile(file, flags); + }); + expect(findBlueprintNativeReceipt(f.input)).toBeNull(); + expect(read).not.toHaveBeenCalled(); + expect(() => fs.fstatSync(open.mock.results[0]!.value)).toThrow(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }, 5_000); + + it("refuses a file replaced after opening without reading the replacement", async () => { + const f = await recoveryFixture(); + const open = fs.openSync; + try { + const opened = vi.spyOn(fs, "openSync").mockImplementationOnce((file, flags) => { + const descriptor = open(file, flags); + fs.renameSync(f.file, path.join(f.home, "retained.json")); + fs.writeFileSync(f.file, f.plan); + return descriptor; + }); + const read = vi.spyOn(fs, "readSync"); + expect(() => findBlueprintNativeReceipt(f.input)).toThrow("changed while it was read"); + expect(read).not.toHaveBeenCalled(); + expect(() => fs.fstatSync(opened.mock.results[0]!.value)).toThrow(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("refuses a file replaced after reading and closes the original descriptor", async () => { + const f = await recoveryFixture(); + const stat = fs.fstatSync; + try { + vi.spyOn(fs, "fstatSync") + .mockImplementationOnce((descriptor) => stat(descriptor)) + .mockImplementationOnce((descriptor) => { + fs.renameSync(f.file, path.join(f.home, "retained.json")); + fs.writeFileSync(f.file, f.plan); + return stat(descriptor); + }); + const open = vi.spyOn(fs, "openSync"); + expect(() => findBlueprintNativeReceipt(f.input)).toThrow("changed while it was read"); + expect(() => fs.fstatSync(open.mock.results[0]!.value)).toThrow(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("refuses two different complete receipts for the same sandbox", async () => { + const f = await recoveryFixture(); + try { + const other = path.join(path.dirname(f.directory), "nc-other"); + fs.mkdirSync(other); + const receipt = f.receipts.at(-1)!; + fs.writeFileSync( + path.join(other, "plan.json"), + JSON.stringify({ + native_inference: { + ...receipt, + providerAttachment: { ...receipt.providerAttachment, providerId: "other-provider-id" }, + }, + }), + ); + expect(() => findBlueprintNativeReceipt(f.input)).toThrow("Conflicting blueprint"); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("refuses a run directory replaced by a symbolic link before reading", async () => { + const f = await recoveryFixture(); + const open = fs.openSync; + try { + const outside = path.join(f.home, "outside"); + fs.mkdirSync(outside); + fs.writeFileSync(path.join(outside, "plan.json"), f.plan); + vi.spyOn(fs, "openSync").mockImplementationOnce((file, flags) => { + fs.renameSync(f.directory, path.join(f.home, "retained")); + fs.symlinkSync(outside, f.directory); + return open(file, flags); + }); + const read = vi.spyOn(fs, "readSync"); + expect(() => findBlueprintNativeReceipt(f.input)).toThrow("changed while it was read"); + expect(read).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("bounds a receipt that grows after descriptor validation and refuses it", async () => { + const f = await recoveryFixture(); + const stat = fs.fstatSync; + try { + vi.spyOn(fs, "fstatSync").mockImplementationOnce((descriptor) => { + const before = stat(descriptor); + fs.appendFileSync(f.file, " ".repeat(2 * 1024 * 1024)); + return before; + }); + const read = vi.spyOn(fs, "readSync"); + const open = vi.spyOn(fs, "openSync"); + expect(() => findBlueprintNativeReceipt(f.input)).toThrow("changed while it was read"); + expect(read).toHaveBeenCalledTimes(1); + expect(read.mock.results[0]?.value).toBe(1024 * 1024 + 1); + expect(() => fs.fstatSync(open.mock.results[0]!.value)).toThrow(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it.each([ + "{ malformed historical JSON", + JSON.stringify({ native_inference: { gatewayName: "other", sandboxName: "chosen-sandbox" } }), + ])("ignores an unrelated unusable run while preserving valid authority", async (otherPlan) => { + const f = fixture(); + await createBlueprintNativeInference(f.input, f.deps); + const home = fs.mkdtempSync(path.join(os.tmpdir(), "blueprint-recovery-")); + vi.spyOn(os, "homedir").mockReturnValue(home); + const runs = path.join(home, ".nemoclaw/state/runs"); + try { + fs.mkdirSync(path.join(runs, "nc-first"), { recursive: true }); + fs.mkdirSync(path.join(runs, "nc-second")); + fs.writeFileSync(path.join(runs, "nc-first/plan.json"), otherPlan); + fs.writeFileSync( + path.join(runs, "nc-second/plan.json"), + JSON.stringify({ native_inference: f.receipts.at(-1) }), + ); + const receipt = findBlueprintNativeReceipt(f.input); + expect(receipt).toEqual(f.receipts.at(-1)); + await expect( + observeBlueprintCreatedInference({ ...f.input, endpointUrl: undefined }, receipt!, f.deps), + ).resolves.toContain("provider-id-1"); + } finally { + fs.rmSync(home, { recursive: true, force: true }); + } + }); + + it.each(["incomplete", "malformed"])( + "refuses a matching %s run despite another complete receipt", + async (kind) => { + const f = fixture(); + await createBlueprintNativeInference(f.input, f.deps); + const home = fs.mkdtempSync(path.join(os.tmpdir(), "blueprint-recovery-")); + vi.spyOn(os, "homedir").mockReturnValue(home); + const runs = path.join(home, ".nemoclaw/state/runs"); + try { + fs.mkdirSync(path.join(runs, "nc-first"), { recursive: true }); + fs.mkdirSync(path.join(runs, "nc-second")); + const partial = + kind === "incomplete" ? f.receipts[0] : { ...f.receipts.at(-1), schemaVersion: 7 }; + fs.writeFileSync( + path.join(runs, "nc-first/plan.json"), + JSON.stringify({ native_inference: partial }), + ); + fs.writeFileSync( + path.join(runs, "nc-second/plan.json"), + JSON.stringify({ native_inference: f.receipts.at(-1) }), + ); + expect(() => findBlueprintNativeReceipt(f.input)).toThrow(/incomplete|Invalid/); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + } finally { + fs.rmSync(home, { recursive: true, force: true }); + } + }, + ); +}); + +async function interruptedRunFixture() { + const f = fixture({ configFailure: true }); + await expect(createBlueprintNativeInference(f.input, f.deps)).rejects.toThrow( + "configuration failed", + ); + const home = fs.mkdtempSync(path.join(os.tmpdir(), "blueprint-resume-")); + vi.spyOn(os, "homedir").mockReturnValue(home); + const input = { + runId: "nc-interrupted", + gateway: { name: f.input.gatewayName, host: "127.0.0.1", port: 8080 }, + sandboxName: f.input.sandboxName, + revalidate: vi.fn(async () => {}), + }; + const directory = path.join(home, ".nemoclaw/state/runs", input.runId); + fs.mkdirSync(directory, { recursive: true }); + const file = path.join(directory, "plan.json"); + const plan = { + run_id: input.runId, + gateway: input.gateway, + sandbox_name: input.sandboxName, + sandbox_created_by_apply: true, + inference_provider_created_by_apply: true, + native_inference: f.receipts.at(-1)!, + profile: "default", + }; + fs.writeFileSync(file, JSON.stringify(plan), { mode: 0o600 }); + f.deps.writeConfig.mockReset().mockImplementation(() => {}); + vi.mocked(f.adapter.createProvider).mockClear(); + vi.mocked(f.adapter.attachProvider).mockClear(); + f.input.createSandbox.mockClear(); + const resume = () => resumeBlueprintNativeInferenceRun(input, f.deps); + return { ...f, home, directory, file, plan, resume, resumeInput: input }; +} + +type ResumeFaultCase = [ + string, + (fixture: Awaited>) => void, +]; + +describe("blueprint configuration resume", () => { + it.each([ + ["symlink", (directory: string, retained: string) => fs.symlinkSync(retained, directory), 1], + ["directory", (directory: string) => fs.mkdirSync(directory), 1], + ["file", (directory: string) => fs.writeFileSync(directory, "unrelated file"), 1], + ] as const)( + "refuses a %s replacing the run directory after receipt publication", + async (_replacement, replace, closedDescriptors) => { + const f = await interruptedRunFixture(); + const rename = fs.renameSync; + const retained = path.join(f.home, "retained-run"); + const sync = vi.spyOn(fs, "fsyncSync"); + const close = vi.spyOn(fs, "closeSync"); + let replaced = false; + try { + vi.spyOn(fs, "renameSync").mockImplementationOnce((from, to) => { + rename(from, to); + rename(f.directory, retained); + replace(f.directory, retained); + replaced = true; + sync.mockClear(); + close.mockClear(); + }); + await expect(f.resume()).rejects.toThrow(); + expect(replaced).toBe(true); + expect(sync).not.toHaveBeenCalled(); + expect(close).toHaveBeenCalledTimes(closedDescriptors); + expect(JSON.parse(fs.readFileSync(path.join(retained, "plan.json"), "utf8"))).toEqual({ + ...f.plan, + native_inference: { ...f.plan.native_inference, configPublished: true }, + }); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }, + ); + + it("completes a failed configuration and permits later reuse without recreating resources", async () => { + const f = await interruptedRunFixture(); + try { + expect(() => findBlueprintNativeReceipt(f.input)).toThrow("incomplete"); + await f.resume(); + const completedFd = fs.openSync(f.file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + try { + const completed = JSON.parse(fs.readFileSync(completedFd, "utf8")); + expect(completed).toEqual({ + ...f.plan, + native_inference: { ...f.plan.native_inference, configPublished: true }, + }); + expect(fs.fstatSync(completedFd).mode & 0o777).toBe(0o600); + } finally { + fs.closeSync(completedFd); + } + expect(f.deps.writeConfig).toHaveBeenCalledWith( + f.input.sandboxName, + expect.arrayContaining([ + { dotpath: "agents.defaults.model.primary", value: "inference/custom-model" }, + ]), + f.input.gatewayName, + ); + await expect( + observeBlueprintCreatedInference(f.input, findBlueprintNativeReceipt(f.input)!, f.deps), + ).resolves.toContain("provider-id-1"); + await f.resume(); + expect(f.deps.writeConfig).toHaveBeenCalledTimes(1); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.input.createSandbox).not.toHaveBeenCalled(); + const finalFd = fs.openSync(f.file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + try { + expect(fs.readFileSync(finalFd, "utf8")).not.toContain("test-host-secret"); + } finally { + fs.closeSync(finalFd); + } + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it.each([ + ["sandbox", (f) => f.replaceSandbox()], + ["provider", (f) => f.replaceProvider()], + [ + "attachment", + (f) => { + vi.mocked(f.adapter.listProviderAttachments).mockResolvedValue({ + ok: true, + value: { names: [] }, + }); + }, + ], + [ + "profile", + (f) => { + vi.mocked(f.adapter.inspectProviderProfile).mockResolvedValue({ + ok: false, + error: { kind: "command", reason: "profile_incompatible", message: "changed" }, + }); + }, + ], + [ + "credential", + (f) => { + f.deps.readHandle.mockRejectedValue(new Error("credential handle unavailable")); + }, + ], + [ + "gateway", + (f) => { + f.resumeInput.revalidate.mockRejectedValue(new Error("gateway changed")); + }, + ], + [ + "receipt", + (f) => { + f.resumeInput.revalidate.mockImplementationOnce(async () => { + fs.writeFileSync(f.file, JSON.stringify({ ...f.plan, profile: "changed" })); + }); + }, + ], + ])("refuses changed %s authority before configuration", async (_kind, arrange) => { + const f = await interruptedRunFixture(); + try { + arrange(f); + await expect(f.resume()).rejects.toThrow(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect( + JSON.parse(fs.readFileSync(f.file, "utf8")).native_inference.configPublished, + ).toBeUndefined(); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it.each([ + [ + "missing sandbox", + (f) => { + fs.writeFileSync( + f.file, + JSON.stringify({ + ...f.plan, + native_inference: { ...f.plan.native_inference, sandboxId: undefined }, + }), + ); + }, + ], + [ + "missing provider", + (f) => { + fs.writeFileSync( + f.file, + JSON.stringify({ + ...f.plan, + native_inference: { ...f.plan.native_inference, providerAttachment: undefined }, + }), + ); + }, + ], + [ + "wrong run", + (f) => { + fs.writeFileSync(f.file, JSON.stringify({ ...f.plan, run_id: "another-run" })); + }, + ], + [ + "wrong gateway", + (f) => { + fs.writeFileSync( + f.file, + JSON.stringify({ ...f.plan, gateway: { ...f.plan.gateway, port: 9090 } }), + ); + }, + ], + [ + "unowned", + (f) => { + fs.writeFileSync(f.file, JSON.stringify({ ...f.plan, sandbox_created_by_apply: false })); + }, + ], + [ + "corrupt", + (f) => { + fs.writeFileSync(f.file, "{"); + }, + ], + [ + "symlink", + (f) => { + fs.renameSync(f.file, `${f.file}.original`); + fs.symlinkSync(`${f.file}.original`, f.file); + }, + ], + [ + "hardlink", + (f) => { + fs.linkSync(f.file, `${f.file}.link`); + }, + ], + [ + "rolled back", + (f) => { + fs.writeFileSync(path.join(f.directory, "rolled_back"), "done"); + }, + ], + ])("refuses %s receipt before any remote action", async (_kind, arrange) => { + const f = await interruptedRunFixture(); + try { + arrange(f); + f.deps.capture.mockClear(); + await expect(f.resume()).rejects.toThrow(); + expect(f.deps.capture).not.toHaveBeenCalled(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(f.resumeInput.revalidate).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it.each([ + [ + "write", + (f) => { + f.deps.writeConfig.mockImplementationOnce(() => { + throw new Error("write failed"); + }); + }, + ], + [ + "post-write identity", + (f) => { + f.deps.writeConfig.mockImplementationOnce(() => f.replaceSandbox()); + }, + ], + [ + "receipt persistence", + () => { + vi.spyOn(fs, "renameSync").mockImplementationOnce(() => { + throw new Error("rename failed"); + }); + }, + ], + ])("preserves incomplete authority after %s fails", async (_kind, arrange) => { + const f = await interruptedRunFixture(); + try { + arrange(f); + await expect(f.resume()).rejects.toThrow(); + expect(JSON.parse(fs.readFileSync(f.file, "utf8"))).toEqual(f.plan); + expect(fs.readdirSync(f.directory)).toEqual(["plan.json"]); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); + + it("refuses a replacement sandbox while the handle is being read", async () => { + const f = await interruptedRunFixture(); + try { + f.deps.readHandle.mockImplementationOnce(async () => { + f.replaceSandbox(); + return "openshell:resolve:env:v1_OPENAI_API_KEY"; + }); + await expect(f.resume()).rejects.toThrow("identity changed"); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(JSON.parse(fs.readFileSync(f.file, "utf8"))).toEqual(f.plan); + } finally { + fs.rmSync(f.home, { recursive: true, force: true }); + } + }); +}); + +function timeoutFixture() { + const input = { + gatewayName: "chosen-gateway", + sandboxName: "chosen-sandbox", + provider: "chosen-provider", + model: "custom-model", + api: "openai-completions", + timeoutSeconds: 300, + }; + const config = { + agents: { defaults: { model: { primary: "inference/custom-model" }, timeoutSeconds: 600 } }, + models: { + providers: { + inference: { + api: "openai-completions", + baseUrl: "http://8.8.8.8:8000/v1", + apiKey: "openshell:resolve:env:v1_OPENAI_API_KEY", + timeoutSeconds: 180, + }, + }, + }, + }; + const deps = { + readConfig: vi.fn(() => config), + configTarget: vi.fn(() => ({ + agentName: "openclaw", + configPath: "/sandbox/.openclaw/openclaw.json", + configDir: "/sandbox/.openclaw", + configFile: "openclaw.json", + format: "json", + })), + writeConfig: vi.fn(), + }; + return { + input, + config, + deps, + revalidate: vi.fn(), + observe: vi.fn(async () => "immutable-identity"), + }; +} + +describe("blueprint native timeout reuse", () => { + it("updates only the selected provider request timeout under repeated authority checks", async () => { + const f = timeoutFixture(); + await reconcileBlueprintNativeTimeout(f.input, f.observe, f.revalidate, f.deps); + expect(f.deps.writeConfig).toHaveBeenCalledWith( + "chosen-sandbox", + [{ dotpath: "models.providers.inference.timeoutSeconds", value: 300 }], + "chosen-gateway", + ); + expect(f.config.agents.defaults.timeoutSeconds).toBe(600); + expect(f.observe).toHaveBeenCalledTimes(3); + }); + it.each([ + ["openai-completions", "http://8.8.8.8:8000/v1", "http://8.8.8.8:8000/v1/"], + ["anthropic-messages", "http://8.8.8.8:8000", "http://8.8.8.8:8000/v1"], + ])( + "normalizes the %s provider endpoint before timeout reuse", + async (api, configuredUrl, requestedUrl) => { + const f = timeoutFixture(); + f.config.models.providers.inference.api = api; + f.config.models.providers.inference.baseUrl = configuredUrl; + await reconcileBlueprintNativeTimeout( + { ...f.input, api, endpointUrl: requestedUrl }, + f.observe, + f.revalidate, + f.deps, + ); + expect(f.deps.writeConfig).toHaveBeenCalledTimes(1); + expect(f.observe).toHaveBeenCalledTimes(3); + }, + ); + + it("preserves an already matching timeout without writing config", async () => { + const f = timeoutFixture(); + await reconcileBlueprintNativeTimeout( + { ...f.input, timeoutSeconds: 180 }, + f.observe, + f.revalidate, + f.deps, + ); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + }); + it.each([ + { model: "other-model" }, + { api: "anthropic-messages" }, + { endpointUrl: "http://9.9.9.9/v1" }, + { timeoutSeconds: 0 }, + ])("refuses mismatched selection before writes: %j", async (override) => { + const f = timeoutFixture(); + await expect( + reconcileBlueprintNativeTimeout({ ...f.input, ...override }, f.observe, f.revalidate, f.deps), + ).rejects.toThrow(); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + }); + it("refuses identity replacement between config read and write", async () => { + const f = timeoutFixture(); + f.observe.mockResolvedValueOnce("original").mockResolvedValueOnce("replacement"); + await expect( + reconcileBlueprintNativeTimeout(f.input, f.observe, f.revalidate, f.deps), + ).rejects.toThrow("before timeout update"); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + }); + + it("refuses a timeout write when the sandbox changes as its observer returns", async () => { + const created = fixture(); + const authority = await createBlueprintNativeInference(created.input, created.deps); + const receipt = created.receipts.at(-1)!; + const f = timeoutFixture(); + let replaced = false; + const observe = vi + .fn(() => authority.observe()) + .mockImplementationOnce(() => authority.observe()) + .mockImplementationOnce(async () => { + const identity = await authority.observe(); + queueMicrotask(() => { + created.replaceSandbox(); + replaced = true; + }); + return identity; + }); + await expect( + reconcileBlueprintNativeTimeout( + f.input, + observe, + () => requireCreatedSandbox(receipt, created.deps), + f.deps, + ), + ).rejects.toThrow("identity changed"); + expect(replaced).toBe(true); + expect(f.deps.writeConfig).not.toHaveBeenCalled(); + expect(created.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(created.adapter.detachProvider).not.toHaveBeenCalled(); + }); +}); diff --git a/src/lib/inference/native-provider/blueprint.ts b/src/lib/inference/native-provider/blueprint.ts new file mode 100644 index 00000000000..66581185078 --- /dev/null +++ b/src/lib/inference/native-provider/blueprint.ts @@ -0,0 +1,551 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { createHash, randomBytes } from "node:crypto"; +import fs from "node:fs"; +import { isIP } from "node:net"; +import os from "node:os"; +import path from "node:path"; +import YAML from "yaml"; +import { isPrivateIp } from "../../private-networks"; +import { isValidProviderName } from "../../../../nemoclaw/dist/shared/sandbox-name.cjs"; +import { normalizeProviderBaseUrl } from "../../core/provider-endpoint"; +import { unsafeEndpointUrlViolation } from "../../core/endpoint-url-safety"; +import { createManagedProviderAdapter } from "../../adapters/openshell/managed-provider-adapter"; +import { parseCheckedInProviderProfileContract } from "../../adapters/openshell/provider-profile"; +import { captureSanitizedResolvedOpenshell } from "../../adapters/openshell/sanitized-capture"; +import { + observeCreatedOpenShellSandboxId, + settleCreatedOpenShellSandboxId, +} from "../../adapters/openshell/sandbox-identity"; +import { + readSandboxConfig, + resolveAgentConfig, + setOpenClawConfigValues, +} from "../../sandbox/config"; +import { readNativeCredentialHandle } from "../../actions/inference-set-provider"; +import { boundCustomNativeProfile, customNativeProvider } from "./custom"; +import { normalizeNativeProviderAttachment } from "./contract"; +import { nativeProviderLifecycle } from "./index"; +import { + ensureNativeProvider, + ensureNativeProviderAttached, + verifyNativeProviderAttachment, + type NativeProviderAttachment, +} from "./lifecycle"; + +export type BlueprintNativeSelection = Readonly<{ + gatewayName: string; + sandboxName: string; + provider: string; + model: string; + api: string; + endpointUrl?: string; + requiresCredential?: boolean; +}>; + +/** Authority belongs to this blueprint run, not the managed sandbox registry. */ +export type BlueprintNativeReceipt = BlueprintNativeSelection & + Readonly<{ + schemaVersion: 1; + endpointUrl: string; + api: "openai-completions" | "anthropic-messages"; + createAttemptNonce: string; + credentialEnv: string | null; + timeoutSeconds?: number; + providerAttachment?: NativeProviderAttachment; + sandboxId?: string; + configPublished?: true; + }>; + +export type BlueprintNativeCreateInput = BlueprintNativeSelection & + Readonly<{ + credentialValue: string; + timeoutSeconds?: number; + persist: (receipt: BlueprintNativeReceipt) => void; + createSandbox: (providerName: string, createAttemptNonce: string) => Promise; + prepareSandboxPolicy: () => Promise; + }>; + +const NAME = /^[a-zA-Z0-9][a-zA-Z0-9_-]{0,127}$/u; + +/** Recheck the plugin's existing public, literal-address admission at the host boundary. */ +function admitBlueprintSelection(input: BlueprintNativeSelection): { + endpointUrl: string; + api: BlueprintNativeReceipt["api"]; +} { + if ( + typeof input.gatewayName !== "string" || + typeof input.sandboxName !== "string" || + typeof input.provider !== "string" || + !NAME.test(input.gatewayName) || + !NAME.test(input.sandboxName) || + !isValidProviderName(input.provider) || + typeof input.model !== "string" || + !input.model || + input.model.length > 1024 || + /[\u0000-\u001f\u007f]/u.test(input.model) || + !["openai-completions", "anthropic-messages"].includes(input.api) + ) + throw new Error("Invalid blueprint native inference selection."); + const endpoint = new URL(input.endpointUrl ?? ""); + const address = endpoint.hostname.replace(/^\[|\]$/gu, ""); + if ( + unsafeEndpointUrlViolation(input.endpointUrl ?? "") || + /%2f|%5c|%2e/iu.test(input.endpointUrl ?? "") || + !isIP(address) || + isPrivateIp(address) + ) + throw new Error("Blueprint native inference requires its admitted public endpoint address."); + const api = input.api as BlueprintNativeReceipt["api"]; + const endpointUrl = normalizeProviderBaseUrl( + endpoint.href, + api === "anthropic-messages" ? "anthropic" : "openai", + ); + // The existing custom profile owner checks URL syntax, encoded path escapes, and exact IPs. + customNativeProvider({ + provider: + api === "anthropic-messages" ? "compatible-anthropic-endpoint" : "compatible-endpoint", + endpointUrl, + api, + credentialEnv: "OPENAI_API_KEY", + allowedIps: [address], + }); + return { endpointUrl, api }; +} + +function blueprintProfile(receipt: BlueprintNativeReceipt) { + const admitted = admitBlueprintSelection(receipt); + if ( + receipt.schemaVersion !== 1 || + admitted.endpointUrl !== receipt.endpointUrl || + !/^[a-f0-9]{62}$/u.test(receipt.createAttemptNonce) || + (receipt.credentialEnv !== null && receipt.credentialEnv !== "OPENAI_API_KEY") || + (receipt.timeoutSeconds !== undefined && + (!Number.isSafeInteger(receipt.timeoutSeconds) || receipt.timeoutSeconds <= 0)) + ) + throw new Error("Invalid blueprint native inference receipt."); + if (receipt.credentialEnv !== null && new URL(receipt.endpointUrl).protocol !== "https:") + throw new Error("Blueprint inference endpoints with credentials require HTTPS."); + const definition = customNativeProvider({ + provider: + receipt.api === "anthropic-messages" + ? "compatible-anthropic-endpoint" + : "compatible-endpoint", + endpointUrl: receipt.endpointUrl, + api: receipt.api, + credentialEnv: "OPENAI_API_KEY", + allowedIps: [new URL(receipt.endpointUrl).hostname.replace(/^\[|\]$/gu, "")], + }); + const template = nativeProviderLifecycle(definition).nativeProviderProfilePath(); + const document = YAML.parse(boundCustomNativeProfile(definition, template)); + // The run nonce prevents a same-name provider from masquerading as this operation's provider. + document.id = `nemoclaw-blueprint-${createHash("sha256") + .update( + JSON.stringify({ + endpoint: receipt.endpointUrl, + api: receipt.api, + credentialEnv: receipt.credentialEnv, + nonce: receipt.createAttemptNonce, + }), + ) + .digest("hex") + .slice(0, 32)}-v1`; + document.credentials = + receipt.credentialEnv === null + ? [] + : document.credentials.map((credential: Record) => ({ + ...credential, + env_vars: [receipt.credentialEnv], + })); + const source = YAML.stringify(document); + const contract = parseCheckedInProviderProfileContract(source); + if (!contract) throw new Error("Blueprint native profile could not be validated."); + return { + source, + contract, + profile: { + profileId: document.id as string, + providerName: receipt.provider, + credentialEnv: receipt.credentialEnv, + label: "blueprint", + profilePath: "", + }, + }; +} + +function capture(args: string[], options?: Record): string { + const result = captureSanitizedResolvedOpenshell(args, { + ...options, + ignoreError: true, + includeStreams: true, + includeStderr: true, + timeout: typeof options?.timeout === "number" ? options.timeout : 30_000, + maxBuffer: 1024 * 1024, + }); + if (result.status !== 0 || result.error || result.signal) + throw new Error("Blueprint native sandbox identity could not be inspected."); + return result.stdout ?? ""; +} + +const defaultDependencies = { + adapter: createManagedProviderAdapter, + capture, + writeConfig: setOpenClawConfigValues, + readHandle: readNativeCredentialHandle, +}; +type Dependencies = typeof defaultDependencies; + +export function requireCreatedSandbox( + receipt: BlueprintNativeReceipt, + deps: Dependencies = defaultDependencies, +): void { + const observed = observeCreatedOpenShellSandboxId( + { ...receipt, runCaptureOpenshell: deps.capture }, + 15_000, + ); + if (observed.state !== "matched" || observed.sandboxId !== receipt.sandboxId) + throw new Error("Blueprint sandbox identity changed; no later resource was modified."); +} + +const MAX_BLUEPRINT_RECEIPT_BYTES = 1024 * 1024; + +export function readBlueprintRunPlan(file: string, directory: fs.Stats): unknown | null { + const noFollow = fs.constants.O_NOFOLLOW; + if (typeof noFollow !== "number") + throw new Error("Blueprint run receipts require no-follow file reads."); + let descriptor: number; + try { + descriptor = fs.openSync( + file, + fs.constants.O_RDONLY | noFollow | (fs.constants.O_NONBLOCK ?? 0), + ); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code === "ENOENT" || code === "ELOOP") return null; + throw error; + } + try { + const before = fs.fstatSync(descriptor); + if (!before.isFile() || before.size > MAX_BLUEPRINT_RECEIPT_BYTES) return null; + const requireSamePath = (): void => { + const currentDirectory = fs.lstatSync(path.dirname(file)); + const currentFile = fs.lstatSync(file); + if ( + !currentDirectory.isDirectory() || + currentDirectory.dev !== directory.dev || + currentDirectory.ino !== directory.ino || + !currentFile.isFile() || + currentFile.dev !== before.dev || + currentFile.ino !== before.ino + ) + throw new Error("Blueprint run receipt changed while it was read."); + }; + requireSamePath(); + const contents = Buffer.alloc(MAX_BLUEPRINT_RECEIPT_BYTES + 1); + let offset = 0; + while (offset < contents.length) { + const count = fs.readSync(descriptor, contents, offset, contents.length - offset, null); + if (count === 0) break; + offset += count; + } + const after = fs.fstatSync(descriptor); + requireSamePath(); + if ( + offset > MAX_BLUEPRINT_RECEIPT_BYTES || + offset !== before.size || + after.size !== before.size || + after.mtimeMs !== before.mtimeMs || + after.ctimeMs !== before.ctimeMs + ) + throw new Error("Blueprint run receipt changed while it was read."); + try { + return JSON.parse(contents.subarray(0, offset).toString("utf8")); + } catch (error) { + if (error instanceof SyntaxError) return null; + throw error; + } + } finally { + fs.closeSync(descriptor); + } +} + +/** Recover only this plugin's nonsecret run receipts; live identities remain authoritative. */ +export function findBlueprintNativeReceipt( + requested: BlueprintNativeSelection, +): BlueprintNativeReceipt | null { + const runs = path.join(os.homedir(), ".nemoclaw/state/runs"); + if (!fs.existsSync(runs)) return null; + const matches = new Map(); + for (const name of fs.readdirSync(runs)) { + if (!/^[a-zA-Z0-9_-]+$/u.test(name)) continue; + const directory = path.join(runs, name); + const directoryMetadata = fs.lstatSync(directory); + if (!directoryMetadata.isDirectory()) continue; + const file = path.join(directory, "plan.json"); + const plan = readBlueprintRunPlan(file, directoryMetadata); + if (!plan || typeof plan !== "object" || Array.isArray(plan)) continue; + const receipt = (plan as Record).native_inference as + | BlueprintNativeReceipt + | undefined; + if ( + receipt?.gatewayName !== requested.gatewayName || + receipt?.sandboxName !== requested.sandboxName + ) + continue; + blueprintProfile(receipt); + if (!receipt.providerAttachment || !receipt.sandboxId || receipt.configPublished !== true) + throw new Error( + "An incomplete blueprint run owns this sandbox selection. Preserve its receipt for recovery.", + ); + matches.set(JSON.stringify(receipt), receipt); + } + if (matches.size > 1) + throw new Error("Conflicting blueprint native inference receipts require recovery."); + return matches.values().next().value ?? null; +} + +/** Reuse requires completed configuration publication and current resource authority. */ +export async function observeBlueprintCreatedInference( + requested: BlueprintNativeSelection, + receipt: BlueprintNativeReceipt, + deps: Dependencies = defaultDependencies, +): Promise { + if (receipt.configPublished !== true) + throw new Error( + "Blueprint native configuration publication is incomplete. Preserve its receipt for recovery.", + ); + return verifyBlueprintCreatedInference(requested, receipt, deps); +} + +/** Verify the generated profile and both immutable resource identities without mutations. */ +async function verifyBlueprintCreatedInference( + requested: BlueprintNativeSelection, + receipt: BlueprintNativeReceipt, + deps: Dependencies = defaultDependencies, +): Promise { + const admitted = admitBlueprintSelection({ + ...requested, + endpointUrl: requested.endpointUrl ?? receipt.endpointUrl, + }); + if ( + requested.gatewayName !== receipt.gatewayName || + requested.sandboxName !== receipt.sandboxName || + requested.provider !== receipt.provider || + requested.model !== receipt.model || + admitted.api !== receipt.api || + admitted.endpointUrl !== receipt.endpointUrl || + !receipt.providerAttachment || + !receipt.sandboxId || + (requested.requiresCredential === true && receipt.credentialEnv === null) + ) + throw new Error("Blueprint native receipt does not match the requested inference selection."); + const { profile, contract } = blueprintProfile(receipt); + const expected = normalizeNativeProviderAttachment(receipt.providerAttachment, profile); + if (!expected) throw new Error("Invalid blueprint native provider receipt."); + const adapter = deps.adapter(); + const target = { kind: "named", gatewayName: receipt.gatewayName } as const; + requireCreatedSandbox(receipt, deps); + const inspected = await adapter.inspectProviderProfile({ + target, + profileType: profile.profileId, + expectedProfile: contract, + }); + if (!inspected.ok) throw new Error("Blueprint native provider profile changed."); + await verifyNativeProviderAttachment({ + profile, + adapter, + target, + sandboxName: receipt.sandboxName, + expected, + requireUnexpiredCredential: true, + }); + requireCreatedSandbox(receipt, deps); + return JSON.stringify({ + gatewayName: receipt.gatewayName, + sandboxName: receipt.sandboxName, + provider: receipt.provider, + model: receipt.model, + endpointUrl: receipt.endpointUrl, + api: receipt.api, + createAttemptNonce: receipt.createAttemptNonce, + sandboxId: receipt.sandboxId, + providerAttachment: expected, + credentialEnv: receipt.credentialEnv, + timeoutSeconds: receipt.timeoutSeconds, + }); +} + +/** Create only new resources, recording ownership before each later effect. */ +export async function createBlueprintNativeInference( + input: BlueprintNativeCreateInput, + deps: Dependencies = defaultDependencies, +): Promise<{ identity: string; observe: () => Promise }> { + const admitted = admitBlueprintSelection(input); + let receipt: BlueprintNativeReceipt = { + gatewayName: input.gatewayName, + sandboxName: input.sandboxName, + provider: input.provider, + model: input.model, + ...admitted, + schemaVersion: 1, + createAttemptNonce: randomBytes(31).toString("hex"), + credentialEnv: input.credentialValue ? "OPENAI_API_KEY" : null, + ...(input.timeoutSeconds === undefined ? {} : { timeoutSeconds: input.timeoutSeconds }), + }; + const { profile, source } = blueprintProfile(receipt); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-blueprint-profile-")); + try { + const profilePath = path.join(directory, "profile.yaml"); + fs.writeFileSync(profilePath, source, { mode: 0o600 }); + input.persist(receipt); + await input.createSandbox(receipt.provider, receipt.createAttemptNonce); + const sandboxId = settleCreatedOpenShellSandboxId({ + ...receipt, + runCaptureOpenshell: deps.capture, + sleep: (ms) => { + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); + }, + }); + receipt = { ...receipt, sandboxId }; + input.persist(receipt); + await input.prepareSandboxPolicy(); + requireCreatedSandbox(receipt, deps); + const providerAttachment = await ensureNativeProvider({ + profile: { ...profile, profilePath }, + adapter: deps.adapter(), + target: { kind: "named", gatewayName: receipt.gatewayName }, + credentialValue: input.credentialValue || null, + }); + receipt = { ...receipt, providerAttachment }; + input.persist(receipt); + requireCreatedSandbox(receipt, deps); + await ensureNativeProviderAttached({ + profile, + adapter: deps.adapter(), + target: { kind: "named", gatewayName: receipt.gatewayName }, + sandboxName: receipt.sandboxName, + expected: providerAttachment, + }); + const identity = await publishBlueprintNativeConfiguration(receipt, async () => {}, deps); + receipt = { ...receipt, configPublished: true }; + input.persist(receipt); + return { identity, observe: () => observeBlueprintCreatedInference(input, receipt, deps) }; + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} + +/** Complete only configuration; never create or reattach resources during recovery. */ +export async function publishBlueprintNativeConfiguration( + receipt: BlueprintNativeReceipt, + revalidate: () => Promise, + deps: Dependencies = defaultDependencies, +): Promise { + const verify = async () => { + await revalidate(); + const identity = await verifyBlueprintCreatedInference(receipt, receipt, deps); + await revalidate(); + return identity; + }; + await verify(); + let apiKey = "unused"; + if (receipt.credentialEnv) { + apiKey = await deps.readHandle(receipt.sandboxName, receipt.gatewayName, { + script: + 'if [ -z "${OPENAI_API_KEY:-}" ]; then printf "%s\\n" NEMOCLAW_NATIVE_PROVIDER_HANDLE_PENDING >&2; exit 75; fi; printf "%s" "$OPENAI_API_KEY"', + pattern: /^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_OPENAI_API_KEY$/u, + providerLabel: "blueprint", + sleep: (ms) => new Promise((resolve) => setTimeout(resolve, ms)), + }); + } + await verify(); + // Awaiting verification yields before this synchronous configuration write. + requireCreatedSandbox(receipt, deps); + deps.writeConfig( + receipt.sandboxName, + [ + { + dotpath: "models.providers.inference", + value: { + baseUrl: receipt.endpointUrl, + api: receipt.api, + apiKey, + ...(receipt.credentialEnv === null ? { authHeader: false } : {}), + ...(receipt.timeoutSeconds === undefined + ? {} + : { timeoutSeconds: receipt.timeoutSeconds }), + models: [{ id: receipt.model, name: receipt.model }], + }, + }, + { dotpath: "agents.defaults.model.primary", value: `inference/${receipt.model}` }, + ], + receipt.gatewayName, + ); + return verify(); +} + +/** Update only the selected OpenClaw provider timeout after independent authority checks. */ +export async function reconcileBlueprintNativeTimeout( + requested: BlueprintNativeSelection & Readonly<{ timeoutSeconds: number }>, + observe: () => Promise, + revalidate: () => void, + deps = { + readConfig: readSandboxConfig, + configTarget: resolveAgentConfig, + writeConfig: setOpenClawConfigValues, + }, +): Promise { + if (!Number.isSafeInteger(requested.timeoutSeconds) || requested.timeoutSeconds <= 0) + throw new Error("Invalid blueprint inference timeout."); + const identity = await observe(); + const target = deps.configTarget(requested.sandboxName); + if (target.agentName !== "openclaw") throw new Error("Blueprint inference requires OpenClaw."); + const config = deps.readConfig(requested.sandboxName, target, requested.gatewayName) as { + agents?: { defaults?: { model?: { primary?: unknown } } }; + models?: { providers?: Record }; + }; + const primary = config?.agents?.defaults?.model?.primary; + if (typeof primary !== "string") + throw new Error("Native inference model configuration is missing."); + const separator = primary.indexOf("/"); + const providerKey = primary.slice(0, separator); + const configured = config?.models?.providers?.[providerKey]; + if ( + separator <= 0 || + !/^[a-zA-Z0-9_-]+$/u.test(providerKey) || + primary.slice(separator + 1) !== requested.model || + !configured || + typeof configured !== "object" || + Array.isArray(configured) || + (configured as Record).api !== requested.api || + (requested.endpointUrl && + (typeof (configured as Record).baseUrl !== "string" || + normalizeProviderBaseUrl( + (configured as Record).baseUrl as string, + requested.api === "anthropic-messages" ? "anthropic" : "openai", + ) !== + normalizeProviderBaseUrl( + requested.endpointUrl, + requested.api === "anthropic-messages" ? "anthropic" : "openai", + ))) + ) + throw new Error("Native inference configuration does not match the blueprint selection."); + if (identity !== (await observe())) + throw new Error("Native inference identity changed before timeout update."); + if ((configured as Record).timeoutSeconds !== requested.timeoutSeconds) { + revalidate(); + deps.writeConfig( + requested.sandboxName, + [ + { + dotpath: `models.providers.${providerKey}.timeoutSeconds`, + value: requested.timeoutSeconds, + }, + ], + requested.gatewayName, + ); + } + if (identity !== (await observe())) + throw new Error("Native inference identity changed during timeout update."); +} diff --git a/src/lib/inference/native-provider/catalog.ts b/src/lib/inference/native-provider/catalog.ts new file mode 100644 index 00000000000..bacc9ae7fac --- /dev/null +++ b/src/lib/inference/native-provider/catalog.ts @@ -0,0 +1,105 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { unsafeEndpointUrlViolation } from "../../core/endpoint-url-safety.ts"; +import type { NativeProviderDefinition } from "./contract.ts"; + +export function isCustomHostedProvider(provider: string | null | undefined): boolean { + return provider === "compatible-endpoint" || provider === "compatible-anthropic-endpoint"; +} + +/** Resolve the native binding and agent configuration from the same API selection. */ +export function resolveCustomNativeApi( + provider: string | null | undefined, + preferredApi: string | null | undefined, + nativeProvider: boolean, +): string | null { + if (!nativeProvider || !isCustomHostedProvider(provider)) return preferredApi ?? null; + return ( + preferredApi || + (provider === "compatible-anthropic-endpoint" ? "anthropic-messages" : "openai-completions") + ); +} + +export function customNativeCredentialEnv(provider: string): string { + if (provider === "compatible-endpoint") return "COMPATIBLE_API_KEY"; + if (provider === "compatible-anthropic-endpoint") return "COMPATIBLE_ANTHROPIC_API_KEY"; + throw new Error("Unsupported custom hosted provider"); +} + +export type HostedProviderDefinition = NativeProviderDefinition & + Readonly<{ + endpoint: string; + api: "openai-completions" | "openai-responses" | "anthropic-messages"; + }>; + +/** Existing fixed hosted selections only; custom and local endpoints have separate owners. */ +export const HOSTED_NATIVE_PROVIDERS = [ + { + logicalProvider: "openai-api", + label: "OpenAI", + profileId: "nemoclaw-openai-inference-v1", + providerName: "nemoclaw-openai-api-v1", + credentialEnv: "OPENAI_API_KEY", + endpoint: "https://api.openai.com/v1", + api: "openai-completions", + }, + { + logicalProvider: "anthropic-prod", + label: "Anthropic", + profileId: "nemoclaw-anthropic-inference-v1", + providerName: "nemoclaw-anthropic-prod-v1", + credentialEnv: "ANTHROPIC_API_KEY", + endpoint: "https://api.anthropic.com", + api: "anthropic-messages", + }, + { + logicalProvider: "gemini-api", + label: "Gemini", + profileId: "nemoclaw-gemini-inference-v1", + providerName: "nemoclaw-gemini-api-v1", + credentialEnv: "GEMINI_API_KEY", + endpoint: "https://generativelanguage.googleapis.com/v1beta/openai/", + api: "openai-completions", + }, + { + logicalProvider: "openrouter-api", + label: "OpenRouter", + profileId: "nemoclaw-openrouter-inference-v1", + providerName: "nemoclaw-openrouter-api-v1", + credentialEnv: "OPENROUTER_API_KEY", + endpoint: "https://openrouter.ai/api/v1", + api: "openai-completions", + }, + { + logicalProvider: "hermes-provider", + label: "Hermes Provider", + profileId: "nemoclaw-hermes-inference-v1", + providerName: "nemoclaw-hermes-provider-v1", + credentialEnv: "OPENAI_API_KEY", + endpoint: "https://inference-api.nousresearch.com/v1", + api: "openai-completions", + }, +] as const satisfies readonly HostedProviderDefinition[]; + +/** Parse public image configuration; host admission separately checks private destinations. */ +export function recordedHermesEndpointUrl(value: string): URL { + const endpoint = new URL(value); + if ( + !["http:", "https:"].includes(endpoint.protocol) || + endpoint.username || + endpoint.password || + endpoint.search || + endpoint.hash || + unsafeEndpointUrlViolation(value) || + /%2f|%5c/iu.test(endpoint.pathname) + ) + throw new Error("Recorded Hermes inference endpoint is not an admitted HTTP(S) destination"); + return endpoint; +} + +export function fixedHostedNativeProvider( + provider: string | null | undefined, +): HostedProviderDefinition | undefined { + return HOSTED_NATIVE_PROVIDERS.find((entry) => entry.logicalProvider === provider?.trim()); +} diff --git a/src/lib/inference/native-provider/contract.ts b/src/lib/inference/native-provider/contract.ts index 733c0189e40..24ca46b79ee 100644 --- a/src/lib/inference/native-provider/contract.ts +++ b/src/lib/inference/native-provider/contract.ts @@ -66,6 +66,9 @@ export function normalizeNativeProviderAttachment

&2; exit 2`; return [ - `case "${value}" in *[!a-zA-Z0-9:_]*) exit 2 ;; esac`, - `printf '%s' "${value}" | LC_ALL=C grep -Eq '^openshell:resolve:env:((v[0-9]{1,20}|s[a-f0-9]{64})_)?${env}$' || exit 2`, + `[ -n "${value}" ] || { printf '%s\\n' '${NATIVE_HOSTED_CREDENTIAL_MISSING}' >&2; exit 2; }`, + `case "${value}" in *[!a-zA-Z0-9:_]*) ${reject} ;; esac`, + `printf '%s' "${value}" | LC_ALL=C grep -Eq '^openshell:resolve:env:((v[0-9]{1,20}|s[a-f0-9]{64})_)?${env}$' || { ${reject}; }`, `AUTH_HEADER="${header} ${value}"`, ].join("; "); } diff --git a/src/lib/inference/native-provider/custom-setup.ts b/src/lib/inference/native-provider/custom-setup.ts index c1db79cdbdd..9008f7945ce 100644 --- a/src/lib/inference/native-provider/custom-setup.ts +++ b/src/lib/inference/native-provider/custom-setup.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; +import { isAllowedOpenShellSandboxBridgeUrl } from "../../private-networks"; import { unsafeEndpointUrlViolation } from "../../core/endpoint-url-safety"; import { normalizeProviderBaseUrl } from "../../core/provider-endpoint"; import { @@ -10,7 +11,7 @@ import { } from "../../security/trusted-private-endpoint"; import { parseTrustedPrivateInferenceHostsFromEnv } from "../endpoint-ssrf-preflight"; import type { NativeProviderAttachment } from "./contract"; -import { customNativeProvider, isHostedCustomEndpoint, type CustomNativeBinding } from "./custom"; +import { customNativeProvider, isCustomNativeEndpoint, type CustomNativeBinding } from "./custom"; import { requireHostedProviderAttachment } from "./hosted-attachment"; import { nativeProviderLifecycle } from "./index"; import { @@ -19,14 +20,13 @@ import { hasBedrockRuntimeAwsAuthEnv, } from "../bedrock-runtime"; import { ensureBedrockRuntimeAdapter } from "../bedrock-runtime-adapter"; -import { PERSONAL_POLICY_TIER_NAME } from "../../policy/tiers"; import { assertOpenShellPolicyEndpointPins } from "../../adapters/openshell/policy-boundary"; import { cliOpenShellSandboxPolicyReader } from "../../adapters/openshell/sandbox-policy-cli"; import type { OpenShellSandboxPolicyReader } from "../../adapters/openshell/sandbox-policy"; import { isRouteOnlySandboxReservation } from "../../state/registry/route-reservation"; import type { SandboxEntry } from "../../state/registry/types"; -export { isHostedCustomEndpoint }; +export { isCustomNativeEndpoint }; /** Read retained rules before re-onboarding can prepare a custom provider. */ export async function readCustomNativeSandboxPolicy(input: { @@ -53,14 +53,7 @@ export async function readCustomNativeSandboxPolicy(input: { return policy.value.document; } -/** - * Existing shared-route sandboxes retain their route until recreated. - * - * A Personal sandbox keeps a new custom endpoint on the shared inference.local - * route. OpenShell rejects Personal's hostless port-443 rule beside an attached - * provider endpoint on the same port (NVIDIA/OpenShell#4662), so a native - * provider would make the Personal policy unappliable. - */ +/** Existing shared-route sandboxes retain their route until recreated. */ export function usesCustomNativeOnboarding( provider: string, endpointUrl: string | null, @@ -72,14 +65,13 @@ export function usesCustomNativeOnboarding( } | null | undefined, - policyTier: string | undefined = process.env.NEMOCLAW_POLICY_TIER, ): boolean { - const personal = policyTier?.trim().toLowerCase() === PERSONAL_POLICY_TIER_NAME; return Boolean( sandboxName && - isHostedCustomEndpoint(provider, endpointUrl) && - (recorded?.nativeHostedProviderAttachment?.custom || - (!personal && (!recorded || recorded.pendingRouteReservation))), + isCustomNativeEndpoint(provider, endpointUrl) && + (!recorded || + recorded.pendingRouteReservation || + recorded.nativeHostedProviderAttachment?.custom), ); } @@ -126,6 +118,13 @@ export async function prepareCustomNativeProvider(input: { input.endpointUrl, input.provider === "compatible-anthropic-endpoint" ? "anthropic" : "openai", ); + const bridge = + input.provider === "compatible-anthropic-endpoint" && + isAllowedOpenShellSandboxBridgeUrl(new URL(endpointUrl)); + if (bridge) + throw new Error( + "Custom native inference cannot use an unverified OpenShell host bridge endpoint", + ); const classified = input.provider === "compatible-anthropic-endpoint" ? classifyCustomAnthropicEndpoint(input.endpointUrl) diff --git a/src/lib/inference/native-provider/custom.test.ts b/src/lib/inference/native-provider/custom.test.ts index 1effbd18e16..b51f3a9640f 100644 --- a/src/lib/inference/native-provider/custom.test.ts +++ b/src/lib/inference/native-provider/custom.test.ts @@ -143,6 +143,7 @@ describe("custom native endpoint ownership", () => { (rule: { allow: { path: string } }) => !rule.allow.path.includes("*"), ), ).toBe(true); + expect(profile.endpoints[0]).not.toHaveProperty("tls"); expect(profile.binaries).not.toContain("*"); expect(profile.credentials[0].env_vars).toEqual([definition.credentialEnv]); const config = getSandboxInferenceConfig("selected-model", provider, api, true, endpointUrl); @@ -173,6 +174,82 @@ describe("custom native endpoint ownership", () => { }, ); + it("rejects an unverified Anthropic bridge before provider mutation", async () => { + const h = harness(); + const bridge = { + ...binding, + provider: "compatible-anthropic-endpoint", + endpointUrl: "http://host.openshell.internal:49152", + allowedIps: [], + }; + await expect(prepareCustomNativeProvider({ ...h.input, ...bridge })).rejects.toThrow( + "cannot use an unverified OpenShell host bridge endpoint", + ); + expect(h.lookup).not.toHaveBeenCalled(); + expect(h.adapter.importProviderProfile).not.toHaveBeenCalled(); + expect(h.adapter.createProvider).not.toHaveBeenCalled(); + }); + + it("rejects a saved unverified Anthropic bridge before recovery or profile reconstruction", () => { + // This receipt was accepted by the earlier bridge exception. + const custom = { + provider: "compatible-anthropic-endpoint", + endpointUrl: "http://host.openshell.internal:49152", + api: "anthropic-messages", + credentialEnv: "SYNTHETIC_TEST_KEY", + allowedIps: [], + } as const; + const receipt = { + schemaVersion: 1, + profileId: "nemoclaw-custom-b2b8be7bf9dee6b604ca6933660085f7-v1", + providerName: "nemoclaw-custom-b2b8be7bf9dee6b604ca6933660085f7-v1", + providerId: "synthetic-provider-id", + endpointUrl: custom.endpointUrl, + custom, + } as const; + expect(() => customNativeProvider(custom)).toThrow("requires a hosted endpoint"); + expect(() => + recordedNativeProviderAttachment({ + provider: custom.provider, + endpointUrl: custom.endpointUrl, + nativeHostedProviderAttachment: receipt, + }), + ).toThrow("Invalid native hosted provider attachment"); + expect(() => + boundCustomNativeProfile( + { + ...receipt, + logicalProvider: custom.provider, + label: "custom hosted", + credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", + }, + "unread-template.yaml", + ), + ).toThrow("requires a hosted endpoint"); + }); + + it.each([ + "http://host.openshell.internal:80", + "https://host.openshell.internal:49152", + "http://host.docker.internal:49152", + "http://host.openshell.internal:18789", + "http://host.openshell.internal:8080", + ])( + "rejects unsupported or protected Anthropic bridge %s before mutation", + async (endpointUrl) => { + const h = harness(); + await expect( + prepareCustomNativeProvider({ + ...h.input, + provider: "compatible-anthropic-endpoint", + endpointUrl, + }), + ).rejects.toThrow(); + expect(h.adapter.importProviderProfile).not.toHaveBeenCalled(); + expect(h.adapter.createProvider).not.toHaveBeenCalled(); + }, + ); + it("keeps endpoint, API, credential source and pin authorities distinct", () => { const names = [ binding, @@ -348,6 +425,28 @@ describe("custom native endpoint ownership", () => { expect(onCredentialPersisted).not.toHaveBeenCalled(); }); + it("removes a new attachment when its profile changes before verification", async () => { + const h = harness(); + const receipt = await prepareCustomNativeProvider(h.input); + const provider = nativeProviderLifecycle(customNativeProvider(binding)); + h.adapter.importProviderProfile.mockResolvedValueOnce({ ok: true }).mockResolvedValueOnce({ + ok: false, + error: { kind: "command", reason: "profile_incompatible", message: "collision" }, + }); + await expect( + provider.ensureNativeProviderAttached({ + adapter: h.input.adapter, + target: { kind: "named", gatewayName: "gateway" }, + sandboxName: "selected", + expected: receipt, + }), + ).rejects.toThrow("conflicts with NemoClaw's checked-in security boundary"); + expect(h.adapter.attachProvider).toHaveBeenCalledOnce(); + expect(h.adapter.detachProvider).toHaveBeenCalledOnce(); + expect(h.attachments.get("selected")).toEqual([]); + expect(h.attachments.get("other")).toEqual(["unrelated-provider"]); + }); + it("rejects profile collisions before provider or attachment changes", async () => { const h = harness(); h.adapter.importProviderProfile.mockResolvedValue({ @@ -395,28 +494,6 @@ describe("custom native endpoint ownership", () => { expect( usesCustomNativeOnboarding(binding.provider, "http://127.0.0.1:8080/v1", "alpha", null), ).toBe(false); - // OpenShell rejects Personal's hostless web rule beside a port-443 provider endpoint. - expect( - usesCustomNativeOnboarding( - binding.provider, - binding.endpointUrl, - "alpha", - null, - " Personal ", - ), - ).toBe(false); - expect( - usesCustomNativeOnboarding( - binding.provider, - binding.endpointUrl, - "alpha", - { pendingRouteReservation: true }, - "personal", - ), - ).toBe(false); - expect( - usesCustomNativeOnboarding(binding.provider, binding.endpointUrl, "alpha", null, "balanced"), - ).toBe(true); const h = harness(); const receipt = await prepareCustomNativeProvider(h.input); expect( diff --git a/src/lib/inference/native-provider/custom.ts b/src/lib/inference/native-provider/custom.ts index d5a990ea77b..88d432ba1ad 100644 --- a/src/lib/inference/native-provider/custom.ts +++ b/src/lib/inference/native-provider/custom.ts @@ -14,7 +14,11 @@ import { import { normalizeProviderBaseUrl } from "../../core/provider-endpoint"; import type { NativeProviderDefinition } from "./contract"; import { nativeHostedAgentConfig } from "./agent-config"; -import { isPrivateIp, isLoopbackHostname } from "../../private-networks"; +import { + isPrivateIp, + isLoopbackHostname, + isAllowedOpenShellSandboxBridgeUrl, +} from "../../private-networks"; import { isOperatorTrustablePrivateIp, isOpenShellManagedHost, @@ -36,8 +40,8 @@ export type CustomNativeBinding = Readonly<{ adapterEndpointUrl?: string; }>; -/** Hosted selections exclude the existing loopback and managed bridge routes. */ -export function isHostedCustomEndpoint( +/** Native custom routes retain the existing sandbox bridge admission boundary. */ +export function isCustomNativeEndpoint( provider: string, endpointUrl: string | null | undefined, ): boolean { @@ -47,7 +51,11 @@ export function isHostedCustomEndpoint( ) return false; const host = new URL(endpointUrl).hostname; - return !isLoopbackHostname(host) && !isOpenShellManagedHost(host); + return ( + (provider === "compatible-anthropic-endpoint" && + isAllowedOpenShellSandboxBridgeUrl(new URL(endpointUrl))) || + (!isLoopbackHostname(host) && !isOpenShellManagedHost(host)) + ); } export function isCustomNativeProviderName(name: string): boolean { @@ -166,7 +174,6 @@ export function boundCustomNativeProfile( port: Number(endpoint.port || (endpoint.protocol === "https:" ? 443 : 80)), ...(expected.allowedIps ? { allowed_ips: expected.allowedIps } : {}), protocol: "rest", - ...(endpoint.protocol === "https:" ? { tls: "terminate" } : {}), enforcement: "enforce", rules: [ { allow: { method: "GET", path: `${apiPrefix}/models` } }, diff --git a/src/lib/inference/native-provider/hermes-binding.test.ts b/src/lib/inference/native-provider/hermes-binding.test.ts index 9879e93ff6a..86c1c8cb4f9 100644 --- a/src/lib/inference/native-provider/hermes-binding.test.ts +++ b/src/lib/inference/native-provider/hermes-binding.test.ts @@ -2,7 +2,11 @@ // SPDX-License-Identifier: Apache-2.0 import fs from "node:fs"; -import { describe, expect, it, vi } from "vitest"; +import { createHash } from "node:crypto"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it, vi, onTestFinished } from "vitest"; +import { nativeProviderLifecycle } from "./index"; import YAML from "yaml"; import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; import { hostedNativeProvider } from "./hosted"; @@ -98,61 +102,96 @@ describe("Hermes authenticated endpoint binding", () => { expect(importProviderProfile).not.toHaveBeenCalled(); }); - it("passes the API key only to the restricted provider and keeps returned state credential-free", async () => { - let present = false; - const createProvider = vi.fn(async () => { - present = true; - return { ok: true }; - }); - const importProviderProfile = vi.fn( - async ({ profilePath }) => { - expect(YAML.parse(fs.readFileSync(profilePath, "utf8")).id).toBe(definition.profileId); - expect(YAML.parse(fs.readFileSync(profilePath, "utf8")).endpoints[0].allowed_ips).toEqual([ - "8.8.8.8", - ]); + it.each([ + { endpoint, retainedHermesEndpoint: false, allowedIps: ["8.8.8.8"] }, + { + endpoint: "http://retained.example/team/v1", + retainedHermesEndpoint: true, + allowedIps: ["8.8.8.8"], + }, + { endpoint: "https://8.8.8.8/team/v1", retainedHermesEndpoint: true, allowedIps: ["8.8.8.8"] }, + { + endpoint: "https://[2001:4860:4860::8888]/team/v1", + retainedHermesEndpoint: true, + allowedIps: ["2001:4860:4860::8888"], + }, + { + endpoint: "http://host.openshell.internal:42103/team/v1", + retainedHermesEndpoint: true, + allowedIps: undefined, + }, + ])( + "passes the API key only to the restricted $endpoint provider", + async ({ endpoint, retainedHermesEndpoint, allowedIps }) => { + const definition = { ...hostedNativeProvider("hermes-provider", endpoint)!, allowedIps }; + const receipt = { + schemaVersion: 1 as const, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "identity", + endpointUrl: endpoint, + ...(allowedIps ? { allowedIps } : {}), + }; + let present = false; + const createProvider = vi.fn(async () => { + present = true; return { ok: true }; - }, - ); - const adapter = { - createProvider, - importProviderProfile, - ensureProviderPolicyComposition: async () => ({ ok: true, value: undefined }), - getProvider: async () => - present - ? { - ok: true, - value: { - name: definition.providerName, - type: definition.profileId, - credentialKeys: ["OPENAI_API_KEY"], - configKeys: [], - revision: { id: "identity", resourceVersion: 1 }, - }, - } - : { ok: false, error: { kind: "command", reason: "not_found", message: "absent" } }, - } as unknown as OpenShellProviderAdapter; - const writeAuthority = vi.fn(); - const result = await prepareHostedNativeProvider({ - provider: "hermes-provider", - gatewayName: "gateway", - endpointUrl: endpoint, - credentialValue: "host-secret", - lookup: async () => [{ address: "8.8.8.8", family: 4 }], - adapter, - readAuthority: () => undefined, - writeAuthority, - }); - expect(result).toEqual(receipt); - expect(createProvider).toHaveBeenCalledWith( - expect.objectContaining({ - name: definition.providerName, - credentials: [{ name: "OPENAI_API_KEY", value: "host-secret" }], - config: [], - }), - ); - expect(JSON.stringify(writeAuthority.mock.calls)).not.toContain("host-secret"); - expect(fs.existsSync(importProviderProfile.mock.calls[0][0].profilePath)).toBe(false); - }); + }); + const importProviderProfile = vi.fn( + async ({ profilePath }) => { + expect(YAML.parse(fs.readFileSync(profilePath, "utf8")).id).toBe(definition.profileId); + expect(YAML.parse(fs.readFileSync(profilePath, "utf8")).endpoints[0].allowed_ips).toEqual( + allowedIps, + ); + return { ok: true }; + }, + ); + const adapter = { + createProvider, + importProviderProfile, + ensureProviderPolicyComposition: async () => ({ ok: true, value: undefined }), + getProvider: async () => + present + ? { + ok: true, + value: { + name: definition.providerName, + type: definition.profileId, + credentialKeys: ["OPENAI_API_KEY"], + configKeys: [], + revision: { id: "identity", resourceVersion: 1 }, + }, + } + : { ok: false, error: { kind: "command", reason: "not_found", message: "absent" } }, + } as unknown as OpenShellProviderAdapter; + const writeAuthority = vi.fn(); + const result = await prepareHostedNativeProvider({ + provider: "hermes-provider", + gatewayName: "gateway", + endpointUrl: endpoint, + retainedHermesEndpoint, + credentialValue: "host-secret", + lookup: async () => [{ address: "8.8.8.8", family: 4 }], + adapter, + readAuthority: () => undefined, + writeAuthority, + }); + expect(result).toEqual(receipt); + expect(requireHostedProviderAttachment(result, "hermes-provider")).toEqual(result); + expect(nativeHostedAgentConfig("hermes-provider", endpoint)?.credentialEnv).toBe( + "OPENAI_API_KEY", + ); + expect(createProvider).toHaveBeenCalledWith( + expect.objectContaining({ + name: definition.providerName, + credentials: [{ name: "OPENAI_API_KEY", value: "host-secret" }], + config: [], + }), + ); + expect(JSON.stringify(writeAuthority.mock.calls)).not.toContain("host-secret"); + expect(fs.existsSync(importProviderProfile.mock.calls[0][0].profilePath)).toBe(false); + }, + ); it("keeps approved pins on restart instead of accepting rebound DNS", async () => { const lookup = vi.fn(async () => [{ address: "10.0.0.1", family: 4 }]); @@ -212,3 +251,127 @@ describe("Hermes authenticated endpoint binding", () => { expect(fs.existsSync(importedPath)).toBe(false); }); }); + +it.each([ + "http://127.0.0.1:8000/v1", + "http://10.0.0.1:8000/v1", + "http://169.254.169.254/v1", + "http://other.internal:42103/v1", + "https://host.openshell.internal:42103/v1", + "http://host.openshell.internal:80/v1", + "http://host.openshell.internal:8080/v1", + "http://host.openshell.internal:4000/v1", + "http://host.openshell.internal:11435/v1", +])("rejects retained Hermes endpoint %s before credential registration", async (endpointUrl) => { + const importProviderProfile = vi.fn(); + await expect( + prepareHostedNativeProvider({ + provider: "hermes-provider", + gatewayName: "gateway", + endpointUrl, + retainedHermesEndpoint: true, + credentialValue: "host-secret", + adapter: { importProviderProfile } as unknown as OpenShellProviderAdapter, + readAuthority: () => undefined, + writeAuthority: vi.fn(), + }), + ).rejects.toThrow(); + expect(importProviderProfile).not.toHaveBeenCalled(); +}); + +it.each([8080, 4000, 11435])( + "rejects a protected Hermes profile port %s for every lifecycle operation", + (port) => { + const definition = hostedNativeProvider( + "hermes-provider", + `http://host.openshell.internal:${port}/v1`, + )!; + expect(() => boundHermesNativeProfile(definition)).toThrow("protected host port"); + }, +); + +it("normalizes the recorded host alias before binding its runtime destination", () => { + const definition = hostedNativeProvider( + "hermes-provider", + "http://host.openshell.internal.:42103/team/v1", + )!; + expect(definition.endpoint).toBe("http://host.openshell.internal:42103/team/v1"); + expect(YAML.parse(boundHermesNativeProfile(definition)).endpoints[0].host).toBe( + "host.openshell.internal", + ); +}); + +it.each([ + { service: "gateway", directory: ".nemoclaw/gateways/42103", routerPort: null }, + { service: "Model Router", directory: ".nemoclaw", routerPort: 42103 }, +])( + "rejects retained Hermes access after its port is reserved for $service", + async ({ directory, routerPort }) => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-reserved-port-")); + onTestFinished(() => fs.rmSync(home, { recursive: true, force: true })); + vi.stubEnv("HOME", home); + vi.stubEnv("NEMOCLAW_GATEWAY_PORT", ""); + vi.stubEnv("NEMOCLAW_MODEL_ROUTER_PORT", ""); + const endpointUrl = "http://host.openshell.internal:42103/team/v1"; + const definition = hostedNativeProvider("hermes-provider", endpointUrl)!; + const expected = { + schemaVersion: 1 as const, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "retained-identity", + endpointUrl, + }; + expect(YAML.parse(boundHermesNativeProfile(definition)).endpoints[0].port).toBe(42103); + const root = path.join(home, directory); + fs.mkdirSync(root, { recursive: true }); + fs.writeFileSync( + path.join(root, "onboard-session.json"), + JSON.stringify({ routerPort, routerPid: null, routerCredentialHash: null }), + ); + const importProviderProfile = vi.fn(); + const input = { + adapter: { importProviderProfile } as unknown as OpenShellProviderAdapter, + target: { kind: "named" as const, gatewayName: "gateway" }, + sandboxName: "sandbox", + expected, + }; + const lifecycle = nativeProviderLifecycle(definition); + await expect(lifecycle.verifyNativeProviderAttachment(input)).rejects.toThrow( + "protected host port", + ); + await expect(lifecycle.ensureNativeProviderAttached(input)).rejects.toThrow( + "protected host port", + ); + expect(() => buildNativeHostedSandboxPolicy("{}", definition.providerName, expected)).toThrow( + "protected host port", + ); + expect(importProviderProfile).not.toHaveBeenCalled(); + }, +); + +it.each([ + "http://127.0.0.1:8000/v1", + "http://10.0.0.1:8000/v1", + "http://169.254.169.254/v1", + "http://[::1]:8000/v1", + "http://[::ffff:127.0.0.1]:8000/v1", + "http://other.internal:42103/v1", + "http://host.openshell.internal:8080/v1", +])("cannot turn image configuration for %s into an attached destination", (endpointUrl) => { + // Environment-driven image config is public data, not authority to register a route. + expect(nativeHostedAgentConfig("hermes-provider", endpointUrl)?.apiKey).toBe( + "openshell:resolve:env:OPENAI_API_KEY", + ); + const suffix = createHash("sha256").update(endpointUrl).digest("hex").slice(0, 20); + const forged = { + schemaVersion: 1 as const, + profileId: `nemoclaw-hermes-inference-${suffix}-v1`, + providerName: `nemoclaw-hermes-${suffix}-v1`, + providerId: "forged-identity", + endpointUrl, + allowedIps: ["8.8.8.8"], + }; + expect(() => requireHostedProviderAttachment(forged, "hermes-provider")).toThrow(); + expect(() => buildNativeHostedSandboxPolicy("{}", forged.providerName, forged)).toThrow(); + expect(() => boundHermesNativeProfile({ ...definition, ...forged })).toThrow(); +}); diff --git a/src/lib/inference/native-provider/hermes-pins.ts b/src/lib/inference/native-provider/hermes-pins.ts index ec1a75d5969..226ccb3b1bc 100644 --- a/src/lib/inference/native-provider/hermes-pins.ts +++ b/src/lib/inference/native-provider/hermes-pins.ts @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { isIP } from "node:net"; -import { isPrivateIp } from "../../private-networks"; +import { isPrivateIp, isAllowedOpenShellSandboxBridgeUrl } from "../../private-networks"; /** Durable Hermes policy accepts exact public IPs, never CIDRs or private ranges. */ export function requireHermesPublicPins(value: unknown): readonly string[] { @@ -14,3 +14,16 @@ export function requireHermesPublicPins(value: unknown): readonly string[] { throw new Error("Hermes endpoint requires recorded public IP addresses"); return [...new Set(value as string[])].sort(); } + +/** The reserved alias is pinned by OpenShell; public endpoints retain exact public pins. */ +export function requireHermesEndpointPins( + endpointUrl: string, + value: unknown, +): readonly string[] | undefined { + if (isAllowedOpenShellSandboxBridgeUrl(new URL(endpointUrl))) { + if (value !== undefined) + throw new Error("Hermes host alias cannot override the runtime destination"); + return undefined; + } + return requireHermesPublicPins(value); +} diff --git a/src/lib/inference/native-provider/hermes-profile.ts b/src/lib/inference/native-provider/hermes-profile.ts index e8eccb7e17f..7365d414e78 100644 --- a/src/lib/inference/native-provider/hermes-profile.ts +++ b/src/lib/inference/native-provider/hermes-profile.ts @@ -7,10 +7,11 @@ import path from "node:path"; import YAML from "yaml"; import { REPOSITORY_ROOT } from "../../core/repository-root"; import type { NativeProviderDefinition } from "./contract"; -import { requireHermesPublicPins } from "./hermes-pins"; +import { requireHermesEndpointPins } from "./hermes-pins"; import { hostedNativeProvider } from "./hosted"; +import { isAllowedOpenShellSandboxBridgeUrl } from "../../private-networks"; -/** Reuse the checked-in Hermes boundary, replacing only its authenticated destination. */ +/** Reuse the checked-in Hermes boundary for its recorded endpoint. */ export function boundHermesNativeProfile(definition: NativeProviderDefinition): string { const expected = hostedNativeProvider("hermes-provider", definition.endpointUrl); if ( @@ -30,13 +31,21 @@ export function boundHermesNativeProfile(definition: NativeProviderDefinition): ), ); const endpoint = new URL(expected.endpoint); + if (isAllowedOpenShellSandboxBridgeUrl(endpoint)) { + // Load host port ownership only when validating a host route; it initializes registry locks. + const { isProtectedLocalInferencePort } = + require("../gateway/local-endpoint-ports") as typeof import("../gateway/local-endpoint-ports"); + if (isProtectedLocalInferencePort(Number(endpoint.port))) + throw new Error("The recorded Hermes endpoint uses a protected host port"); + } const prefix = endpoint.pathname.replace(/\/+$/, ""); + const allowedIps = requireHermesEndpointPins(expected.endpoint, definition.allowedIps); profile.id = expected.profileId; profile.endpoints = [ { - host: endpoint.hostname, - port: Number(endpoint.port || 443), - allowed_ips: requireHermesPublicPins(definition.allowedIps), + host: endpoint.hostname.replace(/^\[|\]$/gu, ""), + port: Number(endpoint.port || (endpoint.protocol === "http:" ? 80 : 443)), + ...(allowedIps ? { allowed_ips: allowedIps } : {}), protocol: "rest", enforcement: "enforce", rules: [ diff --git a/src/lib/inference/native-provider/hosted-attachment.ts b/src/lib/inference/native-provider/hosted-attachment.ts index 854130a919f..5f2eb35cac0 100644 --- a/src/lib/inference/native-provider/hosted-attachment.ts +++ b/src/lib/inference/native-provider/hosted-attachment.ts @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { requireHermesPublicPins } from "./hermes-pins"; +import { requireHermesEndpointPins } from "./hermes-pins"; import { customNativeProvider, isCustomNativeProviderName } from "./custom"; import { normalizeNativeProviderAttachment, type NativeProviderAttachment } from "./contract"; import { @@ -33,7 +33,10 @@ export function hostedNativeProviderForAttachment( : undefined, )!; if (definition.endpointUrl) { - definition = { ...definition, allowedIps: requireHermesPublicPins(receipt.allowedIps) }; + definition = { + ...definition, + allowedIps: requireHermesEndpointPins(definition.endpointUrl, receipt.allowedIps), + }; } } catch { continue; diff --git a/src/lib/inference/native-provider/hosted.test.ts b/src/lib/inference/native-provider/hosted.test.ts index 420155dc2f6..5a046005d6d 100644 --- a/src/lib/inference/native-provider/hosted.test.ts +++ b/src/lib/inference/native-provider/hosted.test.ts @@ -106,6 +106,7 @@ describe.each(HOSTED_NATIVE_PROVIDERS)("native $label lifecycle", (definition) = "/usr/bin/node", "/opt/hermes/.venv/bin/python", "/opt/hermes/.venv/bin/python3", + "/usr/bin/python3.13", "/opt/venv/bin/python3", "/usr/local/bin/curl", "/usr/bin/curl", diff --git a/src/lib/inference/native-provider/hosted.ts b/src/lib/inference/native-provider/hosted.ts index f796637f6a1..29c98b219e6 100644 --- a/src/lib/inference/native-provider/hosted.ts +++ b/src/lib/inference/native-provider/hosted.ts @@ -2,86 +2,21 @@ // SPDX-License-Identifier: Apache-2.0 import { createHash } from "node:crypto"; +import { isAllowedOpenShellSandboxBridgeUrl, isPrivateHostname } from "../../private-networks"; +import { + fixedHostedNativeProvider, + recordedHermesEndpointUrl, + type HostedProviderDefinition, +} from "./catalog.ts"; +export { + HOSTED_NATIVE_PROVIDERS, + customNativeCredentialEnv, + isCustomHostedProvider, + resolveCustomNativeApi, + type HostedProviderDefinition, +} from "./catalog.ts"; -import { type NativeProviderAttachment, type NativeProviderDefinition } from "./contract.ts"; - -export function isCustomHostedProvider(provider: string | null | undefined): boolean { - return provider === "compatible-endpoint" || provider === "compatible-anthropic-endpoint"; -} - -/** Resolve the native binding and agent configuration from the same API selection. */ -export function resolveCustomNativeApi( - provider: string | null | undefined, - preferredApi: string | null | undefined, - nativeProvider: boolean, -): string | null { - if (!nativeProvider || !isCustomHostedProvider(provider)) return preferredApi ?? null; - return ( - preferredApi || - (provider === "compatible-anthropic-endpoint" ? "anthropic-messages" : "openai-completions") - ); -} - -export function customNativeCredentialEnv(provider: string): string { - if (provider === "compatible-endpoint") return "COMPATIBLE_API_KEY"; - if (provider === "compatible-anthropic-endpoint") return "COMPATIBLE_ANTHROPIC_API_KEY"; - throw new Error("Unsupported custom hosted provider"); -} - -export type HostedProviderDefinition = NativeProviderDefinition & - Readonly<{ - endpoint: string; - api: "openai-completions" | "openai-responses" | "anthropic-messages"; - }>; - -/** Existing fixed hosted selections only; custom and local endpoints have separate owners. */ -export const HOSTED_NATIVE_PROVIDERS = [ - { - logicalProvider: "openai-api", - label: "OpenAI", - profileId: "nemoclaw-openai-inference-v1", - providerName: "nemoclaw-openai-api-v1", - credentialEnv: "OPENAI_API_KEY", - endpoint: "https://api.openai.com/v1", - api: "openai-completions", - }, - { - logicalProvider: "anthropic-prod", - label: "Anthropic", - profileId: "nemoclaw-anthropic-inference-v1", - providerName: "nemoclaw-anthropic-prod-v1", - credentialEnv: "ANTHROPIC_API_KEY", - endpoint: "https://api.anthropic.com", - api: "anthropic-messages", - }, - { - logicalProvider: "gemini-api", - label: "Gemini", - profileId: "nemoclaw-gemini-inference-v1", - providerName: "nemoclaw-gemini-api-v1", - credentialEnv: "GEMINI_API_KEY", - endpoint: "https://generativelanguage.googleapis.com/v1beta/openai/", - api: "openai-completions", - }, - { - logicalProvider: "openrouter-api", - label: "OpenRouter", - profileId: "nemoclaw-openrouter-inference-v1", - providerName: "nemoclaw-openrouter-api-v1", - credentialEnv: "OPENROUTER_API_KEY", - endpoint: "https://openrouter.ai/api/v1", - api: "openai-completions", - }, - { - logicalProvider: "hermes-provider", - label: "Hermes Provider", - profileId: "nemoclaw-hermes-inference-v1", - providerName: "nemoclaw-hermes-provider-v1", - credentialEnv: "OPENAI_API_KEY", - endpoint: "https://inference-api.nousresearch.com/v1", - api: "openai-completions", - }, -] as const satisfies readonly HostedProviderDefinition[]; +import { type NativeProviderAttachment } from "./contract.ts"; /** Only the authenticated Hermes provider may bind a returned endpoint. */ export function canonicalHermesNativeEndpoint(value: string): string { @@ -105,16 +40,23 @@ export function canonicalHermesNativeEndpoint(value: string): string { return `${endpoint.origin}${endpoint.pathname.replace(/\/+$/, "")}`; } +/** Restore an existing Hermes endpoint without admitting other private destinations. */ +export function canonicalRecordedHermesEndpoint(value: string): string { + const endpoint = recordedHermesEndpointUrl(value); + if (isPrivateHostname(endpoint.hostname) && !isAllowedOpenShellSandboxBridgeUrl(endpoint)) + throw new Error("Recorded Hermes inference endpoint is not an admitted HTTP(S) destination"); + if (isAllowedOpenShellSandboxBridgeUrl(endpoint)) endpoint.hostname = "host.openshell.internal"; + return `${endpoint.origin}${endpoint.pathname.replace(/\/+$/, "")}`; +} + export function hostedNativeProvider( provider: string | null | undefined, endpointUrl?: string | null, ): HostedProviderDefinition | undefined { - const definition = HOSTED_NATIVE_PROVIDERS.find( - (entry) => entry.logicalProvider === provider?.trim(), - ); + const definition = fixedHostedNativeProvider(provider); if (!definition || definition.logicalProvider !== "hermes-provider" || !endpointUrl) return definition; - const endpoint = canonicalHermesNativeEndpoint(endpointUrl); + const endpoint = canonicalRecordedHermesEndpoint(endpointUrl); if (endpoint === definition.endpoint) return definition; const suffix = createHash("sha256").update(endpoint).digest("hex").slice(0, 20); return { @@ -126,12 +68,12 @@ export function hostedNativeProvider( }; } -/** Keep explicit user-supplied Hermes endpoints on their existing workflow. */ +/** Existing Hermes endpoints now use an endpoint-bound provider. */ export function usesNativeHermesEndpoint( endpoint: string | null | undefined, - authority?: NativeProviderAttachment, + _authority?: NativeProviderAttachment, ): boolean { - const fixed = hostedNativeProvider("hermes-provider")!; - const normalized = endpoint?.replace(/\/+$/, ""); - return !normalized || normalized === fixed.endpoint || normalized === authority?.endpointUrl; + if (!endpoint) return true; + canonicalRecordedHermesEndpoint(endpoint); + return true; } diff --git a/src/lib/inference/native-provider/index.ts b/src/lib/inference/native-provider/index.ts index 89547d65359..7cfc1def02a 100644 --- a/src/lib/inference/native-provider/index.ts +++ b/src/lib/inference/native-provider/index.ts @@ -8,26 +8,20 @@ import { withCustomNativeProfile } from "./custom"; import type { OpenShellGatewayTarget, OpenShellProviderAdapter, - OpenShellProviderError, OpenShellProviderMetadata, } from "../../adapters/openshell/sandbox-observer"; import { REPOSITORY_ROOT } from "../../core/repository-root"; -import { isValidName } from "../../sandbox-name-contract"; -import { unownedNativeProviderMessage } from "./lifecycle"; import { normalizeNativeProviderAttachment, type NativeProviderAttachment, type NativeProviderDefinition, } from "./contract"; -export class NativeProviderError extends Error { - constructor(message: string) { - super(message); - this.name = "NativeProviderError"; - } -} +import * as lifecycle from "./lifecycle"; +import { NativeProviderError } from "./lifecycle"; +export { NativeProviderError } from "./lifecycle"; -/** The shared lifecycle for NemoClaw's fixed hosted provider profiles. */ +/** Bind hosted profile materialization and receipts to the shared provider lifecycle. */ export class NativeProviderLifecycle { constructor(readonly definition: NativeProviderDefinition) {} private requireExpectedReceipt(receipt: NativeProviderAttachment): void { @@ -47,6 +41,34 @@ export class NativeProviderLifecycle( + operation: (profile: lifecycle.NativeProviderProfile) => Promise, + profilePath = this.nativeProviderProfilePath(), + ): Promise { + const run = (boundPath: string) => operation(this.profile(boundPath)); + if (this.definition.custom) return withCustomNativeProfile(this.definition, run, profilePath); + if (this.definition.endpointUrl) return withHermesNativeProfile(this.definition, run); + return run(profilePath); + } + + private bindReceipt( + receipt: lifecycle.NativeProviderAttachment, + ): NativeProviderAttachment { + return { + schemaVersion: 1, + profileId: this.definition.profileId, + providerName: this.definition.providerName, + providerId: receipt.providerId, + ...(this.definition.endpointUrl ? { endpointUrl: this.definition.endpointUrl } : {}), + ...(this.definition.allowedIps ? { allowedIps: [...this.definition.allowedIps] } : {}), + ...(this.definition.custom ? { custom: this.definition.custom } : {}), + }; + } + isNativeProvider(provider: string | null | undefined): boolean { return provider?.trim() === this.definition.logicalProvider; } @@ -76,126 +98,30 @@ export class NativeProviderLifecycle { - const imported = this.definition.custom - ? await withCustomNativeProfile( - this.definition, - (boundPath) => adapter.importProviderProfile({ target, profilePath: boundPath }), - profilePath, - ) - : this.definition.endpointUrl - ? await withHermesNativeProfile(this.definition, (boundPath) => - adapter.importProviderProfile({ target, profilePath: boundPath }), - ) - : await adapter.importProviderProfile({ target, profilePath }); - if (imported.ok) return; - const collision = - imported.error.kind === "command" && imported.error.reason === "profile_incompatible"; - throw new NativeProviderError( - collision - ? `OpenShell provider profile '${this.definition.profileId}' conflicts with NemoClaw's checked-in security boundary. No provider was changed.` - : `Could not verify OpenShell provider profile '${this.definition.profileId}': ${this.providerErrorDetail(imported.error)}`, - ); - } - - exactNativeProvider(metadata: OpenShellProviderMetadata): boolean { - return ( - metadata.name === this.definition.providerName && - metadata.type === this.definition.profileId && - metadata.configKeys.length === 0 && - metadata.credentialKeys.length === 1 && - metadata.credentialKeys[0] === this.definition.credentialEnv && - Boolean(metadata.revision?.id) + await this.withProfile( + (profile) => lifecycle.requireNativeProviderProfileBoundary(adapter, target, profile), + profilePath, ); } nativeProviderAttachmentFromMetadata( metadata: OpenShellProviderMetadata, ): NativeProviderAttachment { - if (!this.exactNativeProvider(metadata) || !metadata.revision) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' does not match the NemoClaw-owned ${this.definition.label} inference boundary. No provider was changed.`, - ); - } - return { - schemaVersion: 1, - profileId: this.definition.profileId, - providerName: this.definition.providerName, - providerId: metadata.revision.id, - ...(this.definition.endpointUrl ? { endpointUrl: this.definition.endpointUrl } : {}), - ...(this.definition.allowedIps ? { allowedIps: [...this.definition.allowedIps] } : {}), - ...(this.definition.custom ? { custom: this.definition.custom } : {}), - }; + return this.bindReceipt( + lifecycle.nativeProviderAttachmentFromMetadata(metadata, this.profile()), + ); } async inspectNativeProvider( adapter: OpenShellProviderAdapter, target: OpenShellGatewayTarget, ): Promise { - const observed = await adapter.getProvider({ - target, - providerName: this.definition.providerName, - }); - if (observed.ok) return observed.value; - if (observed.error.kind === "command" && observed.error.reason === "not_found") return null; - throw new NativeProviderError( - `Could not inspect OpenShell provider '${this.definition.providerName}': ${this.providerErrorDetail(observed.error)}`, - ); - } - - mutationOutcomeMayBeAmbiguous(error: OpenShellProviderError): boolean { - return ( - error.kind === "timeout" || - (error.kind === "transport" && - (error.reason === "connection_loss" || error.reason === "unreachable")) || - (error.kind === "command" && error.reason === "uncertain") - ); - } - - authorityPersistenceRecovery(gatewayName: string): string { - return `Run 'nemoclaw credentials reset ${this.definition.custom ? this.definition.providerName : this.definition.logicalProvider} --yes' against gateway '${gatewayName}', then retry.`; - } - - async removeNewNativeProvider(input: { - adapter: OpenShellProviderAdapter; - target: OpenShellGatewayTarget; - expected: NativeProviderAttachment; - }): Promise { - this.requireExpectedReceipt(input.expected); - const provider = await this.inspectNativeProvider(input.adapter, input.target); - if (!provider) return; - const current = this.nativeProviderAttachmentFromMetadata(provider); - if (current.providerId !== input.expected.providerId) { - throw new NativeProviderError( - `Refusing to remove OpenShell provider '${this.definition.providerName}' because its identity changed.`, - ); - } - const removed = await input.adapter.deleteProvider({ - target: input.target, - providerName: this.definition.providerName, - }); - if (!removed.ok && !this.mutationOutcomeMayBeAmbiguous(removed.error)) { - throw new NativeProviderError( - `Could not remove OpenShell provider '${this.definition.providerName}': ${this.providerErrorDetail(removed.error)}`, - ); - } - const after = await this.inspectNativeProvider(input.adapter, input.target); - if (after) { - const observed = this.nativeProviderAttachmentFromMetadata(after); - const identity = - observed.providerId === input.expected.providerId ? "still exists" : "changed identity"; - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' ${identity} after cleanup.`, - ); - } + return lifecycle.inspectNativeProvider(adapter, target, this.profile()); } /** Record provider authority or remove only the new, unreferenced provider. */ @@ -215,43 +141,12 @@ export class NativeProviderLifecycle { this.requireExpectedReceipt(input.receipt); if (input.existing) this.requireExpectedReceipt(input.existing); - try { - input.writeAuthority(input.gatewayName, input.receipt); - return; - } catch (writeError) { - const writeDetail = writeError instanceof Error ? writeError.message : String(writeError); - let observed: NativeProviderAttachment | undefined; - try { - observed = input.readAuthority(input.gatewayName); - } catch (readError) { - const readDetail = readError instanceof Error ? readError.message : String(readError); - throw new NativeProviderError( - `NemoClaw could not record or confirm ownership of OpenShell provider '${this.definition.providerName}' for gateway '${input.gatewayName}'. The provider was retained. ${this.authorityPersistenceRecovery(input.gatewayName)}\n Write failure: ${writeDetail}\n Read failure: ${readDetail}`, - ); - } - if (observed?.providerId === input.receipt.providerId) return; - if (input.existing || observed) { - throw new NativeProviderError( - `NemoClaw could not record ownership of OpenShell provider '${this.definition.providerName}' for gateway '${input.gatewayName}'. The provider was retained because this operation cannot prove that it is unreferenced. ${this.authorityPersistenceRecovery(input.gatewayName)}\n ${writeDetail}`, - ); - } - try { - await this.removeNewNativeProvider({ - adapter: input.adapter, - target: input.target, - expected: input.receipt, - }); - } catch (cleanupError) { - const cleanupDetail = - cleanupError instanceof Error ? cleanupError.message : String(cleanupError); - throw new NativeProviderError( - `NemoClaw could not record ownership of OpenShell provider '${this.definition.providerName}' for gateway '${input.gatewayName}', and cleanup did not complete. ${this.authorityPersistenceRecovery(input.gatewayName)}\n Write failure: ${writeDetail}\n Cleanup failure: ${cleanupDetail}`, - ); - } - throw new NativeProviderError( - `NemoClaw could not record ownership of OpenShell provider '${this.definition.providerName}' for gateway '${input.gatewayName}'. The newly created provider was removed. Retry the command.\n ${writeDetail}`, - ); - } + await lifecycle.persistNativeProviderAuthority({ + ...input, + profile: this.profile(), + recoveryGuidance: `Run 'nemoclaw credentials reset ${this.definition.custom ? this.definition.providerName : this.definition.logicalProvider} --yes' against gateway '${input.gatewayName}', then retry.`, + writeAuthority: (gatewayName) => input.writeAuthority(gatewayName, input.receipt), + }); } /** Ensure the restricted provider profile and provider without retrying an ambiguous mutation. */ @@ -266,109 +161,26 @@ export class NativeProviderLifecycle> { if (input.expected) this.requireExpectedReceipt(input.expected); - const { adapter, target } = input; - await this.requireNativeProviderProfileBoundary( - adapter, - target, - input.profilePath ?? this.nativeProviderProfilePath(), + return this.withProfile( + async (profile) => + this.bindReceipt( + await lifecycle.ensureNativeProvider({ + ...input, + profile, + profilePath: profile.profilePath, + prepareProviderPolicy: async () => { + const policy = await input.adapter.ensureProviderPolicyComposition({ + target: input.target, + }); + if (!policy.ok) + throw new NativeProviderError( + `Could not activate native ${this.definition.label} provider policy: ${lifecycle.providerErrorDetail(policy.error)}`, + ); + }, + }), + ), + input.profilePath, ); - - const activatePolicy = async () => { - const policy = await adapter.ensureProviderPolicyComposition({ target }); - if (!policy.ok) { - throw new NativeProviderError( - `Could not activate native ${this.definition.label} provider policy: ${this.providerErrorDetail(policy.error)}`, - ); - } - }; - - const before = await this.inspectNativeProvider(adapter, target); - if (before) { - const receipt = this.nativeProviderAttachmentFromMetadata(before); - if (!input.expected) { - throw new NativeProviderError( - unownedNativeProviderMessage(this.definition.providerName, target), - ); - } - if (input.expected.providerId !== receipt.providerId) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' changed identity. Recreate the sandbox before using native ${this.definition.label} inference. No provider was changed.`, - ); - } - await activatePolicy(); - if (!input.credentialValue) return receipt; - const updated = await adapter.updateProvider({ - target, - providerName: this.definition.providerName, - credentials: [{ name: this.definition.credentialEnv, value: input.credentialValue }], - config: [], - }); - if (!updated.ok && this.mutationOutcomeMayBeAmbiguous(updated.error)) { - throw new NativeProviderError( - `OpenShell did not confirm whether provider '${this.definition.providerName}' accepted its credential update. No provider receipt was recorded.`, - ); - } - if (!updated.ok) { - throw new NativeProviderError( - `Could not update OpenShell provider '${this.definition.providerName}': ${this.providerErrorDetail(updated.error)}`, - ); - } - const observed = await this.inspectNativeProvider(adapter, target); - if (!observed) { - throw new NativeProviderError( - `OpenShell did not confirm provider '${this.definition.providerName}' after its credential update.`, - ); - } - const observedReceipt = this.nativeProviderAttachmentFromMetadata(observed); - if (observedReceipt.providerId !== input.expected.providerId) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' changed identity during its credential update. No provider receipt was recorded.`, - ); - } - return observedReceipt; - } - - // The registry name is untrusted input; print it as a command only when it is a valid sandbox name. - const sandbox = input.recordedSandboxName; - if (input.expected && sandbox && isValidName(sandbox)) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' is missing, and only the NemoClaw registry record for sandbox '${sandbox}' shows that NemoClaw owned it. If that sandbox no longer exists on this gateway, run 'nemoclaw ${sandbox} destroy' to clear the stale record, then onboard again. Otherwise, recreate the sandbox before using native ${this.definition.label} inference. No provider was changed.`, - ); - } - if (input.expected) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' is missing. If this gateway has no sandboxes after a reinstall, complete uninstall without --keep-openshell, then onboard again. Otherwise, recreate the sandbox before using native ${this.definition.label} inference. No provider was changed.`, - ); - } - - if (!input.credentialValue && !input.reuseExistingCredential) { - throw new NativeProviderError( - `A host credential is required to create OpenShell provider '${this.definition.providerName}'.`, - ); - } - await activatePolicy(); - const created = await adapter.createProvider({ - target, - name: this.definition.providerName, - type: this.definition.profileId, - credentials: input.credentialValue - ? [{ name: this.definition.credentialEnv, value: input.credentialValue }] - : [], - config: [], - fromExisting: !input.credentialValue, - }); - if (!created.ok && !this.mutationOutcomeMayBeAmbiguous(created.error)) { - throw new NativeProviderError( - `Could not create OpenShell provider '${this.definition.providerName}': ${this.providerErrorDetail(created.error)}`, - ); - } - const observed = await this.inspectNativeProvider(adapter, target); - if (!observed) { - throw new NativeProviderError( - `OpenShell did not confirm provider '${this.definition.providerName}' after creation.`, - ); - } - return this.nativeProviderAttachmentFromMetadata(observed); } /** Prove that the exact NemoClaw-owned provider is attached to one sandbox. */ @@ -379,34 +191,14 @@ export class NativeProviderLifecycle; }): Promise> { if (input.expected) this.requireExpectedReceipt(input.expected); - await this.requireNativeProviderProfileBoundary(input.adapter, input.target); - const provider = await this.inspectNativeProvider(input.adapter, input.target); - if (!provider) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' is missing. Recreate the sandbox to restore native ${this.definition.label} inference.`, - ); - } - const receipt = this.nativeProviderAttachmentFromMetadata(provider); - if (input.expected && input.expected.providerId !== receipt.providerId) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' changed identity. Recreate the sandbox before using native ${this.definition.label} inference.`, - ); - } - const attachments = await input.adapter.listProviderAttachments({ - target: input.target, - sandboxName: input.sandboxName, - }); - if (!attachments.ok) { - throw new NativeProviderError( - `Could not inspect provider attachments for sandbox '${input.sandboxName}': ${this.providerErrorDetail(attachments.error)}`, - ); - } - if (!attachments.value.names.includes(this.definition.providerName)) { - throw new NativeProviderError( - `Sandbox '${input.sandboxName}' does not have its native ${this.definition.label} inference provider attached. Recreate the sandbox; NemoClaw does not migrate existing beta sandboxes automatically.`, - ); - } - return receipt; + return this.bindReceipt( + await lifecycle.verifyNativeProviderAttachment({ + ...input, + profile: this.profile(), + verifyProfileBoundary: () => + this.requireNativeProviderProfileBoundary(input.adapter, input.target), + }), + ); } /** Attach native provider access and reconcile an ambiguous command through observation. */ @@ -416,61 +208,14 @@ export class NativeProviderLifecycle; }): Promise<{ receipt: NativeProviderAttachment; changed: boolean }> { - if (input.expected) this.requireExpectedReceipt(input.expected); - await this.requireNativeProviderProfileBoundary(input.adapter, input.target); - const provider = await this.inspectNativeProvider(input.adapter, input.target); - if (!provider) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' is missing. Recreate the sandbox to restore native ${this.definition.label} inference.`, - ); - } - const receipt = this.nativeProviderAttachmentFromMetadata(provider); - if (receipt.providerId !== input.expected.providerId) { - throw new NativeProviderError( - `OpenShell provider '${this.definition.providerName}' changed identity. Recreate the sandbox before using native ${this.definition.label} inference.`, - ); - } - const before = await input.adapter.listProviderAttachments({ - target: input.target, - sandboxName: input.sandboxName, - }); - if (!before.ok) { - throw new NativeProviderError( - `Could not inspect provider attachments for sandbox '${input.sandboxName}': ${this.providerErrorDetail(before.error)}`, - ); - } - if (before.value.names.includes(this.definition.providerName)) { - return { - receipt: await this.verifyNativeProviderAttachment(input), - changed: false, - }; - } - const attached = await input.adapter.attachProvider({ - target: input.target, - sandboxName: input.sandboxName, - providerName: this.definition.providerName, + this.requireExpectedReceipt(input.expected); + const result = await lifecycle.ensureNativeProviderAttached({ + ...input, + profile: this.profile(), + verifyProfileBoundary: () => + this.requireNativeProviderProfileBoundary(input.adapter, input.target), }); - if (!attached.ok && !this.mutationOutcomeMayBeAmbiguous(attached.error)) { - throw new NativeProviderError( - `Could not attach native ${this.definition.label} provider to sandbox '${input.sandboxName}': ${this.providerErrorDetail(attached.error)}`, - ); - } - try { - return { - receipt: await this.verifyNativeProviderAttachment(input), - changed: true, - }; - } catch (error) { - try { - await this.detachNativeProvider(input); - } catch (detachError) { - const detail = error instanceof Error ? error.message : String(error); - const detachDetail = - detachError instanceof Error ? detachError.message : String(detachError); - throw new NativeProviderError(`${detail}\n ${detachDetail}`); - } - throw error; - } + return { ...result, receipt: this.bindReceipt(result.receipt) }; } /** Detach only the recorded native provider identity and prove absence. */ @@ -481,33 +226,7 @@ export class NativeProviderLifecycle; }): Promise { this.requireExpectedReceipt(input.expected); - const provider = await this.inspectNativeProvider(input.adapter, input.target); - if (!provider) return; - const current = this.nativeProviderAttachmentFromMetadata(provider); - if (current.providerId !== input.expected.providerId) { - throw new NativeProviderError( - `Refusing to detach OpenShell provider '${this.definition.providerName}' because its identity changed.`, - ); - } - const detached = await input.adapter.detachProvider({ - target: input.target, - sandboxName: input.sandboxName, - providerName: this.definition.providerName, - }); - if (!detached.ok && !this.mutationOutcomeMayBeAmbiguous(detached.error)) { - throw new NativeProviderError( - `Could not detach native ${this.definition.label} provider from sandbox '${input.sandboxName}': ${this.providerErrorDetail(detached.error)}`, - ); - } - const after = await input.adapter.listProviderAttachments({ - target: input.target, - sandboxName: input.sandboxName, - }); - if (!after.ok || after.value.names.includes(this.definition.providerName)) { - throw new NativeProviderError( - `OpenShell did not confirm removal of native provider access from sandbox '${input.sandboxName}'.`, - ); - } + await lifecycle.detachNativeProvider({ ...input, profile: this.profile() }); } } diff --git a/src/lib/inference/native-provider/lifecycle.ts b/src/lib/inference/native-provider/lifecycle.ts index 46cf91b03d9..6d82b9b95a3 100644 --- a/src/lib/inference/native-provider/lifecycle.ts +++ b/src/lib/inference/native-provider/lifecycle.ts @@ -7,6 +7,7 @@ import type { OpenShellProviderError, OpenShellProviderMetadata, } from "../../adapters/openshell/sandbox-observer"; +import { isValidName } from "../../sandbox-name-contract"; import { shellQuote } from "../../core/shell-quote"; export type NativeProviderAttachment = Readonly<{ @@ -18,7 +19,8 @@ export type NativeProviderAttachment = Readonly<{ export type NativeProviderProfile = Readonly<{ profileId: string; providerName: string; - credentialEnv: string; + /** Null only for an admitted profile declaring no credentials. */ + credentialEnv: string | null; label: string; profilePath: string; }>; @@ -65,7 +67,7 @@ function assertExpectedProfile( } } -function providerErrorDetail(error: OpenShellProviderError): string { +export function providerErrorDetail(error: OpenShellProviderError): string { return error.message.trim() || "OpenShell did not provide a diagnostic."; } @@ -77,8 +79,10 @@ function exactNativeProvider( metadata.name === profile.providerName && metadata.type === profile.profileId && metadata.configKeys.length === 0 && - metadata.credentialKeys.length === 1 && - metadata.credentialKeys[0] === profile.credentialEnv && + (profile.credentialEnv === null + ? metadata.credentialKeys.length === 0 + : metadata.credentialKeys.length === 1 && + metadata.credentialKeys[0] === profile.credentialEnv) && Boolean(metadata.revision?.id) ); } @@ -100,7 +104,7 @@ export function nativeProviderAttachmentFromMetadata( }; } -async function inspectNativeProvider( +export async function inspectNativeProvider( adapter: OpenShellProviderAdapter, target: OpenShellGatewayTarget, profile: NativeProviderProfile, @@ -215,6 +219,26 @@ export async function persistNativeProviderAuthority(input: { } } +/** Verify the imported profile before provider or attachment changes. */ +export async function requireNativeProviderProfileBoundary( + adapter: OpenShellProviderAdapter, + target: OpenShellGatewayTarget, + profile: NativeProviderProfile, +): Promise { + const imported = await adapter.importProviderProfile({ + target, + profilePath: profile.profilePath, + }); + if (imported.ok) return; + const collision = + imported.error.kind === "command" && imported.error.reason === "profile_incompatible"; + throw new NativeProviderError( + collision + ? `OpenShell provider profile '${profile.profileId}' conflicts with NemoClaw's checked-in security boundary. No provider was changed.` + : `Could not verify OpenShell provider profile '${profile.profileId}': ${providerErrorDetail(imported.error)}`, + ); +} + /** Ensure the least-privilege hosted profile and provider without retrying an ambiguous mutation. */ export async function ensureNativeProvider(input: { profile: NativeProviderProfile; @@ -225,23 +249,20 @@ export async function ensureNativeProvider(input: { /** Provider-owned policy preparation after identity and credential checks. */ prepareProviderPolicy?: () => Promise; expected?: NativeProviderAttachment; + /** Registry sandbox whose receipt is the only ownership evidence for `expected`. */ + recordedSandboxName?: string; profilePath?: string; }): Promise { const { adapter, target, profile } = input; + if (profile.credentialEnv === null && (input.credentialValue || input.reuseExistingCredential)) + throw new NativeProviderError( + "An anonymous native profile cannot receive or inherit credentials.", + ); assertExpectedProfile(profile, input.expected); - const imported = await adapter.importProviderProfile({ - target, + await requireNativeProviderProfileBoundary(adapter, target, { + ...profile, profilePath: input.profilePath ?? profile.profilePath, }); - if (!imported.ok) { - const collision = - imported.error.kind === "command" && imported.error.reason === "profile_incompatible"; - throw new NativeProviderError( - collision - ? `OpenShell provider profile '${profile.profileId}' conflicts with NemoClaw's checked-in security boundary. No provider was changed.` - : `Could not prepare OpenShell provider profile '${profile.profileId}': ${providerErrorDetail(imported.error)}`, - ); - } const before = await inspectNativeProvider(adapter, target, profile); if (before) { @@ -255,7 +276,7 @@ export async function ensureNativeProvider(input: { ); } await input.prepareProviderPolicy?.(); - if (!input.credentialValue) return receipt; + if (!input.credentialValue || profile.credentialEnv === null) return receipt; const updated = await adapter.updateProvider({ target, providerName: profile.providerName, @@ -287,13 +308,19 @@ export async function ensureNativeProvider(input: { return confirmed; } + const recordedSandboxName = input.recordedSandboxName; + if (input.expected && recordedSandboxName && isValidName(recordedSandboxName)) { + throw new NativeProviderError( + `OpenShell provider '${profile.providerName}' is missing, and only the NemoClaw registry record for sandbox '${recordedSandboxName}' shows that NemoClaw owned it. If that sandbox no longer exists on this gateway, run 'nemoclaw ${recordedSandboxName} destroy' to clear the stale record, then onboard again. Otherwise, recreate the sandbox before using native ${profile.label} inference. No provider was changed.`, + ); + } if (input.expected) { throw new NativeProviderError( - `OpenShell provider '${profile.providerName}' is missing. Recreate the sandbox before using native ${profile.label} inference. No provider was changed.`, + `OpenShell provider '${profile.providerName}' is missing. If this gateway has no sandboxes after a reinstall, complete uninstall without --keep-openshell, then onboard again. Otherwise, recreate the sandbox before using native ${profile.label} inference. No provider was changed.`, ); } - if (!input.credentialValue && !input.reuseExistingCredential) { + if (profile.credentialEnv !== null && !input.credentialValue && !input.reuseExistingCredential) { throw new NativeProviderError( `A host credential is required to create OpenShell provider '${profile.providerName}'.`, ); @@ -303,11 +330,12 @@ export async function ensureNativeProvider(input: { target, name: profile.providerName, type: profile.profileId, - credentials: input.credentialValue - ? [{ name: profile.credentialEnv, value: input.credentialValue }] - : [], + credentials: + input.credentialValue && profile.credentialEnv !== null + ? [{ name: profile.credentialEnv, value: input.credentialValue }] + : [], config: [], - fromExisting: !input.credentialValue, + fromExisting: profile.credentialEnv !== null && !input.credentialValue, }); if (!created.ok && !mutationOutcomeMayBeAmbiguous(created.error)) { throw new NativeProviderError( @@ -331,11 +359,14 @@ export async function verifyNativeProviderAttachment(input: { sandboxName: string; expected?: NativeProviderAttachment; requireUnexpiredCredential?: boolean; + /** Recheck a materialized hosted profile before observing its attachment. */ + verifyProfileBoundary?: () => Promise; }): Promise { const profile = input.profile; if (!profile) throw new NativeProviderError("Native inference requires a recorded provider identity."); assertExpectedProfile(profile, input.expected); + await input.verifyProfileBoundary?.(); const provider = await inspectNativeProvider( input.adapter, input.target, @@ -367,7 +398,7 @@ export async function verifyNativeProviderAttachment(input: { `Sandbox '${input.sandboxName}' does not have its native ${profile.label} inference provider attached. Recreate the sandbox; NemoClaw does not migrate existing beta sandboxes automatically.`, ); } - if (input.requireUnexpiredCredential) { + if (input.requireUnexpiredCredential && profile.credentialEnv !== null) { // Expiry inventory is read separately from provider metadata. Bind that // observation to the same immutable provider and resource version. const current = await inspectNativeProvider(input.adapter, input.target, profile); @@ -394,11 +425,14 @@ export async function ensureNativeProviderAttached(input: { target: OpenShellGatewayTarget; sandboxName: string; expected: NativeProviderAttachment; + /** Check before attaching and again during verification, with cleanup on failure. */ + verifyProfileBoundary?: () => Promise; }): Promise<{ receipt: NativeProviderAttachment; changed: boolean }> { const profile = input.profile; if (!profile) throw new NativeProviderError("Native inference requires a recorded provider identity."); assertExpectedProfile(profile, input.expected); + await input.verifyProfileBoundary?.(); const provider = await inspectNativeProvider(input.adapter, input.target, profile); if ( !provider || diff --git a/src/lib/inference/native-provider/network-policy.test.ts b/src/lib/inference/native-provider/network-policy.test.ts index d07b0c5f442..47adc5ef38c 100644 --- a/src/lib/inference/native-provider/network-policy.test.ts +++ b/src/lib/inference/native-provider/network-policy.test.ts @@ -24,6 +24,9 @@ describe.each(HOSTED_NATIVE_PROVIDERS)("$label sandbox policy", (provider) => { enforcement: "enforce", }), ]); + expect(parsed.network_policies.native_hosted_inference.binaries).toContainEqual({ + path: "/usr/bin/python3.13", + }); expect(buildNativeHostedSandboxPolicy(result, provider.providerName)).toBe(result); }); it("refuses an existing policy granting different provider access (#12589)", () => { diff --git a/src/lib/inference/native-provider/network-policy.ts b/src/lib/inference/native-provider/network-policy.ts index 9a34758706f..1f9e43197d5 100644 --- a/src/lib/inference/native-provider/network-policy.ts +++ b/src/lib/inference/native-provider/network-policy.ts @@ -10,7 +10,7 @@ import { assertOpenShellPolicyEndpointPins, parseOpenShellPolicy, } from "../../adapters/openshell/policy-boundary"; -import { ROOT } from "../../state/paths"; +import { REPOSITORY_ROOT } from "../../core/repository-root"; import { nativeProviderLifecycle } from "./index"; import { boundHermesNativeProfile } from "./hermes-profile"; import { boundCustomNativeProfile } from "./custom"; @@ -77,7 +77,7 @@ export function buildNativeHostedSandboxPolicy( if (usesAttachedCustomNvidiaPolicy(providerName, receipt, agentName)) { const bundled = parseOpenShellPolicy( fs.readFileSync( - path.join(ROOT, "nemoclaw-blueprint", "policies", "openclaw-sandbox.yaml"), + path.join(REPOSITORY_ROOT, "nemoclaw-blueprint", "policies", "openclaw-sandbox.yaml"), "utf8", ), ).policy.network_policies?.nvidia; diff --git a/src/lib/inference/native-provider/recorded-selection.ts b/src/lib/inference/native-provider/recorded-selection.ts index 2d6c667dea5..784e77bfbe4 100644 --- a/src/lib/inference/native-provider/recorded-selection.ts +++ b/src/lib/inference/native-provider/recorded-selection.ts @@ -26,6 +26,15 @@ export function recordedNativeProviderAttachment(selection: { selection.nativeHostedProviderAttachment, selection.provider, ); + if (hosted && selection.provider === "hermes-provider") { + const endpoint = hostedNativeProviderForAttachment(hosted)!.endpoint; + if ( + (selection.endpointUrl && + selection.endpointUrl.replace(/\/+$/, "") !== endpoint.replace(/\/+$/, "")) || + (selection.preferredInferenceApi && selection.preferredInferenceApi !== "openai-completions") + ) + throw new Error("Recorded Hermes selection does not match its endpoint or API authority"); + } if (hosted?.custom) { const binding = hosted.custom; const api = diff --git a/src/lib/inference/native-provider/registered.test.ts b/src/lib/inference/native-provider/registered.test.ts new file mode 100644 index 00000000000..59c485f2144 --- /dev/null +++ b/src/lib/inference/native-provider/registered.test.ts @@ -0,0 +1,394 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { readVerifiedRegisteredCredentialHandle } from "../../actions/inference-set/registered"; +import { inspectInferenceProviderProfile } from "../../adapters/openshell/provider-profile"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; +import { + attachRegisteredInferenceProvider, + detachRegisteredInferenceProvider, + recordedRegisteredInferenceProviderAttachment, + registeredInferenceProviderRevision, + resolveRegisteredInferenceProvider, + verifyRegisteredInferenceProviderAttachment, +} from "./registered"; + +describe("operator inference provider attachments", () => { + it.each([false, true])( + "preserves the provider while selecting an attachment (already attached: %s)", + async (alreadyAttached) => { + const f = registeredInferenceFixture(alreadyAttached); + const selected = await attachRegisteredInferenceProvider({ + adapter: f.adapter, + expected: f.binding, + }); + expect(selected.changed).toBe(!alreadyAttached); + expect(selected.attachment.attachedBySelection).toBe(!alreadyAttached); + await verifyRegisteredInferenceProviderAttachment({ + adapter: f.adapter, + expected: selected.attachment, + }); + await detachRegisteredInferenceProvider({ + adapter: f.adapter, + expected: selected.attachment, + }); + expect(f.state.attached).toBe(alreadyAttached); + expect(f.adapter.attachProvider).toHaveBeenCalledTimes(alreadyAttached ? 0 : 1); + expect(f.adapter.detachProvider).toHaveBeenCalledTimes(alreadyAttached ? 0 : 1); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + }, + ); + + it.each([ + ["provider identity", { providerId: "other-id" }], + ["profile identity", { profileId: "other-profile" }], + ["profile definition", { profileDigest: "b".repeat(64) }], + ["destination", { endpointUrl: "https://unrelated.example/v1" }], + ["credential environment", { credentialEnv: "UNRELATED_KEY" }], + ["API authentication", { inferenceApi: "anthropic-messages" as const }], + ])("rejects mismatched %s before attachment", async (_label, changed) => { + const f = registeredInferenceFixture(); + await expect( + attachRegisteredInferenceProvider({ + adapter: f.adapter, + expected: { ...f.binding, ...changed }, + }), + ).rejects.toThrow(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + }); + + it("removes only the new attachment when the provider revision changes during attachment", async () => { + const f = registeredInferenceFixture(); + f.adapter.attachProvider.mockImplementation(async () => { + f.state.attached = true; + f.state.revision++; + return { ok: true }; + }); + await expect( + attachRegisteredInferenceProvider({ adapter: f.adapter, expected: f.binding }), + ).rejects.toThrow("changed during attachment"); + expect(f.adapter.detachProvider).toHaveBeenCalledOnce(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.state.attached).toBe(false); + }); + + it("refuses to detach a replaced provider after attachment failure", async () => { + const f = registeredInferenceFixture(); + f.adapter.attachProvider.mockImplementation(async () => { + f.state.attached = true; + f.state.id = "replacement"; + return { ok: true }; + }); + await expect( + attachRegisteredInferenceProvider({ adapter: f.adapter, expected: f.binding }), + ).rejects.toThrow("identity changed"); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + }); + + it("accepts credential rotation while retaining the immutable binding", async () => { + const f = registeredInferenceFixture(true); + f.state.revision++; + await expect( + resolveRegisteredInferenceProvider({ ...f.binding, adapter: f.adapter, previous: f.binding }), + ).resolves.toEqual(f.binding); + }); +}); + +describe("registered provider profile authentication", () => { + it.each([ + ["**.example", "deep.api.example"], + ["api.*.example", "api.internal.example"], + ["*-api.example", "tenant-api.example"], + ["api?.example", "api1.example"], + ["api[0-9].example", "api1.example"], + ["api[!a-z].example", "api1.example"], + ])("rejects unsafe policy at matching host pattern %s", async (pattern, host) => { + const f = registeredInferenceFixture(); + f.profile.endpoints[0]!.host = host; + f.profile.endpoints.push({ ...f.profile.endpoints[0]!, host: pattern, enforcement: "audit" }); + await expect( + resolveRegisteredInferenceProvider({ + ...f.binding, + endpointUrl: `https://${host}/v1`, + adapter: f.adapter, + }), + ).rejects.toThrow("enforced REST policy"); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it("rejects unsafe policy whose port list includes the selected port", async () => { + const f = registeredInferenceFixture(); + f.profile.endpoints.push({ + ...f.profile.endpoints[0]!, + port: 8443, + enforcement: "audit", + ...{ ports: [443] }, + }); + await expect( + resolveRegisteredInferenceProvider({ ...f.binding, adapter: f.adapter }), + ).rejects.toThrow("enforced REST policy"); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it("ignores unsafe policy whose port list excludes the selected port", async () => { + const f = registeredInferenceFixture(); + f.profile.endpoints.push({ + ...f.profile.endpoints[0]!, + enforcement: "audit", + ...{ ports: [8443] }, + }); + await expect( + resolveRegisteredInferenceProvider({ ...f.binding, adapter: f.adapter }), + ).resolves.toMatchObject({ providerName: f.binding.providerName }); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it("accepts a declared port list without a singular port", async () => { + const f = registeredInferenceFixture(); + Object.assign(f.profile.endpoints[0]!, { port: undefined, ports: [443] }); + const binding = await resolveRegisteredInferenceProvider({ ...f.binding, adapter: f.adapter }); + const selected = await attachRegisteredInferenceProvider({ + adapter: f.adapter, + expected: binding, + }); + await expect( + detachRegisteredInferenceProvider({ adapter: f.adapter, expected: selected.attachment }), + ).resolves.toBe(true); + }); + + it("retains owned cleanup for a legacy singular-port binding", async () => { + const f = registeredInferenceFixture(true); + Object.assign(f.profile.endpoints[0]!, { ports: [8443] }); + const binding = { + ...f.binding, + attachedBySelection: true, + profileDigest: inspectInferenceProviderProfile(JSON.stringify(f.profile))!.digest, + }; + await expect( + attachRegisteredInferenceProvider({ adapter: f.adapter, expected: binding }), + ).rejects.toThrow("enforced REST policy"); + await expect( + detachRegisteredInferenceProvider({ adapter: f.adapter, expected: binding }), + ).resolves.toBe(true); + }); + + it.each([ + ["audit policy", { enforcement: "audit" }], + ["omitted enforcement", { enforcement: undefined }], + ["empty enforcement", { enforcement: "" }], + ["skipped TLS inspection", { tls: "skip" }], + ["invalid TLS mode", { tls: null }], + ["uninspected credentials", { allow_uninspected_credentials: true }], + ["invalid credential inspection flag", { allow_uninspected_credentials: "false" }], + ["TCP transport", { protocol: "tcp" }], + ["omitted protocol", { protocol: undefined }], + ])("rejects %s before active use while permitting owned detach", async (_label, fields) => { + const f = registeredInferenceFixture(true); + Object.assign(f.profile.endpoints[0]!, fields); + const binding = { + ...f.binding, + profileDigest: inspectInferenceProviderProfile(JSON.stringify(f.profile))!.digest, + attachedBySelection: true, + }; + const input = { adapter: f.adapter, expected: binding }; + await expect( + resolveRegisteredInferenceProvider({ ...binding, adapter: f.adapter }), + ).rejects.toThrow("enforced REST policy"); + await expect(attachRegisteredInferenceProvider(input)).rejects.toThrow("enforced REST policy"); + await expect(verifyRegisteredInferenceProviderAttachment(input)).rejects.toThrow( + "enforced REST policy", + ); + await expect(registeredInferenceProviderRevision(input)).rejects.toThrow( + "enforced REST policy", + ); + const read = vi.fn(async () => f.handle); + await expect( + readVerifiedRegisteredCredentialHandle({ + adapter: f.adapter, + attachment: binding, + read, + sleep: async () => {}, + }), + ).rejects.toThrow("enforced REST policy"); + expect(read).not.toHaveBeenCalled(); + expect(f.adapter.ensureProviderPolicyComposition).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.state.attached).toBe(true); + await expect(detachRegisteredInferenceProvider(input)).resolves.toBe(true); + expect(f.state.attached).toBe(false); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + }); + + it.each([0, 1])("rejects an unsafe matching wildcard endpoint at index %s", async (index) => { + const f = registeredInferenceFixture(); + const wildcard = { ...f.profile.endpoints[0]!, host: "*.example", enforcement: "audit" }; + f.profile.endpoints.splice(index, 0, wildcard); + await expect( + resolveRegisteredInferenceProvider({ ...f.binding, adapter: f.adapter }), + ).rejects.toThrow("enforced REST policy"); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }); + + it.each(["rest", "REST"])( + "preserves enforced operator rules and address restrictions with %s", + async (protocol) => { + const f = registeredInferenceFixture(); + Object.assign(f.profile.endpoints[0]!, { + protocol, + access: "", + rules: [{ allow: { method: "POST", path: "/v1/chat/completions" } }], + allowed_ips: ["10.0.0.0/8"], + tls: "", + allow_uninspected_credentials: false, + }); + f.profile.endpoints.push({ + ...f.profile.endpoints[0]!, + host: "unrelated.example", + enforcement: "audit", + }); + const before = JSON.stringify(f.profile); + const binding = await resolveRegisteredInferenceProvider({ + ...f.binding, + adapter: f.adapter, + }); + const selected = await attachRegisteredInferenceProvider({ + adapter: f.adapter, + expected: binding, + }); + const read = vi.fn(async () => f.handle); + await expect( + readVerifiedRegisteredCredentialHandle({ + adapter: f.adapter, + attachment: selected.attachment, + read, + sleep: async () => {}, + }), + ).resolves.toBe(f.handle); + expect(JSON.stringify(f.profile)).toBe(before); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + await expect( + detachRegisteredInferenceProvider({ adapter: f.adapter, expected: selected.attachment }), + ).resolves.toBe(true); + }, + ); + + it.each([ + ["openai-completions", "bearer", "authorization"], + ["openai-responses", "bearer", "authorization"], + ["anthropic-messages", "header", "x-api-key"], + ] as const)( + "rejects HTTP before attachment or credential handle access for %s", + async (inferenceApi, authStyle, headerName) => { + const f = registeredInferenceFixture(); + f.profile.credentials[0]!.auth_style = authStyle; + f.profile.credentials[0]!.header_name = headerName; + const binding = { + ...f.binding, + endpointUrl: "http://native.example:443/v1", + inferenceApi, + profileDigest: inspectInferenceProviderProfile(JSON.stringify(f.profile))!.digest, + }; + await expect( + resolveRegisteredInferenceProvider({ ...binding, adapter: f.adapter }), + ).rejects.toThrow("HTTPS endpoint"); + await expect( + attachRegisteredInferenceProvider({ adapter: f.adapter, expected: binding }), + ).rejects.toThrow("HTTPS endpoint"); + const read = vi.fn(async () => f.handle); + await expect( + readVerifiedRegisteredCredentialHandle({ + adapter: f.adapter, + attachment: binding, + read, + sleep: async () => {}, + }), + ).rejects.toThrow("HTTPS endpoint"); + expect(read).not.toHaveBeenCalled(); + expect(f.adapter.ensureProviderPolicyComposition).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.state.attached).toBe(false); + }, + ); + + it("preserves retained HTTP identity for inspection and owned detachment", async () => { + const f = registeredInferenceFixture(true); + const binding = { + ...f.binding, + endpointUrl: "http://native.example:443/v1", + attachedBySelection: true, + }; + expect( + recordedRegisteredInferenceProviderAttachment({ + name: binding.sandboxName, + gatewayName: binding.gatewayName, + provider: binding.providerName, + endpointUrl: binding.endpointUrl, + preferredInferenceApi: binding.inferenceApi, + credentialEnv: binding.credentialEnv, + registeredInferenceProviderAttachment: binding, + }), + ).toEqual(binding); + await expect( + verifyRegisteredInferenceProviderAttachment({ adapter: f.adapter, expected: binding }), + ).rejects.toThrow("HTTPS endpoint"); + await expect( + detachRegisteredInferenceProvider({ adapter: f.adapter, expected: binding }), + ).resolves.toBe(true); + expect(f.state.attached).toBe(false); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + }); + + it.each(["10.0.0.8", "inference.corp.example"])( + "preserves operator HTTPS inference at %s", + async (host) => { + const f = registeredInferenceFixture(); + f.profile.endpoints[0]!.host = host; + const binding = await resolveRegisteredInferenceProvider({ + ...f.binding, + adapter: f.adapter, + endpointUrl: `https://${host}/v1`, + }); + await expect( + attachRegisteredInferenceProvider({ adapter: f.adapter, expected: binding }), + ).resolves.toMatchObject({ changed: true }); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + }, + ); + + it("accepts a declared Anthropic header credential without changing the provider", async () => { + const f = registeredInferenceFixture(); + f.profile.credentials[0]!.auth_style = "header"; + f.profile.credentials[0]!.header_name = "x-api-key"; + const binding = await resolveRegisteredInferenceProvider({ + ...f.binding, + adapter: f.adapter, + inferenceApi: "anthropic-messages", + }); + await expect( + attachRegisteredInferenceProvider({ adapter: f.adapter, expected: binding }), + ).resolves.toMatchObject({ changed: true }); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + }); + + it.each([0, -1, Number.NaN])( + "rejects unavailable revision %s before attachment", + async (revision) => { + const f = registeredInferenceFixture(); + f.state.revision = revision; + await expect( + attachRegisteredInferenceProvider({ adapter: f.adapter, expected: f.binding }), + ).rejects.toThrow(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/src/lib/inference/native-provider/registered.ts b/src/lib/inference/native-provider/registered.ts new file mode 100644 index 00000000000..dbcfc05ec54 --- /dev/null +++ b/src/lib/inference/native-provider/registered.ts @@ -0,0 +1,392 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import type { + OpenShellProviderAdapter, + OpenShellProviderMetadata, +} from "../../adapters/openshell/provider-adapter"; +import type { InferenceProviderProfileProjection } from "../../adapters/openshell/provider-profile"; +import { unsafeEndpointUrlViolation } from "../../core/endpoint-url-safety"; +import { openShellHostMatches } from "../../adapters/openshell/policy-boundary"; + +export type RegisteredInferenceApi = + | "openai-completions" + | "openai-responses" + | "anthropic-messages"; + +/** Selection of an operator provider grants no ownership of that provider. */ +export type RegisteredInferenceProviderAttachment = Readonly<{ + schemaVersion: 1; + gatewayName: string; + sandboxName: string; + providerName: string; + providerId: string; + profileId: string; + profileDigest: string; + endpointUrl: string; + inferenceApi: RegisteredInferenceApi; + credentialEnv: string; + /** Only an attachment made by inference selection may be removed on departure. */ + attachedBySelection: boolean; +}>; + +export function registeredInferenceEndpoint(value: string): URL { + const endpoint = recordedInferenceEndpoint(value); + if (endpoint.protocol !== "https:") + throw new Error("Registered inference requires its explicit native HTTPS endpoint"); + return endpoint; +} + +// Retained HTTP records must remain readable for inspection and owned detachment. +function recordedInferenceEndpoint(value: string): URL { + if (unsafeEndpointUrlViolation(value)) + throw new Error("Registered inference endpoint is invalid"); + const endpoint = new URL(value); + if ( + !["http:", "https:"].includes(endpoint.protocol) || + endpoint.username || + endpoint.password || + endpoint.search || + endpoint.hash || + ["localhost", "127.0.0.1", "[::1]", "inference.local"].includes(endpoint.hostname) || + /%2f|%5c/iu.test(endpoint.pathname) + ) + throw new Error("Registered inference requires its explicit native HTTP(S) endpoint"); + return endpoint; +} + +export function normalizeRegisteredInferenceProviderAttachment( + value: unknown, +): RegisteredInferenceProviderAttachment | undefined { + if (!value || typeof value !== "object" || Array.isArray(value)) return undefined; + const row = value as Record; + if (row.schemaVersion !== 1 || typeof row.attachedBySelection !== "boolean") return undefined; + for (const key of ["gatewayName", "sandboxName", "providerName", "profileId"]) { + if (typeof row[key] !== "string" || !/^[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}$/u.test(row[key])) + return undefined; + } + if ( + typeof row.providerId !== "string" || + !row.providerId.trim() || + row.providerId.length > 256 || + /\s/u.test(row.providerId) || + typeof row.profileDigest !== "string" || + !/^[a-f0-9]{64}$/u.test(row.profileDigest) || + typeof row.credentialEnv !== "string" || + !/^[A-Z][A-Z0-9_]{0,255}$/u.test(row.credentialEnv) + ) + return undefined; + if ( + !["openai-completions", "openai-responses", "anthropic-messages"].includes( + String(row.inferenceApi), + ) || + typeof row.endpointUrl !== "string" + ) + return undefined; + try { + recordedInferenceEndpoint(row.endpointUrl); + } catch { + return undefined; + } + return { + schemaVersion: 1, + gatewayName: row.gatewayName as string, + sandboxName: row.sandboxName as string, + providerName: row.providerName as string, + providerId: row.providerId, + profileId: row.profileId as string, + profileDigest: row.profileDigest, + endpointUrl: row.endpointUrl, + inferenceApi: row.inferenceApi as RegisteredInferenceApi, + credentialEnv: row.credentialEnv, + attachedBySelection: row.attachedBySelection, + }; +} + +function requireBinding(value: unknown): RegisteredInferenceProviderAttachment { + const binding = normalizeRegisteredInferenceProviderAttachment(value); + if (!binding) throw new Error("Registered inference provider binding is invalid"); + return binding; +} + +function matchingDeclaredEndpoints( + endpoint: URL, + projection: InferenceProviderProfileProjection, +): InferenceProviderProfileProjection["endpoints"] { + const port = Number(endpoint.port || (endpoint.protocol === "https:" ? 443 : 80)); + return projection.endpoints.filter( + (declared) => + declared.ports.includes(port) && openShellHostMatches(declared.host, endpoint.hostname), + ); +} + +function assertProjection( + binding: RegisteredInferenceProviderAttachment, + metadata: OpenShellProviderMetadata, + projection: InferenceProviderProfileProjection | undefined, +): asserts projection is InferenceProviderProfileProjection { + if ( + !projection || + projection.digest !== binding.profileDigest || + !projection.endpoints.some((declared) => { + const endpoint = recordedInferenceEndpoint(binding.endpointUrl); + const port = Number(endpoint.port || (endpoint.protocol === "https:" ? 443 : 80)); + // Keep retained bindings detachable under the previous host and singular-port projection. + return ( + (declared.port === port || declared.ports.includes(port)) && + (declared.host === endpoint.hostname || + (declared.host.startsWith("*.") && endpoint.hostname.endsWith(declared.host.slice(1))) || + openShellHostMatches(declared.host, endpoint.hostname)) + ); + }) + ) + throw new Error("Registered provider profile does not match the selected inference endpoint"); + const matches = projection.credentials.filter((credential) => + credential.envVars.includes(binding.credentialEnv), + ); + const expectedHeader = + binding.inferenceApi === "anthropic-messages" ? "x-api-key" : "authorization"; + const expectedStyle = binding.inferenceApi === "anthropic-messages" ? "header" : "bearer"; + if ( + !metadata.credentialKeys.includes(binding.credentialEnv) || + matches.length !== 1 || + matches[0]!.headerName !== expectedHeader || + matches[0]!.authStyle !== expectedStyle + ) + throw new Error("Registered provider credential does not match the selected inference API"); +} + +function requireEnforcedCredentialTransport( + binding: RegisteredInferenceProviderAttachment, + projection: InferenceProviderProfileProjection, +): void { + const endpoints = matchingDeclaredEndpoints( + registeredInferenceEndpoint(binding.endpointUrl), + projection, + ); + if (!endpoints.length || endpoints.some((endpoint) => endpoint.enforcesCredentialedRest !== true)) + throw new Error( + "Registered inference requires enforced REST policy with TLS inspection and inspected credentials", + ); +} + +async function inspectBinding( + adapter: OpenShellProviderAdapter, + expected: RegisteredInferenceProviderAttachment, +): Promise<{ + metadata: OpenShellProviderMetadata; + projection: InferenceProviderProfileProjection; +}> { + const target = { kind: "named", gatewayName: expected.gatewayName } as const; + const provider = await adapter.getProvider({ target, providerName: expected.providerName }); + if ( + !provider.ok || + provider.value.name !== expected.providerName || + provider.value.type !== expected.profileId || + provider.value.revision?.id !== expected.providerId || + !Number.isSafeInteger(provider.value.revision.resourceVersion) || + provider.value.revision.resourceVersion < 1 + ) + throw new Error("Registered inference provider identity changed"); + const profile = await adapter.inspectProviderProfile({ + target, + profileType: expected.profileId, + includeInferenceProjection: true, + }); + if (!profile.ok) throw new Error("Registered inference provider profile could not be verified"); + const projection = profile.value.inference; + assertProjection(expected, provider.value, projection); + return { metadata: provider.value, projection }; +} + +async function inspectActiveBinding( + adapter: OpenShellProviderAdapter, + expected: RegisteredInferenceProviderAttachment, +): Promise { + const { metadata, projection } = await inspectBinding(adapter, expected); + requireEnforcedCredentialTransport(expected, projection); + return metadata; +} + +export async function resolveRegisteredInferenceProvider(input: { + adapter: OpenShellProviderAdapter; + gatewayName: string; + sandboxName: string; + providerName: string; + endpointUrl: string; + inferenceApi: string; + credentialEnv: string; + previous?: RegisteredInferenceProviderAttachment; +}): Promise { + registeredInferenceEndpoint(input.endpointUrl); + const target = { kind: "named", gatewayName: input.gatewayName } as const; + const observed = await input.adapter.getProvider({ target, providerName: input.providerName }); + if ( + !observed.ok || + !observed.value.revision?.id || + observed.value.name !== input.providerName || + !Number.isSafeInteger(observed.value.revision.resourceVersion) || + observed.value.revision.resourceVersion < 1 + ) + throw new Error("Registered inference provider identity is unavailable"); + const profile = await input.adapter.inspectProviderProfile({ + target, + profileType: observed.value.type, + includeInferenceProjection: true, + }); + if (!profile.ok || !profile.value.inference) + throw new Error("Registered provider has no declared inference profile"); + const binding = requireBinding({ + schemaVersion: 1, + gatewayName: input.gatewayName, + sandboxName: input.sandboxName, + providerName: input.providerName, + providerId: observed.value.revision.id, + profileId: observed.value.type, + profileDigest: profile.value.inference.digest, + endpointUrl: input.endpointUrl, + inferenceApi: input.inferenceApi, + credentialEnv: input.credentialEnv, + attachedBySelection: input.previous?.attachedBySelection ?? false, + }); + if ( + input.previous && + (input.previous.providerId !== binding.providerId || + input.previous.profileId !== binding.profileId || + input.previous.profileDigest !== binding.profileDigest) + ) + throw new Error("Recorded registered inference provider identity or profile changed"); + assertProjection(binding, observed.value, profile.value.inference); + requireEnforcedCredentialTransport(binding, profile.value.inference); + return binding; +} + +export async function verifyRegisteredInferenceProviderAttachment(input: { + revalidateSandboxIdentity?: (operation: string) => void; + adapter: OpenShellProviderAdapter; + expected: RegisteredInferenceProviderAttachment; +}): Promise { + const expected = requireBinding(input.expected); + registeredInferenceEndpoint(expected.endpointUrl); + await inspectActiveBinding(input.adapter, expected); + const attachments = await input.adapter.listProviderAttachments({ + target: { kind: "named", gatewayName: expected.gatewayName }, + sandboxName: expected.sandboxName, + }); + if (!attachments.ok || !attachments.value.names.includes(expected.providerName)) + throw new Error("Registered inference provider is not attached to the selected sandbox"); + input.revalidateSandboxIdentity?.("verify registered inference attachment"); +} + +export async function attachRegisteredInferenceProvider(input: { + adapter: OpenShellProviderAdapter; + expected: RegisteredInferenceProviderAttachment; + revalidateSandboxIdentity?: (operation: string) => void; +}): Promise<{ attachment: RegisteredInferenceProviderAttachment; changed: boolean }> { + const expected = requireBinding(input.expected); + registeredInferenceEndpoint(expected.endpointUrl); + const before = await inspectActiveBinding(input.adapter, expected); + const request = { + target: { kind: "named", gatewayName: expected.gatewayName } as const, + sandboxName: expected.sandboxName, + providerName: expected.providerName, + }; + const attachments = await input.adapter.listProviderAttachments(request); + if (!attachments.ok) throw new Error("Registered provider attachments could not be inspected"); + if (attachments.value.names.includes(expected.providerName)) { + await verifyRegisteredInferenceProviderAttachment({ ...input, expected }); + return { attachment: expected, changed: false }; + } + input.revalidateSandboxIdentity?.("compose registered inference provider policy"); + const policy = await input.adapter.ensureProviderPolicyComposition(request); + if (!policy.ok) throw new Error("Registered provider policy composition could not be verified"); + input.revalidateSandboxIdentity?.("attaching registered inference provider"); + const attached = await input.adapter.attachProvider(request); + // An uncertain runtime mutation must never become an inferred successful attachment. + if (!attached.ok) + throw new Error( + `Registered provider attachment failed: ${attached.error.kind}. Inspect the sandbox attachment before retrying.`, + ); + const attachment = { ...expected, attachedBySelection: true }; + try { + const after = await inspectActiveBinding(input.adapter, expected); + if (before.revision?.resourceVersion !== after.revision?.resourceVersion) + throw new Error("Registered provider changed during attachment"); + await verifyRegisteredInferenceProviderAttachment({ ...input, expected: attachment }); + return { attachment, changed: true }; + } catch (error) { + input.revalidateSandboxIdentity?.("rolling back unverified registered inference attachment"); + await detachRegisteredInferenceProvider({ ...input, expected: attachment }); + throw error; + } +} + +/** Remove only selection-created access; the operator provider is never mutated. */ +export async function detachRegisteredInferenceProvider(input: { + revalidateSandboxIdentity?: (operation: string) => void; + adapter: OpenShellProviderAdapter; + expected: RegisteredInferenceProviderAttachment; +}): Promise { + const expected = requireBinding(input.expected); + if (!expected.attachedBySelection) return false; + await inspectBinding(input.adapter, expected); + const request = { + target: { kind: "named", gatewayName: expected.gatewayName } as const, + sandboxName: expected.sandboxName, + providerName: expected.providerName, + }; + const before = await input.adapter.listProviderAttachments(request); + if (!before.ok) throw new Error("Registered provider attachments could not be inspected"); + if (!before.value.names.includes(expected.providerName)) return false; + input.revalidateSandboxIdentity?.("detach registered inference provider"); + const detached = await input.adapter.detachProvider(request); + if (!detached.ok) throw new Error("Registered provider attachment removal was not confirmed"); + const after = await input.adapter.listProviderAttachments(request); + if (!after.ok || after.value.names.includes(expected.providerName)) + throw new Error("Registered provider attachment absence was not confirmed"); + input.revalidateSandboxIdentity?.("verify registered inference provider detach"); + return true; +} + +/** A recorded selection binds one operator provider to this exact sandbox route. */ +export function recordedRegisteredInferenceProviderAttachment(entry: { + name?: string; + gatewayName?: string | null; + provider?: string | null; + endpointUrl?: string | null; + preferredInferenceApi?: string | null; + credentialEnv?: string | null; + registeredInferenceProviderAttachment?: unknown; + nativeLocalProviderAttachment?: unknown; + nativeNvidiaProviderAttachment?: unknown; + nativeHostedProviderAttachment?: unknown; +}): RegisteredInferenceProviderAttachment | undefined { + if (entry.registeredInferenceProviderAttachment === undefined) return undefined; + const binding = requireBinding(entry.registeredInferenceProviderAttachment); + if ( + entry.name !== binding.sandboxName || + entry.gatewayName !== binding.gatewayName || + entry.provider !== binding.providerName || + entry.endpointUrl !== binding.endpointUrl || + entry.preferredInferenceApi !== binding.inferenceApi || + entry.credentialEnv !== binding.credentialEnv || + entry.nativeLocalProviderAttachment !== undefined || + entry.nativeNvidiaProviderAttachment !== undefined || + entry.nativeHostedProviderAttachment !== undefined + ) + throw new Error("Recorded registered inference selection does not match its sandbox route"); + return binding; +} + +export async function registeredInferenceProviderRevision(input: { + adapter: OpenShellProviderAdapter; + expected: RegisteredInferenceProviderAttachment; +}): Promise { + const expected = requireBinding(input.expected); + registeredInferenceEndpoint(expected.endpointUrl); + const metadata = await inspectActiveBinding(input.adapter, expected); + const revision = metadata.revision?.resourceVersion; + if (!Number.isSafeInteger(revision) || !revision || revision < 1) + throw new Error("Registered provider revision is unavailable"); + return revision; +} diff --git a/src/lib/inference/native-provider/setup.ts b/src/lib/inference/native-provider/setup.ts index 28c3e24009b..411bfb1d168 100644 --- a/src/lib/inference/native-provider/setup.ts +++ b/src/lib/inference/native-provider/setup.ts @@ -9,9 +9,11 @@ import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider import type { SandboxEntry } from "../../state/registry/types"; import type { NativeProviderAttachment } from "./contract"; import { nativeProviderLifecycle } from "./index"; -import { hostedNativeProvider } from "./hosted"; +import { hostedNativeProvider, canonicalHermesNativeEndpoint } from "./hosted"; import { requireHermesPublicPins } from "./hermes-pins"; import { requireHostedProviderAttachment } from "./hosted-attachment"; +import { isAllowedOpenShellSandboxBridgeUrl } from "../../private-networks"; +import { isProtectedLocalInferencePort } from "../gateway/local-endpoint-ports"; /** Prepare gateway ownership; attachment remains the selected sandbox transaction's job. */ export async function prepareHostedNativeProvider(input: { @@ -20,6 +22,8 @@ export async function prepareHostedNativeProvider(input: { gatewayName: string; credentialValue: string | null; endpointUrl?: string | null; + /** Only callers restoring an existing endpoint may retain its HTTP or host-alias route. */ + retainedHermesEndpoint?: boolean; lookup?: EndpointDnsLookupFn; reuseExistingCredential?: boolean; recordedSandbox?: SandboxEntry | null; @@ -34,7 +38,10 @@ export async function prepareHostedNativeProvider(input: { input.recordedSandbox?.provider === input.provider ? input.recordedSandbox.nativeHostedProviderAttachment?.endpointUrl : undefined; - let definition = hostedNativeProvider(input.provider, input.endpointUrl ?? recordedEndpoint); + const endpointUrl = input.endpointUrl ?? recordedEndpoint; + if (input.provider === "hermes-provider" && endpointUrl && !input.retainedHermesEndpoint) + canonicalHermesNativeEndpoint(endpointUrl); + let definition = hostedNativeProvider(input.provider, endpointUrl); if (!definition) throw new Error("Unsupported fixed hosted provider"); const recorded = input.recordedSandbox; const recordedAttachment = @@ -47,7 +54,11 @@ export async function prepareHostedNativeProvider(input: { definition.endpointUrl ?? (input.provider === "hermes-provider" ? definition.endpoint : undefined), ); - if (definition.endpointUrl) { + const bridge = + definition.endpointUrl && isAllowedOpenShellSandboxBridgeUrl(new URL(definition.endpointUrl)); + if (bridge && isProtectedLocalInferencePort(Number(new URL(definition.endpointUrl!).port))) + throw new Error("The recorded Hermes endpoint uses a protected host port"); + if (definition.endpointUrl && !bridge) { const prior = gatewayAuthority ?? (recordedAttachment?.providerName === definition.providerName @@ -100,7 +111,7 @@ export async function prepareHostedNativeProvider(input: { } export { - isHostedCustomEndpoint, + isCustomNativeEndpoint, prepareCustomNativeProvider, readCustomNativeSandboxPolicy, usesCustomNativeOnboarding, diff --git a/src/lib/inference/nim-context-window.test.ts b/src/lib/inference/nim-context-window.test.ts new file mode 100644 index 00000000000..85f96aa4af1 --- /dev/null +++ b/src/lib/inference/nim-context-window.test.ts @@ -0,0 +1,38 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { expect, it, vi } from "vitest"; +import { getNimContextWindow, getNimInferenceEndpoints, getServedModelId } from "./nim"; + +const capture = vi.hoisted(() => vi.fn()); +vi.mock("../runner", () => ({ runCapture: capture })); + +it("reads NIM model and context from its bounded unauthenticated loopback endpoint", () => { + capture.mockReturnValue(JSON.stringify({ data: [{ id: "nim/model", max_model_len: 65536 }] })); + vi.stubEnv("NGC_API_KEY", "host-ngc-secret"); + expect(getServedModelId()).toBe("nim/model"); + expect(getNimContextWindow("nim/model")).toBe(65536); + expect(capture).toHaveBeenLastCalledWith( + [ + "curl", + "-sf", + "--connect-timeout", + "5", + "--max-time", + "5", + `${getNimInferenceEndpoints().validationBaseUrl}/models`, + ], + { ignoreError: true }, + ); +}); + +it.each([ + null, + "not-json", + JSON.stringify({ data: [] }), + JSON.stringify({ data: [{ id: "different-model", max_model_len: 262144 }] }), + JSON.stringify({ data: [{ id: "nim/model" }] }), +])("omits unavailable or mismatched NIM context metadata: %s", (body) => { + capture.mockReturnValue(body); + expect(getNimContextWindow("nim/model")).toBeNull(); +}); diff --git a/src/lib/inference/nim.ts b/src/lib/inference/nim.ts index e4baa2bb7f9..c87236064ed 100644 --- a/src/lib/inference/nim.ts +++ b/src/lib/inference/nim.ts @@ -24,6 +24,7 @@ import { sleepSeconds } from "../core/wait"; import { isWsl as detectWsl } from "../platform"; import { runCapture } from "../runner"; import { isSafeModelId } from "../validation"; +import { resolveVllmContextWindowFromModels } from "./vllm-runtime-context"; import { buildModelListProbeCommand, isOpenAiModelListResponse } from "./serving/model-inventory"; import { classifyNvidiaFirmwareProducts, @@ -453,9 +454,16 @@ export function parseServedModelId(modelsJson: string): string | null { return null; } -// Model id a running local NIM actually serves; null if unreachable/empty. -export function getServedModelId(port = VLLM_PORT): string | null { - const out = runCapture( +/** NIM owns this published port; managed vLLM receipts cannot select its endpoint. */ +export function getNimInferenceEndpoints() { + return { + baseUrl: `http://host.openshell.internal:${VLLM_PORT}/v1`, + validationBaseUrl: `http://127.0.0.1:${VLLM_PORT}/v1`, + }; +} + +function readNimModels(port: number): string | null { + return runCapture( [ "curl", ...buildValidatedCurlCommandArgs([ @@ -469,9 +477,25 @@ export function getServedModelId(port = VLLM_PORT): string | null { ], { ignoreError: true }, ); +} + +// Model id a running local NIM actually serves; null if unreachable/empty. +export function getServedModelId(port = VLLM_PORT): string | null { + const out = readNimModels(port); return out ? parseServedModelId(out) : null; } +/** Read context metadata only from the NIM-owned listener and selected model. */ +export function getNimContextWindow(model: string): number | null { + const raw = readNimModels(VLLM_PORT); + if (!raw || parseServedModelId(raw) !== model) return null; + try { + return resolveVllmContextWindowFromModels(JSON.parse(raw), model); + } catch { + return null; + } +} + // Adopt the id NIM serves (from /v1/models) when it differs from the catalog // name and is safe; the catalog id otherwise 404s on validation. See #3885. export function adoptServedModelId(catalogModel: string | null, port = VLLM_PORT): string | null { diff --git a/src/lib/inference/nvidia-probe-recovery.test.ts b/src/lib/inference/nvidia-probe-recovery.test.ts new file mode 100644 index 00000000000..9d593d7ad89 --- /dev/null +++ b/src/lib/inference/nvidia-probe-recovery.test.ts @@ -0,0 +1,134 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import http from "node:http"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { probeOpenAiLikeEndpointWithValidationSession } from "./openai-validation-session"; +import { + createOpenAiValidationTestDeps, + useOpenAiValidationTestServers, +} from "./openai-validation-session.test-helpers"; +import { + HARNESS_COUNTER, + makeFakeCurlScript, + withFakeCurlProbe, +} from "./onboard-probes-curl-harness"; + +const { probeOpenAiLikeEndpoint } = require("./onboard-probes"); +const listen = useOpenAiValidationTestServers(); + +beforeEach(() => vi.stubEnv("NEMOCLAW_TEST_NO_SLEEP", "1")); + +describe("NVIDIA Endpoints validation recovery", () => { + it("recovers from HTTP 500 without changing the native request or resolving another address (#12603)", async () => { + const requests: { authorization: string | undefined; url: string | undefined; body: string }[] = + []; + const server = http.createServer((request, response) => { + let body = ""; + request.setEncoding("utf8"); + request.on("data", (chunk) => { + body += chunk; + }); + request.on("end", () => { + requests.push({ authorization: request.headers.authorization, url: request.url, body }); + response.statusCode = requests.length === 1 ? 500 : 200; + response.end('{"choices":[{"message":{"content":"OK"}}]}'); + }); + }); + const port = await listen(server); + const deps = createOpenAiValidationTestDeps(); + const result = await probeOpenAiLikeEndpointWithValidationSession( + `http://provider.example.test:${port}/v1`, + "test-model", + "fixture-key", + { provider: "nvidia-prod", skipResponsesProbe: true }, + deps, + ); + expect(result).toMatchObject({ ok: true, api: "openai-completions" }); + expect(requests).toEqual( + Array(2).fill({ + authorization: "Bearer fixture-key", + url: "/v1/chat/completions", + body: JSON.stringify({ model: "test-model", messages: [] }), + }), + ); + expect(deps.sessionOptions?.lookup).toHaveBeenCalledTimes(1); + expect(deps.legacyProbe).not.toHaveBeenCalled(); + }); + + it.each([ + { + provider: "nvidia-prod", + status: 500, + attempts: 4, + legacy: 0, + expected: { ok: false, failures: [{ name: "Chat Completions API", httpStatus: 500 }] }, + }, + { provider: "custom", status: 500, attempts: 4, legacy: 1, expected: { ok: false } }, + { provider: "nvidia-prod", status: 401, attempts: 1, legacy: 1, expected: { ok: false } }, + { provider: "nvidia-prod", status: 403, attempts: 1, legacy: 1, expected: { ok: false } }, + ])( + "limits native $provider HTTP $status to $attempts requests (#12603)", + async ({ provider, status, attempts, legacy, expected }) => { + let calls = 0; + const server = http.createServer((_request, response) => { + calls++; + response.statusCode = status; + response.end("provider failure"); + }); + const port = await listen(server); + const deps = createOpenAiValidationTestDeps( + vi.fn(() => ({ ok: false, message: "rejected" })), + ); + const result = await probeOpenAiLikeEndpointWithValidationSession( + `http://provider.example.test:${port}/v1`, + "test-model", + "fixture-key", + { provider, skipResponsesProbe: true }, + deps, + ); + expect(result).toMatchObject(expected); + expect(calls).toBe(attempts); + expect(deps.legacyProbe).toHaveBeenCalledTimes(legacy); + }, + ); + + it.each([ + { provider: "nvidia-prod", status: 500, recoverAt: 2, attempts: 2, ok: true }, + { provider: "nvidia-prod", status: 500, recoverAt: 99, attempts: 4, ok: false }, + { provider: "custom", status: 500, recoverAt: 2, attempts: 2, ok: true }, + { provider: "nvidia-prod", status: 401, recoverAt: 2, attempts: 1, ok: false }, + { provider: "nvidia-prod", status: 403, recoverAt: 2, attempts: 1, ok: false }, + ])( + "limits curl $provider HTTP $status recovery to $attempts requests (#12603)", + ({ provider, status, recoverAt, attempts, ok }) => { + withFakeCurlProbe( + { + script: makeFakeCurlScript(` +n=$(cat "${HARNESS_COUNTER}") +n=$((n + 1)) +printf '%s' "$n" > "${HARNESS_COUNTER}" +if [ "$n" -ge ${recoverAt} ]; then + printf '%s' '{"choices":[{"message":{"content":"OK"}}]}' > "$outfile" + printf '200' +else + printf '%s' 'provider failure' > "$outfile" + printf '${status}' +fi +`), + }, + ({ counter }) => { + const result = probeOpenAiLikeEndpoint( + "https://integrate.api.nvidia.com/v1", + "test-model", + "fixture-key", + { provider, skipResponsesProbe: true }, + ); + expect(result.ok).toBe(ok); + expect(Number(fs.readFileSync(counter, "utf8"))).toBe(attempts); + }, + ); + }, + ); +}); diff --git a/src/lib/inference/openai-validation-session.ts b/src/lib/inference/openai-validation-session.ts index 4f32c839cef..572472dd4bd 100644 --- a/src/lib/inference/openai-validation-session.ts +++ b/src/lib/inference/openai-validation-session.ts @@ -27,6 +27,7 @@ import { RETRIABLE_HTTP_PROBE_STATUSES } from "./probe/transient-http-policy"; const RETRY_DELAYS_MS = [5_000, 15_000, 30_000]; export interface OpenAiValidationOptions { + provider?: string; authMode?: "bearer" | "query-param"; extraHeaders?: readonly string[]; requireResponsesToolCalling?: boolean; @@ -103,9 +104,10 @@ function failedChatValidation( result: CurlProbeResult, failureMessage: string, diagnosticCodes?: string[], + probeName = "Chat Completions API with tool calling", ): OpenAiValidationResult { const failure = { - name: "Chat Completions API with tool calling", + name: probeName, httpStatus: result.httpStatus, curlStatus: result.curlStatus, message: failureMessage, @@ -415,8 +417,16 @@ export async function probeOpenAiLikeEndpointWithValidationSession( : nativeFailureFallback("native_chat_failure"); } if (!chat.ok) { - return retriedReasoningTruncation - ? failedChatValidation(chat, chat.message) + // Replaying an exhausted NVIDIA HTTP 500 through curl would reset the + // four-request budget without changing the request or endpoint. + return retriedReasoningTruncation || + (options.provider === "nvidia-prod" && chat.httpStatus === 500) + ? failedChatValidation( + chat, + chat.message, + undefined, + requireToolCall ? "Chat Completions API with tool calling" : "Chat Completions API", + ) : nativeFailureFallback("native_terminal_http_failure"); } if (requireToolCall) { diff --git a/src/lib/inference/probe-retry.ts b/src/lib/inference/probe-retry.ts index fd935b09ff1..2a788b32232 100644 --- a/src/lib/inference/probe-retry.ts +++ b/src/lib/inference/probe-retry.ts @@ -19,7 +19,7 @@ const CURL_TIMEOUT_STATUS = 28; const NODE_SPAWN_TIMEOUT_STATUS = -110; // Which statuses are transient is shared policy, not a wizard-local decision: -// the sandbox status probe classifies the same four (#10709). The definition +// the sandbox status probe classifies the same statuses (#10709). The definition // and its rationale live in ./probe/transient-http-policy; this module keeps // re-exporting the set so existing consumers are unaffected. const { RETRIABLE_HTTP_PROBE_STATUSES } = require("./probe/transient-http-policy"); diff --git a/src/lib/inference/selection.ts b/src/lib/inference/selection.ts index efd12a7a844..aa20725916e 100644 --- a/src/lib/inference/selection.ts +++ b/src/lib/inference/selection.ts @@ -7,6 +7,9 @@ import { type ModelSelectionProvenance, } from "../domain/telemetry/provenance"; import { usesNativeLocalInferenceForAgent } from "./native-local/contract"; +import { isCustomNativeEndpoint } from "./native-provider/custom"; +import { recordedNativeProviderAttachment } from "./native-provider/recorded-selection"; +import { recordedRegisteredInferenceProviderAttachment } from "./native-provider/registered"; export { isPersistedPiSharedLocalInferenceRoute, @@ -24,7 +27,6 @@ export function hasLegacySharedLocalInferenceRoute(recorded: { }): boolean { return ( recorded.nativeLocalProviderAttachment === undefined && - !recorded.nimContainer && usesNativeLocalInferenceForAgent( recorded.agent ?? "openclaw", recorded.provider, @@ -34,6 +36,17 @@ export function hasLegacySharedLocalInferenceRoute(recorded: { ); } +/** Identify retained custom routes that still need the removed shared inference service. */ +export function hasLegacySharedCustomInferenceRoute( + recorded: Parameters[0], +): boolean { + return ( + isCustomNativeEndpoint(recorded.provider ?? "", recorded.endpointUrl) && + !recordedNativeProviderAttachment(recorded) && + !recordedRegisteredInferenceProviderAttachment(recorded) + ); +} + export { type ReasoningEffortRequest, resolveReasoningEffortRequest, @@ -168,3 +181,5 @@ export function inferenceSelectionRegistryFields( ): InferenceSelection { return normalizeInferenceSelection(input); } + +export { recordedRegisteredInferenceProviderAttachment }; diff --git a/src/lib/inference/serving/vllm-export-runtime.test.ts b/src/lib/inference/serving/vllm-export-runtime.test.ts index 25d0cd3584c..d818e277f02 100644 --- a/src/lib/inference/serving/vllm-export-runtime.test.ts +++ b/src/lib/inference/serving/vllm-export-runtime.test.ts @@ -57,12 +57,6 @@ function fixture() { directory, ); const image = { Id: imageId, Os: "linux", Architecture: "amd64", Environment: ["PATH=/usr/bin"] }; - const network = { - Id: "c".repeat(64), - Name: "openshell-docker", - Driver: "bridge", - Config: [{ Gateway: "172.18.0.1", Subnet: "172.18.0.0/16" }], - }; const container = { Id: containerId, Name: "/nemoclaw-vllm", @@ -84,17 +78,12 @@ function fixture() { }, NetworkSettings: { Ports: { - "8000/tcp": [ - { HostIp: "127.0.0.1", HostPort: "18000" }, - { HostIp: "172.18.0.1", HostPort: "18000" }, - ], + "8000/tcp": [{ HostIp: "127.0.0.1", HostPort: "18000" }], }, }, }; const capture = vi.fn>((args) => { - return JSON.stringify( - { image, network, container }[args[0] as "image" | "network" | "container"], - ); + return JSON.stringify({ image, container }[args[0] as "image" | "container"]); }); const loadApiKey = vi.fn(() => key); const options = { @@ -103,7 +92,7 @@ function fixture() { architecture: "x64", authentication: { stateDir: directory, loadApiKey }, }; - return { provenance, directory, image, network, container, capture, loadApiKey, options }; + return { provenance, directory, image, container, capture, loadApiKey, options }; } describe("fixed managed vLLM export observation", () => { @@ -113,7 +102,6 @@ describe("fixed managed vLLM export observation", () => { expect(result).toMatchObject({ containerId, imageId, - networkId: f.network.Id, startedAt: f.container.StartedAt, serving: { profile: { id: EXPORTED_VLLM_PROFILE_ID }, @@ -133,14 +121,9 @@ describe("fixed managed vLLM export observation", () => { expect(serialized).not.toContain(f.directory); expect(f.capture.mock.calls.map(([args]) => args.slice(0, 2))).toEqual([ ["image", "inspect"], - ["network", "inspect"], ["container", "inspect"], ]); - expect(f.capture.mock.calls.map(([args]) => args[2])).toEqual([ - "--format", - "--format", - "--format", - ]); + expect(f.capture.mock.calls.map(([args]) => args[2])).toEqual(["--format", "--format"]); expect(JSON.stringify(f.capture.mock.calls.map(([args]) => args))).not.toContain(key); const boundedCapture = expect.objectContaining({ env: expect.objectContaining({ DOCKER_CONTEXT: "default" }), @@ -150,7 +133,6 @@ describe("fixed managed vLLM export observation", () => { expect(f.capture.mock.calls.map(([, options]) => options)).toEqual([ boundedCapture, boundedCapture, - boundedCapture, ]); }); @@ -225,9 +207,12 @@ describe("fixed managed vLLM export observation", () => { }, ], [ - "different bridge listener", + "additional private listener", (f: ReturnType) => { - f.network.Config[0]!.Gateway = "172.19.0.1"; + f.container.NetworkSettings.Ports["8000/tcp"].push({ + HostIp: "172.19.0.1", + HostPort: "18000", + }); }, ], [ @@ -237,9 +222,9 @@ describe("fixed managed vLLM export observation", () => { }, ], [ - "unsafe bridge", + "metadata listener", (f: ReturnType) => { - f.network.Config[0]!.Gateway = "169.254.169.254"; + f.container.NetworkSettings.Ports["8000/tcp"][0]!.HostIp = "169.254.169.254"; }, ], [ diff --git a/src/lib/inference/serving/vllm-export-runtime.ts b/src/lib/inference/serving/vllm-export-runtime.ts index 086faa1b720..6edccfecdfe 100644 --- a/src/lib/inference/serving/vllm-export-runtime.ts +++ b/src/lib/inference/serving/vllm-export-runtime.ts @@ -38,7 +38,6 @@ import { recoverHostLocalManagedVllmEndpoint, type RecoverHostLocalManagedVllmOptions, } from "./vllm-host-local-lifecycle"; -import { validateManagedVllmBridgeHost } from "./vllm-host-local-network"; const { Type } = require("typebox") as typeof TypeBoxModule; const { Check } = require("typebox/value") as typeof TypeBoxValueModule; @@ -56,15 +55,6 @@ const ImageSchema = Type.Object( }, { additionalProperties: false }, ); -const NetworkSchema = Type.Object( - { - Id, - Name: Type.Literal("openshell-docker"), - Driver: Type.Literal("bridge"), - Config: Type.Array(Type.Object({ Gateway: Text, Subnet: Text }), { minItems: 1, maxItems: 1 }), - }, - { additionalProperties: false }, -); const ContainerSchema = Type.Object( { Id, @@ -84,8 +74,8 @@ const ContainerSchema = Type.Object( Ports: Type.Object( { "8000/tcp": Type.Array(Type.Object({ HostIp: Text, HostPort: Text }), { - minItems: 2, - maxItems: 2, + minItems: 1, + maxItems: 1, }), }, { additionalProperties: false }, @@ -269,15 +259,6 @@ export function observeManagedVllmForExport( image.Environment.some((value) => value.startsWith("VLLM_API_KEY=")) ) fail(); - const network = parse( - inspect( - "network", - "openshell-docker", - '{"Id":{{json .Id}},"Name":{{json .Name}},"Driver":{{json .Driver}},"Config":{{json .IPAM.Config}}}', - ), - NetworkSchema, - ); - const bridge = validateManagedVllmBridgeHost(network.Config[0]!.Gateway); const row = parse( inspect( "container", @@ -289,7 +270,6 @@ export function observeManagedVllmForExport( const recovered = recoverHostLocalManagedVllmEndpoint({ ...options.authentication, dockerInspect: () => JSON.stringify([row]), - resolveBridgeHost: () => bridge, }); if (!recovered || recovered.containerId !== row.Id || row.Image !== image.Id) fail(); if ( @@ -321,7 +301,6 @@ export function observeManagedVllmForExport( serving, containerId: row.Id, imageId: row.Image, - networkId: network.Id, startedAt: row.StartedAt, }; } catch { diff --git a/src/lib/inference/serving/vllm-host-local-lifecycle.test.ts b/src/lib/inference/serving/vllm-host-local-lifecycle.test.ts index 6e359366ec7..910518b0331 100644 --- a/src/lib/inference/serving/vllm-host-local-lifecycle.test.ts +++ b/src/lib/inference/serving/vllm-host-local-lifecycle.test.ts @@ -23,20 +23,6 @@ import { recoverHostLocalManagedVllmEndpoint, } from "./vllm-host-local-lifecycle"; -const networkMocks = vi.hoisted(() => ({ - resolveBridgeHost: vi.fn( - ( - _capture: NonNullable, - _dockerEnv?: Record, - ) => "172.18.0.1", - ), -})); - -vi.mock("./vllm-host-local-network", async (importOriginal) => ({ - ...(await importOriginal()), - resolveManagedVllmBridgeHost: networkMocks.resolveBridgeHost, -})); - const API_KEY = "b".repeat(64); const IDENTITY = { catalogDigest: `sha256:${"1".repeat(64)}`, @@ -62,7 +48,6 @@ function stateDir(): string { afterEach(() => { vi.unstubAllEnvs(); - networkMocks.resolveBridgeHost.mockClear(); for (const directory of temporaryDirectories.splice(0)) { fs.rmSync(directory, { force: true, recursive: true }); } @@ -72,7 +57,7 @@ function inspect( key = API_KEY, fingerprint = runtimeAuthFingerprint(key), labels: Record = {}, - bridgeHost = "172.18.0.1", + bridgeHost = "127.0.0.1", hostPort = "8000", ) { return JSON.stringify([ @@ -92,7 +77,7 @@ function inspect( Ports: { "8000/tcp": [ { HostIp: "127.0.0.1", HostPort: hostPort }, - { HostIp: bridgeHost, HostPort: hostPort }, + ...(bridgeHost === "127.0.0.1" ? [] : [{ HostIp: bridgeHost, HostPort: hostPort }]), ], }, }, @@ -101,7 +86,7 @@ function inspect( } describe("host-local managed vLLM recovery", () => { - it("pins inspection and bridge discovery to the physical default Docker daemon", () => { + it("pins inspection to the physical default Docker daemon", () => { vi.stubEnv("DOCKER_CONTEXT", "remote-context"); vi.stubEnv("DOCKER_HOST", "tcp://remote.example:2376"); vi.stubEnv("DOCKER_CONFIG", "/tmp/remote-docker-config"); @@ -126,11 +111,9 @@ describe("host-local managed vLLM recovery", () => { expect(dockerOptions?.env).toMatchObject({ DOCKER_CONTEXT: "default" }); expect(dockerOptions?.env).not.toHaveProperty("DOCKER_HOST"); expect(dockerOptions?.env).not.toHaveProperty("DOCKER_CONFIG"); - expect(networkMocks.resolveBridgeHost).toHaveBeenCalledWith(capture, dockerOptions?.env); - expect(networkMocks.resolveBridgeHost.mock.calls[0]?.[1]).toBe(dockerOptions?.env); }); - it("recovers one owned endpoint with loopback and current OpenShell bridge bindings", () => { + it("recovers one owned endpoint with one loopback binding", () => { const observed = vi.fn(); expect( recoverHostLocalManagedVllmEndpoint({ @@ -150,7 +133,7 @@ describe("host-local managed vLLM recovery", () => { expect( recoverHostLocalManagedVllmEndpoint({ dockerInspect: () => - inspect(API_KEY, runtimeAuthFingerprint(API_KEY), {}, "172.18.0.1", "19000"), + inspect(API_KEY, runtimeAuthFingerprint(API_KEY), {}, "127.0.0.1", "19000"), loadApiKey: () => API_KEY, }), ).toEqual({ @@ -166,7 +149,7 @@ describe("host-local managed vLLM recovery", () => { expect(() => recoverHostLocalManagedVllmEndpoint({ dockerInspect: () => - inspect(API_KEY, runtimeAuthFingerprint(API_KEY), {}, "172.18.0.1", hostPort), + inspect(API_KEY, runtimeAuthFingerprint(API_KEY), {}, "127.0.0.1", hostPort), loadApiKey: () => API_KEY, }), ).toThrow("unsafe or incomplete"); @@ -182,7 +165,7 @@ describe("host-local managed vLLM recovery", () => { ).toThrow("missing or mismatched"); }); - it("rejects a private binding that is not the current OpenShell bridge (#8379)", () => { + it("rejects an additional private binding (#8379)", () => { expect(() => recoverHostLocalManagedVllmEndpoint({ dockerInspect: () => inspect(API_KEY, runtimeAuthFingerprint(API_KEY), {}, "172.19.0.1"), diff --git a/src/lib/inference/serving/vllm-host-local-lifecycle.ts b/src/lib/inference/serving/vllm-host-local-lifecycle.ts index 44c2eeb2053..36777385906 100644 --- a/src/lib/inference/serving/vllm-host-local-lifecycle.ts +++ b/src/lib/inference/serving/vllm-host-local-lifecycle.ts @@ -13,13 +13,6 @@ import { loadManagedVllmApiKey, managedVllmApiKeyPath, managedVllmStateDir } fro import { buildLocalManagedVllmDockerEnv } from "../vllm-docker-env"; import { assertNoManagedDistributedVllmRuntimeReceipts } from "./managed-runtime-receipts"; import { runtimeAuthFingerprint } from "./runtime-auth-fingerprint"; -import { - resolveManagedVllmBridgeHost, - validateManagedVllmBridgeHost, -} from "./vllm-host-local-network"; - -export { resolveManagedVllmBridgeHost }; - export const HOST_LOCAL_VLLM_CONTAINER_NAME = "nemoclaw-vllm" as const; export const HOST_LOCAL_VLLM_MANAGED_LABEL = "com.nvidia.nemoclaw.managed-vllm" as const; export const HOST_LOCAL_VLLM_AUTH_LABEL = "com.nvidia.nemoclaw.managed-vllm-auth" as const; @@ -63,7 +56,6 @@ export interface RecoverHostLocalManagedVllmOptions { dockerInspect?: () => string; loadApiKey?: () => string | null; onManagedContainerObserved?: () => void; - resolveBridgeHost?: () => string; stateDir?: string; } @@ -296,11 +288,8 @@ export class ManagedVllmInspectionError extends Error { } } -/** Reject unsafe owned bearerless bindings and select their private probe address. */ -export function getLegacyHostLocalManagedVllmBridgeHost( - servingPort: number, - capture: typeof dockerCapture = dockerCapture, -): string | null { +/** Reject unsafe owned bearerless bindings and select their loopback probe address. */ +export function getLegacyHostLocalManagedVllmLoopbackHost(servingPort: number): string | null { const dockerEnv = buildLocalManagedVllmDockerEnv(); const inspection = dockerRun(["container", "inspect", HOST_LOCAL_VLLM_CONTAINER_NAME], { env: dockerEnv, @@ -398,11 +387,10 @@ export function getLegacyHostLocalManagedVllmBridgeHost( ports && typeof ports === "object" && !Array.isArray(ports) ? (ports as Record)[`${String(HOST_LOCAL_VLLM_CONTAINER_PORT)}/tcp`] : null; - const bridge = resolveManagedVllmBridgeHost(capture, dockerEnv); - const expectedHosts = new Set(["127.0.0.1", bridge]); + const expectedHosts = new Set(["127.0.0.1"]); if ( !Array.isArray(bindings) || - bindings.length !== 2 || + bindings.length !== 1 || !bindings.every( (binding) => binding && @@ -413,7 +401,7 @@ export function getLegacyHostLocalManagedVllmBridgeHost( ) { throw new UnsafeLegacyVllmBindingsError(); } - return bridge; + return "127.0.0.1"; } /** Recover only the exact authenticated host-local container with bounded host bindings. */ @@ -465,17 +453,6 @@ export function recoverHostLocalManagedVllmEndpoint( typeof binding === "object" && (binding as { HostIp?: unknown }).HostIp === "127.0.0.1", ); - const expectedBridgeHost = validateManagedVllmBridgeHost( - options.resolveBridgeHost - ? options.resolveBridgeHost() - : resolveManagedVllmBridgeHost(capture, dockerEnv), - ); - const bridgeBinding = bindings.find( - (binding) => - binding && - typeof binding === "object" && - (binding as { HostIp?: unknown }).HostIp === expectedBridgeHost, - ); const env = Array.isArray(row.Config?.Env) ? row.Config.Env : []; const configuredKeyRows = env.filter( (value): value is string => typeof value === "string" && value.startsWith("VLLM_API_KEY="), @@ -486,27 +463,18 @@ export function recoverHostLocalManagedVllmEndpoint( /^\d{4,5}$/u.test((loopbackBinding as { HostPort: string }).HostPort) ? Number((loopbackBinding as { HostPort: string }).HostPort) : Number.NaN; - const bridgeHostPort = - bridgeBinding && - typeof (bridgeBinding as { HostPort?: unknown }).HostPort === "string" && - /^\d{4,5}$/u.test((bridgeBinding as { HostPort: string }).HostPort) - ? Number((bridgeBinding as { HostPort: string }).HostPort) - : Number.NaN; if ( row.Id === undefined || typeof row.Id !== "string" || !/^[a-f0-9]{12,64}$/.test(row.Id) || row.Name !== `/${HOST_LOCAL_VLLM_CONTAINER_NAME}` || row.State?.Running !== true || - bindings.length !== 2 || + bindings.length !== 1 || !loopbackBinding || - !bridgeBinding || !Number.isSafeInteger(loopbackHostPort) || loopbackHostPort < 1024 || loopbackHostPort > 65_535 || String(loopbackHostPort) !== (loopbackBinding as { HostPort: string }).HostPort || - bridgeHostPort !== loopbackHostPort || - String(bridgeHostPort) !== (bridgeBinding as { HostPort: string }).HostPort || configuredKeyRows.length !== 1 || typeof authFingerprint !== "string" ) { diff --git a/src/lib/inference/serving/vllm-host-local-network.test.ts b/src/lib/inference/serving/vllm-host-local-network.test.ts deleted file mode 100644 index aa4140cb2e8..00000000000 --- a/src/lib/inference/serving/vllm-host-local-network.test.ts +++ /dev/null @@ -1,33 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -import { describe, expect, it, vi } from "vitest"; - -import type { dockerCapture } from "../../adapters/docker/local-model-runtime"; -import { resolveManagedVllmBridgeHost } from "./vllm-host-local-network"; - -function capture(value: string): typeof dockerCapture { - return vi.fn(() => value) as unknown as typeof dockerCapture; -} - -describe("managed host-local vLLM bridge", () => { - it("resolves one private OpenShell bridge gateway (#8379)", () => { - const run = capture(JSON.stringify([{ Subnet: "172.18.0.0/16", Gateway: "172.18.0.1" }])); - const dockerEnv = { DOCKER_CONTEXT: "default", PATH: "/usr/bin" }; - expect(resolveManagedVllmBridgeHost(run, dockerEnv)).toBe("172.18.0.1"); - expect(run).toHaveBeenCalledWith( - ["network", "inspect", "--format", "{{json .IPAM.Config}}", "openshell-docker"], - { env: dockerEnv, replaceEnv: true, ignoreError: true, timeout: 10_000 }, - ); - }); - - it.each([ - ["invalid JSON", "not-json"], - ["missing gateway", JSON.stringify([{ Subnet: "172.18.0.0/16" }])], - ["public gateway", JSON.stringify([{ Gateway: "203.0.113.1" }])], - ["zero gateway", JSON.stringify([{ Gateway: "0.0.0.0" }])], - ["multiple gateways", JSON.stringify([{ Gateway: "172.18.0.1" }, { Gateway: "172.19.0.1" }])], - ])("rejects %s", (_label, value) => { - expect(() => resolveManagedVllmBridgeHost(capture(value))).toThrow(/OpenShell bridge|private/); - }); -}); diff --git a/src/lib/inference/serving/vllm-host-local-network.ts b/src/lib/inference/serving/vllm-host-local-network.ts deleted file mode 100644 index 220b679e920..00000000000 --- a/src/lib/inference/serving/vllm-host-local-network.ts +++ /dev/null @@ -1,61 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -import net from "node:net"; - -import { dockerCapture } from "../../adapters/docker/local-model-runtime"; -import { buildLocalManagedVllmDockerEnv } from "../vllm-docker-env"; -export function validateManagedVllmBridgeHost(value: string): string { - const [first, second] = value.split(".").map(Number); - const privateIpv4 = - net.isIP(value) === 4 && - (first === 10 || - (first === 172 && second! >= 16 && second! <= 31) || - (first === 192 && second === 168)); - if (!privateIpv4) { - throw new Error("Managed host-local vLLM requires one private OpenShell bridge address"); - } - return value; -} - -function isManagedVllmBridgeHost(value: unknown): value is string { - if (typeof value !== "string") return false; - try { - validateManagedVllmBridgeHost(value); - return true; - } catch { - return false; - } -} - -export function resolveManagedVllmBridgeHost( - capture: typeof dockerCapture = dockerCapture, - dockerEnv: Record = buildLocalManagedVllmDockerEnv(), -): string { - const raw = capture( - ["network", "inspect", "--format", "{{json .IPAM.Config}}", "openshell-docker"], - { - env: dockerEnv, - replaceEnv: true, - ignoreError: true, - timeout: 10_000, - }, - ).trim(); - let parsed: unknown; - try { - parsed = JSON.parse(raw); - } catch { - throw new Error("Managed host-local vLLM could not inspect the OpenShell bridge"); - } - const gateways = Array.isArray(parsed) - ? parsed.flatMap((entry) => { - if (!entry || typeof entry !== "object" || Array.isArray(entry)) return []; - const gateway = (entry as Record).Gateway; - return isManagedVllmBridgeHost(gateway) ? [gateway] : []; - }) - : []; - if (gateways.length !== 1) { - throw new Error("Managed host-local vLLM requires one private OpenShell bridge address"); - } - return validateManagedVllmBridgeHost(gateways[0]!); -} diff --git a/src/lib/inference/serving/vllm-host-local-probe.test.ts b/src/lib/inference/serving/vllm-host-local-probe.test.ts index f4a871ef408..011ca70a714 100644 --- a/src/lib/inference/serving/vllm-host-local-probe.test.ts +++ b/src/lib/inference/serving/vllm-host-local-probe.test.ts @@ -21,7 +21,6 @@ import { } from "../local"; import { MANAGED_VLLM_API_KEY_FILE, managedVllmStateDir } from "../vllm-api-key"; -import type { dockerCapture as captureDocker } from "../../adapters/docker/local-model-runtime"; import { DUAL_STATION_VLLM_RUNTIME_RECEIPT_FILE, MANAGED_CLUSTER_VLLM_RUNTIME_RECEIPT_FILE, @@ -29,10 +28,9 @@ import { import { runtimeAuthFingerprint } from "./runtime-auth-fingerprint"; import { HOST_LOCAL_VLLM_RUNTIME_RECEIPT_FILE, - getLegacyHostLocalManagedVllmBridgeHost, + getLegacyHostLocalManagedVllmLoopbackHost, ManagedVllmInspectionError, recoverHostLocalManagedVllmEndpoint, - resolveManagedVllmBridgeHost, } from "./vllm-host-local-lifecycle"; const subprocess = vi.hoisted(() => ({ @@ -54,7 +52,7 @@ vi.mock("node:child_process", async (importOriginal) => ({ spawnSync: subprocess.spawnSync, })); -function ownedContainer(bridge = "172.18.0.1") { +function ownedContainer() { return { Id: "a".repeat(64), Name: "/nemoclaw-vllm", @@ -65,22 +63,12 @@ function ownedContainer(bridge = "172.18.0.1") { }, NetworkSettings: { Ports: { - "8000/tcp": [ - { HostIp: "127.0.0.1", HostPort: "18080" }, - { HostIp: bridge, HostPort: "18080" }, - ], + "8000/tcp": [{ HostIp: "127.0.0.1", HostPort: "18080" }], }, }, }; } -function dockerCapture(row: ReturnType, gateway = "172.18.0.1") { - interceptDocker(row); - return vi.fn((argv, _options) => - argv[0] === "container" ? JSON.stringify([row]) : JSON.stringify([{ Gateway: gateway }]), - ); -} - function interceptDocker( local: ReturnType | null, remote: ReturnType | null = null, @@ -161,26 +149,18 @@ describe("host-local managed vLLM container probe", () => { ])("rejects local publication on %s despite a %s remote container (#14083)", (host, remote) => { const row = ownedContainer(); row.NetworkSettings.Ports["8000/tcp"] = [{ HostIp: host, HostPort: "18080" }]; - interceptDocker(row, remote === "safe" ? ownedContainer("172.19.0.1") : null); - expect(() => getLegacyHostLocalManagedVllmBridgeHost(18080)).toThrow( + interceptDocker(row, remote === "safe" ? ownedContainer() : null); + expect(() => getLegacyHostLocalManagedVllmLoopbackHost(18080)).toThrow( "Managed bearerless vLLM has unsafe host bindings. Follow 'Handle a Running vLLM Server' in the vLLM setup guide before recreating the owned `nemoclaw-vllm` container. Preserve the original model and port settings.", ); - expectLocalSpawns(2); - }); - - it("inspects the default local daemon and its bridge despite ambient Docker selectors (#14083)", () => { - interceptDocker(ownedContainer(), ownedContainer("172.19.0.1")); - expect(getLegacyHostLocalManagedVllmBridgeHost(18080)).toBe("172.18.0.1"); - expectLocalSpawns(2); - expect(subprocess.spawnSync.mock.calls[1]?.[2].env).toEqual( - subprocess.spawnSync.mock.calls[0]?.[2].env, - ); + expectLocalSpawns(1); }); - it("pins standalone host-local bridge discovery to the same local daemon", () => { - interceptDocker(ownedContainer()); - expect(resolveManagedVllmBridgeHost()).toBe("172.18.0.1"); + it("inspects loopback publication on the default local daemon despite ambient Docker selectors (#14083)", () => { + interceptDocker(ownedContainer(), ownedContainer()); + expect(getLegacyHostLocalManagedVllmLoopbackHost(18080)).toBe("127.0.0.1"); expectLocalSpawns(1); + expect(subprocess.spawnSync.mock.calls[0]?.[1]).not.toContain("network"); }); it.each([ @@ -384,7 +364,7 @@ describe("host-local managed vLLM container probe", () => { : { status: 1, stdout: "", stderr: "Cannot connect to the Docker daemon" }, ); const probe = vi.fn(); - expect(() => getLegacyHostLocalManagedVllmBridgeHost(8000)).toThrow( + expect(() => getLegacyHostLocalManagedVllmLoopbackHost(8000)).toThrow( ManagedVllmInspectionError, ); expect(() => getManagedVllmProviderBinding()).toThrow(); @@ -413,17 +393,6 @@ describe("host-local managed vLLM container probe", () => { expect(probe).not.toHaveBeenCalled(); }); - it("preserves an explicitly selected bridge environment for generic installation", () => { - interceptDocker(null); - const env = { DOCKER_CONTEXT: "requested-builder", PATH: "/usr/bin" }; - expect(resolveManagedVllmBridgeHost(undefined, env)).toBe("172.19.0.1"); - expect(subprocess.spawnSync).toHaveBeenCalledExactlyOnceWith( - "docker", - ["network", "inspect", "--format", "{{json .IPAM.Config}}", "openshell-docker"], - expect.objectContaining({ env, shell: false, timeout: 10_000 }), - ); - }); - it("preserves local daemon authority for authenticated recovery too", () => { const apiKey = "b".repeat(64); const row = ownedContainer(); @@ -435,7 +404,7 @@ describe("host-local managed vLLM container probe", () => { apiKey, containerId: row.Id, }); - expectLocalSpawns(2); + expectLocalSpawns(1); }); it.each([ @@ -464,11 +433,12 @@ describe("host-local managed vLLM container probe", () => { }, ], ] satisfies Array<[string, (row: ReturnType) => void]>)( - "does not select the bridge for a %s", + "does not select loopback for a %s", (_label, change) => { const row = ownedContainer(); change(row); - expect(getLegacyHostLocalManagedVllmBridgeHost(18080, dockerCapture(row))).toBeNull(); + interceptDocker(row); + expect(getLegacyHostLocalManagedVllmLoopbackHost(18080)).toBeNull(); }, ); @@ -477,20 +447,22 @@ describe("host-local managed vLLM container probe", () => { ["a mismatched host port", { HostIp: "172.18.0.1", HostPort: "19000" }], ] as const)("rejects an owned container with %s", (_label, binding) => { const row = ownedContainer(); - row.NetworkSettings.Ports["8000/tcp"][1] = binding; - expect(() => getLegacyHostLocalManagedVllmBridgeHost(18080, dockerCapture(row))).toThrow( - "unsafe host bindings", - ); + row.NetworkSettings.Ports["8000/tcp"].push(binding); + interceptDocker(row); + expect(() => getLegacyHostLocalManagedVllmLoopbackHost(18080)).toThrow("unsafe host bindings"); }); - it("rejects a public OpenShell gateway", () => { - expect(() => - getLegacyHostLocalManagedVllmBridgeHost( - 18080, - dockerCapture(ownedContainer(), "203.0.113.1"), - ), - ).toThrow("requires one private OpenShell bridge address"); - }); + it.each(["172.18.0.1", "203.0.113.1", "::1"])( + "rejects a non-IPv4-loopback binding at %s", + (address) => { + const row = ownedContainer(); + row.NetworkSettings.Ports["8000/tcp"] = [{ HostIp: address, HostPort: "18080" }]; + interceptDocker(row); + expect(() => getLegacyHostLocalManagedVllmLoopbackHost(18080)).toThrow( + "unsafe host bindings", + ); + }, + ); it("preserves the legacy probe when the local container is confirmed absent", () => { interceptDocker(null, ownedContainer()); @@ -500,7 +472,7 @@ describe("host-local managed vLLM container probe", () => { ? { status: 1, stdout: "[]", stderr: "Error: No such object: nemoclaw-vllm" } : capture(file, args, options), ); - expect(getLegacyHostLocalManagedVllmBridgeHost(18080)).toBeNull(); + expect(getLegacyHostLocalManagedVllmLoopbackHost(18080)).toBeNull(); expectLocalSpawns(1); }); @@ -518,7 +490,7 @@ describe("host-local managed vLLM container probe", () => { ])("rejects an inspection with $state instead of treating it as absence", (result) => { interceptDocker(null, ownedContainer()); subprocess.spawnSync.mockReturnValue(result as ReturnType); - expect(() => getLegacyHostLocalManagedVllmBridgeHost(18080)).toThrow( + expect(() => getLegacyHostLocalManagedVllmLoopbackHost(18080)).toThrow( "Managed vLLM container inspection failed", ); expectLocalSpawns(1); diff --git a/src/lib/inference/serving/vllm-managed-support.ts b/src/lib/inference/serving/vllm-managed-support.ts index b9a3a88fbcc..481d5b6eb08 100644 --- a/src/lib/inference/serving/vllm-managed-support.ts +++ b/src/lib/inference/serving/vllm-managed-support.ts @@ -25,7 +25,3 @@ export { persistHostLocalVllmRuntimeReceipt, recoverHostLocalManagedVllmEndpoint, } from "./vllm-host-local-lifecycle.js"; -export { - resolveManagedVllmBridgeHost, - validateManagedVllmBridgeHost, -} from "./vllm-host-local-network.js"; diff --git a/src/lib/inference/vllm-configured-port.test.ts b/src/lib/inference/vllm-configured-port.test.ts index a3636ed9a1d..8235d85768e 100644 --- a/src/lib/inference/vllm-configured-port.test.ts +++ b/src/lib/inference/vllm-configured-port.test.ts @@ -8,27 +8,11 @@ vi.mock("../core/ports", async (importOriginal) => ({ VLLM_PORT: 19_000, })); -import { buildVllmRunArgs, detectVllmProfile } from "./vllm"; +import { buildVllmRunArgs } from "./vllm"; describe("managed vLLM configured host port", () => { - it.each([undefined, "0.0.0.0", "::", "203.0.113.1"])( - "rejects bridge address %s without managed bearer authentication (#14083)", - (bridgeHost) => { - const profile = detectVllmProfile({ platform: "linux", type: "nvidia" })!; - expect(() => - buildVllmRunArgs( - profile, - { ...profile.defaultModel, managedBearerAuth: undefined }, - profile.dockerRunFlags, - {}, - bridgeHost, - ), - ).toThrow("requires one private OpenShell bridge address"); - }, - ); - it.each([undefined, true] as const)( - "publishes the configured port on loopback and the OpenShell bridge when managed bearer authentication is %s (#14083)", + "publishes the configured port only on loopback when managed bearer authentication is %s (#14083)", (managedBearerAuth) => { const model = { id: "test/model", @@ -55,16 +39,11 @@ describe("managed vLLM configured host port", () => { loadTimeoutSec: 60, }; - const args = buildVllmRunArgs( - profile, - model, - profile.dockerRunFlags, - { VLLM_API_KEY: "a".repeat(64) }, - "172.18.0.1", - ); + const args = buildVllmRunArgs(profile, model, profile.dockerRunFlags, { + VLLM_API_KEY: "a".repeat(64), + }); expect(args.flatMap((value, index) => (value === "-p" ? [args[index + 1]] : []))).toEqual([ "127.0.0.1:19000:8000", - "172.18.0.1:19000:8000", ]); }, ); diff --git a/src/lib/inference/vllm-fixed-catalog-install.test.ts b/src/lib/inference/vllm-fixed-catalog-install.test.ts index 0a0951038ab..a63f8195088 100644 --- a/src/lib/inference/vllm-fixed-catalog-install.test.ts +++ b/src/lib/inference/vllm-fixed-catalog-install.test.ts @@ -262,12 +262,12 @@ describe("fixed catalog vLLM installs", () => { ); }); - it("rejects an unsafe bridge before downloads or container replacement (#14083)", async () => { + it("rejects unreadable ownership before downloads or container replacement (#14083)", async () => { const profile = detectVllmProfile({ platform: "linux", type: "nvidia" })!; mockSuccessfulVllmInstall(mocks, profile.containerName); const capture = mocks.dockerCapture.getMockImplementation()!; mocks.dockerCapture.mockImplementation((args, ...rest) => - args[0] === "network" ? JSON.stringify([{ Gateway: "0.0.0.0" }]) : capture(args, ...rest), + args[0] === "container" && args[1] === "ls" ? "invalid inspection" : capture(args, ...rest), ); const result = await installVllm(profile, { @@ -283,17 +283,17 @@ describe("fixed catalog vLLM installs", () => { expect(mocks.dockerForceRm).not.toHaveBeenCalled(); expect(mocks.dockerRunDetached).not.toHaveBeenCalled(); expect(spies.errSpy.mock.calls.map((call) => String(call[0])).join("\n")).toContain( - "requires one private OpenShell bridge address", + "Could not verify ownership", ); }); - it("rechecks the bridge after downloads before replacing a host-local container", async () => { + it("rechecks ownership after downloads before replacing a host-local container", async () => { const profile = detectVllmProfile({ platform: "linux", type: "nvidia" })!; mockSuccessfulVllmInstall(mocks, profile.containerName); const capture = mocks.dockerCapture.getMockImplementation()!; - const bridge = vi.fn().mockReturnValueOnce("172.18.0.1").mockReturnValue("0.0.0.0"); + const inspection = vi.fn().mockReturnValueOnce("").mockReturnValue("invalid inspection"); mocks.dockerCapture.mockImplementation((args, ...rest) => - args[0] === "network" ? JSON.stringify([{ Gateway: bridge() }]) : capture(args, ...rest), + args[0] === "container" && args[1] === "ls" ? inspection() : capture(args, ...rest), ); const result = await installVllm(profile, { @@ -407,7 +407,6 @@ describe("fixed catalog vLLM installs", () => { nonInteractive: true, promptFn: vi.fn(), readinessReports, - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(spies.errSpy).not.toHaveBeenCalledWith( @@ -441,7 +440,6 @@ describe("fixed catalog vLLM installs", () => { nonInteractive: true, promptFn: vi.fn(), readinessReports, - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(result).toEqual({ ok: true }); @@ -481,7 +479,6 @@ describe("fixed catalog vLLM installs", () => { promptFn: vi.fn(), modelIntent: checkpointModel, readinessReports, - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(result, spies.errSpy.mock.calls.flat().join("\n")).toEqual({ ok: true }); const [args, options] = mocks.dockerRunDetached.mock.calls[0]!; @@ -579,7 +576,6 @@ describe("fixed catalog vLLM installs", () => { modelIntent, readinessReports, beforeInstall, - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(spies.errSpy).not.toHaveBeenCalledWith( @@ -737,7 +733,6 @@ describe("fixed catalog vLLM installs", () => { nonInteractive: false, promptFn, readinessReports, - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(result).toEqual({ ok: true }); @@ -775,7 +770,6 @@ describe("fixed catalog vLLM installs", () => { nonInteractive: true, promptFn: vi.fn(), readinessReports: vllmInstallTestReadiness(profile, modelIntent), - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(result).toEqual({ ok: true }); diff --git a/src/lib/inference/vllm-install.test-support.ts b/src/lib/inference/vllm-install.test-support.ts index e4a87059216..00518833fc0 100644 --- a/src/lib/inference/vllm-install.test-support.ts +++ b/src/lib/inference/vllm-install.test-support.ts @@ -198,7 +198,6 @@ export function withVllmInstallTestReadiness( options: InstallVllmOptions, ): InstallVllmOptions { return { - resolveManagedBridgeHost: () => "172.18.0.1", ...options, readinessReports: vllmInstallTestReadiness(profile), }; diff --git a/src/lib/inference/vllm-serving-port.test.ts b/src/lib/inference/vllm-serving-port.test.ts index 2c0614d9cec..15dfd4480e6 100644 --- a/src/lib/inference/vllm-serving-port.test.ts +++ b/src/lib/inference/vllm-serving-port.test.ts @@ -168,7 +168,6 @@ describe("managed vLLM serving-port guard (#8685)", () => { hasImage: true, nonInteractive: false, promptFn, - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(result).toEqual({ ok: true }); @@ -479,7 +478,6 @@ describe("managed vLLM serving-port guard (#8685)", () => { const runArgs = mocks.dockerRunDetached.mock.calls[0]![0] as string[]; expect(runArgs.flatMap((arg, i) => (arg === "-p" ? [runArgs[i + 1]] : []))).toEqual([ "127.0.0.1:8000:8000", - "172.18.0.1:8000:8000", ]); expect(errSpy.mock.calls.flat().join("\n")).not.toContain("another process"); const dockerOptions = [ diff --git a/src/lib/inference/vllm.test.ts b/src/lib/inference/vllm.test.ts index 1160684e407..4365d97a51a 100644 --- a/src/lib/inference/vllm.test.ts +++ b/src/lib/inference/vllm.test.ts @@ -259,11 +259,9 @@ describe("vLLM profile detection", () => { resolved.model, runtime.dockerRunFlags, {} as NodeJS.ProcessEnv, - "172.18.0.1", ); expect(args.flatMap((value, index) => (value === "-p" ? [args[index + 1]] : []))).toEqual([ "127.0.0.1:8000:8000", - "172.18.0.1:8000:8000", ]); expect(args).toEqual( expect.arrayContaining([ @@ -305,16 +303,12 @@ describe("vLLM profile detection", () => { expect(runtime.modelDownloadSizeBytes).toBe(25_447_097_878); const apiKey = "a".repeat(64); - const args = buildVllmRunArgs( - runtime, - resolved.model, - runtime.dockerRunFlags, - { VLLM_API_KEY: apiKey }, - "172.18.0.1", - ); + const args = buildVllmRunArgs(runtime, resolved.model, runtime.dockerRunFlags, { + VLLM_API_KEY: apiKey, + }); const bindings = args.flatMap((value, index) => (value === "-p" ? [args[index + 1]] : [])); - expect(bindings).toEqual(["127.0.0.1:8000:8000", "172.18.0.1:8000:8000"]); + expect(bindings).toEqual(["127.0.0.1:8000:8000"]); expect(args).toEqual(expect.arrayContaining(["--env", "VLLM_API_KEY", runtime.image])); expect(args).not.toContain(apiKey); }); @@ -420,26 +414,14 @@ describe("vLLM run command", () => { it("adds Docker init so restarts can reap the server process (#7219)", () => { const profile = detectVllmProfile({ platform: "spark", type: "nvidia" }); expect(profile).not.toBeNull(); - const args = buildVllmRunArgs( - profile!, - profile!.defaultModel, - profile!.dockerRunFlags, - {}, - "172.18.0.1", - ); + const args = buildVllmRunArgs(profile!, profile!.defaultModel, profile!.dockerRunFlags, {}); expect(args.slice(0, 4)).toEqual(["--pull=never", "--init", "--restart", "unless-stopped"]); }); it("adds --restart unless-stopped so the container survives a host reboot (#4886)", () => { const profile = detectVllmProfile({ platform: "spark", type: "nvidia" }); expect(profile).not.toBeNull(); - const args = buildVllmRunArgs( - profile!, - profile!.defaultModel, - profile!.dockerRunFlags, - {}, - "172.18.0.1", - ); + const args = buildVllmRunArgs(profile!, profile!.defaultModel, profile!.dockerRunFlags, {}); expect(args).toEqual(expect.arrayContaining(["--restart", "unless-stopped"])); expect(args).toContain("--name"); expect(args[args.indexOf("--name") + 1]).toBe(profile!.containerName); @@ -448,10 +430,21 @@ describe("vLLM run command", () => { ); expect(args.flatMap((value, index) => (value === "-p" ? [args[index + 1]] : []))).toEqual([ "127.0.0.1:8000:8000", - "172.18.0.1:8000:8000", ]); }); + it("publishes authenticated managed vLLM only on loopback (#8379)", () => { + const profile = detectVllmProfile({ platform: "spark", type: "nvidia" })!; + const model = { ...profile.defaultModel, managedBearerAuth: true as const }; + const args = buildVllmRunArgs(profile, model, profile.dockerRunFlags, { + VLLM_API_KEY: "a".repeat(64), + }); + const bindings = args.flatMap((value, index) => (value === "-p" ? [args[index + 1]] : [])); + + expect(bindings).toEqual(["127.0.0.1:8000:8000"]); + expect(bindings.join(" ")).not.toMatch(/(?:0\.0\.0\.0|\[?::\]?):8000/u); + }); + it("labels catalog-selected host-local containers with immutable recipe provenance (#8246)", () => { const profile = detectVllmProfile({ platform: "spark", type: "nvidia" }); expect(profile).not.toBeNull(); @@ -470,7 +463,6 @@ describe("vLLM run command", () => { catalogProfile.defaultModel, catalogProfile.dockerRunFlags, {}, - "172.18.0.1", ); expect(args).toEqual( @@ -495,7 +487,6 @@ describe("vLLM run command", () => { profile!.defaultModel, ["--gpus", '"device=0,1"', "--ipc=host"], { VLLM_API_KEY: "a".repeat(64) }, - "172.18.0.1", ); expect(args).toEqual(expect.arrayContaining(["--gpus", '"device=0,1"', "--ipc=host"])); expect(args).toContain(profile!.image); @@ -507,15 +498,9 @@ describe("vLLM run command", () => { const profile = detectVllmProfile({ platform: "spark", type: "nvidia" }); expect(profile).not.toBeNull(); const labelValue = "profile=$(touch /tmp/nemoclaw-vllm-pwn)"; - const args = buildVllmRunArgs( - profile!, - profile!.defaultModel, - ["--label", labelValue], - { - HF_TOKEN: "hf_test", - } as NodeJS.ProcessEnv, - "172.18.0.1", - ); + const args = buildVllmRunArgs(profile!, profile!.defaultModel, ["--label", labelValue], { + HF_TOKEN: "hf_test", + } as NodeJS.ProcessEnv); expect(args).toEqual(expect.arrayContaining(["--label", labelValue])); expect(args).not.toContain(`--label ${labelValue}`); @@ -1018,7 +1003,6 @@ describe("installVllm model resolution", () => { hasImage: true, nonInteractive: true, promptFn: vi.fn(), - resolveManagedBridgeHost: () => "172.18.0.1", }); expect(result).toEqual({ ok: false }); expect(errSpy).toHaveBeenCalledWith(expect.stringContaining("unauthenticated model inventory")); @@ -1129,13 +1113,11 @@ describe("installVllm model resolution", () => { recipeId: servingCatalog.recipeId, }); mockSuccessfulVllmInstall(mocks, profile.containerName); - const resolveManagedBridgeHost = vi.fn(() => "172.18.0.1"); const result = await installVllm(baseProfile, { hasImage: false, nonInteractive: true, promptFn: vi.fn(), - resolveManagedBridgeHost, }); expect(result).toEqual({ ok: false }); @@ -1160,12 +1142,7 @@ describe("installVllm model resolution", () => { expect(options.env.DOCKER_HOST).toBeUndefined(); expect(options.env.DOCKER_CONFIG).toBeUndefined(); }); - expect(resolveManagedBridgeHost).toHaveBeenCalledWith( - expect.objectContaining({ - DOCKER_CONTEXT: "default", - VLLM_API_KEY: "b".repeat(64), - }), - ); + expect(mocks.dockerCapture.mock.calls.some(([args]) => args[0] === "network")).toBe(false); }); it("fails before image pull when the host Hugging Face cache cannot be created", async () => { diff --git a/src/lib/inference/vllm.ts b/src/lib/inference/vllm.ts index ccaba9c3d53..13311936fa6 100644 --- a/src/lib/inference/vllm.ts +++ b/src/lib/inference/vllm.ts @@ -47,12 +47,10 @@ import { recoverHostLocalManagedVllmEndpoint, recoverInstalledManagedClusterVllmEndpoint, resolveHostLocalVllmSelection, - resolveManagedVllmBridgeHost, resolveNvidiaSmiCommand, resolveVllmInstallModel, runtimeAuthFingerprint, tryInstallManagedClusterManagedVllm, - validateManagedVllmBridgeHost, } from "./serving/vllm-managed-support"; import { assertGatedModelAccess, @@ -787,7 +785,6 @@ export function buildVllmRunArgs( model: VllmModelDef, runFlags: readonly string[], env: NodeJS.ProcessEnv = process.env, - managedBridgeHost?: string, ): string[] { assertVllmRegistryDigestRef(profile.image); const image = validateDockerArg(profile.image, "vLLM image"); @@ -797,7 +794,6 @@ export function buildVllmRunArgs( if (model.managedBearerAuth && !/^[a-f0-9]{64}$/.test(managedApiKey)) { throw new Error("Managed host-local vLLM requires a valid host-global API key"); } - const managedPublishHost = validateManagedVllmBridgeHost(managedBridgeHost ?? ""); return [ "--pull=never", "--init", @@ -830,8 +826,6 @@ export function buildVllmRunArgs( : []), "-p", `127.0.0.1:${String(VLLM_PORT)}:8000`, - "-p", - `${managedPublishHost}:${String(VLLM_PORT)}:8000`, "--name", containerName, "--entrypoint", @@ -1208,31 +1202,10 @@ export async function persistConfiguredManagedVllmRuntimeReceipt(): Promise, - resolveBridgeHost: (dockerEnv: Record) => string = (env) => - resolveManagedVllmBridgeHost(dockerCapture, env), - expectedReplacementContainerId?: string, -): { ok: true } | { ok: false; reason: string } { - try { - validateManagedVllmBridgeHost(resolveBridgeHost(dockerEnv)); - } catch (error) { - return { ok: false, reason: (error as Error).message }; - } - return vllmContainerReplacementTarget( - profile.containerName, - dockerEnv, - expectedReplacementContainerId, - ); -} - function startContainer( profile: VllmProfile, model: VllmModelDef, dockerEnv: Record, - resolveBridgeHost: (dockerEnv: Record) => string = (env) => - resolveManagedVllmBridgeHost(dockerCapture, env), expectedReplacementContainerId?: string, ): { ok: true; containerId: string } | { ok: false; reason: string } { emit(`Starting vLLM container (${profile.containerName})`); @@ -1249,13 +1222,7 @@ function startContainer( ? {} : { [VLLM_EXTRA_ARGS_ENV]: process.env[VLLM_EXTRA_ARGS_ENV] }), }; - runArgs = buildVllmRunArgs( - profile, - model, - resolvedFlags, - commandEnv, - resolveBridgeHost(dockerEnv), - ); + runArgs = buildVllmRunArgs(profile, model, resolvedFlags, commandEnv); } catch (err) { return { ok: false, reason: (err as Error).message }; } @@ -1811,7 +1778,6 @@ export interface InstallVllmOptions { checkpointInstallIntent?: (modelId: string) => void; /** Secret-free model restored from an unfinished onboarding checkpoint. */ modelIntent?: string; - resolveManagedBridgeHost?: (dockerEnv: Record) => string; /** Reuse an already-collected readiness snapshot instead of probing the host again. */ readinessReports?: readonly ManagedInferenceReadinessSource[]; /** @@ -2496,10 +2462,9 @@ async function runVllmInstall( return { ok: false }; } } else { - const preflight = preflightHostLocalVllmContainer( - runtimeProfile, + const preflight = vllmContainerReplacementTarget( + runtimeProfile.containerName, localDockerEnv, - opts.resolveManagedBridgeHost, recoveredHostLocalContainerId, ); if (!preflight.ok) { @@ -2783,7 +2748,6 @@ async function runVllmInstall( runtimeProfile, model, localDockerEnv, - opts.resolveManagedBridgeHost, recoveredHostLocalContainerId, ); if (!start.ok) { diff --git a/src/lib/inference/vllm/gpu-device.test.ts b/src/lib/inference/vllm/gpu-device.test.ts index e18af8db737..61a0fedbf65 100644 --- a/src/lib/inference/vllm/gpu-device.test.ts +++ b/src/lib/inference/vllm/gpu-device.test.ts @@ -18,7 +18,6 @@ describe("managed vLLM GPU selection", () => { selected.defaultModel, selected.buildDockerRunFlags?.() ?? selected.dockerRunFlags, {}, - "172.18.0.1", ); expect(args.slice(args.indexOf("--gpus"), args.indexOf("--gpus") + 2)).toEqual([ diff --git a/src/lib/messaging/applier/__test-helpers__/provider-fixtures.ts b/src/lib/messaging/applier/__test-helpers__/provider-fixtures.ts new file mode 100644 index 00000000000..e59973875e0 --- /dev/null +++ b/src/lib/messaging/applier/__test-helpers__/provider-fixtures.ts @@ -0,0 +1,94 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { vi } from "vitest"; +import type { OpenShellProviderAdapter } from "../../../adapters/openshell/provider-adapter"; +import type { SandboxMessagingPlan } from "../../manifest"; +import type { MessagingCredentialProviderEphemeralInput } from "../types"; + +export const target = { kind: "named", gatewayName: "nemoclaw" } as const; +export const plan: SandboxMessagingPlan = { + schemaVersion: 1, + sandboxName: "alpha", + agent: "openclaw", + workflow: "onboard", + channels: [], + disabledChannels: [], + credentialBindings: [], + networkPolicy: { presets: [], entries: [] }, + agentRender: [], + buildSteps: [], + stateUpdates: [], + healthChecks: [], +}; + +export function definition( + overrides: Partial = {}, +): MessagingCredentialProviderEphemeralInput { + return { + channelId: "telegram", + credentialId: "TELEGRAM_BOT_TOKEN", + providerName: "alpha-telegram-bridge", + providerType: "nemoclaw-messaging-v1", + credentials: [{ name: "TELEGRAM_BOT_TOKEN", value: "telegram-secret" }], + profile: { profilePath: "/repo/messaging.yaml", profileType: "nemoclaw-messaging-v1" }, + ...overrides, + }; +} + +export function metadata(input: MessagingCredentialProviderEphemeralInput) { + return { + name: input.providerName, + type: input.providerType, + credentialKeys: input.credentials.map(({ name }) => name), + configKeys: [], + }; +} + +export function providerAdapter( + overrides: Partial = {}, +): OpenShellProviderAdapter { + return { + ensureProviderPolicyComposition: vi + .fn() + .mockResolvedValue({ ok: true, value: undefined }), + listProviders: vi + .fn() + .mockResolvedValue({ ok: true, value: { names: [] } }), + createProvider: vi + .fn() + .mockResolvedValue({ ok: true }), + getProvider: vi.fn().mockResolvedValue({ + ok: false, + error: { kind: "command", reason: "not_found", message: "provider not found" }, + }), + updateProvider: vi + .fn() + .mockResolvedValue({ ok: true }), + importProviderProfile: vi + .fn() + .mockResolvedValue({ ok: true }), + inspectProviderProfile: vi + .fn() + .mockResolvedValue({ ok: true, value: { credentialKeys: [] } }), + deleteProvider: vi + .fn() + .mockResolvedValue({ ok: true }), + detachProvider: vi + .fn() + .mockResolvedValue({ ok: true, value: { changed: true } }), + attachProvider: vi + .fn() + .mockResolvedValue({ ok: true }), + listProviderAttachments: vi + .fn() + .mockResolvedValue({ ok: true, value: { names: [] } }), + configureProviderRefresh: vi + .fn() + .mockResolvedValue({ ok: true }), + getProviderRefreshStatus: vi + .fn() + .mockResolvedValue({ ok: true, value: { status: "refreshed" } }), + ...overrides, + }; +} diff --git a/src/lib/messaging/applier/credential-family-profile.test.ts b/src/lib/messaging/applier/credential-family-profile.test.ts new file mode 100644 index 00000000000..bd52754719c --- /dev/null +++ b/src/lib/messaging/applier/credential-family-profile.test.ts @@ -0,0 +1,404 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; +import { + EXTRA_PLACEHOLDER_KEYS_ENV, + registerExtraPlaceholderProviders, +} from "../../onboard/extra-placeholder-keys"; +import { + applyCredentialsAtOpenShell, + isMessagingProviderBindingConflict, +} from "./openshell-provider"; +import { buildMessagingProviderApplication } from "./provider-application"; +import { + target, + plan, + definition, + metadata, + providerAdapter, +} from "./__test-helpers__/provider-fixtures"; + +import { REPOSITORY_ROOT } from "../../core/repository-root"; +import { exportedProviderProfileMatchesContract } from "../../adapters/openshell/provider-profile"; +import { + buildCredentialFamilyProfile, + credentialFamilyProfileContract, + credentialFamilyProfileType, + matchesCredentialFamilyProfileType, + withCredentialFamilyProfile, +} from "./credential-family-profile"; + +const profilePath = path.join(REPOSITORY_ROOT, "nemoclaw-blueprint/provider-profiles/brave.yaml"); +const primary = "BRAVE_API_KEY"; +const keys = [primary, `${primary}_TEAM`]; + +describe("additional provider credential declarations", () => { + it("preserves the owned endpoint and executable boundaries while declaring the selected key", () => { + const original = { + id: "brave", + display_name: "Brave Search", + description: "Brave Search API access", + category: "agent", + credentials: [ + { + name: "api_key", + description: "Brave Search API key", + env_vars: [primary], + required: true, + auth_style: "header", + header_name: "x-subscription-token", + query_param: "", + }, + ], + endpoints: [ + { + host: "api.search.brave.com", + port: 443, + protocol: "rest", + access: "read-write", + enforcement: "enforce", + }, + ], + binaries: ["/usr/local/bin/node", "/usr/bin/node", "/usr/local/bin/curl", "/usr/bin/curl"], + inference_capable: false, + }; + const source = JSON.stringify(original); + const generated = buildCredentialFamilyProfile(source, primary, keys); + const profile = JSON.parse(generated.source); + expect({ ...profile, id: original.id, credentials: original.credentials }).toEqual(original); + expect(profile.credentials.at(-1)).toEqual({ + name: "BRAVE_API_KEY_TEAM", + env_vars: ["BRAVE_API_KEY_TEAM"], + required: false, + auth_style: "header", + header_name: "x-subscription-token", + query_param: "", + }); + const expected = credentialFamilyProfileContract(profilePath, primary, keys); + expect(exportedProviderProfileMatchesContract(generated.source, expected)).toBe(true); + profile.endpoints.push({ host: "unowned.example", port: 443 }); + expect(exportedProviderProfileMatchesContract(JSON.stringify(profile), expected)).toBe(false); + }); + + it("binds the profile to credential names regardless of input order", () => { + const type = credentialFamilyProfileType("brave", primary, keys); + expect(credentialFamilyProfileType("brave", primary, [...keys].reverse())).toBe(type); + expect(matchesCredentialFamilyProfileType(type, "brave", primary, keys)).toBe(true); + expect(matchesCredentialFamilyProfileType(type, "brave", primary, [primary])).toBe(false); + expect(matchesCredentialFamilyProfileType(type, "tavily", primary, keys)).toBe(false); + }); + + it.each([ + [primary, "GITHUB_TOKEN"], + [primary, `${primary}_`], + [primary, `${primary}_TEAM`, `${primary}_TEAM`], + [primary, `${primary}_TEAM\nOTHER_KEY`], + [`${primary}_TEAM`], + ])("rejects names outside the selected credential family: %j", (...invalid) => { + expect(() => + buildCredentialFamilyProfile(fs.readFileSync(profilePath, "utf8"), primary, invalid), + ).toThrow(); + }); + + it("removes the private profile file when its consumer fails", async () => { + let temporaryPath = ""; + await expect( + withCredentialFamilyProfile(profilePath, primary, keys, async (profile) => { + temporaryPath = profile.profilePath; + expect(fs.statSync(temporaryPath).mode & 0o777).toBe(0o600); + expect(fs.statSync(path.dirname(temporaryPath)).mode & 0o777).toBe(0o700); + throw new Error("import failed"); + }), + ).rejects.toThrow("import failed"); + expect(fs.existsSync(path.dirname(temporaryPath))).toBe(false); + }); +}); + +describe("additional credential provider lifecycle", () => { + it.each([ + [false, "nemoclaw-mcp-v1"], + [true, "nemoclaw-mcp-v1"], + [false, "nemoclaw-mcp-v1-0123456789abcdef0123456789abcdef"], + [true, "nemoclaw-mcp-v1-0123456789abcdef0123456789abcdef"], + ] as const)( + "preserves stored MCP credentials when the missing key is declared: %s, profile: %s", + async (declareMissing, providerType) => { + const requested = definition({ + profile: { + profilePath: path.join( + REPOSITORY_ROOT, + "nemoclaw-blueprint/provider-profiles/nemoclaw-messaging-v1.yaml", + ), + profileType: "nemoclaw-messaging-v1", + }, + }); + const completeCredentials = [ + ...requested.credentials, + { name: "TELEGRAM_BOT_TOKEN_TEAM", value: "telegram-team-secret" }, + ]; + const existing = metadata({ + ...requested, + providerType, + credentials: completeCredentials, + }); + const adapter = providerAdapter({ + getProvider: vi + .fn() + .mockResolvedValue({ ok: true, value: existing }), + }); + await expect( + applyCredentialsAtOpenShell(plan, { + target, + providerAdapter: adapter, + definitions: [ + { + ...requested, + credentials: declareMissing + ? [...requested.credentials, { name: "TELEGRAM_BOT_TOKEN_TEAM", value: null }] + : requested.credentials, + }, + ], + replaceExisting: true, + }), + ).rejects.toThrow("cannot be replaced without every stored credential"); + expect(adapter.importProviderProfile).not.toHaveBeenCalled(); + expect(adapter.deleteProvider).not.toHaveBeenCalled(); + expect(adapter.createProvider).not.toHaveBeenCalled(); + expect(adapter.updateProvider).not.toHaveBeenCalled(); + + const complete = { ...requested, credentials: completeCredentials }; + const replacementType = credentialFamilyProfileType( + requested.providerType, + requested.credentialId, + completeCredentials.map(({ name }) => name), + ); + vi.mocked(adapter.getProvider) + .mockReset() + .mockResolvedValueOnce({ ok: true, value: existing }) + .mockResolvedValueOnce({ + ok: true, + value: metadata({ ...complete, providerType: replacementType }), + }); + await applyCredentialsAtOpenShell(plan, { + target, + providerAdapter: adapter, + definitions: [complete], + replaceExisting: true, + }); + expect(adapter.deleteProvider).toHaveBeenCalledExactlyOnceWith({ + target, + providerName: requested.providerName, + }); + expect(adapter.createProvider).toHaveBeenCalledExactlyOnceWith({ + target, + name: requested.providerName, + type: replacementType, + credentials: completeCredentials, + config: [], + fromExisting: false, + }); + expect(adapter.updateProvider).not.toHaveBeenCalled(); + }, + ); + + it("omits an absent optional credential when creating a provider (#11190)", async () => { + const tokenDefs = [ + { + name: "alpha-telegram-bridge", + envKey: "TELEGRAM_BOT_TOKEN", + token: "telegram-secret", + providerType: "nemoclaw-messaging-v1", + }, + ]; + const warnings: string[] = []; + const acceptedKeys = registerExtraPlaceholderProviders( + tokenDefs, + (message) => { + warnings.push(message); + }, + { + env: { + [EXTRA_PLACEHOLDER_KEYS_ENV]: + "TELEGRAM_BOT_TOKEN_AGENT_A TELEGRAM_BOT_TOKEN_AGENT_MISSING GITHUB_TOKEN", + TELEGRAM_BOT_TOKEN_AGENT_A: "telegram-agent-a-secret", + TELEGRAM_BOT_TOKEN_AGENT_MISSING: undefined, + GITHUB_TOKEN: "arbitrary-host-secret", + }, + getCredential: () => null, + normalizeCredentialValue: (value) => value?.trim() ?? "", + }, + ); + const application = buildMessagingProviderApplication({ + tokenDefs, + root: REPOSITORY_ROOT, + agent: "openclaw", + getCredential: () => null, + profiles: [], + }); + const builtDefinition = application.definitions[0]!; + const expected = { + ...builtDefinition, + credentials: [ + builtDefinition.credentials[2]!, + builtDefinition.credentials[0]!, + builtDefinition.credentials[1]!, + ], + }; + const createdCredentials = expected.credentials.filter(({ value }) => value !== null); + const createdType = credentialFamilyProfileType( + expected.providerType, + "TELEGRAM_BOT_TOKEN", + createdCredentials.map(({ name }) => name), + ); + const adapter = providerAdapter({ + getProvider: vi + .fn() + .mockResolvedValueOnce({ + ok: false, + error: { kind: "command", reason: "not_found", message: "provider not found" }, + }) + .mockResolvedValueOnce({ + ok: true, + value: metadata({ + ...expected, + providerType: createdType, + credentials: createdCredentials, + }), + }), + }); + + await applyCredentialsAtOpenShell(plan, { + providerAdapter: adapter, + target, + definitions: [expected], + requireCompleteBindings: true, + }); + + expect(adapter.createProvider).toHaveBeenCalledWith({ + target, + name: expected.providerName, + type: createdType, + credentials: createdCredentials, + config: [], + fromExisting: false, + }); + expect(acceptedKeys).toEqual([ + "TELEGRAM_BOT_TOKEN_AGENT_A", + "TELEGRAM_BOT_TOKEN_AGENT_MISSING", + ]); + expect(warnings.some((warning) => warning.includes('"GITHUB_TOKEN"'))).toBe(true); + expect(JSON.stringify(application)).not.toContain("arbitrary-host-secret"); + + vi.mocked(adapter.getProvider).mockResolvedValue({ + ok: true, + value: metadata({ ...expected, providerType: createdType, credentials: createdCredentials }), + }); + vi.mocked(adapter.createProvider).mockClear(); + const retryDefinition = { + ...expected, + credentials: expected.credentials.map(({ name }) => ({ name, value: null })), + }; + + const retry = await applyCredentialsAtOpenShell(plan, { + providerAdapter: adapter, + target, + definitions: [retryDefinition], + requireCompleteBindings: true, + }); + + expect(retry.reused).toEqual([ + expect.objectContaining({ providerName: expected.providerName }), + ]); + expect(adapter.createProvider).not.toHaveBeenCalled(); + expect(adapter.updateProvider).not.toHaveBeenCalled(); + expect(adapter.deleteProvider).not.toHaveBeenCalled(); + + const expandedCredentials = expected.credentials.map((credential) => + credential.name === "TELEGRAM_BOT_TOKEN_AGENT_MISSING" + ? { ...credential, value: "telegram-agent-missing-secret" } + : credential, + ); + const expandedType = credentialFamilyProfileType( + expected.providerType, + "TELEGRAM_BOT_TOKEN", + expandedCredentials.map(({ name }) => name), + ); + const existing = metadata({ + ...expected, + providerType: createdType, + credentials: createdCredentials, + }); + vi.mocked(adapter.getProvider).mockResolvedValue({ ok: true, value: existing }); + + await expect( + applyCredentialsAtOpenShell(plan, { + providerAdapter: adapter, + target, + definitions: [{ ...expected, credentials: expandedCredentials }], + requireCompleteBindings: true, + }), + ).rejects.toSatisfy(isMessagingProviderBindingConflict); + expect(adapter.deleteProvider).not.toHaveBeenCalled(); + + vi.mocked(adapter.importProviderProfile).mockClear(); + await expect( + applyCredentialsAtOpenShell(plan, { + providerAdapter: adapter, + target, + definitions: [ + { + ...expected, + credentials: expandedCredentials.map((credential) => + credential.name === "TELEGRAM_BOT_TOKEN_AGENT_A" + ? { ...credential, value: null } + : credential, + ), + }, + ], + requireCompleteBindings: true, + replaceExisting: true, + }), + ).rejects.toThrow("cannot be replaced without every stored credential"); + expect(adapter.importProviderProfile).not.toHaveBeenCalled(); + expect(adapter.deleteProvider).not.toHaveBeenCalled(); + expect(adapter.createProvider).not.toHaveBeenCalled(); + expect(adapter.updateProvider).not.toHaveBeenCalled(); + + vi.mocked(adapter.getProvider) + .mockReset() + .mockResolvedValueOnce({ ok: true, value: existing }) + .mockResolvedValueOnce({ + ok: true, + value: metadata({ + ...expected, + providerType: expandedType, + credentials: expandedCredentials, + }), + }); + await applyCredentialsAtOpenShell(plan, { + providerAdapter: adapter, + target, + definitions: [{ ...expected, credentials: expandedCredentials }], + requireCompleteBindings: true, + replaceExisting: true, + }); + + expect(adapter.createProvider).toHaveBeenCalledWith({ + target, + name: expected.providerName, + type: expandedType, + credentials: expandedCredentials, + config: [], + fromExisting: false, + }); + expect(adapter.updateProvider).not.toHaveBeenCalled(); + expect(adapter.deleteProvider).toHaveBeenCalledExactlyOnceWith({ + target, + providerName: expected.providerName, + }); + }); +}); diff --git a/src/lib/messaging/applier/credential-family-profile.ts b/src/lib/messaging/applier/credential-family-profile.ts new file mode 100644 index 00000000000..f9254555159 --- /dev/null +++ b/src/lib/messaging/applier/credential-family-profile.ts @@ -0,0 +1,155 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import YAML from "yaml"; + +import { parseCheckedInProviderProfileContract } from "../../adapters/openshell/provider-profile"; + +const CREDENTIAL_KEY = /^[A-Z][A-Z0-9_]{0,127}$/u; + +/** Build a stable identifier from validated names, never from credential values. */ +function profileIdentityDigest(value: string): string { + return createHash("sha256").update(value).digest("hex"); +} + +function familyKeys(primary: string, keys: readonly string[]): string[] { + if ( + !CREDENTIAL_KEY.test(primary) || + keys.length > 33 || + !keys.includes(primary) || + new Set(keys).size !== keys.length || + keys.some( + (key) => + !CREDENTIAL_KEY.test(key) || + (key !== primary && (!key.startsWith(`${primary}_`) || key.length <= primary.length + 1)), + ) + ) { + throw new Error("Provider credential names do not match the selected credential family."); + } + return [...keys].sort(); +} + +/** Bind an immutable profile to the selected names, without hashing credential values. */ +export function credentialFamilyProfileType( + baseType: string, + primary: string, + keys: readonly string[], +): string { + const sorted = familyKeys(primary, keys); + if (!/^[a-z][a-z0-9-]*$/u.test(baseType) || baseType === "generic") { + throw new Error("Additional credentials require an explicit owned provider profile."); + } + if (sorted.length === 1) return baseType; + const digest = profileIdentityDigest(JSON.stringify([baseType, primary, sorted])).slice(0, 32); + return `${baseType}-${digest}`; +} + +export function matchesCredentialFamilyProfileType( + actualType: string, + baseType: string, + primary: string, + keys: readonly string[], +): boolean { + try { + familyKeys(primary, keys); + if (baseType === "generic") return actualType === baseType; + return actualType === credentialFamilyProfileType(baseType, primary, keys); + } catch { + return false; + } +} + +export function credentialFamilyBaseType(profileType: string): string { + return profileType.replace(/-[a-f0-9]{32}$/u, ""); +} + +export function credentialFamilyPrimary(keys: readonly string[]): string | null { + const primary = keys.find((key) => + keys.every((other) => other === key || other.startsWith(`${key}_`)), + ); + if (!primary) return null; + try { + familyKeys(primary, keys); + return primary; + } catch { + return null; + } +} + +/** Preserve the checked-in profile's network and execution authority. */ +export function buildCredentialFamilyProfile( + source: string, + primary: string, + keys: readonly string[], +): { readonly profileType: string; readonly source: string } { + const contract = parseCheckedInProviderProfileContract(source); + if (!contract) throw new Error("The owned provider profile is invalid."); + const sorted = familyKeys(primary, keys); + const primaryCredential = contract.boundary.credentials.find( + (credential) => Array.isArray(credential.env_vars) && credential.env_vars.includes(primary), + ); + if (!primaryCredential || primaryCredential.refresh) { + throw new Error("Additional credentials require a declared static credential."); + } + const profileType = credentialFamilyProfileType(contract.profileId, primary, sorted); + if (profileType === contract.profileId) return { profileType, source }; + const profile = YAML.parse(source) as Record; + const credentials = profile.credentials as Record[]; + const declared = new Set( + contract.boundary.credentials.flatMap((credential) => credential.env_vars as string[]), + ); + profile.id = profileType; + profile.credentials = [ + ...credentials, + ...sorted + .filter((key) => key !== primary && !declared.has(key)) + .map((key) => ({ + name: key, + env_vars: [key], + required: false, + auth_style: primaryCredential.auth_style, + header_name: primaryCredential.header_name, + query_param: primaryCredential.query_param ?? "", + })), + ]; + return { profileType, source: JSON.stringify(profile) }; +} + +export function credentialFamilyProfileContract( + profilePath: string, + primary: string, + keys: readonly string[], +) { + const generated = buildCredentialFamilyProfile( + fs.readFileSync(profilePath, "utf8"), + primary, + keys, + ); + const contract = parseCheckedInProviderProfileContract(generated.source); + if (!contract) throw new Error("The generated credential profile is invalid."); + return contract; +} + +/** Keep generated profile files private and remove them after import or verification. */ +export async function withCredentialFamilyProfile( + profilePath: string, + primary: string, + keys: readonly string[], + use: (profile: { readonly profileType: string; readonly profilePath: string }) => T | Promise, +): Promise { + const source = fs.readFileSync(profilePath, "utf8"); + const generated = buildCredentialFamilyProfile(source, primary, keys); + if (keys.length === 1) return use({ profileType: generated.profileType, profilePath }); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-credential-profile-")); + try { + const generatedPath = path.join(directory, "profile.json"); + fs.writeFileSync(generatedPath, generated.source, { mode: 0o600, flag: "wx" }); + return await use({ profileType: generated.profileType, profilePath: generatedPath }); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} diff --git a/src/lib/messaging/applier/openshell-provider.test.ts b/src/lib/messaging/applier/openshell-provider.test.ts index 6adda8f9192..ee865ba05b5 100644 --- a/src/lib/messaging/applier/openshell-provider.test.ts +++ b/src/lib/messaging/applier/openshell-provider.test.ts @@ -2,15 +2,16 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import { + target, + plan, + definition, + metadata, + providerAdapter, +} from "./__test-helpers__/provider-fixtures"; import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; -import { namedOpenShellGateway } from "../../adapters/openshell/sandbox-observer"; -import { - EXTRA_PLACEHOLDER_KEYS_ENV, - registerExtraPlaceholderProviders, -} from "../../onboard/extra-placeholder-keys"; import type { MessagingBridgeProfile } from "../../onboard/messaging-bridge-provider"; -import type { SandboxMessagingPlan } from "../manifest"; import { applyCredentialsAtOpenShell, cleanupProvidersAtOpenShell, @@ -19,97 +20,7 @@ import { type MessagingProviderApplyError, } from "./openshell-provider"; import { buildMessagingProviderApplication } from "./provider-application"; -import type { - MessagingCredentialProviderEphemeralInput, - MessagingProviderRefreshEphemeralInput, -} from "./types"; - -const target = namedOpenShellGateway("nemoclaw"); -const plan: SandboxMessagingPlan = { - schemaVersion: 1, - sandboxName: "alpha", - agent: "openclaw", - workflow: "onboard", - channels: [], - disabledChannels: [], - credentialBindings: [], - networkPolicy: { presets: [], entries: [] }, - agentRender: [], - buildSteps: [], - stateUpdates: [], - healthChecks: [], -}; - -function definition( - overrides: Partial = {}, -): MessagingCredentialProviderEphemeralInput { - return { - channelId: "telegram", - credentialId: "TELEGRAM_BOT_TOKEN", - providerName: "alpha-telegram-bridge", - providerType: "nemoclaw-mcp-v1", - credentials: [{ name: "TELEGRAM_BOT_TOKEN", value: "telegram-secret" }], - profile: { profilePath: "/repo/messaging.yaml", profileType: "nemoclaw-mcp-v1" }, - ...overrides, - }; -} - -function metadata(input: MessagingCredentialProviderEphemeralInput) { - return { - name: input.providerName, - type: input.providerType, - credentialKeys: input.credentials.map(({ name }) => name), - configKeys: [], - }; -} - -function providerAdapter( - overrides: Partial = {}, -): OpenShellProviderAdapter { - return { - ensureProviderPolicyComposition: vi - .fn() - .mockResolvedValue({ ok: true, value: undefined }), - listProviders: vi - .fn() - .mockResolvedValue({ ok: true, value: { names: [] } }), - createProvider: vi - .fn() - .mockResolvedValue({ ok: true }), - getProvider: vi.fn().mockResolvedValue({ - ok: false, - error: { kind: "command", reason: "not_found", message: "provider not found" }, - }), - updateProvider: vi - .fn() - .mockResolvedValue({ ok: true }), - importProviderProfile: vi - .fn() - .mockResolvedValue({ ok: true }), - inspectProviderProfile: vi - .fn() - .mockResolvedValue({ ok: true, value: { credentialKeys: [] } }), - deleteProvider: vi - .fn() - .mockResolvedValue({ ok: true }), - detachProvider: vi - .fn() - .mockResolvedValue({ ok: true, value: { changed: true } }), - attachProvider: vi - .fn() - .mockResolvedValue({ ok: true }), - listProviderAttachments: vi - .fn() - .mockResolvedValue({ ok: true, value: { names: [] } }), - configureProviderRefresh: vi - .fn() - .mockResolvedValue({ ok: true }), - getProviderRefreshStatus: vi - .fn() - .mockResolvedValue({ ok: true, value: { status: "refreshed" } }), - ...overrides, - }; -} +import type { MessagingProviderRefreshEphemeralInput } from "./types"; describe("messaging OpenShell provider application", () => { it("reuses an exact provider through typed adapter calls (#9806)", async () => { @@ -231,142 +142,6 @@ describe("messaging OpenShell provider application", () => { expect(adapter.updateProvider).not.toHaveBeenCalled(); }); - it("omits an absent optional credential when creating a provider (#11190)", async () => { - const tokenDefs = [ - { - name: "alpha-telegram-bridge", - envKey: "TELEGRAM_BOT_TOKEN", - token: "telegram-secret", - providerType: "nemoclaw-mcp-v1", - }, - ]; - const warnings: string[] = []; - const acceptedKeys = registerExtraPlaceholderProviders( - tokenDefs, - (message) => { - warnings.push(message); - }, - { - env: { - [EXTRA_PLACEHOLDER_KEYS_ENV]: - "TELEGRAM_BOT_TOKEN_AGENT_A TELEGRAM_BOT_TOKEN_AGENT_MISSING GITHUB_TOKEN", - TELEGRAM_BOT_TOKEN_AGENT_A: "telegram-agent-a-secret", - TELEGRAM_BOT_TOKEN_AGENT_MISSING: undefined, - GITHUB_TOKEN: "arbitrary-host-secret", - }, - getCredential: () => null, - normalizeCredentialValue: (value) => value?.trim() ?? "", - }, - ); - const application = buildMessagingProviderApplication({ - tokenDefs, - root: "/repo", - agent: "openclaw", - getCredential: () => null, - profiles: [], - }); - const builtDefinition = application.definitions[0]!; - const expected = { - ...builtDefinition, - credentials: [ - builtDefinition.credentials[2]!, - builtDefinition.credentials[0]!, - builtDefinition.credentials[1]!, - ], - }; - const createdCredentials = expected.credentials.filter(({ value }) => value !== null); - const adapter = providerAdapter({ - getProvider: vi - .fn() - .mockResolvedValueOnce({ - ok: false, - error: { kind: "command", reason: "not_found", message: "provider not found" }, - }) - .mockResolvedValueOnce({ - ok: true, - value: metadata({ ...expected, credentials: createdCredentials }), - }), - }); - - await applyCredentialsAtOpenShell(plan, { - providerAdapter: adapter, - target, - definitions: [expected], - requireCompleteBindings: true, - }); - - expect(adapter.createProvider).toHaveBeenCalledWith({ - target, - name: expected.providerName, - type: expected.providerType, - credentials: createdCredentials, - config: [], - fromExisting: false, - }); - expect(acceptedKeys).toEqual([ - "TELEGRAM_BOT_TOKEN_AGENT_A", - "TELEGRAM_BOT_TOKEN_AGENT_MISSING", - ]); - expect(warnings.some((warning) => warning.includes('"GITHUB_TOKEN"'))).toBe(true); - expect(JSON.stringify(application)).not.toContain("arbitrary-host-secret"); - - vi.mocked(adapter.getProvider).mockResolvedValue({ - ok: true, - value: metadata({ ...expected, credentials: createdCredentials }), - }); - vi.mocked(adapter.createProvider).mockClear(); - const retryDefinition = { - ...expected, - credentials: expected.credentials.map(({ name }) => ({ name, value: null })), - }; - - const retry = await applyCredentialsAtOpenShell(plan, { - providerAdapter: adapter, - target, - definitions: [retryDefinition], - requireCompleteBindings: true, - }); - - expect(retry.reused).toEqual([ - expect.objectContaining({ providerName: expected.providerName }), - ]); - expect(adapter.createProvider).not.toHaveBeenCalled(); - expect(adapter.updateProvider).not.toHaveBeenCalled(); - expect(adapter.deleteProvider).not.toHaveBeenCalled(); - - const expandedCredentials = expected.credentials.map((credential) => - credential.name === "TELEGRAM_BOT_TOKEN_AGENT_MISSING" - ? { ...credential, value: "telegram-agent-missing-secret" } - : credential, - ); - vi.mocked(adapter.getProvider) - .mockReset() - .mockResolvedValueOnce({ - ok: true, - value: metadata({ ...expected, credentials: createdCredentials }), - }) - .mockResolvedValueOnce({ - ok: true, - value: metadata({ ...expected, credentials: expandedCredentials }), - }); - - await applyCredentialsAtOpenShell(plan, { - providerAdapter: adapter, - target, - definitions: [{ ...expected, credentials: expandedCredentials }], - requireCompleteBindings: true, - }); - - expect(adapter.updateProvider).toHaveBeenCalledWith({ - target, - providerName: expected.providerName, - credentials: expandedCredentials, - config: [], - }); - expect(adapter.createProvider).not.toHaveBeenCalled(); - expect(adapter.deleteProvider).not.toHaveBeenCalled(); - }); - it("rejects provider creation when only an optional credential is present (#11190)", async () => { const expected = definition({ credentials: [ diff --git a/src/lib/messaging/applier/openshell-provider.ts b/src/lib/messaging/applier/openshell-provider.ts index 80663fb2b5b..9796e09d74c 100644 --- a/src/lib/messaging/applier/openshell-provider.ts +++ b/src/lib/messaging/applier/openshell-provider.ts @@ -25,6 +25,11 @@ import { MESSAGING_CREDENTIAL_PROVIDER_TYPE, } from "../provider-profile"; import { filterEnabledPlanEntries } from "./plan-filter"; +import { + credentialFamilyProfileType, + matchesCredentialFamilyProfileType, + withCredentialFamilyProfile, +} from "./credential-family-profile"; import type { MessagingCredentialApplyOptions, MessagingCredentialApplyResult, @@ -93,23 +98,26 @@ export async function applyCredentialsAtOpenShell( ): Promise { const target = options.target ?? selectedOpenShellGateway(); const providerAdapter = resolveProviderAdapter(options); - const definitions = + const requestedDefinitions = options.definitions ?? definitionsFromPlan( options.env ?? process.env, filterEnabledPlanEntries(plan, plan.credentialBindings), ); const refreshes = options.refreshes ?? []; - assertUniqueDefinitions(definitions); - assertRefreshDefinitions(refreshes, definitions); + assertUniqueDefinitions(requestedDefinitions); + assertRefreshDefinitions(refreshes, requestedDefinitions); assertAuthorizedAttachment(options); + const definitions: MessagingCredentialProviderEphemeralInput[] = []; const states = new Map(); - for (const definition of definitions) { + for (const requested of requestedDefinitions) { const observed = await providerAdapter.getProvider({ target, - providerName: definition.providerName, + providerName: requested.providerName, }); + const definition = resolveCredentialFamilyDefinition(requested, observed); + definitions.push(definition); const state = classifyProviderDefinition(observed, definition, true); const primaryCredentialKey = requiredCredentialKey(definition)!; const hasAnyCredential = definition.credentials.some(({ value }) => Boolean(value)); @@ -125,6 +133,28 @@ export async function applyCredentialsAtOpenShell( if (state === "collision" && (!options.replaceExisting || !hasPrimaryCredential)) { throw bindingConflict(definition); } + if ( + state === "collision" && + observed.ok && + (definition.profile?.credentialFamily || + (/^nemoclaw-mcp-v1(?:-[a-f0-9]{32})?$/u.test(observed.value.type) && + requested.providerType === MESSAGING_CREDENTIAL_PROVIDER_TYPE) || + (observed.value.type !== requested.providerType && + matchesCredentialFamilyProfileType( + observed.value.type, + requested.providerType, + primaryCredentialKey, + observed.value.credentialKeys, + ))) && + observed.value.credentialKeys.some( + (key) => !definition.credentials.some(({ name, value }) => name === key && Boolean(value)), + ) + ) { + throw new MessagingProviderApplyError({ + message: `Messaging provider '${definition.providerName}' cannot be replaced without every stored credential. Supply its credential values again before changing its profile.`, + bindingConflict: true, + }); + } if (state === "missing" && hasAnyCredential && !hasPrimaryCredential) { throw new MessagingProviderApplyError({ message: `Messaging provider '${definition.providerName}' is missing required credential '${primaryCredentialKey}' for creation.`, @@ -617,11 +647,25 @@ async function prepareProfiles( ): Promise { const profiles = new Map( definitions.flatMap(({ profile }) => - profile ? [[profile.profileType, profile.profilePath] as const] : [], + profile ? [[profile.profileType, profile] as const] : [], ), ); - for (const [profileType, profilePath] of profiles) { - const imported = await providerAdapter.importProviderProfile({ target, profilePath }); + for (const [profileType, profile] of profiles) { + const importProfile = (profilePath: string) => + providerAdapter.importProviderProfile({ target, profilePath }); + const imported = profile.credentialFamily + ? await withCredentialFamilyProfile( + profile.profilePath, + profile.credentialFamily.primary, + profile.credentialFamily.keys, + (generated) => { + if (generated.profileType !== profileType) { + throw new Error("The owned credential profile changed before registration."); + } + return importProfile(generated.profilePath); + }, + ) + : await importProfile(profile.profilePath); if (!imported.ok) { throw new MessagingProviderApplyError({ message: `Could not prepare messaging provider profile '${profileType}': ${providerErrorMessage(imported.error)}`, @@ -630,6 +674,51 @@ async function prepareProfiles( } } +function resolveCredentialFamilyDefinition( + definition: MessagingCredentialProviderEphemeralInput, + observed: OpenShellProviderResult, +): MessagingCredentialProviderEphemeralInput { + const primary = requiredCredentialKey(definition); + if (!definition.profile || !primary || definition.credentials.length === 1) return definition; + const declared = new Set(definition.credentials.map(({ name }) => name)); + const keys = new Set([ + primary, + ...definition.credentials.filter(({ value }) => Boolean(value)).map(({ name }) => name), + ]); + if ( + observed.ok && + observed.value.name === definition.providerName && + observed.value.configKeys.length === 0 && + observed.value.credentialKeys.includes(primary) && + observed.value.credentialKeys.every((key) => declared.has(key)) + ) { + const retainedType = credentialFamilyProfileType( + definition.providerType, + primary, + observed.value.credentialKeys, + ); + if (observed.value.type === retainedType) { + for (const key of observed.value.credentialKeys) keys.add(key); + } + } + if (keys.size === 1) return definition; + const credentialKeys = [...keys].sort(); + const providerType = credentialFamilyProfileType( + definition.providerType, + primary, + credentialKeys, + ); + return { + ...definition, + providerType, + profile: { + ...definition.profile, + profileType: providerType, + credentialFamily: { primary, keys: credentialKeys }, + }, + }; +} + function readCredentialEnv(env: NodeJS.ProcessEnv, envKey: string): string | null { const raw = env[envKey]; if (typeof raw !== "string") return null; diff --git a/src/lib/messaging/applier/setup-applier.test.ts b/src/lib/messaging/applier/setup-applier.test.ts index 1bef4c035ce..7b4610cb202 100644 --- a/src/lib/messaging/applier/setup-applier.test.ts +++ b/src/lib/messaging/applier/setup-applier.test.ts @@ -3,6 +3,9 @@ import { describe, expect, it, vi } from "vitest"; import YAML from "yaml"; +import fs from "node:fs"; +import { REPOSITORY_ROOT } from "../../core/repository-root"; +import { messagingCredentialProviderProfilePath } from "../provider-profile"; import { createBuiltInChannelManifestRegistry, @@ -40,13 +43,9 @@ const TEST_CREDENTIALS: Readonly> = { const EXACT_MESSAGING_PROFILE = { status: 0, - stdout: JSON.stringify({ - id: "nemoclaw-mcp-v1", - credentials: [], - endpoints: [], - binaries: [], - inference_capable: false, - }), + stdout: JSON.stringify( + YAML.parse(fs.readFileSync(messagingCredentialProviderProfilePath(REPOSITORY_ROOT), "utf8")), + ), }; const ALL_CHANNEL_ENV = { @@ -424,7 +423,7 @@ describe("MessagingSetupApplier", () => { return credentialKey ? { status: 0, - stdout: `Name: ${name}\nType: nemoclaw-mcp-v1\nCredential keys: ${credentialKey}\nConfig keys: \n`, + stdout: `Name: ${name}\nType: nemoclaw-messaging-v1\nCredential keys: ${credentialKey}\nConfig keys: \n`, } : { status: 1, stderr: `provider '${name}' not found` }; } @@ -447,18 +446,18 @@ describe("MessagingSetupApplier", () => { ["provider", "get", "demo-telegram-bridge"], ["provider", "get", "demo-slack-bridge"], ["provider", "get", "demo-slack-app"], - ["provider", "profile", "export", "nemoclaw-mcp-v1", "--output", "json"], + ["provider", "profile", "export", "nemoclaw-messaging-v1", "--output", "json"], [ "provider", "create", "--name", "demo-telegram-bridge", "--type", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "--credential", "TELEGRAM_BOT_TOKEN", ], - ["provider", "profile", "export", "nemoclaw-mcp-v1", "--output", "json"], + ["provider", "profile", "export", "nemoclaw-messaging-v1", "--output", "json"], ["provider", "get", "demo-telegram-bridge"], ["provider", "update", "demo-slack-bridge", "--credential", "SLACK_BOT_TOKEN"], ["provider", "get", "demo-slack-bridge"], @@ -468,11 +467,11 @@ describe("MessagingSetupApplier", () => { "--name", "demo-slack-app", "--type", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "--credential", "SLACK_APP_TOKEN", ], - ["provider", "profile", "export", "nemoclaw-mcp-v1", "--output", "json"], + ["provider", "profile", "export", "nemoclaw-messaging-v1", "--output", "json"], ["provider", "get", "demo-slack-app"], ]); expect(calls[4]?.env).toEqual({ TELEGRAM_BOT_TOKEN: "123456:telegram-token" }); @@ -538,7 +537,7 @@ describe("MessagingSetupApplier", () => { return { status: 0, stdout: JSON.stringify({ - id: "nemoclaw-mcp-v1", + id: "nemoclaw-messaging-v1", credentials: [], endpoints: ["https://foreign.invalid"], binaries: [], @@ -549,7 +548,7 @@ describe("MessagingSetupApplier", () => { return { status: 0, stdout: - "Name: demo-telegram-bridge\nType: nemoclaw-mcp-v1\nCredential keys: TELEGRAM_BOT_TOKEN\nConfig keys: \n", + "Name: demo-telegram-bridge\nType: nemoclaw-messaging-v1\nCredential keys: TELEGRAM_BOT_TOKEN\nConfig keys: \n", }; } }, @@ -850,7 +849,7 @@ describe("MessagingSetupApplier", () => { return credentialKey ? { status: 0, - stdout: `Name: ${name}\nType: nemoclaw-mcp-v1\nCredential keys: ${credentialKey}\nConfig keys: \n`, + stdout: `Name: ${name}\nType: nemoclaw-messaging-v1\nCredential keys: ${credentialKey}\nConfig keys: \n`, } : { status: 1, stderr: `provider '${name}' not found` }; } @@ -999,7 +998,7 @@ describe("MessagingSetupApplier", () => { return credentialKey ? { status: 0, - stdout: `Name: ${name}\nType: nemoclaw-mcp-v1\nCredential keys: ${credentialKey}\nConfig keys: \n`, + stdout: `Name: ${name}\nType: nemoclaw-messaging-v1\nCredential keys: ${credentialKey}\nConfig keys: \n`, } : { status: 1, stderr: `provider '${name}' not found` }; } diff --git a/src/lib/messaging/applier/types.ts b/src/lib/messaging/applier/types.ts index a09b283e5c3..2b328d1b5d7 100644 --- a/src/lib/messaging/applier/types.ts +++ b/src/lib/messaging/applier/types.ts @@ -75,6 +75,7 @@ export type MessagingOpenShellRunner = ( export type MessagingCredentialProviderProfile = Readonly<{ profilePath: string; profileType: string; + credentialFamily?: Readonly<{ primary: string; keys: readonly string[] }>; }>; /** diff --git a/src/lib/messaging/channels/googlechat/manifest.ts b/src/lib/messaging/channels/googlechat/manifest.ts index ef95747be7a..df42df059f9 100644 --- a/src/lib/messaging/channels/googlechat/manifest.ts +++ b/src/lib/messaging/channels/googlechat/manifest.ts @@ -291,19 +291,7 @@ export const googlechatManifest = { configKeys: ["googlechat"], logPatterns: ["googlechat"], }, - // Interim sandbox-DNS workaround: the sandbox netns is DNS-less (all - // resolution goes through the L7 proxy), but OpenClaw's Google Chat fetches - // default to STRICT SSRF mode, which does a LOCAL getaddrinfo first and - // fails with EAI_AGAIN. This boot preload rewrites the plugin's googleapis - // fetches (inbound cert verify + all outbound sends) to the guard's - // first-class `trusted_env_proxy` mode, so they skip the local resolve and - // route by hostname through the L7 proxy (which resolves + enforces policy). - // No sentinel IP. It replaces the older googlechat-dns-resolve.ts sentinel - // shim, and is exactly the upstream OpenClaw fix (trusted-env-proxy fetch, - // like web_fetch openclaw#50650) applied in the plugin bundle; remove once - // that lands upstream. See runtime/googlechat-trusted-proxy-fetch.ts. - // - // Second boot preload: move OUTBOUND auth off the in-sandbox SA key. By + // Move outbound auth off the in-sandbox service-account key. By // default @openclaw/googlechat signs an auth JWT with the SA private key // in-process, which forces the key to live in the sandbox. This preload // rewrites the plugin's single token producer to return the OpenShell @@ -311,15 +299,6 @@ export const googlechatManifest = { // L7 proxy injects the real bearer outbound and the key never enters the // sandbox. See runtime/googlechat-outbound-auth.ts. nodePreloads: [ - { - module: "googlechat-trusted-proxy-fetch", - injectInto: ["boot"], - optional: false, - installMessage: - "[channels] Installing Google Chat trusted-proxy-fetch patch (route googleapis via trusted env proxy)", - installedMessage: - "[channels] Google Chat trusted-proxy-fetch patch installed (NODE_OPTIONS updated)", - }, { module: "googlechat-outbound-auth", injectInto: ["boot"], diff --git a/src/lib/messaging/channels/googlechat/runtime-contract.test.ts b/src/lib/messaging/channels/googlechat/runtime-contract.test.ts index a78460f8502..2160914bbd7 100644 --- a/src/lib/messaging/channels/googlechat/runtime-contract.test.ts +++ b/src/lib/messaging/channels/googlechat/runtime-contract.test.ts @@ -47,11 +47,11 @@ describe("googlechat runtime security contract", () => { expect(renderFragmentValue("gateway.reload")).toEqual({ mode: "off" }); }); - it.each(["googlechat-trusted-proxy-fetch", "googlechat-outbound-auth"])( - "requires both outbound-security boot preloads and marks them non-optional [%s]", + it.each(["googlechat-outbound-auth"])( + "requires the outbound credential preload at boot [%s]", (module) => { const preloads = googlechatManifest.runtime?.openclaw?.nodePreloads; - expect(preloads).toBeDefined(); + expect(preloads?.map((entry) => entry.module)).toEqual(["googlechat-outbound-auth"]); const preload = preloads?.find((entry) => entry.module === module); expect(preload, `missing required preload ${module}`).toBeDefined(); diff --git a/src/lib/messaging/channels/googlechat/runtime/googlechat-outbound-auth.ts b/src/lib/messaging/channels/googlechat/runtime/googlechat-outbound-auth.ts index 3346a8434d3..4a5067f6e75 100644 --- a/src/lib/messaging/channels/googlechat/runtime/googlechat-outbound-auth.ts +++ b/src/lib/messaging/channels/googlechat/runtime/googlechat-outbound-auth.ts @@ -45,8 +45,7 @@ // // INBOUND webhook JWT verification is untouched: it uses Google's PUBLIC certs // + appPrincipal (no SA material) and a different code path, so it is unaffected -// by this patch (its cert fetch is handled by the separate -// googlechat-trusted-proxy-fetch patch). +// by this patch. Certificate fetches use the native OpenClaw fetch guard. // // ── Contract with the B-side wiring ────────────────────────────────────────── // - The provider's injectable credential key MUST be `GOOGLE_CHAT_ACCESS_TOKEN`. diff --git a/src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.test.ts b/src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.test.ts deleted file mode 100644 index 827ba250726..00000000000 --- a/src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.test.ts +++ /dev/null @@ -1,158 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -import { describe, expect, it } from "vitest"; -import { outboundAuthPatchInternals } from "./googlechat-outbound-auth"; -import { trustedProxyFetchPatchInternals } from "./googlechat-trusted-proxy-fetch"; - -type CommonJsLoader = ( - this: unknown, - mod: { _compile: (source: string, filename: string) => unknown }, - filename: string, -) => unknown; - -const { patchSource, isPatchError, isOpenClawGooglechatFile, createComposableJsLoader } = - trustedProxyFetchPatchInternals as { - patchSource: (source: string, filename: string) => string; - isPatchError: (reason: unknown) => boolean; - isOpenClawGooglechatFile: (filename: string) => boolean; - createComposableJsLoader: (loader: CommonJsLoader) => CommonJsLoader; - }; -const { createComposableJsLoader: createOutboundAuthJsLoader } = outboundAuthPatchInternals as { - createComposableJsLoader: (loader: CommonJsLoader) => CommonJsLoader; -}; - -const FILE = "/x/node_modules/@openclaw/googlechat/dist/api-XXXX.js"; -const PATCH_MARKER = "nemoclaw:gc-trusted-proxy"; - -// Representative slice of the plugin's google-auth/api bundle carrying all three -// fetch call sites the patch targets, plus the BUNDLE_MARKER that identifies it. -const BUNDLE = [ - 'const GOOGLE_AUTH_AUDIT_CONTEXT = "googlechat.auth";', - "function createGoogleAuthFetch(opts) {", - " return fetchWithSsrFGuard({", - " auditContext: GOOGLE_AUTH_AUDIT_CONTEXT,", - " dispatcherPolicy: opts.dispatcherPolicy,", - " });", - "}", - "function fetchChatCerts() {", - " return fetchWithSsrFGuard({", - " url: CHAT_CERTS_URL,", - ' auditContext: "googlechat.auth.certs",', - " });", - "}", - "function withGoogleChatResponse(url, init) {", - " return fetchWithSsrFGuard({", - " url,", - " init,", - " });", - "}", - "async function getGoogleChatAccessToken(account) {", - " return (await account.auth.getClient()).getAccessToken();", - "}", -].join("\n"); - -describe("googlechat trusted-proxy-fetch patch", () => { - it("matches the external-extension install path, not only the package path", () => { - // Bundled/package load path (pre-2026.6.10). - expect( - isOpenClawGooglechatFile("/x/node_modules/@openclaw/googlechat/dist/channel.adapters-A.js"), - ).toBe(true); - // External-extension install path (openclaw plugins install → 2026.6.10 sandbox). - expect( - isOpenClawGooglechatFile( - "/sandbox/.openclaw/extensions/googlechat/dist/channel.adapters-DqnXEL1u.js", - ), - ).toBe(true); - // Other channels and non-.js files stay untouched. - expect(isOpenClawGooglechatFile("/sandbox/.openclaw/extensions/slack/dist/index.js")).toBe( - false, - ); - expect(isOpenClawGooglechatFile("/x/extensions/googlechat/dist/index.ts")).toBe(false); - }); - - it("rewrites all three googleapis fetch sites when the bundle matches", () => { - const patched = patchSource(BUNDLE, FILE); - expect(patched).not.toBe(BUNDLE); - expect(patched).toContain(PATCH_MARKER); - // Site A: proxy-mode-conditional spread injected. - expect(patched).toContain('.dispatcherPolicy?.mode === "explicit-proxy"'); - // Both trusted modes appear (Site A ternary + Sites B/C). - expect(patched).toContain("trusted_explicit_proxy"); - expect(patched).toContain("trusted_env_proxy"); - }); - - it("maps Site A explicit-proxy to trusted_explicit_proxy and keeps the dispatcherPolicy", () => { - const patched = patchSource(BUNDLE, FILE); - expect(patched).toContain( - '...(opts.dispatcherPolicy?.mode === "explicit-proxy" ' + - '? { mode: "trusted_explicit_proxy", dispatcherPolicy: opts.dispatcherPolicy } ' + - ': { mode: "trusted_env_proxy" })', - ); - }); - - it("adds trusted_env_proxy to Site B (fetchChatCerts) and Site C (outbound wrapper)", () => { - const patched = patchSource(BUNDLE, FILE); - // Site B: mode inserted between the url and auditContext of the certs fetch. - expect(patched).toContain('url: CHAT_CERTS_URL, mode: "trusted_env_proxy",'); - // Site C: mode inserted at the head of the outbound-wrapper options. - expect(patched).toMatch(/fetchWithSsrFGuard\(\{\s*mode: "trusted_env_proxy",\s*url,/); - }); - - it("is idempotent — already-patched source is returned unchanged", () => { - const once = patchSource(BUNDLE, FILE); - expect(patchSource(once, FILE)).toBe(once); - }); - - it("passes through files that are not the google-auth/api bundle", () => { - const other = "export function unrelated() {\n return fetchWithSsrFGuard({ url, });\n}"; - expect(patchSource(other, FILE)).toBe(other); - }); - - it("throws naming the drifted anchor when a call site is missing", () => { - // Bundle marker present + Sites A/B, but the outbound wrapper (Site C) drifted away. - const drift = BUNDLE.replace( - "function withGoogleChatResponse(url, init) {\n return fetchWithSsrFGuard({\n url,\n init,\n });\n}", - "", - ); - expect(() => patchSource(drift, FILE)).toThrow(/withGoogleChatResponse outbound fetch/); - expect(() => patchSource(drift, FILE)).toThrow(/not recognized/); - let caught: unknown; - try { - patchSource(drift, FILE); - } catch (error) { - caught = error; - } - expect(isPatchError(caught)).toBe(true); - }); - - it("isPatchError distinguishes anchor-drift errors from unrelated ones", () => { - expect(isPatchError(new Error("some unrelated failure"))).toBe(false); - expect( - isPatchError( - new Error("OpenClaw Google Chat trusted-proxy fetch anchors not recognized in x.js [..]"), - ), - ).toBe(true); - }); - - it.each([ - ["trusted proxy then outbound auth", createOutboundAuthJsLoader, createComposableJsLoader], - ["outbound auth then trusted proxy", createComposableJsLoader, createOutboundAuthJsLoader], - ])("composes the CommonJS rewrites in either preload order: %s", (_label, outer, inner) => { - let compiledSource = ""; - const mod = { - _compile(source: string) { - compiledSource = source; - }, - }; - const baseLoader: CommonJsLoader = (loadedModule, filename) => - loadedModule._compile(BUNDLE, filename); - - outer(inner(baseLoader))(mod, FILE); - - expect(compiledSource).toContain(PATCH_MARKER); - expect(compiledSource).toContain( - "nemoclaw: googlechat outbound bearer via gateway-minted credential", - ); - }); -}); diff --git a/src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.ts b/src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.ts deleted file mode 100644 index a2351bbd750..00000000000 --- a/src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.ts +++ /dev/null @@ -1,330 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 -// -// googlechat-trusted-proxy-fetch.ts — load-time patch that routes the Google -// Chat plugin's googleapis fetches through the TRUSTED ENV PROXY instead of -// resolving hostnames locally. It replaces the earlier DNS-resolve shim, which -// answered the googleapis hosts with a public sentinel IP so the SSRF gate's -// local getaddrinfo would pass; that shim has been removed. -// -// ── Workaround Analysis ────────────────────────────────────────────────────── -// 1. What: rewrites the plugin's three googleapis fetch call sites (dist bundle) -// to a trusted guard mode so they route by hostname through the env proxy. -// 2. Why: the sandbox netns is DNS-less; the plugin's fetches default to STRICT -// SSRF mode (local getaddrinfo), which fails EAI_AGAIN with no local DNS. -// 3. Alts: no config/env toggle exists to request trusted mode; the prior -// DNS-sentinel-IP shim was removed as hackier; upstream fix (5) unshipped. -// 4. Risk: version-sensitive (keys on bundle anchors) but bounded — plugin is -// integrity-version-pinned, anchors are unit-tested, and drift fails loud -// (named patch error), never a silent downgrade; Google hosts only, so no -// SSRF widening. -// 5. Exit: upstream @openclaw/googlechat requests trusted-env-proxy mode under a -// managed/env proxy (mirror of web_fetch openclaw#50650) — see below. -// -// ── The story: what this changes, and why ──────────────────────────────────── -// The sandbox netns is DNS-less BY DESIGN — all name resolution and egress go -// through the L7 proxy (10.200.0.1:3128). OpenClaw's SSRF fetch guard -// (`fetchWithSsrFGuard`) defaults to STRICT mode, which does a LOCAL -// `getaddrinfo` (SSRF IP-pinning) before connecting. In the DNS-less netns that -// local lookup fails with EAI_AGAIN, so every googleapis fetch dies. -// -// The guard already supports the correct behavior via `mode: "trusted_env_proxy"`: -// it SKIPS the local resolve, keeps the pre-DNS hostname/policy checks, and hands -// the hostname to the env proxy (which resolves + enforces policy). But the -// @openclaw/googlechat plugin does not request that mode: -// • createGoogleAuthFetch() (inbound cert verify + google-auth) passes a proxy -// `dispatcherPolicy` (gaxios injects an env-derived proxy agent, so the mode -// is "explicit-proxy"); the guard's plain `canUseTrustedEnvProxy` gate -// requires `!dispatcherPolicy`, so it falls to the local-resolve pin branch. -// • fetchChatCerts() (inbound chat-cert verify) and withGoogleChatResponse() -// (ALL outbound sends/edits/uploads) pass neither `mode` nor a policy, so -// they default to STRICT + local resolve too. -// -// This preload rewrites those three call sites in the plugin's dist bundle to a -// trusted guard mode (see the per-site mapping below): explicit-proxy → -// `trusted_explicit_proxy` (keep the dispatcherPolicy), otherwise -// `trusted_env_proxy`. The guard's own `shouldUseEnvHttpProxyForUrl` / -// explicit-proxy checks mean the trusted path only activates when a proxy is -// actually configured (the sandbox); outside a proxy (dev/CI) it degrades to the -// normal pinned path, so behavior is unchanged there. All three call sites target -// Google hosts only (google-auth is confined by GOOGLE_AUTH_POLICY; the others -// hit fixed googleapis/chat URLs), so this does not broaden SSRF exposure for any -// user-influenced URL. -// -// Net effect: inbound cert verification and outbound replies both route by -// hostname through the trusted proxy — no local getaddrinfo, no sentinel IP. -// -// ── Long-term fix (remove this once it lands) ───────────────────────────────── -// This is exactly the upstream OpenClaw change (mirror of web_fetch openclaw#50650): -// make @openclaw/googlechat's createGoogleAuthFetch / fetchChatCerts / outbound -// request wrapper use trusted-env-proxy mode under a managed/env proxy. When that -// ships, delete this preload and its `runtime.openclaw.nodePreloads` entry. -// -// Mechanism mirrors slack-channel-guard.ts / googlechat-outbound-auth.ts -// (load-time source rewrite of the @openclaw/googlechat dist bundle via the -// module loader hooks). It targets the SAME dist chunk that -// googlechat-outbound-auth already patches, so the CommonJS wrappers compose via -// Module._compile and both transforms apply regardless of preload order. - -// Test seam: the self-installing IIFE below publishes its pure source-rewrite -// helpers here so unit tests can exercise the anchor rewrites, drift handling, and -// idempotency directly. Requiring this module still installs the loader hooks, but -// that install is inert for files outside @openclaw/googlechat. -type CommonJsModuleLike = { - _compile?: (source: unknown, filename?: string) => unknown; -}; - -type CommonJsLoader = ( - this: unknown, - mod: CommonJsModuleLike, - filename: string, - ...args: unknown[] -) => unknown; - -type ModuleLoadResult = Record & { source?: unknown }; -type NextModuleLoad = (urlValue: string, context: unknown) => ModuleLoadResult; - -type TrustedProxyFetchPatchInternals = { - patchSource: (source: string, filename: string) => string; - isPatchError: (reason: unknown) => boolean; - isOpenClawGooglechatFile: (filename: unknown) => boolean; - createComposableJsLoader: (loader: CommonJsLoader) => CommonJsLoader; -}; - -export const trustedProxyFetchPatchInternals = {} as TrustedProxyFetchPatchInternals; - -(function () { - "use strict"; - - // Idempotency / shape markers. - var PATCH_MARKER = "nemoclaw:gc-trusted-proxy"; - // Only the dist chunk that defines the google-auth fetch factory is a target; - // this constant is defined right beside it, so its presence identifies the bundle. - var BUNDLE_MARKER = "GOOGLE_AUTH_AUDIT_CONTEXT"; - - var processWithPatchMarker = process as NodeJS.Process & { - __nemoclawGooglechatTrustedProxyFetchInstalled?: boolean; - }; - if (processWithPatchMarker.__nemoclawGooglechatTrustedProxyFetchInstalled) return; - try { - Object.defineProperty( - processWithPatchMarker, - "__nemoclawGooglechatTrustedProxyFetchInstalled", - { - value: true, - }, - ); - } catch (_e) { - processWithPatchMarker.__nemoclawGooglechatTrustedProxyFetchInstalled = true; - } - - function isOpenClawGooglechatFile(filename: unknown): boolean { - var normalized = String(filename || "").replace(/\\/g, "/"); - if (!normalized.endsWith(".js")) return false; - // The plugin loads either from a package path (/@openclaw/googlechat/) or, when - // installed as an external extension (openclaw plugins install), flat from - // ~/.openclaw/extensions/googlechat/dist/. Match both; the BUNDLE_MARKER gate in - // patchTrustedProxyFetchSource still confines the rewrite to the google-auth chunk. - return ( - normalized.indexOf("/@openclaw/googlechat/") !== -1 || - normalized.indexOf("/extensions/googlechat/") !== -1 - ); - } - - // Site A — createGoogleAuthFetch: the guarded fetch passes an env-proxy - // dispatcherPolicy that blocks the trusted-env gate. When the resolved policy - // is env-proxy, request trusted-env-proxy mode and drop the dispatcherPolicy so - // the guard skips the local resolve; otherwise leave the call untouched. - var ANCHOR_A = /dispatcherPolicy:\s*([A-Za-z_$][\w$]*)\.dispatcherPolicy\s*,/; - - // Site B — fetchChatCerts: STRICT + no dispatcherPolicy. Add trusted-env-proxy mode. - var ANCHOR_B = /(url:\s*CHAT_CERTS_URL\s*,)(\s*auditContext:\s*"googlechat\.auth\.certs")/; - - // Site C — withGoogleChatResponse (the single outbound wrapper): STRICT + no - // dispatcherPolicy. Add trusted-env-proxy mode at the head of the options. - // Matched by the `url` shorthand immediately after the call open-brace, which - // uniquely identifies this call (the google-auth call opens with auditContext). - var ANCHOR_C = /fetchWithSsrFGuard\(\{(\s*)url\s*,/; - - function patchTrustedProxyFetchSource(source: string, filename: string): string { - // Not the plugin's google-auth/api bundle — pass through untouched. - if (source.indexOf(BUNDLE_MARKER) === -1) return source; - // Already patched (idempotent across repeated --require of this preload). - if (source.indexOf(PATCH_MARKER) !== -1) return source; - - var missing: string[] = []; - if (!ANCHOR_A.test(source)) missing.push("createGoogleAuthFetch dispatcherPolicy"); - if (!ANCHOR_B.test(source)) missing.push("fetchChatCerts (googlechat.auth.certs)"); - if (!ANCHOR_C.test(source)) missing.push("withGoogleChatResponse outbound fetch"); - if (missing.length > 0) { - // The bundle defines the google-auth factory but one or more call sites - // drifted (e.g. a new plugin version). Fail loud rather than silently - // leave any googleapis fetch on the local-resolve path (which would break - // once the DNS shim is retired). - throw new Error( - "OpenClaw Google Chat trusted-proxy fetch anchors not recognized in " + - filename + - " [" + - missing.join("; ") + - "]; trusted-proxy-fetch patch not applied", - ); - } - - var patched = source; - // Site A (createGoogleAuthFetch): the google-auth transport carries a proxy - // dispatcherPolicy derived from HTTPS_PROXY. gaxios injects a proxy AGENT, so - // resolveGoogleAuthDispatcherPolicy returns mode "explicit-proxy" (not - // "env-proxy"). Map each proxy mode to its trusted (no-local-resolve) guard - // mode: explicit-proxy → trusted_explicit_proxy (KEEP the dispatcherPolicy, - // which sets allowPrivateProxy for the 10.200.0.1 proxy); anything else - // (env-proxy / direct / none) → trusted_env_proxy (drop the dispatcherPolicy; - // the guard's shouldUseEnvHttpProxyForUrl gate uses HTTPS_PROXY, which the - // sandbox always sets). Both branches skip the STRICT local getaddrinfo. - patched = patched.replace( - ANCHOR_A, - '...($1.dispatcherPolicy?.mode === "explicit-proxy" ' + - '? { mode: "trusted_explicit_proxy", dispatcherPolicy: $1.dispatcherPolicy } ' + - ': { mode: "trusted_env_proxy" }), /* ' + - PATCH_MARKER + - " */", - ); - // Site B (fetchChatCerts) + Site C (withGoogleChatResponse outbound): STRICT, - // no dispatcherPolicy → just request trusted_env_proxy (guard skips the local - // resolve when HTTPS_PROXY is configured; degrades to the normal pinned path - // otherwise). - patched = patched.replace(ANCHOR_B, '$1 mode: "trusted_env_proxy",$2'); - patched = patched.replace(ANCHOR_C, 'fetchWithSsrFGuard({$1mode: "trusted_env_proxy",$1url,'); - // Confirm the rewrite actually happened (the "active" banner only proves the - // loader hook was installed, not that a file was patched). - process.stderr.write( - "[channels] [googlechat] trusted-proxy-fetch: rewrote googleapis fetch sites in " + - filename + - "\n", - ); - return patched; - } - - function errorMessage(reason: unknown): string { - if (typeof reason === "object" && reason !== null && "message" in reason) { - return String(Reflect.get(reason, "message") ?? ""); - } - return String(reason ?? ""); - } - - function isTrustedProxyFetchPatchError(reason: unknown): boolean { - var msg = errorMessage(reason); - return ( - msg.indexOf("OpenClaw Google Chat trusted-proxy fetch anchors") !== -1 && - msg.indexOf("not recognized") !== -1 - ); - } - - function fileNameFromModuleUrl(urlValue: unknown): string { - if (typeof urlValue !== "string" || !urlValue.startsWith("file:")) return ""; - try { - return require("url").fileURLToPath(urlValue); - } catch (_e) { - return ""; - } - } - - function sourceToText(source: unknown): string | null { - if (typeof source === "string") return source; - if (typeof Buffer !== "undefined") { - if (Buffer.isBuffer(source)) return source.toString("utf8"); - if (source instanceof Uint8Array) return Buffer.from(source).toString("utf8"); - if (source instanceof ArrayBuffer) return Buffer.from(source).toString("utf8"); - } - return null; - } - - // Compose with other CommonJS source-rewrite preloads by intercepting - // Module._compile instead of reading + compiling the file directly. Calling the - // previous loader lets each wrapper transform the same source in sequence, - // regardless of NODE_OPTIONS preload order. - function createComposableJsLoader(originalJsLoader: CommonJsLoader): CommonJsLoader { - return function nemoclawGooglechatTrustedProxyJsLoader( - this: unknown, - mod: CommonJsModuleLike, - filename: string, - ...args: unknown[] - ) { - if (!isOpenClawGooglechatFile(filename) || typeof mod._compile !== "function") { - return originalJsLoader.call(this, mod, filename, ...args); - } - - var originalCompile = mod._compile; - mod._compile = function nemoclawGooglechatTrustedProxyCompile( - this: unknown, - source: unknown, - loadedFilename?: string, - ) { - var sourceText = sourceToText(source); - var patched = - sourceText === null - ? source - : patchTrustedProxyFetchSource(sourceText, loadedFilename || filename); - return originalCompile.call(this, patched, loadedFilename); - }; - try { - return originalJsLoader.call(this, mod, filename, ...args); - } finally { - mod._compile = originalCompile; - } - }; - } - - function installTrustedProxyFetchPatch() { - var Module = require("module"); - var originalJsLoader = Module._extensions && Module._extensions[".js"]; - if (typeof originalJsLoader === "function") { - Module._extensions[".js"] = createComposableJsLoader(originalJsLoader); - } - - if (typeof Module.registerHooks === "function") { - Module.registerHooks({ - load: function nemoclawGooglechatTrustedProxyLoadHook( - urlValue: string, - context: unknown, - nextLoad: NextModuleLoad, - ) { - var result = nextLoad(urlValue, context); - var filename = fileNameFromModuleUrl(urlValue); - if (!isOpenClawGooglechatFile(filename)) return result; - var sourceText = sourceToText(result && result.source); - if (sourceText === null) return result; - var patched = patchTrustedProxyFetchSource(sourceText, filename); - if (patched === sourceText) return result; - return Object.assign({}, result, { source: patched }); - }, - }); - } - } - - trustedProxyFetchPatchInternals.patchSource = patchTrustedProxyFetchSource; - trustedProxyFetchPatchInternals.isPatchError = isTrustedProxyFetchPatchError; - trustedProxyFetchPatchInternals.isOpenClawGooglechatFile = isOpenClawGooglechatFile; - trustedProxyFetchPatchInternals.createComposableJsLoader = createComposableJsLoader; - - try { - installTrustedProxyFetchPatch(); - process.stderr.write( - "[channels] [googlechat] trusted-proxy-fetch patch active " + - "(googleapis fetches routed via trusted env proxy; no local DNS resolve)\n", - ); - } catch (e) { - if (isTrustedProxyFetchPatchError(e)) { - process.stderr.write( - "[channels] [googlechat] trusted-proxy-fetch patch NOT applied: " + errorMessage(e) + "\n", - ); - } else { - // Any other failure: never break gateway boot, but still surface it so an - // inactive patch is diagnosable instead of failing silently. - process.stderr.write( - "[channels] [googlechat] trusted-proxy-fetch patch NOT applied (unexpected preload error): " + - errorMessage(e) + - "\n", - ); - } - } -})(); diff --git a/src/lib/messaging/channels/telegram/manifest.ts b/src/lib/messaging/channels/telegram/manifest.ts index 91030ef4909..b4646556261 100644 --- a/src/lib/messaging/channels/telegram/manifest.ts +++ b/src/lib/messaging/channels/telegram/manifest.ts @@ -119,7 +119,6 @@ export const telegramManifest = { healthMonitor: { enabled: false, }, - proxy: "{{proxyUrl}}", groupPolicy: "{{telegramConfig.groupPolicy}}", dmPolicy: "{{allowedIds.telegram.dmPolicy}}", allowFrom: "{{allowedIds.telegram.values}}", diff --git a/src/lib/messaging/channels/telegram/template-resolver.ts b/src/lib/messaging/channels/telegram/template-resolver.ts index ef0b5bf6845..edd0bea6dfb 100644 --- a/src/lib/messaging/channels/telegram/template-resolver.ts +++ b/src/lib/messaging/channels/telegram/template-resolver.ts @@ -14,8 +14,6 @@ import { stateValue, } from "../template-resolver-utils"; -const DEFAULT_PROXY_HOST = "10.200.0.1"; -const DEFAULT_PROXY_PORT = "3128"; const DEFAULT_TELEGRAM_GROUP_POLICY = "open"; const TELEGRAM_GROUP_POLICIES = new Set(["open", "allowlist", "disabled"]); @@ -23,7 +21,6 @@ export const resolveTelegramTemplateReference: BuiltInRenderTemplateResolver = ( reference, context, ) => { - if (reference === "proxyUrl") return resolvedRenderTemplateReference(proxyUrl(context.env)); if (reference === "telegramConfig.groupPolicy") { return resolvedRenderTemplateReference(telegramGroupPolicy(context)); } @@ -51,12 +48,6 @@ export const resolveTelegramTemplateReference: BuiltInRenderTemplateResolver = ( } }; -function proxyUrl(env: RenderTemplateContext["env"]): string { - const host = nonEmptyString(env?.NEMOCLAW_PROXY_HOST) ?? DEFAULT_PROXY_HOST; - const port = nonEmptyString(env?.NEMOCLAW_PROXY_PORT) ?? DEFAULT_PROXY_PORT; - return `http://${host}:${port}`; -} - function telegramGroupPolicy(context: RenderTemplateContext): string { const value = nonEmptyString(stateValue(context, "telegramConfig.groupPolicy")); return value && TELEGRAM_GROUP_POLICIES.has(value) ? value : DEFAULT_TELEGRAM_GROUP_POLICY; diff --git a/src/lib/messaging/provider-profile.test.ts b/src/lib/messaging/provider-profile.test.ts index fa5e9b73ea8..d6a9f5df154 100644 --- a/src/lib/messaging/provider-profile.test.ts +++ b/src/lib/messaging/provider-profile.test.ts @@ -2,6 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 import path from "node:path"; +import fs from "node:fs"; +import YAML from "yaml"; +import { listMessagingCredentialMetadata } from "./channels"; import { describe, expect, it, vi } from "vitest"; import { REPOSITORY_ROOT } from "../core/repository-root"; @@ -11,10 +14,62 @@ import { messagingCredentialProviderProfilePath, } from "./provider-profile"; +const ownedProfile = YAML.parse( + fs.readFileSync(messagingCredentialProviderProfilePath(REPOSITORY_ROOT), "utf8"), +); + describe("messaging credential provider profile", () => { + it("declares exactly the manifest credential keys without endpoint or executable grants", () => { + const keys = listMessagingCredentialMetadata() + .map(({ providerEnvKey }) => providerEnvKey) + .sort(); + expect( + ownedProfile.credentials + .map(({ env_vars }: { env_vars: string[] }) => env_vars) + .flat() + .sort(), + ).toEqual(keys); + expect( + ownedProfile.credentials.map( + ({ name, required, auth_style, header_name, query_param }: Record) => ({ + name, + required, + auth_style, + header_name, + query_param, + }), + ), + ).toEqual( + listMessagingCredentialMetadata().map(({ providerEnvKey: name }) => ({ + name, + required: false, + auth_style: "", + header_name: "", + query_param: "", + })), + ); + expect(ownedProfile).toMatchObject({ endpoints: [], binaries: [], inference_capable: false }); + expect(ownedProfile.files).toBeUndefined(); + }); + + it("rejects a catalog profile that still has no credential declarations", () => { + const runOpenshell = vi + .fn() + .mockReturnValue({ status: 0, stdout: JSON.stringify({ ...ownedProfile, credentials: [] }) }); + expect(() => + ensureMessagingCredentialProviderProfile({ root: REPOSITORY_ROOT, runOpenshell }), + ).toThrow(/does not match/); + expect(runOpenshell).toHaveBeenCalledOnce(); + }); + it("resolves the checked-in profile from the source repository root (#9875)", () => { expect(messagingCredentialProviderProfilePath(REPOSITORY_ROOT)).toBe( - path.join(REPOSITORY_ROOT, "nemoclaw-blueprint", "provider-profiles", "nemoclaw-mcp-v1.yaml"), + path.join( + REPOSITORY_ROOT, + "nemoclaw-blueprint", + "provider-profiles", + "nemoclaw-messaging-v1.yaml", + ), ); }); @@ -25,13 +80,7 @@ describe("messaging credential provider profile", () => { .mockReturnValueOnce({ status: 0 }) .mockReturnValueOnce({ status: 0, - stdout: JSON.stringify({ - id: MESSAGING_CREDENTIAL_PROVIDER_TYPE, - credentials: [], - endpoints: [], - binaries: [], - inference_capable: false, - }), + stdout: JSON.stringify(ownedProfile), }); ensureMessagingCredentialProviderProfile({ root: REPOSITORY_ROOT, runOpenshell }); @@ -47,7 +96,7 @@ describe("messaging credential provider profile", () => { REPOSITORY_ROOT, "nemoclaw-blueprint", "provider-profiles", - "nemoclaw-mcp-v1.yaml", + "nemoclaw-messaging-v1.yaml", ), ], { diff --git a/src/lib/messaging/provider-profile.ts b/src/lib/messaging/provider-profile.ts index 9aa1a07440e..a774f9017ab 100644 --- a/src/lib/messaging/provider-profile.ts +++ b/src/lib/messaging/provider-profile.ts @@ -7,7 +7,7 @@ import { registerCheckedInProviderProfile, } from "../adapters/openshell/provider-profile-registration"; -export const MESSAGING_CREDENTIAL_PROVIDER_TYPE = "nemoclaw-mcp-v1"; // gitleaks:allow +export const MESSAGING_CREDENTIAL_PROVIDER_TYPE = "nemoclaw-messaging-v1"; // gitleaks:allow export function messagingCredentialProviderProfilePath(root: string): string { return endpointlessProviderProfilePath(root, MESSAGING_CREDENTIAL_PROVIDER_TYPE); diff --git a/src/lib/onboard.ts b/src/lib/onboard.ts index 58730397263..c33cf3fe139 100644 --- a/src/lib/onboard.ts +++ b/src/lib/onboard.ts @@ -1704,7 +1704,7 @@ async function handleNimLocalSelection( state.model = nim.expectedServedModelId(catalogModel); state.provider = "vllm-local"; state.credentialEnv = null; - state.endpointUrl = getLocalProviderBaseUrl(state.provider); + state.endpointUrl = nim.getNimInferenceEndpoints().baseUrl; state.preferredInferenceApi = "openai-completions"; if (!state.endpointUrl) { console.error(" Local NVIDIA NIM base URL could not be determined."); @@ -1778,7 +1778,7 @@ async function handleNimLocalSelection( state.model = nim.adoptServedModelId(catalogModel); state.assertRouteCompatible?.(); - const nimValidationUrl = getLocalProviderValidationBaseUrl(state.provider) || state.endpointUrl; + const nimValidationUrl = nim.getNimInferenceEndpoints().validationBaseUrl; const validation = await validateOpenAiLikeSelection( "Local NVIDIA NIM", nimValidationUrl, @@ -2267,6 +2267,7 @@ function getSetupNimDeps(): SetupNimDeps { const setupNim = setupNimFlow.createSetupNim(getSetupNimDeps()); // ── Step 4: Inference provider ─────────────────────────────────── const getSetupInferenceDeps = (): SetupInferenceDeps => ({ + getInstalledOpenshellVersion, checkGatewayRouteCompatibility, withGatewayRouteMutationLock: gatewayRouteMutationLock.withGatewayRouteMutationLock, withSandboxMutationLock: sandboxMutationLock.withSandboxMutationLock, @@ -2955,8 +2956,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise { startRecordedStep, recordStepComplete, recordStepRejected, - toSessionUpdates: (updates) => - toSessionUpdates(updates as Parameters[0]), + toSessionUpdates, skippedStepMessage, recordStateSkipped, recordRepairEvent, @@ -2966,6 +2966,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise { repairLocalInferenceSystemdOverrideOrExit, isNonInteractive, getOpenshellBinary, + getInstalledOpenshellVersion, needsBedrockRuntimeAdapter: (providerName, url) => providerName === "compatible-anthropic-endpoint" && bedrockRuntimeOnboard.needsBedrockRuntimeAdapter(url), @@ -3119,7 +3120,6 @@ async function runOnboard(opts: OnboardOptions = {}): Promise { ); const finalFlowContext = prepareFinalOnboardFlowContext(coreFlowResult); let liveFinalFlowContext: InitialOnboardFlowContext = finalFlowContext; - const finalSandboxRegistration = registry.getSandbox(finalFlowContext.sandboxName); const finalFlowPhases = createFinalOnboardFlowPhases< InitialOnboardFlowContext, import("./dashboard/contract").DashboardDeliveryChain, @@ -3128,7 +3128,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise { branchState: agent ? "agent_setup" : "openclaw", managedOpenclawStartup: managedWorkloadOnboard.shouldUseManagedOpenclawStartup( !agent, - finalSandboxRegistration, + registry.getSandbox(finalFlowContext.sandboxName), ), portableRuntimeContext: agent?.name === "hermes" ? lockedRuntime.portableRuntimeContext : null, @@ -3136,6 +3136,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise { agentSetupDeps: { handleAgentSetup: agentOnboard.handleAgentSetup, agentSetupContext: () => ({ + getSandbox: registry.getSandbox, step, sandboxCommandExecutor: sandboxExec, gatewayName: GATEWAY_NAME!, @@ -3215,8 +3216,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise { executeSandboxCommandForVerification(sandbox, script, sandboxExec), probeHostPort: onboardDashboard.probeVerificationHostPort, getMessagingChannels: () => liveFinalFlowContext.selectedMessagingChannels || [], - providerExistsInGateway: (providerName: string) => - providerExistsInGateway(providerName), + providerExistsInGateway, probeInferenceInvocation: () => verifyDeploymentModule.probeOnboardInferenceInvocation(inference.invocation), }, diff --git a/src/lib/onboard/__test-helpers__/compatible-endpoint-smoke-helpers.ts b/src/lib/onboard/__test-helpers__/compatible-endpoint-smoke-helpers.ts index e4ca30f8d86..6076a8e93c3 100644 --- a/src/lib/onboard/__test-helpers__/compatible-endpoint-smoke-helpers.ts +++ b/src/lib/onboard/__test-helpers__/compatible-endpoint-smoke-helpers.ts @@ -3,6 +3,7 @@ import { spawnSync } from "node:child_process"; import { buildProviderNeutralInferenceSandboxSmokeScript } from "../compatible-endpoint-smoke"; +import type { NativeLocalProviderAttachment } from "../../inference/native-local/contract"; import fs from "node:fs"; import path from "node:path"; @@ -127,7 +128,7 @@ export function runSmokeScript(script: string, tmpDir: string, binDir: string) { }); } -type ProviderNeutralAuthority = NonNullable< +export type ProviderNeutralAuthority = NonNullable< Parameters[1] >; @@ -162,6 +163,7 @@ export function providerNeutralResponses( export function runProviderNeutralScript(options: { authority: ProviderNeutralAuthority; + local?: NativeLocalProviderAttachment; model?: string; script?: string; responses?: unknown[]; @@ -180,6 +182,7 @@ export function runProviderNeutralScript(options: { inferenceUrl?: string; expectedAuthorization?: string; runtimeEnvironment?: NodeJS.ProcessEnv; + inferenceError?: "refused" | "dns" | "permission" | "timeout" | "unknown"; }) { const model = options.model ?? "qwen3.5-9b"; const directAuthority = `host.openshell.internal:${String(options.authority.directHostPort)}`; @@ -195,6 +198,12 @@ export function runProviderNeutralScript(options: { const denialBody = options.denialOversized ? 'b"x" * 1048577' : `bytes(${JSON.stringify(denialBytes)})`; + const inferenceUrl = + options.inferenceUrl ?? + `${options.local?.endpointUrl ?? "https://inference.local/v1"}/chat/completions`; + const expectedAuthorization = + options.expectedAuthorization ?? + (options.local ? "Bearer openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN" : undefined); const prelude = ` import atexit import errno @@ -224,6 +233,7 @@ managed_proxy_request_count = [] managed_proxy_request_urls = [] sleep_delays = [] direct_error = ${options.directError === undefined ? "None" : JSON.stringify(options.directError)} +inference_error = ${options.inferenceError === undefined ? "None" : JSON.stringify(options.inferenceError)} def emit_request_evidence(): print("INFERENCE_REQUEST_TOKENS=" + ",".join(str(value) for value in inference_request_tokens)) @@ -259,8 +269,8 @@ class FakeDirectResponse: class FakeOpener: def open(self, request, timeout): - if request.full_url == ${JSON.stringify(options.inferenceUrl ?? "https://inference.local/v1/chat/completions")}: - assert request.get_header("Authorization") == ${options.expectedAuthorization ? JSON.stringify(options.expectedAuthorization) : "None"} + if request.full_url == ${JSON.stringify(inferenceUrl)}: + assert request.get_header("Authorization") == ${expectedAuthorization ? JSON.stringify(expectedAuthorization) : "None"} if not responses: raise RuntimeError("test response queue exhausted") request_data = json.loads(request.data.decode("utf-8")) @@ -271,6 +281,16 @@ class FakeOpener: raise TimeoutError("timed out after %s seconds" % timeout) if responses[0] == "__wrapped_timeout__": raise urllib.error.URLError(TimeoutError("timed out after %s seconds" % timeout)) + if inference_error: + private_detail = "private-credential-value" + reason = { + "refused": ConnectionRefusedError(errno.ECONNREFUSED, private_detail), + "dns": socket.gaierror(socket.EAI_NONAME, private_detail), + "permission": PermissionError(errno.EACCES, private_detail), + "timeout": TimeoutError(errno.ETIMEDOUT, private_detail), + "unknown": private_detail, + }[inference_error] + raise urllib.error.URLError(reason) return FakeResponse(200, responses.pop(0)) raise RuntimeError("unexpected direct request through inference opener") @@ -348,12 +368,22 @@ urllib.request.build_opener = build_test_opener time.sleep = lambda seconds: sleep_delays.append(seconds) `; const script = - options.script ?? buildProviderNeutralInferenceSandboxSmokeScript(model, options.authority); + options.script ?? + buildProviderNeutralInferenceSandboxSmokeScript(model, options.authority, options.local); return spawnSync("python3", ["-c", `${prelude}\n${script}`], { encoding: "utf8", env: options.managedProxyResponses === undefined - ? { ...process.env, ...options.runtimeEnvironment } + ? { + ...process.env, + ...(options.local + ? { + NEMOCLAW_LOCAL_INFERENCE_TOKEN: + "openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + } + : {}), + ...options.runtimeEnvironment, + } : { ...process.env, HTTPS_PROXY: "http://openshell-runtime-proxy.invalid:8080", diff --git a/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts b/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts index 948c6c406af..a86cfd28e17 100644 --- a/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/__test-helpers__/sandbox-gpu-create-flow.ts @@ -370,3 +370,14 @@ export function createGpuFlowTestHarness(mocks: Record { expect(targetDeps.inferenceRouteState).not.toHaveBeenCalled(); }); }); + +it("checks the immutable external provider instead of the removed shared route during recreate preflight", async () => { + const f = registeredInferenceFixture(false); + vi.spyOn(managedProvider, "createManagedProviderAdapter").mockReturnValue(f.adapter); + const targetDeps = deps({ + inferenceRouteState: vi.fn(() => { + throw new Error("retired route"); + }), + }); + const registeredTarget = { + ...target, + sandboxName: f.binding.sandboxName, + targetGatewayName: f.binding.gatewayName, + controlUiPort: null, + provider: f.binding.providerName, + registeredInferenceProviderAttachment: f.binding, + }; + await expect( + preflightAuthoritativeRebuildTarget(registeredTarget, targetDeps), + ).resolves.toBeUndefined(); + f.state.id = "replacement"; + await expect(preflightAuthoritativeRebuildTarget(registeredTarget, targetDeps)).rejects.toThrow(); + expect(targetDeps.inferenceRouteState).not.toHaveBeenCalled(); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); +}); diff --git a/src/lib/onboard/authoritative-rebuild-target.ts b/src/lib/onboard/authoritative-rebuild-target.ts index 4e447d253b5..44533765b83 100644 --- a/src/lib/onboard/authoritative-rebuild-target.ts +++ b/src/lib/onboard/authoritative-rebuild-target.ts @@ -1,6 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + registeredInferenceProviderRevision, + type RegisteredInferenceProviderAttachment, +} from "../inference/native-provider/registered"; + import { replaceOpenShellRuntimeSelectionEnv, snapshotOpenShellEnv, @@ -104,6 +109,7 @@ export type AuthoritativeRebuildPreflightOptions = Pick< /** Internal prepared-backup recovery defers route repair to authoritative onboard. */ deferInferenceRouteUntilOnboard?: true; agent?: string | null; + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; model: string; provider: string; @@ -164,6 +170,7 @@ export function resolveAuthoritativeOnboardGatewayBinding( export type AuthoritativeRebuildTarget = { deferInferenceRouteUntilOnboard?: true; agent?: string | null; + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; sandboxName: string; provider: string; @@ -372,7 +379,19 @@ export async function preflightAuthoritativeRebuildTarget< // before recreating the sandbox. A gateway that cannot answer at all leaves // the route unknown, which onboarding resolves the same way. Only a gateway // that answers with a different route contradicts the rebuild target. - if (target.nativeLocalProviderAttachment !== undefined) { + if (target.registeredInferenceProviderAttachment !== undefined) { + const expected = target.registeredInferenceProviderAttachment; + if ( + expected.gatewayName !== target.targetGatewayName || + expected.sandboxName !== target.sandboxName || + expected.providerName !== target.provider + ) + fail("Registered provider binding does not match the rebuild target."); + await registeredInferenceProviderRevision({ + adapter: createManagedProviderAdapter(), + expected, + }); + } else if (target.nativeLocalProviderAttachment !== undefined) { await verifyRecordedNativeLocalProviderAttachment({ adapter: createManagedProviderAdapter(), expected: target.nativeLocalProviderAttachment, diff --git a/src/lib/onboard/checkpoint-replay.test.ts b/src/lib/onboard/checkpoint-replay.test.ts index 5f2e4b3f316..6362f1c221e 100644 --- a/src/lib/onboard/checkpoint-replay.test.ts +++ b/src/lib/onboard/checkpoint-replay.test.ts @@ -449,29 +449,29 @@ describe("requiredMessagingProviderBindings", () => { expect(collectRequiredMessagingProviderBindings("alpha", plan, new Set(["slack"]))).toEqual([ { name: "alpha-slack-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "SLACK_BOT_TOKEN", }, { name: "alpha-slack-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "SLACK_APP_TOKEN", }, ]); expect(requiredMessagingProviderBindings("alpha", plan)).toEqual([ { name: "alpha-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "TELEGRAM_BOT_TOKEN", }, { name: "alpha-slack-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "SLACK_BOT_TOKEN", }, { name: "alpha-slack-app", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "SLACK_APP_TOKEN", }, ]); diff --git a/src/lib/onboard/command/error-reporting.test.ts b/src/lib/onboard/command/error-reporting.test.ts index 4db512f1b7c..24c30f99cd0 100644 --- a/src/lib/onboard/command/error-reporting.test.ts +++ b/src/lib/onboard/command/error-reporting.test.ts @@ -4,7 +4,11 @@ import { expect, it, vi } from "vitest"; import { handleOnboardCommandError } from "./error-reporting"; import { requireNativeProviderPolicy } from "../../adapters/openshell/provider-policy"; -import { fixture } from "../../adapters/openshell/provider-policy.test-support"; +// Exercise the retained legacy refusal producer independently of the migrated runtime pin. +vi.mock("../openshell-version", async (original) => ({ + ...(await original()), + SUPPORTED_OPENSHELL_FALLBACK_VERSION: "0.0.116", +})); it("reports nested onboarding causes without replacing the failure", () => { const primary = new Error("Created sandbox identity changed"); @@ -68,12 +72,22 @@ it("keeps cancellation and ordinary failure output unchanged", () => { expect(error).toHaveBeenCalledExactlyOnceWith(" Installation cancelled"); }); -it("prints the providers_v2 refusal with the administrator command on its own line (#13811)", async () => { - const refusal = await requireNativeProviderPolicy("selected", fixture("false")).catch( - (e: unknown) => e, - ); +it("prints the legacy providers_v2 refusal with the administrator command on its own line (#13811)", async () => { + const command = vi.fn(async () => ({ + status: 0, + stdout: JSON.stringify({ scope: "global", settings: { providers_v2_enabled: "false" } }), + })); + const refusal = await requireNativeProviderPolicy("selected", command).catch((e: unknown) => e); const error = vi.fn(); expect(handleOnboardCommandError(refusal, { error }, null)).toBe(1); + expect(command).toHaveBeenCalledExactlyOnceWith([ + "settings", + "get", + "-g", + "selected", + "--global", + "--json", + ]); const lines = (error.mock.calls.flat() as string[]).join("\n").split("\n"); expect(lines).toContain( " openshell settings set -g selected --global --key providers_v2_enabled --value true --yes", diff --git a/src/lib/onboard/compatible-endpoint-shell-smoke.test.ts b/src/lib/onboard/compatible-endpoint-shell-smoke.test.ts new file mode 100644 index 00000000000..06696ef47df --- /dev/null +++ b/src/lib/onboard/compatible-endpoint-shell-smoke.test.ts @@ -0,0 +1,320 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { + runSmokeScript, + writeFakeCurl, + writeFakeSleep, + writeSmokeConfig, +} from "./__test-helpers__/compatible-endpoint-smoke-helpers"; +import { + buildCompatibleEndpointSandboxSmokeCommand, + buildCompatibleEndpointSandboxSmokeScript, +} from "./compatible-endpoint-smoke"; + +describe("compatible endpoint shell smoke", () => { + it("shell-quotes hostile model text through the generated smoke script", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-quoting-")); + const sentinel = path.join(tmpDir, "model-command-ran"); + const model = "foo'bar`baz$(touch " + sentinel + ")"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, requestFile } = writeFakeCurl( + tmpDir, + `printf '%s\\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}'`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + configPath, + retryDelaySeconds: 0, + }); + const result = runSmokeScript(script, tmpDir, binDir); + expect(result.status).toBe(0); + expect(JSON.parse(fs.readFileSync(requestFile, "utf-8")).model).toBe(model); + expect(fs.existsSync(sentinel)).toBe(false); + }); + + it("retries a reasoning-only length response before failing the sandbox smoke", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-reasoning-")); + const model = "provider/reasoning-model"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + String.raw` +if [ "$count" -eq 1 ]; then + cat <<'JSON' +{"id":"82f5ff","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":"The user asked for PONG."},"finish_reason":"length"}],"usage":{"completion_tokens":32,"reasoning_tokens":32}} +JSON +else + cat <<'JSON' +{"id":"82f5ff","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"PONG"},"finish_reason":"stop"}]} +JSON +fi +`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 2, + configPath, + initialMaxTokens: 32, + retryDelaySeconds: 0, + retryMaxTokens: 512, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(0); + expect(result.stdout).toContain("OPENCLAW_CONFIG_OK"); + expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); + expect(result.stderr).toContain("exhausted max_tokens=32 in reasoning_content"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); + }); + + it("retries a transient non-JSON gateway response", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-transient-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + String.raw` +if [ "$count" -eq 1 ]; then + printf '%s\n' '504 Gateway Time-out' +else + printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' +fi +`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 0, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(0); + expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); + expect(result.stderr).toContain("inference.local returned non-JSON response"); + expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); + }); + + it("retries a parseable JSON HTTP 500 response", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-json-500-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + String.raw` +if [ "$count" -eq 1 ]; then + printf '%s\n' '__HTTP_STATUS__=500' + printf '%s\n' '{"error":{"message":"temporary gateway failure"}}' +else + printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' +fi +`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 0, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(0); + expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); + expect(result.stderr).toContain("transient HTTP 500"); + expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); + }); + + it("backs off for 5s then 10s between three attempts", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-backoff-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir } = writeFakeCurl( + tmpDir, + String.raw` +if [ "$count" -lt 3 ]; then + printf '%s\n' '__HTTP_STATUS__=500' + printf '%s\n' '{"error":{"message":"temporary gateway failure"}}' +else + printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' +fi +`, + ); + const sleepFile = writeFakeSleep(tmpDir, binDir); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 5, + }); + const result = runSmokeScript(script, tmpDir, binDir); + expect(result.status).toBe(0); + expect(result.stderr).toContain("retrying in 5s"); + expect(result.stderr).toContain("retrying in 10s"); + expect(fs.readFileSync(sleepFile, "utf-8")).toBe("5\n10\n"); + }); + + it("does not retry a parseable JSON HTTP 429 response", async () => { + // Fail closed: a blind replay cannot honor Retry-After and amplifies overload. + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-json-429-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + String.raw` +printf '%s\n' '__HTTP_STATUS__=429' +printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' +`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 0, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(1); + expect(result.stdout).not.toContain("INFERENCE_SMOKE_OK"); + expect(result.stderr).toContain("terminal HTTP 429"); + expect(result.stderr).not.toContain("retrying in"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("1"); + }); + + it.each([6, 7, 28, 52, 55, 56])( + "retries transient curl exit %i before succeeding", + (exitCode) => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-curl-retry-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + String.raw` +if [ "$count" -eq 1 ]; then + exit ${exitCode} +fi +printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' +`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 0, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(0); + expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); + expect(result.stderr).toContain(`curl exit ${exitCode}`); + expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); + }, + ); + + it("does not retry a permanent curl exit", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-curl-terminal-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl(tmpDir, "exit 2"); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 0, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(1); + expect(result.stderr).toContain("curl exit 2"); + expect(result.stderr).not.toContain("retrying in"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("1"); + }); + + it("does not retry a permanent JSON response validation failure", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-permanent-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + `printf '%s\\n' '{"error":{"message":"invalid model"}}'`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 0, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(1); + expect(result.stderr).toContain("did not contain non-empty choices[0].message.content"); + expect(result.stderr).not.toContain("retrying in"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("1"); + }); + + it("fails after the bounded transient retry budget", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-exhausted-")); + const model = "nvidia/nemotron-3-ultra"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + "printf '%s\\n' '504 Gateway Time-outAuthorization: Bearer test-secret'", + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 3, + configPath, + retryDelaySeconds: 0, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(1); + expect(result.stderr).toContain("http_status=504"); + expect(result.stderr).toContain("response_bytes="); + expect(result.stderr).not.toContain("test-secret"); + expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); + expect(result.stderr).toContain("smoke attempt 2/3 failed; retrying in 0s"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("3"); + }); + + it("reports a model-output budget problem when the retry also has no assistant content", async () => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-no-content-")); + const model = "provider/reasoning-model"; + const configPath = writeSmokeConfig(tmpDir, model); + const { binDir, callFile } = writeFakeCurl( + tmpDir, + String.raw` +cat <<'JSON' +{"id":"82f5ff","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":"Still reasoning."},"finish_reason":"length"}],"usage":{"completion_tokens":32,"reasoning_tokens":32}} +JSON +`, + ); + const script = buildCompatibleEndpointSandboxSmokeScript(model, { + attempts: 2, + configPath, + initialMaxTokens: 32, + retryDelaySeconds: 0, + retryMaxTokens: 64, + }); + + const result = runSmokeScript(script, tmpDir, binDir); + + expect(result.status).toBe(1); + expect(result.stderr).toContain("initial smoke attempt exhausted max_tokens=32"); + expect(result.stderr).toContain("retry smoke attempt still exhausted max_tokens=64"); + expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); + }); + + it("passes the native multiline script through the OpenShell command argument", async () => { + const command = buildCompatibleEndpointSandboxSmokeCommand("nvidia/model"); + + expect(command).toBe(buildCompatibleEndpointSandboxSmokeScript("nvidia/model")); + expect(command).toContain("\n"); + expect(command).not.toContain("base64.b64decode"); + }); +}); diff --git a/src/lib/onboard/compatible-endpoint-smoke.test.ts b/src/lib/onboard/compatible-endpoint-smoke.test.ts index 23044712fbe..ebb4439cfb5 100644 --- a/src/lib/onboard/compatible-endpoint-smoke.test.ts +++ b/src/lib/onboard/compatible-endpoint-smoke.test.ts @@ -1,23 +1,20 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; import { describe, expect, it, vi } from "vitest"; +import { + nativeLocalIdentity, + type NativeLocalProviderAttachment, +} from "../inference/native-local/contract"; import type { OpenShellSandboxBufferedCommandRequest } from "../adapters/openshell/sandbox-command"; import { providerNeutralResponses, runProviderNeutralScript, - runSmokeScript, - writeFakeCurl, - writeFakeSleep, - writeSmokeConfig, + type ProviderNeutralAuthority, } from "./__test-helpers__/compatible-endpoint-smoke-helpers"; import { createCompatibleEndpointSmoke, - buildCompatibleEndpointSandboxSmokeCommand, buildCompatibleEndpointSandboxSmokeScript, buildProviderNeutralInferenceSandboxSmokeScript, spawnOutputToString, @@ -85,10 +82,6 @@ const providerNeutralCases = ["openclaw", "hermes", "langchain-deepagents-code"] ], ); -type ProviderNeutralAuthority = NonNullable< - Parameters[1] ->; - const allAgentProofAuthorities = [ { agentName: "openclaw", @@ -120,6 +113,136 @@ const allAgentProofAuthorities = [ ] as const satisfies readonly { agentName: string; authority: ProviderNeutralAuthority }[]; describe("compatible endpoint sandbox smoke helpers", () => { + const nativeBinding = { + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated", + gatewayName: "nemoclaw", + sandboxName: "native-denial", + } as const; + const nativeAttachment: NativeLocalProviderAttachment = { + ...nativeBinding, + ...nativeLocalIdentity(nativeBinding), + schemaVersion: 1, + providerId: "native-denial-provider", + }; + // OpenShell 0.1.3 l7/rest.rs deny_response_body; Brev recorded this path-only reason. + const nativeDenial = { + error: "policy_denied", + detail: "POST /v1/nemoclaw-denied not permitted by policy", + layer: "l7", + protocol: "rest", + method: "POST", + path: "/v1/nemoclaw-denied", + host: "host.openshell.internal", + port: 8000, + policy: `_provider_${nativeAttachment.providerName.replaceAll("-", "_")}`, + rule: "POST /v1/nemoclaw-denied", + }; + + it("accepts the native OpenShell L7 denial for the attached provider's forbidden path", () => { + const result = runProviderNeutralScript({ + authority: allAgentProofAuthorities[0].authority, + local: nativeAttachment, + denial: nativeDenial, + }); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); + expect(result.stdout).toContain( + "DIRECT_REQUEST_URLS=http://host.openshell.internal:8000/v1/nemoclaw-denied", + ); + }); + + it.each(["connection-refused", "network-unreachable", "host-unreachable"] as const)( + "checks the native IPv4 policy denial after an IPv6 %s result", + (ipv6Outcome) => { + const result = runProviderNeutralScript({ + authority: allAgentProofAuthorities[0].authority, + local: nativeAttachment, + denial: nativeDenial, + directAddresses: ["fdc4:f303:9324::254", "192.168.65.254"], + directOutcomes: { "fdc4:f303:9324::254": ipv6Outcome }, + }); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); + expect(result.stdout).toContain( + "DIRECT_REQUEST_ADDRESSES=fdc4:f303:9324::254,192.168.65.254", + ); + expect(result.stdout).toContain("DIRECT_REQUEST_HOSTS=host.openshell.internal:8000"); + }, + ); + + it("rejects a native IPv4 denial with the wrong port type after an unreachable IPv6 address", () => { + const result = runProviderNeutralScript({ + authority: allAgentProofAuthorities[0].authority, + local: nativeAttachment, + denial: { ...nativeDenial, port: "8000" }, + directAddresses: ["fdc4:f303:9324::254", "192.168.65.254"], + directOutcomes: { "fdc4:f303:9324::254": "network-unreachable" }, + }); + expect(result.status).toBe(1); + expect(result.stderr).toContain("policy-denial contract v2 format drifted"); + expect(result.stderr).toContain("192.168.65.254 (policy-format-drift)"); + expect(result.stdout).not.toContain("INFERENCE_SMOKE_OK"); + }); + + it.each([ + ["host", "other-host.invalid"], + ["port", 8001], + ["port", "8000"], + ["policy", "_provider_other_sandbox"], + ["layer", "l4"], + ["protocol", "http"], + ["method", "GET"], + ["path", "/v1/chat/completions"], + ["rule", "POST /v1/chat/completions"], + ["detail", "POST host.openshell.internal:8000/v1/nemoclaw-denied not permitted by policy"], + ["host", undefined], + ])("rejects native denial with a mismatched %s field (%s)", (field, value) => { + const result = runProviderNeutralScript({ + authority: allAgentProofAuthorities[0].authority, + local: nativeAttachment, + denial: { ...nativeDenial, [field as string]: value }, + }); + expect(result.status).toBe(1); + expect(result.stderr).toContain("policy-denial contract v2 format drifted"); + expect(result.stdout).not.toContain("INFERENCE_SMOKE_OK"); + }); + + it("rejects an upstream authentication error as a native policy denial", () => { + const result = runProviderNeutralScript({ + authority: allAgentProofAuthorities[0].authority, + local: nativeAttachment, + denial: { ...nativeDenial, error: "authentication_required" }, + }); + expect(result.status).toBe(1); + expect(result.stderr).toContain("was not an OpenShell policy denial"); + }); + + it.each([ + ["refused", "ConnectionRefusedError"], + ["dns", "gaierror"], + ["permission", "PermissionError"], + ["unknown", "unknown"], + ] as const)("reports safe native transport diagnostics for %s", (inferenceError, errorType) => { + const result = runProviderNeutralScript({ + authority: allAgentProofAuthorities[0].authority, + local: nativeAttachment, + inferenceError, + }); + expect(result.status).toBe(1); + expect(result.stderr).toContain( + `Native local inference content proof transport failed after bounded retries (type=${errorType}, errno=`, + ); + expect(result.stderr).toMatch(/errno=(?:-?\d+|unknown)\)/); + expect(result.stderr).not.toContain("private-credential-value"); + expect(result.stdout).toContain("INFERENCE_REQUEST_TOKENS=512,512,512"); + expect(result.stdout).toContain("SLEEP_DELAYS=5,10"); + expect(result.stdout).toContain("DIRECT_REQUEST_COUNT=0"); + expect(result.stdout).not.toContain("INFERENCE_SMOKE_OK"); + }); + const providerMetadata = (name: string): string => [ `Id: provider-${name}`, @@ -367,17 +490,24 @@ describe("compatible endpoint sandbox smoke helpers", () => { }, ); - it.each(["__timeout__", "__wrapped_timeout__"])( - "reports a %s inference timeout once, without a traceback (#13663)", - (timeoutResponse) => { + it.each([ + { timeoutResponse: "__timeout__", local: undefined, inferenceError: undefined }, + { timeoutResponse: "__wrapped_timeout__", local: undefined, inferenceError: undefined }, + { timeoutResponse: "__timeout__", local: nativeAttachment, inferenceError: undefined }, + { timeoutResponse: "unused", local: nativeAttachment, inferenceError: "timeout" as const }, + ])( + "reports a $timeoutResponse inference timeout once, without a traceback (#13663)", + ({ timeoutResponse, local, inferenceError }) => { const authority = allAgentProofAuthorities[0].authority; const result = runProviderNeutralScript({ authority, responses: [timeoutResponse], + local, + inferenceError, script: buildProviderNeutralInferenceSandboxSmokeScript( "qwen3.5-9b", authority, - undefined, + local, 300, ), }); @@ -390,6 +520,9 @@ describe("compatible endpoint sandbox smoke helpers", () => { expect(result.stderr).toContain("retry the failed operation"); expect(result.stderr).not.toContain("resume onboarding"); expect(result.stderr).not.toContain("Traceback"); + expect(result.stderr).not.toContain("private-credential-value"); + expect(result.stdout).toContain("DIRECT_REQUEST_COUNT=0"); + expect(result.stdout).not.toContain("INFERENCE_SMOKE_OK"); expect(result.stdout).toContain("INFERENCE_REQUEST_TOKENS=512\n"); expect(result.stdout).toContain("SLEEP_DELAYS=\n"); }, @@ -1095,305 +1228,4 @@ describe("compatible endpoint sandbox smoke helpers", () => { expect(result.stderr).toContain("OpenShell policy-denial contract v1 format drifted"); expect(result.stderr).not.toContain("was not an OpenShell policy denial"); }); - - it("shell-quotes hostile model text through the generated smoke script", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-quoting-")); - const sentinel = path.join(tmpDir, "model-command-ran"); - const model = "foo'bar`baz$(touch " + sentinel + ")"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, requestFile } = writeFakeCurl( - tmpDir, - `printf '%s\\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}'`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - configPath, - retryDelaySeconds: 0, - }); - const result = runSmokeScript(script, tmpDir, binDir); - expect(result.status).toBe(0); - expect(JSON.parse(fs.readFileSync(requestFile, "utf-8")).model).toBe(model); - expect(fs.existsSync(sentinel)).toBe(false); - }); - - it("retries a reasoning-only length response before failing the sandbox smoke", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-reasoning-")); - const model = "provider/reasoning-model"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - String.raw` -if [ "$count" -eq 1 ]; then - cat <<'JSON' -{"id":"82f5ff","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":"The user asked for PONG."},"finish_reason":"length"}],"usage":{"completion_tokens":32,"reasoning_tokens":32}} -JSON -else - cat <<'JSON' -{"id":"82f5ff","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"PONG"},"finish_reason":"stop"}]} -JSON -fi -`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 2, - configPath, - initialMaxTokens: 32, - retryDelaySeconds: 0, - retryMaxTokens: 512, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(0); - expect(result.stdout).toContain("OPENCLAW_CONFIG_OK"); - expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); - expect(result.stderr).toContain("exhausted max_tokens=32 in reasoning_content"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); - }); - - it("retries a transient non-JSON gateway response", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-transient-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - String.raw` -if [ "$count" -eq 1 ]; then - printf '%s\n' '504 Gateway Time-out' -else - printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' -fi -`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 0, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(0); - expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); - expect(result.stderr).toContain("inference.local returned non-JSON response"); - expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); - }); - - it("retries a parseable JSON HTTP 500 response", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-json-500-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - String.raw` -if [ "$count" -eq 1 ]; then - printf '%s\n' '__HTTP_STATUS__=500' - printf '%s\n' '{"error":{"message":"temporary gateway failure"}}' -else - printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' -fi -`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 0, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(0); - expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); - expect(result.stderr).toContain("transient HTTP 500"); - expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); - }); - - it("backs off for 5s then 10s between three attempts", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-backoff-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir } = writeFakeCurl( - tmpDir, - String.raw` -if [ "$count" -lt 3 ]; then - printf '%s\n' '__HTTP_STATUS__=500' - printf '%s\n' '{"error":{"message":"temporary gateway failure"}}' -else - printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' -fi -`, - ); - const sleepFile = writeFakeSleep(tmpDir, binDir); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 5, - }); - const result = runSmokeScript(script, tmpDir, binDir); - expect(result.status).toBe(0); - expect(result.stderr).toContain("retrying in 5s"); - expect(result.stderr).toContain("retrying in 10s"); - expect(fs.readFileSync(sleepFile, "utf-8")).toBe("5\n10\n"); - }); - - it("does not retry a parseable JSON HTTP 429 response", async () => { - // Fail closed: a blind replay cannot honor Retry-After and amplifies overload. - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-json-429-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - String.raw` -printf '%s\n' '__HTTP_STATUS__=429' -printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' -`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 0, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(1); - expect(result.stdout).not.toContain("INFERENCE_SMOKE_OK"); - expect(result.stderr).toContain("terminal HTTP 429"); - expect(result.stderr).not.toContain("retrying in"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("1"); - }); - - it.each([6, 7, 28, 52, 55, 56])( - "retries transient curl exit %i before succeeding", - (exitCode) => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-curl-retry-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - String.raw` -if [ "$count" -eq 1 ]; then - exit ${exitCode} -fi -printf '%s\n' '{"choices":[{"message":{"content":"PONG"},"finish_reason":"stop"}]}' -`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 0, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(0); - expect(result.stdout).toContain("INFERENCE_SMOKE_OK PONG"); - expect(result.stderr).toContain(`curl exit ${exitCode}`); - expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); - }, - ); - - it("does not retry a permanent curl exit", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-curl-terminal-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl(tmpDir, "exit 2"); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 0, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(1); - expect(result.stderr).toContain("curl exit 2"); - expect(result.stderr).not.toContain("retrying in"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("1"); - }); - - it("does not retry a permanent JSON response validation failure", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-permanent-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - `printf '%s\\n' '{"error":{"message":"invalid model"}}'`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 0, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(1); - expect(result.stderr).toContain("did not contain non-empty choices[0].message.content"); - expect(result.stderr).not.toContain("retrying in"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("1"); - }); - - it("fails after the bounded transient retry budget", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-exhausted-")); - const model = "nvidia/nemotron-3-ultra"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - "printf '%s\\n' '504 Gateway Time-outAuthorization: Bearer test-secret'", - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 3, - configPath, - retryDelaySeconds: 0, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(1); - expect(result.stderr).toContain("http_status=504"); - expect(result.stderr).toContain("response_bytes="); - expect(result.stderr).not.toContain("test-secret"); - expect(result.stderr).toContain("smoke attempt 1/3 failed; retrying in 0s"); - expect(result.stderr).toContain("smoke attempt 2/3 failed; retrying in 0s"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("3"); - }); - - it("reports a model-output budget problem when the retry also has no assistant content", async () => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-compat-smoke-no-content-")); - const model = "provider/reasoning-model"; - const configPath = writeSmokeConfig(tmpDir, model); - const { binDir, callFile } = writeFakeCurl( - tmpDir, - String.raw` -cat <<'JSON' -{"id":"82f5ff","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":"Still reasoning."},"finish_reason":"length"}],"usage":{"completion_tokens":32,"reasoning_tokens":32}} -JSON -`, - ); - const script = buildCompatibleEndpointSandboxSmokeScript(model, { - attempts: 2, - configPath, - initialMaxTokens: 32, - retryDelaySeconds: 0, - retryMaxTokens: 64, - }); - - const result = runSmokeScript(script, tmpDir, binDir); - - expect(result.status).toBe(1); - expect(result.stderr).toContain("initial smoke attempt exhausted max_tokens=32"); - expect(result.stderr).toContain("retry smoke attempt still exhausted max_tokens=64"); - expect(fs.readFileSync(callFile, "utf-8")).toBe("2"); - }); - - it("passes the native multiline script through the OpenShell command argument", async () => { - const command = buildCompatibleEndpointSandboxSmokeCommand("nvidia/model"); - - expect(command).toBe(buildCompatibleEndpointSandboxSmokeScript("nvidia/model")); - expect(command).toContain("\n"); - expect(command).not.toContain("base64.b64decode"); - }); }); diff --git a/src/lib/onboard/compatible-endpoint-smoke.ts b/src/lib/onboard/compatible-endpoint-smoke.ts index 92a4e28b826..e6ceed597cc 100644 --- a/src/lib/onboard/compatible-endpoint-smoke.ts +++ b/src/lib/onboard/compatible-endpoint-smoke.ts @@ -701,8 +701,21 @@ export function buildProviderNeutralInferenceSandboxSmokeScript( const directMethod = JSON.stringify(OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.method); const directDenialError = JSON.stringify(OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.error); const directDenialDetail = JSON.stringify( - `${OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.method} ${directAuthority}${deniedPath} ${OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.detailSuffix}`, + `${OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.method} ${local ? "" : directAuthority}${deniedPath} ${OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.detailSuffix}`, ); + // OpenShell 0.1.3 reports native REST denials with separate endpoint and request fields. + const directDenialFields = local + ? { + layer: "l7", + protocol: "rest", + method: OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.method, + path: deniedPath, + host: new URL(local.endpointUrl).hostname, + port: Number(new URL(local.endpointUrl).port || 80), + policy: `_provider_${local.providerName.replaceAll("-", "_")}`, + rule: `POST ${deniedPath}`, + } + : {}; const authSetup = local ? `credential = os.environ.get("NEMOCLAW_LOCAL_INFERENCE_TOKEN", "") if not re.fullmatch(${JSON.stringify(NATIVE_LOCAL_RUNTIME_REFERENCE_PATTERN)}, credential): @@ -781,7 +794,14 @@ def post_inference(payload, label): if attempt < 2: time.sleep(5 * (attempt + 1)) continue - print("${endpointLabel} %s proof transport failed after bounded retries" % label, file=sys.stderr) + reason = error.reason + error_type = type(reason).__name__ if isinstance(reason, OSError) else "unknown" + error_number = reason.errno if isinstance(reason, OSError) and isinstance(reason.errno, int) else "unknown" + print( + "${endpointLabel} %s proof transport failed after bounded retries (type=%s, errno=%s)" + % (label, error_type, error_number), + file=sys.stderr, + ) sys.exit(1) if not isinstance(response_data, dict) or response_data.get("model") != model: print("${endpointLabel} %s proof returned a different model identity" % label, file=sys.stderr) @@ -868,10 +888,11 @@ if tool_calling_required: direct_url = ${directUrl} direct_method = ${directMethod} -direct_denial_contract_version = ${String(OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.version)} +direct_denial_contract_version = ${String(local ? 2 : OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.version)} direct_denial_http_status = ${String(OPEN_SHELL_DIRECT_POLICY_DENIAL_CONTRACT.httpStatus)} direct_denial_error = ${directDenialError} direct_denial_detail = ${directDenialDetail} +direct_denial_fields = ${JSON.stringify(directDenialFields)} direct_deny_timeout_seconds = ${String(PROVIDER_NEUTRAL_SMOKE_DIRECT_DENY_TIMEOUT_SECONDS)} direct_target = urllib.parse.urlsplit(direct_url) try: @@ -973,7 +994,10 @@ for family, sockaddr in direct_addresses: ): record_direct_outcome(sockaddr, "unexpected-response") fail_direct("direct host inference response was not an OpenShell policy denial") - if detail != direct_denial_detail: + if detail != direct_denial_detail or any( + type(denial.get(field)) is not type(expected) or denial.get(field) != expected + for field, expected in direct_denial_fields.items() + ): record_direct_outcome(sockaddr, "policy-format-drift") fail_direct( "direct host inference OpenShell policy-denial contract v%s format drifted" diff --git a/src/lib/onboard/compute/docker-container-failure-evidence.ts b/src/lib/onboard/compute/docker-container-failure-evidence.ts index 07b9a29ebf1..7276a9a8bb0 100644 --- a/src/lib/onboard/compute/docker-container-failure-evidence.ts +++ b/src/lib/onboard/compute/docker-container-failure-evidence.ts @@ -1,7 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { createDockerGpuDiagnosticRedactor } from "../docker-gpu-diagnostic-redaction"; +import { + createDockerGpuDiagnosticRedactor, + type DockerGpuDiagnosticRedactor, +} from "../docker-gpu-diagnostic-redaction"; import type { DockerContainerInspect, DockerContainerState, @@ -80,6 +83,9 @@ export function captureDockerContainerFailureEvidence( deps: DockerContainerFailureEvidenceDeps, options: { enrichInspect?: boolean; + redactor?: DockerGpuDiagnosticRedactor; + /** Identity-bound inventory captured before any sibling logs are read. */ + inspect?: DockerContainerInspect; /** Shared deadline; later Docker calls get only the remaining time and are skipped after it. */ deadlineMs?: number; now?: () => number; @@ -92,10 +98,10 @@ export function captureDockerContainerFailureEvidence( const remaining = Math.floor(options.deadlineMs - now()); return remaining > 0 ? Math.min(FAILURE_EVIDENCE_CALL_TIMEOUT_MS, remaining) : null; }; - const redactor = createDockerGpuDiagnosticRedactor(); - let state: DockerContainerState | null = null; - let inspect: DockerContainerInspect | null = null; - const stateTimeout = callTimeout(); + const redactor = options.redactor ?? createDockerGpuDiagnosticRedactor(); + let inspect: DockerContainerInspect | null = options.inspect ?? null; + let state: DockerContainerState | null = inspect?.State ?? null; + const stateTimeout = inspect ? null : callTimeout(); try { if (stateTimeout !== null) { const output = capture(["inspect", "--format", "{{json .State}}", containerId], { diff --git a/src/lib/onboard/corporate-ca-validation.ts b/src/lib/onboard/corporate-ca-validation.ts index cea019ed5d3..ffec44e9fda 100644 --- a/src/lib/onboard/corporate-ca-validation.ts +++ b/src/lib/onboard/corporate-ca-validation.ts @@ -26,6 +26,38 @@ export function normalizeCertificateBlocks(blocks: readonly string[]): string { return `${blocks.map((block) => block.trim()).join("\n")}\n`; } +/** Validate already-resolved CA material without accepting keys or other adjacent data. */ +export function validateCorporateCaPem(pem: string): string { + const bytes = Buffer.byteLength(pem, "utf8"); + if (bytes === 0 || bytes > MAX_CORPORATE_CA_BYTES) { + throw new Error( + `corporate CA material must be between 1 and ${String(MAX_CORPORATE_CA_BYTES)} bytes`, + ); + } + const blocks = pem.match(PEM_CERTIFICATE_RE_GLOBAL); + if (!blocks || blocks.length === 0 || blocks.length > MAX_CORPORATE_CA_CERTS) { + throw new Error( + `corporate CA material must contain between 1 and ${String(MAX_CORPORATE_CA_CERTS)} certificates`, + ); + } + for (const block of blocks) { + let certificate: X509Certificate; + try { + certificate = new X509Certificate(block); + } catch { + throw new Error("corporate CA material contains an invalid X.509 certificate"); + } + if (!certificate.ca) { + throw new Error("corporate CA material contains a certificate without CA basic constraints"); + } + } + const normalized = normalizeCertificateBlocks(blocks); + if (pem.trim() !== normalized.trim()) { + throw new Error("corporate CA material must contain only normalized PEM certificate blocks"); + } + return normalized; +} + /** * Validate a candidate corporate CA bundle file and return normalized PEM text. * diff --git a/src/lib/onboard/created-sandbox-finalization.test.ts b/src/lib/onboard/created-sandbox-finalization.test.ts index 78ea9a7a356..5b4e2714867 100644 --- a/src/lib/onboard/created-sandbox-finalization.test.ts +++ b/src/lib/onboard/created-sandbox-finalization.test.ts @@ -9,6 +9,7 @@ import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { writeFakeSsh } from "../../../test/support/native-snapshot"; import * as restoreWindow from "../actions/sandbox/runtime/openclaw-lifecycle"; import type { SandboxEntry } from "../state/registry"; import type { QualifiedSandboxInferenceRouteReservation } from "../state/registry/route-reservation"; @@ -223,32 +224,10 @@ function makeRestoreFixture(): { openshell, '#!/usr/bin/env bash\nprintf "Host openshell-%s\\n HostName 127.0.0.1\\n User sandbox\\n" "${!#}"\n', ); - executable( - path.join(bin, "ssh"), - `#!/usr/bin/env node -const fs = require("node:fs"); -const { spawnSync } = require("node:child_process"); -const command = process.argv.at(-1) - .replaceAll("/sandbox/.deepagents", ${JSON.stringify(liveDir)}); -if (command.includes("printf '%s\\\\0%s\\\\0'")) { - process.stdout.write(Buffer.from("/sandbox\\0/sandbox\\0")); - process.exit(0); -} -if (command.includes('restore_dir "$stage" "$root"')) { - for (const entry of fs.readdirSync(${JSON.stringify(liveRoot)})) { - fs.rmSync(require("node:path").join(${JSON.stringify(liveRoot)}, entry), { recursive: true, force: true }); - } - const result = spawnSync("tar", ["--no-same-owner", "-xf", "-", "-C", ${JSON.stringify(liveRoot)}], { input: fs.readFileSync(0), stdio: ["pipe", "pipe", "pipe"] }); - if (result.stdout) fs.writeSync(1, result.stdout); - if (result.stderr) fs.writeSync(2, result.stderr); - process.exit(result.status ?? 1); -} -const result = spawnSync("bash", ["-c", command], { input: fs.readFileSync(0), stdio: ["pipe", "pipe", "pipe"] }); -if (result.stdout) fs.writeSync(1, result.stdout); -if (result.stderr) fs.writeSync(2, result.stderr); -process.exit(result.status ?? 1); -`, - ); + writeFakeSsh(bin); + vi.stubEnv("NEMOCLAW_TEST_NATIVE_ROOT", liveRoot); + vi.stubEnv("NEMOCLAW_TEST_NATIVE_HOME", "/sandbox"); + vi.stubEnv("NEMOCLAW_TEST_NATIVE_WORKSPACE", "/sandbox"); process.env.NEMOCLAW_OPENSHELL_BIN = openshell; process.env.PATH = `${bin}${path.delimiter}${oldPath}`; return { backupPath, currentPath, oldPath }; @@ -1295,6 +1274,50 @@ describe("restored OpenClaw selection reconciliation", () => { ); }); + it("refreshes a registered route during ordinary rebuild before releasing the gateway", async () => { + const { registeredInferenceFixture } = + await import("../../../test/helpers/registered-inference"); + const binding = { + ...registeredInferenceFixture(true).binding, + sandboxName: options.sandboxName, + gatewayName: options.gatewayName, + }; + const order: string[] = []; + const dependencies = deps(order); + vi.mocked(restoreWindow.finishUnregisteredOpenClawPostRestoreDoctor).mockImplementation( + async () => { + order.push("restart"); + return { ok: true }; + }, + ); + await finalizeCreatedSandbox( + { + ...options, + reconcileOpenClawInference: false, + provider: binding.providerName, + preferredInferenceApi: binding.inferenceApi, + nativeProvider: true, + endpointUrl: binding.endpointUrl, + registeredInferenceProviderAttachment: binding, + }, + dependencies, + ); + expect(order).toEqual(["restore", "selection", "restart", "register"]); + expect(dependencies.writeRestoredOpenclawInferenceRoute).toHaveBeenCalledExactlyOnceWith( + options.sandboxName, + options.model, + binding.providerName, + binding.inferenceApi, + options.gatewayName, + undefined, + { + endpointUrl: binding.endpointUrl, + nativeProvider: true, + registeredInferenceProviderAttachment: binding, + }, + ); + }); + it.each([ { label: "ordinary rebuild", diff --git a/src/lib/onboard/created-sandbox-finalization.ts b/src/lib/onboard/created-sandbox-finalization.ts index 299053ae06b..0e1145e51d5 100644 --- a/src/lib/onboard/created-sandbox-finalization.ts +++ b/src/lib/onboard/created-sandbox-finalization.ts @@ -78,6 +78,7 @@ export type CreatedSandboxFinalizationOptions = { preferredInferenceApi: string | null; endpointUrl?: string | null; nativeProvider?: boolean; + registeredInferenceProviderAttachment?: SandboxEntry["registeredInferenceProviderAttachment"]; }; export type CreatedSandboxFinalizationDeps = { @@ -125,6 +126,7 @@ type RegistrationSeed = Omit< >; export interface CreatedSandboxCompletionOptions { + readonly onRegistered?: () => void; readonly finalization: CreatedSandboxFinalizationOptions; readonly registration: RegistrationSeed; readonly policy: { @@ -579,6 +581,7 @@ export function createCreatedSandboxCompletionActions( prepared, ) : (deps.registerCreatedSandbox ?? registerCreatedSandbox)(input); + options.onRegistered?.(); // Reservation is too early: the former sandbox may still need its provider. // Retirement failure retains the committed selection and cleanup authority. await retireUnselectedCreatedSandboxProviders( @@ -636,6 +639,7 @@ type OnboardCreateIntent = { readonly observabilityEnabled?: boolean; } | null; type OnboardResolvedCreateIntent = { + readonly registeredInferenceProviderAttachment?: SandboxEntry["registeredInferenceProviderAttachment"]; readonly nativeLocalProviderAttachment?: SandboxEntry["nativeLocalProviderAttachment"]; readonly nativeHostedProviderAttachment?: SandboxEntry["nativeHostedProviderAttachment"]; readonly nativeNvidiaProviderAttachment?: SandboxEntry["nativeNvidiaProviderAttachment"]; @@ -647,6 +651,8 @@ type OnboardResolvedCreateIntent = { type OnboardCreateContext = { readonly createIntent: OnboardCreateIntent; readonly resolvedCreateIntent: OnboardResolvedCreateIntent; + readonly getRegisteredInferenceProviderAttachment?: () => SandboxEntry["registeredInferenceProviderAttachment"]; + readonly onRegistered?: () => void; readonly reconcileOpenClawInference?: boolean; }; type OnboardAgentFlags = { @@ -779,8 +785,10 @@ export function createOnboardCreatedSandboxCompletion( } return createCreatedSandboxCompletionActions( { + onRegistered: createContext.onRegistered, finalization: { sandboxName, + gatewayName: gateway.gatewayName, restoreBackupPath, preUpgradeBackup: pendingStateRestoreBackupPath !== null, previousEntry, @@ -792,7 +800,14 @@ export function createOnboardCreatedSandboxCompletion( model, preferredInferenceApi, endpointUrl: resolvedCreateIntent.nativeLocalProviderAttachment?.endpointUrl ?? endpointUrl, + get registeredInferenceProviderAttachment() { + return ( + createContext.getRegisteredInferenceProviderAttachment?.() ?? + resolvedCreateIntent.registeredInferenceProviderAttachment + ); + }, nativeProvider: Boolean( + resolvedCreateIntent.registeredInferenceProviderAttachment || resolvedCreateIntent.nativeLocalProviderAttachment || resolvedCreateIntent.nativeHostedProviderAttachment || resolvedCreateIntent.nativeNvidiaProviderAttachment, @@ -800,6 +815,12 @@ export function createOnboardCreatedSandboxCompletion( }, registration: { sandboxName, + get registeredInferenceProviderAttachment() { + return ( + createContext.getRegisteredInferenceProviderAttachment?.() ?? + resolvedCreateIntent.registeredInferenceProviderAttachment + ); + }, inferenceSelection: selection( sandboxName, provider, @@ -1033,7 +1054,11 @@ export async function finalizeCreatedSandbox( deps.error(` Keep the snapshot for manual recovery: ${options.restoreBackupPath}`); return deps.exitProcess(1); } - if (openClawRestoreWindow && options.reconcileOpenClawInference && !options.customImage) { + if ( + openClawRestoreWindow && + (options.reconcileOpenClawInference || options.registeredInferenceProviderAttachment) && + !options.customImage + ) { try { preparedRegistration = await deps.revalidatePreparedRegistration!(preparedRegistration!); if (!options.gatewayName) @@ -1045,7 +1070,16 @@ export async function finalizeCreatedSandbox( options.preferredInferenceApi, options.gatewayName, deps.revalidateSandboxIdentity, - { endpointUrl: options.endpointUrl, nativeProvider: options.nativeProvider ?? false }, + { + endpointUrl: options.endpointUrl, + nativeProvider: options.nativeProvider ?? false, + ...(options.registeredInferenceProviderAttachment + ? { + registeredInferenceProviderAttachment: + options.registeredInferenceProviderAttachment, + } + : {}), + }, ); } catch (error) { await abortOpenClawRestoreWindow(); diff --git a/src/lib/onboard/created-sandbox-gpu-finalization.test.ts b/src/lib/onboard/created-sandbox-gpu-finalization.test.ts index 51e11d3a262..a7b4407f7e8 100644 --- a/src/lib/onboard/created-sandbox-gpu-finalization.test.ts +++ b/src/lib/onboard/created-sandbox-gpu-finalization.test.ts @@ -4,9 +4,11 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import type { OpenShellSandboxBufferedCommandExecutor } from "../adapters/openshell/sandbox-command"; +import { registeredInferenceFixture } from "../../../test/helpers/registered-inference"; import type { SandboxEntry } from "../state/registry"; import { nativeLocalIdentity, type NativeLocalBinding } from "../inference/native-local/contract"; import { createOnboardCreatedSandboxCompletion } from "./created-sandbox-finalization"; +import * as providerRetirement from "./sandbox-create/provider-retirement"; import { pendingSandboxCreateIdentityForBoundary } from "./sandbox-create/identity-boundary"; import type { SandboxGpuCreateFlowResult } from "./sandbox-gpu-create-flow"; import * as registration from "./sandbox-registration"; @@ -18,6 +20,10 @@ function createCompletionFixture( provider = "ollama-local", route: SandboxGpuCreateFlowResult["route"] = "compatibility", gpuEnabled = true, + createContext: Parameters[7] = { + createIntent: null, + resolvedCreateIntent: { policy: { options: {} } }, + }, localNim = false, ) { const order: string[] = []; @@ -78,15 +84,14 @@ function createCompletionFixture( gatewayName: boundary.gatewayName, sandboxName: boundary.sandboxName, }; - const nativeLocalProviderAttachment = - !localNim && provider.endsWith("-local") - ? { - ...binding, - ...nativeLocalIdentity(binding), - schemaVersion: 1 as const, - providerId: "provider-1", - } - : undefined; + const nativeLocalProviderAttachment = provider.endsWith("-local") + ? { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1 as const, + providerId: "provider-1", + } + : undefined; const verifiedCreate = { checkpoint: pendingSandboxCreateIdentityForBoundary(boundary), reservation: { @@ -126,7 +131,7 @@ function createCompletionFixture( null, { customOpenClawImage: false, isManagedDcodeAgent: false }, selection, - { createIntent: null, resolvedCreateIntent: { policy: { options: {} } } }, + createContext, { gpuEnabled, hostGpuDetected: true, @@ -279,11 +284,18 @@ describe("GPU inference verification in production finalization", () => { }, ); - it("retains the shared route for Local NIM (#14069)", async () => { - const fixture = createCompletionFixture("HTTP_200", "vllm-local", "compatibility", true, true); + it("verifies the native route for Local NVIDIA NIM (#14069)", async () => { + const fixture = createCompletionFixture( + "HTTP_200", + "vllm-local", + "compatibility", + true, + undefined, + true, + ); await fixture.complete(); expect(fixture.runBuffered.mock.calls[0][0].command[2]).toContain( - "https://inference.local/v1/models", + "http://host.openshell.internal:11435/v1/models", ); expect(fixture.order).toEqual(["probe", "commit", "acknowledge", "register"]); }); @@ -354,3 +366,72 @@ describe("GPU inference verification in production finalization", () => { }, ); }); + +it.each([true, false])( + "publishes the current registered provider attachment receipt (owned=%s)", + async (attachedBySelection) => { + const f = registeredInferenceFixture(); + let receipt = f.binding; + const onRegistered = vi.fn(); + const completion = createCompletionFixture( + "HTTP_200", + f.binding.providerName, + "native", + false, + { + createIntent: null, + resolvedCreateIntent: { + policy: { options: {} }, + registeredInferenceProviderAttachment: f.binding, + }, + getRegisteredInferenceProviderAttachment: () => receipt, + onRegistered, + }, + ); + receipt = { ...f.binding, attachedBySelection }; + await completion.complete(); + expect(completion.register.mock.calls[0]![0].registeredInferenceProviderAttachment).toBe( + receipt, + ); + expect(onRegistered).toHaveBeenCalledOnce(); + expect(completion.register.mock.invocationCallOrder[0]).toBeLessThan( + onRegistered.mock.invocationCallOrder[0]!, + ); + }, +); + +it("retains attachment rollback authority if registry publication fails", async () => { + const onRegistered = vi.fn(); + const completion = createCompletionFixture("HTTP_200", "nvidia", "native", false, { + createIntent: null, + resolvedCreateIntent: { policy: { options: {} } }, + onRegistered, + }); + const failure = new Error("publication failed"); + completion.register.mockImplementation(() => { + throw failure; + }); + await expect(completion.complete()).rejects.toBe(failure); + expect(onRegistered).not.toHaveBeenCalled(); +}); + +it("disarms attachment rollback before post-publication provider retirement", async () => { + const onRegistered = vi.fn(); + const completion = createCompletionFixture("HTTP_200", "nvidia", "native", false, { + createIntent: null, + resolvedCreateIntent: { policy: { options: {} } }, + onRegistered, + }); + const failure = new Error("retirement failed"); + const retire = vi + .spyOn(providerRetirement, "retireUnselectedCreatedSandboxProviders") + .mockRejectedValueOnce(failure); + await expect(completion.complete()).rejects.toBe(failure); + expect(onRegistered).toHaveBeenCalledOnce(); + expect(completion.register.mock.invocationCallOrder[0]).toBeLessThan( + onRegistered.mock.invocationCallOrder[0]!, + ); + expect(onRegistered.mock.invocationCallOrder[0]).toBeLessThan( + retire.mock.invocationCallOrder[0]!, + ); +}); diff --git a/src/lib/onboard/credential-provider-registration-provider-type.test.ts b/src/lib/onboard/credential-provider-registration-provider-type.test.ts index d50394a85f6..4bc12d2c6b2 100644 --- a/src/lib/onboard/credential-provider-registration-provider-type.test.ts +++ b/src/lib/onboard/credential-provider-registration-provider-type.test.ts @@ -2,6 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import fs from "node:fs"; +import { REPOSITORY_ROOT } from "../core/repository-root"; +import { buildCredentialFamilyProfile } from "../messaging/applier/credential-family-profile"; import { MESSAGING_CREDENTIAL_PROVIDER_TYPE } from "../messaging/provider-profile"; import type { Session } from "../state/onboard-session"; @@ -14,6 +17,55 @@ import { createCredentialProviderRegistration } from "./credential-provider-regi import type { MessagingTokenDef } from "./messaging-prep"; describe("credential provider type contract", () => { + it.each([false, true])( + "inspects the stored extended Discord profile without requiring a separate base import (drift=%s)", + async (drift) => { + const providerName = "alpha-discord-bridge"; + const baseType = "discord-hermes-static-v1"; + const key = "DISCORD_BOT_TOKEN"; + const keys = [key, `${key}_TEAM`]; + const generated = buildCredentialFamilyProfile( + fs.readFileSync( + `${REPOSITORY_ROOT}/src/lib/messaging/channels/discord/provider-profile/hermes.yaml`, + "utf8", + ), + key, + keys, + ); + const exported = JSON.parse(generated.source); + exported.endpoints.push(...(drift ? [{ host: "unowned.example", port: 443 }] : [])); + const runOpenshell = vi + .fn() + .mockReturnValueOnce(providerMetadata(providerName, generated.profileType, keys.join(", "))) + .mockReturnValueOnce({ status: 0, stdout: JSON.stringify(exported), stderr: "" }) + .mockReturnValue({ status: 1, stdout: "", stderr: "profile not found" }); + const deps = registrationDeps(runOpenshell, { + stagedCredentialProviders: [], + } as unknown as Session); + deps.root = REPOSITORY_ROOT; + const registration = createCredentialProviderRegistration(deps); + + await expect( + registration.inspectGatewayCredential(providerName, baseType, key), + ).resolves.toEqual({ + kind: drift ? "indeterminate" : "exact", + }); + expect(runOpenshell.mock.calls.map(([args]) => args)).toEqual([ + ["provider", "get", "-g", "test-gateway", providerName], + [ + "provider", + "profile", + "-g", + "test-gateway", + "export", + generated.profileType, + "--output", + "json", + ], + ]); + }, + ); + it("rejects an omitted provider type before inspecting or changing the gateway (#13344)", async () => { const session = { stagedCredentialProviders: [] } as unknown as Session; const runOpenshell = vi.fn(); diff --git a/src/lib/onboard/credential-provider-registration.test.ts b/src/lib/onboard/credential-provider-registration.test.ts index 959db75743e..a1807cecbb3 100644 --- a/src/lib/onboard/credential-provider-registration.test.ts +++ b/src/lib/onboard/credential-provider-registration.test.ts @@ -72,7 +72,21 @@ const BRAVE_PROFILE = { const MESSAGING_PROFILE = { id: MESSAGING_CREDENTIAL_PROVIDER_TYPE, - credentials: [], + credentials: [ + "TELEGRAM_BOT_TOKEN", + "DISCORD_BOT_TOKEN", + "WECHAT_BOT_TOKEN", + "SLACK_BOT_TOKEN", + "SLACK_APP_TOKEN", + "MSTEAMS_APP_PASSWORD", + ].map((name) => ({ + name, + env_vars: [name], + required: false, + auth_style: "", + header_name: "", + query_param: "", + })), endpoints: [], binaries: [], inference_capable: false, @@ -300,7 +314,7 @@ describe("credential provider registration", () => { expect( await registration.inspectGatewayCredential( "alpha-telegram-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "TELEGRAM_BOT_TOKEN", ), ).toEqual(expected); @@ -685,7 +699,7 @@ describe("credential provider registration", () => { name: providerName, envKey: "DISCORD_BOT_TOKEN", token: DISCORD_SECRET, - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, ], { allowedSandboxes: ["alpha"] }, @@ -745,7 +759,7 @@ describe("credential provider registration", () => { name: providerName, envKey: "DISCORD_BOT_TOKEN", token: DISCORD_SECRET, - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, ]) .catch((error: unknown) => error); @@ -809,7 +823,7 @@ describe("credential provider registration", () => { name: messagingProviderName, envKey: "DISCORD_BOT_TOKEN", token: DISCORD_SECRET, - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, { name: webSearchProviderName, @@ -826,7 +840,7 @@ describe("credential provider registration", () => { definitions: [ expect.objectContaining({ providerName: messagingProviderName, - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }), expect.objectContaining({ providerName: webSearchProviderName, @@ -898,7 +912,7 @@ describe("credential provider registration", () => { name: typedProviderName, envKey: "DISCORD_BOT_TOKEN", token: DISCORD_SECRET, - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, { name: fallbackProviderName, diff --git a/src/lib/onboard/credential-provider-registration.ts b/src/lib/onboard/credential-provider-registration.ts index d6778f729bf..d197cf7ee94 100644 --- a/src/lib/onboard/credential-provider-registration.ts +++ b/src/lib/onboard/credential-provider-registration.ts @@ -12,7 +12,9 @@ import { import { buildMessagingProviderApplication, hasNemoClawOwnedMessagingProviderProfile, + messagingProviderProfileForType, } from "../messaging/applier/provider-application"; +import { credentialFamilyProfileContract } from "../messaging/applier/credential-family-profile"; import { MessagingSetupApplier } from "../messaging/applier/setup-applier"; import type { SandboxMessagingPlan } from "../messaging/manifest"; import type { CheckpointProviderBinding } from "../state/onboard-checkpoint-types"; @@ -377,12 +379,8 @@ export function createCredentialProviderRegistration(deps: CredentialProviderReg binding: CheckpointProviderBinding, runOpenshell: OpenshellCliHelpers["runOpenshell"], ): Promise { - const profileMatches = messagingBridgeProvider.matchesRegisteredMessagingBridgeProfile( - binding.type, - { root: deps.root, runOpenshell }, - ); - if (profileMatches === false) return { kind: "indeterminate" }; - const observed = await createCliOpenShellProviderAdapter({ run: runOpenshell }).getProvider({ + const adapter = createCliOpenShellProviderAdapter({ run: runOpenshell }); + const observed = await adapter.getProvider({ target: { kind: "selected" }, providerName: binding.name, }); @@ -391,13 +389,44 @@ export function createCredentialProviderRegistration(deps: CredentialProviderReg ? { kind: "missing" } : { kind: "indeterminate" }; } - return gatewayProviderMetadata.matchesGatewayCredentialFamilyProviderBinding(observed.value, { - name: binding.name, - type: binding.type, - credentialKey: binding.credentialEnv, - }) - ? { kind: "exact" } - : { kind: "collision" }; + if ( + !gatewayProviderMetadata.matchesGatewayCredentialFamilyProviderBinding(observed.value, { + name: binding.name, + type: binding.type, + credentialKey: binding.credentialEnv, + }) + ) + return { kind: "collision" }; + if (observed.value.type === binding.type) { + const profileMatches = messagingBridgeProvider.matchesRegisteredMessagingBridgeProfile( + binding.type, + { root: deps.root, runOpenshell }, + ); + return profileMatches === false ? { kind: "indeterminate" } : { kind: "exact" }; + } + const profile = messagingProviderProfileForType( + deps.root, + binding.type, + messagingBridgeProvider + .listMessagingBridgeProfiles({ root: deps.root }) + .find((entry) => entry.profileId === binding.type), + ); + if (!profile) return { kind: "collision" }; + try { + const expectedProfile = credentialFamilyProfileContract( + profile.profilePath, + binding.credentialEnv, + observed.value.credentialKeys, + ); + const verified = await adapter.inspectProviderProfile({ + target: { kind: "selected" }, + profileType: expectedProfile.profileId, + expectedProfile, + }); + return verified.ok ? { kind: "exact" } : { kind: "indeterminate" }; + } catch { + return { kind: "indeterminate" }; + } } async function inspectGatewayCredential( diff --git a/src/lib/onboard/docker-driver-gateway-compat-container.test.ts b/src/lib/onboard/docker-driver-gateway-compat-container.test.ts index dcb3b8b3318..211962889a8 100644 --- a/src/lib/onboard/docker-driver-gateway-compat-container.test.ts +++ b/src/lib/onboard/docker-driver-gateway-compat-container.test.ts @@ -119,9 +119,8 @@ describe("docker-driver-gateway compatibility container", () => { expect(toml).toContain("[openshell.gateway.auth]"); expect(toml).toContain("allow_unauthenticated_users = false"); expect(toml).toContain(`guest_tls_ca = "${path.join(stateDir, "tls", "ca.crt")}"`); - expect(toml).toContain( - `guest_tls_cert = "${path.join(stateDir, "tls", "client", "tls.crt")}"`, - ); + expect(toml).not.toContain("guest_tls_cert"); + expect(toml).not.toContain("guest_tls_key"); expect(launch.env.OPENSHELL_DISABLE_GATEWAY_AUTH).toBeUndefined(); expect(launch.args).not.toContain("OPENSHELL_DISABLE_GATEWAY_AUTH"); expect(fs.existsSync(path.join(stateDir, "jwt", "public.pem"))).toBe(true); @@ -239,7 +238,7 @@ describe("docker-driver-gateway compatibility container", () => { " SECURITY NOTICE: compatibility container uses host networking plus Docker API access; enabled only by NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH=1. Details: https://docs.nvidia.com/nemoclaw/latest/user-guide/openclaw/security/security-controls/gateway-authentication-controls#gateway-compatibility-container.", ]); expect(messages).toContain( - " Gateway auth boundary: host-side OpenShell CLI uses local mTLS; sandbox callbacks use mTLS plus OpenShell gateway JWT.", + " Gateway auth boundary: host-side OpenShell CLI uses local mTLS; sandbox callbacks use server-verified TLS plus OpenShell gateway JWT.", ); }); @@ -272,11 +271,10 @@ describe("docker-driver-gateway compatibility container", () => { }); it("fails closed when the configured Unix socket does not answer as a Docker daemon", async () => { - // Darwin limits AF_UNIX socket paths to 104 bytes. Vitest's nested temp root - // can consume most of that budget before the fixture suffix is appended. - const socketTempRoot = process.platform === "darwin" ? "/tmp" : os.tmpdir(); - const dir = fs.mkdtempSync(path.join(socketTempRoot, "nemoclaw-docker-daemon-probe-")); - const socketPath = path.join(dir, "docker.sock"); + // Keep the fixture suffix short for Darwin's 104-byte socket-path limit + // while respecting the runner's private temporary directory. + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "dd-")); + const socketPath = path.join(dir, "s"); const server = createServer(); await new Promise((resolve, reject) => { server.once("error", reject); diff --git a/src/lib/onboard/docker-driver-gateway-compat.ts b/src/lib/onboard/docker-driver-gateway-compat.ts index 398b500cd9a..ad0ae689b10 100644 --- a/src/lib/onboard/docker-driver-gateway-compat.ts +++ b/src/lib/onboard/docker-driver-gateway-compat.ts @@ -318,7 +318,7 @@ export function logContainerizedDockerDriverGatewayLaunch( " Compatibility gateway bind: 127.0.0.1 main listener plus OpenShell Docker-driver bridge reachability.", ); log( - " Gateway auth boundary: host-side OpenShell CLI uses local mTLS; sandbox callbacks use mTLS plus OpenShell gateway JWT.", + " Gateway auth boundary: host-side OpenShell CLI uses local mTLS; sandbox callbacks use server-verified TLS plus OpenShell gateway JWT.", ); prepareContainerizedDockerDriverGatewayLaunch(launch); } diff --git a/src/lib/onboard/docker-driver-gateway-config-auth-contract.test.ts b/src/lib/onboard/docker-driver-gateway-config-auth-contract.test.ts index ab2f5a42d9e..4b1eb6181ac 100644 --- a/src/lib/onboard/docker-driver-gateway-config-auth-contract.test.ts +++ b/src/lib/onboard/docker-driver-gateway-config-auth-contract.test.ts @@ -6,19 +6,57 @@ import os from "node:os"; import path from "node:path"; import { describe, expect, it } from "vitest"; +import { parse as parseToml } from "smol-toml"; +import { prepareDockerDriverGatewayConfigEnv } from "./docker-driver-gateway-config"; import { - DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS, jwtBundlePaths, mintOpenShellStyleSandboxJwt, - parseTomlInteger, parseTomlString, validateOpenShellStyleSandboxJwt, writeGatewayConfig, } from "../../../test/support/openshell-gateway-config-helpers"; describe("docker-driver-gateway auth contract", () => { - it("emits an OpenShell 0.0.85-compatible sandbox JWT bundle and TTL contract", () => { + it("permits existing create-plan JSON without granting external attachments or host binds", () => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-admission-")); + try { + const env = writeGatewayConfig(stateDir); + const parsed = parseToml(fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8")) as { + openshell: { drivers: { docker: Record } }; + }; + const driver = parsed.openshell.drivers.docker; + expect(driver.allow_driver_config).toBe(true); + expect(driver.enable_bind_mounts).toBeUndefined(); + // No override: pinned OpenShell keeps admission enabled with its required labels. + expect(driver.resource_admission).toBeUndefined(); + expect(env.NEMOCLAW_DOCKER_ENABLE_BIND_MOUNTS).toBeUndefined(); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }); + it.each(["disabled driver JSON", "disabled resource admission"])( + "does not adopt a noncanonical config with %s", + (change) => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-admission-")); + try { + const env = writeGatewayConfig(stateDir); + const before = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8"); + const changed = + change === "disabled driver JSON" + ? before.replace("allow_driver_config = true", "allow_driver_config = false") + : before + "\n[openshell.drivers.docker.resource_admission]\nenabled = false\n"; + fs.writeFileSync(env.OPENSHELL_GATEWAY_CONFIG, changed); + expect(() => + prepareDockerDriverGatewayConfigEnv(env, stateDir, "/usr/bin/openshell-sandbox"), + ).toThrow("the config does not match NemoClaw's generated form"); + expect(fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8")).toBe(changed); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }, + ); + it("emits an OpenShell 0.1.7 sandbox JWT bundle with non-expiring sessions", () => { const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-config-")); try { const env = writeGatewayConfig(stateDir); @@ -27,7 +65,9 @@ describe("docker-driver-gateway auth contract", () => { const publicKeyPath = parseTomlString(toml, "public_key_path"); const kidPath = parseTomlString(toml, "kid_path"); const gatewayId = parseTomlString(toml, "gateway_id"); - const ttlSecs = parseTomlInteger(toml, "ttl_secs"); + const parsed = parseToml(toml) as { + openshell: { gateway: { gateway_jwt: Record } }; + }; const kid = fs.readFileSync(kidPath, "utf-8").trim(); const now = Math.floor(Date.now() / 1000); const sandboxId = "sandbox-contract"; @@ -36,7 +76,7 @@ describe("docker-driver-gateway auth contract", () => { expect(toml).toContain("[openshell.gateway.auth]"); expect(toml).toContain("allow_unauthenticated_users = false"); expect(env.OPENSHELL_DISABLE_GATEWAY_AUTH).toBeUndefined(); - expect(ttlSecs).toBe(DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS); + expect(parsed.openshell.gateway.gateway_jwt.ttl_secs).toBeUndefined(); const token = mintOpenShellStyleSandboxJwt({ signingKeyPath, @@ -44,7 +84,7 @@ describe("docker-driver-gateway auth contract", () => { gatewayId, sandboxId, iat: now, - exp: ttlSecs === 0 ? 0 : now + ttlSecs, + exp: 0, }); const payload = validateOpenShellStyleSandboxJwt({ @@ -60,7 +100,7 @@ describe("docker-driver-gateway auth contract", () => { iss: `openshell-gateway:${gatewayId}`, aud: `openshell-gateway:${gatewayId}`, }); - expect(payload?.exp).toBe(ttlSecs === 0 ? 0 : now + ttlSecs); + expect(payload?.exp).toBe(0); expect(() => validateOpenShellStyleSandboxJwt({ token, @@ -116,14 +156,17 @@ describe("docker-driver-gateway auth contract", () => { } }); - it("emits the complete OpenShell 0.0.85 gateway auth TOML schema", () => { + it("preserves user mTLS without exposing client credentials to OpenShell 0.1.7 sandboxes", () => { const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-config-")); try { const env = writeGatewayConfig(stateDir); const toml = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf-8"); expect(toml).toContain("[openshell.gateway.tls]"); - expect(toml).toContain("require_client_auth = true"); + expect(toml).toContain("version = 2"); + expect(toml).toContain("client_ca_path = "); + expect(toml).not.toContain("require_client_auth"); + expect(toml).not.toContain("[openshell.gateway.oidc]"); expect(toml).toContain("[openshell.gateway.mtls_auth]"); expect(toml).toContain("enabled = true"); expect(toml).toContain("[openshell.gateway.gateway_jwt]"); @@ -131,12 +174,12 @@ describe("docker-driver-gateway auth contract", () => { expect(toml).toContain("public_key_path = "); expect(toml).toContain("kid_path = "); expect(toml).toContain("gateway_id = "); - expect(toml).toContain(`ttl_secs = ${DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS}`); + expect(toml).not.toContain("ttl_secs ="); expect(toml).toContain("[openshell.gateway.auth]"); expect(toml).toContain("allow_unauthenticated_users = false"); expect(toml).toContain("guest_tls_ca = "); - expect(toml).toContain("guest_tls_cert = "); - expect(toml).toContain("guest_tls_key = "); + expect(toml).not.toContain("guest_tls_cert"); + expect(toml).not.toContain("guest_tls_key"); } finally { fs.rmSync(stateDir, { recursive: true, force: true }); } @@ -165,7 +208,7 @@ describe("docker-driver-gateway auth contract", () => { gatewayId: gatewayIdA, sandboxId: sandboxIdA, iat: now, - exp: now + DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS, + exp: 0, }); expect( diff --git a/src/lib/onboard/docker-driver-gateway-config-projected.test.ts b/src/lib/onboard/docker-driver-gateway-config-projected.test.ts index a4e3a1b78b3..1a31d391742 100644 --- a/src/lib/onboard/docker-driver-gateway-config-projected.test.ts +++ b/src/lib/onboard/docker-driver-gateway-config-projected.test.ts @@ -22,7 +22,7 @@ vi.mock("./openshell-version", async (original) => ({ }, })); -describe.each(["0.1.3", "0.1.5"])("OpenShell %s projection", (version) => { +describe.each(["0.1.3", "0.1.5", "0.1.7"])("OpenShell %s projection", (version) => { beforeEach(() => { runtime.version = version; }); diff --git a/src/lib/onboard/docker-driver-gateway-config-toml.test.ts b/src/lib/onboard/docker-driver-gateway-config-toml.test.ts index 03d4c169d10..e6961f3ea23 100644 --- a/src/lib/onboard/docker-driver-gateway-config-toml.test.ts +++ b/src/lib/onboard/docker-driver-gateway-config-toml.test.ts @@ -7,6 +7,7 @@ import os from "node:os"; import path from "node:path"; import { describe, expect, it, vi } from "vitest"; +import { parse as parseToml } from "smol-toml"; import { runOnboardCommand } from "./command"; import { GatewayStateConflictError } from "./errors/gateway-state-conflict"; import { printOnboardResumeHint, resetOnboardResumeHintForTests } from "./resume-hint"; @@ -64,6 +65,42 @@ function podmanGatewayRuntime(env: Record) { }); } +function withLegacyGuestCredentials(toml: string): string { + const tlsDir = path.dirname(parseTomlString(toml, "guest_tls_ca")); + return toml.replace( + /^(guest_tls_ca = .+)$/m, + [ + "$1", + `guest_tls_cert = ${JSON.stringify(path.join(tlsDir, "client", "tls.crt"))}`, + `guest_tls_key = ${JSON.stringify(path.join(tlsDir, "client", "tls.key"))}`, + ].join("\n"), + ); +} + +function schemaOneGatewayConfig(source: string, driver: "docker" | "podman"): string { + const toml = withLegacyGuestCredentials(source); + const guestTls = toml.match(/^guest_tls_.+$/gm) ?? []; + const legacy = + toml + .replace(/^guest_tls_.+\n/gm, "") + .replace("version = 2", "version = 1") + .replace(/^(gateway_id = .+)$/m, "$1\nttl_secs = 0") + .replace(/^allow_driver_config = true\n/m, "") + .replace('compute_driver = "' + driver + '"', 'compute_drivers = ["' + driver + '"]') + .replace(/^sandbox_label = /m, "sandbox_namespace = ") + .replace(/^(client_ca_path = .+)$/m, "$1\nrequire_client_auth = true") + .trimEnd() + + "\n" + + guestTls.join("\n") + + "\n"; + return driver === "docker" + ? legacy.replace( + /^supervisor_image = /m, + 'network_name = "openshell-docker"\nsupervisor_image = ', + ) + : legacy; +} + function writePreScopedGatewayConfig( stateDir: string, includeDefaultNamespace = false, @@ -96,7 +133,7 @@ function writePreScopedGatewayConfig( gatewayId, gatewayRuntime, ); - toml = toml.replace( + toml = schemaOneGatewayConfig(toml, driver).replace( /^sandbox_namespace = .*\n/m, includeDefaultNamespace ? 'sandbox_namespace = "default"\n' : "", ); @@ -107,6 +144,86 @@ function writePreScopedGatewayConfig( } describe("docker-driver-gateway config TOML", () => { + it.each( + (["docker", "podman"] as const).flatMap((driver) => + ([1, 2] as const).flatMap((schema) => + ["guest_tls_ca", "guest_tls_cert", "guest_tls_key"].flatMap((field) => + (["altered", "missing"] as const).map((fault) => ({ driver, schema, field, fault })), + ), + ), + ), + )( + "migrates schema $schema $driver credentials and rejects $fault $field", + ({ driver, schema, field, fault }) => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-schema-one-")); + try { + const env: Record = { + ...baseGatewayEnv(stateDir), + OPENSHELL_DRIVERS: driver, + OPENSHELL_PODMAN_SOCKET: path.join(stateDir, "podman.sock"), + }; + const gatewayRuntime = driver === "podman" ? podmanGatewayRuntime(env) : undefined; + prepareDockerDriverGatewayConfigEnv(env, stateDir, "/usr/bin/openshell-sandbox", { + gatewayRuntime, + }); + const configPath = env.OPENSHELL_GATEWAY_CONFIG; + const before = fs.readFileSync(configPath, "utf-8"); + const bundle = jwtBundlePaths(stateDir); + const keyBefore = fs.readFileSync(bundle.signingKeyPath, "utf-8"); + const clientDir = path.join(stateDir, "tls", "client"); + fs.mkdirSync(clientDir, { recursive: true, mode: 0o700 }); + const clientCert = path.join(clientDir, "tls.crt"); + const clientKey = path.join(clientDir, "tls.key"); + fs.writeFileSync(clientCert, "retained-user-certificate", { mode: 0o600 }); + fs.writeFileSync(clientKey, "retained-user-key", { mode: 0o600 }); + const parsed = parseToml(before) as { + openshell: { + gateway: Record; + drivers: Record>; + }; + }; + expect(parsed.openshell.gateway).toMatchObject({ + guest_tls_ca: path.join(stateDir, "tls", "ca.crt"), + mtls_auth: { enabled: true }, + auth: { allow_unauthenticated_users: false }, + }); + expect(parsed.openshell.gateway).not.toHaveProperty("guest_tls_cert"); + expect(parsed.openshell.gateway).not.toHaveProperty("guest_tls_key"); + expect(parsed.openshell.drivers[driver]).not.toHaveProperty("guest_tls_ca"); + expect(parsed.openshell.drivers[driver]).not.toHaveProperty("guest_tls_cert"); + expect(parsed.openshell.drivers[driver]).not.toHaveProperty("guest_tls_key"); + const legacy = + schema === 1 + ? schemaOneGatewayConfig(before, driver) + : withLegacyGuestCredentials(before); + fs.writeFileSync(configPath, legacy, { mode: 0o600 }); + prepareDockerDriverGatewayConfigEnv(env, stateDir, "/usr/bin/openshell-sandbox", { + gatewayRuntime, + }); + expect(fs.readFileSync(configPath, "utf-8")).toBe(before); + expect(fs.readFileSync(bundle.signingKeyPath, "utf-8")).toBe(keyBefore); + expect(env[NEMOCLAW_OPENSHELL_SANDBOX_NAMESPACE_ENV]).toBe( + driver === "docker" ? gatewayIdForStateDir(stateDir) : undefined, + ); + const replacement = fault === "altered" ? field + ' = "/unowned.pem"\n' : ""; + const invalid = legacy.replace(new RegExp("^" + field + " = .+\\n", "m"), replacement); + fs.writeFileSync(configPath, invalid, { mode: 0o600 }); + expect(() => + prepareDockerDriverGatewayConfigEnv(env, stateDir, "/usr/bin/openshell-sandbox", { + gatewayRuntime, + }), + ).toThrow(GatewayStateConflictError); + expect(fs.readFileSync(configPath, "utf-8")).toBe(invalid); + expect(fs.readFileSync(bundle.signingKeyPath, "utf-8")).toBe(keyBefore); + expect(fs.readFileSync(clientCert, "utf8")).toBe("retained-user-certificate"); + expect(fs.readFileSync(clientKey, "utf8")).toBe("retained-user-key"); + expect(fs.statSync(clientCert).mode & 0o777).toBe(0o600); + expect(fs.statSync(clientKey).mode & 0o777).toBe(0o600); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }, + ); it("renders only the fixed external component interceptor settings (#11340)", () => { const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-component-")); try { @@ -221,8 +338,8 @@ describe("docker-driver-gateway config TOML", () => { expect(toml).toContain(`public_key_path = "${publicKeyPath}"`); expect(toml).toContain(`kid_path = "${kidPath}"`); expect(toml).toContain(`gateway_id = "${gatewayIdForStateDir(stateDir)}"`); - expect(toml).toContain(`sandbox_namespace = "${gatewayIdForStateDir(stateDir)}"`); - expect(toml).toContain(`ttl_secs = ${DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS}`); + expect(toml).toContain(`sandbox_label = "${gatewayIdForStateDir(stateDir)}"`); + expect(toml).not.toContain("ttl_secs ="); expect(toml).toContain("disable_tls = false"); expect(toml).toContain("[openshell.gateway.tls]"); expect(toml).toContain(`cert_path = "${path.join(stateDir, "tls", "server", "tls.crt")}"`); @@ -232,15 +349,11 @@ describe("docker-driver-gateway config TOML", () => { expect(toml).toContain("enabled = true"); expect(toml).toContain("[openshell.gateway.auth]"); expect(toml).toContain("allow_unauthenticated_users = false"); - expect(toml).toContain('compute_drivers = ["docker"]'); + expect(toml).toContain('compute_driver = "docker"'); expect(toml).toContain('grpc_endpoint = "https://127.0.0.1:8080"'); expect(toml).toContain(`guest_tls_ca = "${path.join(stateDir, "tls", "ca.crt")}"`); - expect(toml).toContain( - `guest_tls_cert = "${path.join(stateDir, "tls", "client", "tls.crt")}"`, - ); - expect(toml).toContain( - `guest_tls_key = "${path.join(stateDir, "tls", "client", "tls.key")}"`, - ); + expect(toml).not.toContain("guest_tls_cert"); + expect(toml).not.toContain("guest_tls_key"); expect(toml).toContain('supervisor_bin = "/usr/bin/openshell-sandbox"'); expect(env[NEMOCLAW_OPENSHELL_SANDBOX_NAMESPACE_ENV]).toBe(gatewayIdForStateDir(stateDir)); expect(env.OPENSHELL_DISABLE_GATEWAY_AUTH).toBeUndefined(); @@ -258,9 +371,7 @@ describe("docker-driver-gateway config TOML", () => { gatewayIdForStateDir(stateDir), ); expect(parseTomlString(restartedToml, "gateway_id")).toBe(gatewayIdForStateDir(stateDir)); - expect(parseTomlString(restartedToml, "sandbox_namespace")).toBe( - gatewayIdForStateDir(stateDir), - ); + expect(parseTomlString(restartedToml, "sandbox_label")).toBe(gatewayIdForStateDir(stateDir)); expect(readRegularFileUtf8(signingKeyPath)).toBe(signingKeyBeforeRestart); } finally { fs.rmSync(stateDir, { recursive: true, force: true }); @@ -288,8 +399,8 @@ describe("docker-driver-gateway config TOML", () => { const rewritten = fs.readFileSync(configPath, "utf-8"); expect(parseTomlString(rewritten, "gateway_id")).toBe(gatewayId); - expect(parseTomlString(rewritten, "sandbox_namespace")).toBe("default"); - expect(parseTomlString(rewritten, "network_name")).toBe("openshell-upgraded"); + expect(parseTomlString(rewritten, "sandbox_label")).toBe("default"); + expect(rewritten).not.toContain("network_name ="); expect(parseTomlString(rewritten, "supervisor_bin")).toBe("/opt/openshell/sandbox"); expect(env[NEMOCLAW_OPENSHELL_SANDBOX_NAMESPACE_ENV]).toBe("default"); expect(fs.readFileSync(bundle.signingKeyPath, "utf-8")).toBe(signingKeyBefore); @@ -332,8 +443,7 @@ describe("docker-driver-gateway config TOML", () => { const rewritten = fs.readFileSync(configPath, "utf-8"); expect(parseTomlString(rewritten, "gateway_id")).toBe(gatewayId); - expect(rewritten).toContain(`ttl_secs = ${DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS}`); - expect(rewritten).not.toContain("ttl_secs = 3600"); + expect(rewritten).not.toContain("ttl_secs ="); expect(fs.readFileSync(bundle.signingKeyPath, "utf-8")).toBe(signingKeyBefore); expect( validateOpenShellStyleSandboxJwt({ @@ -368,14 +478,14 @@ describe("docker-driver-gateway config TOML", () => { } }); - it("rejects the legacy JWT TTL on a scoped gateway without mutating the config", () => { + it.each([0, 3600])("rejects explicit schema 2 JWT TTL %i without mutating the config", (ttl) => { const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-scoped-ttl-")); try { const env = writeGatewayConfig(stateDir); const configPath = path.join(stateDir, "openshell-gateway.toml"); const invalidToml = fs .readFileSync(configPath, "utf-8") - .replace(`ttl_secs = ${DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS}`, "ttl_secs = 3600"); + .replace(/^(gateway_id = .+)$/m, "$1\nttl_secs = " + ttl); fs.writeFileSync(configPath, invalidToml, { encoding: "utf-8", mode: 0o600 }); expect(() => @@ -396,7 +506,7 @@ describe("docker-driver-gateway config TOML", () => { const rewritten = fs.readFileSync(configPath, "utf-8"); expect(parseTomlString(rewritten, "gateway_id")).toBe(gatewayId); - expect(parseTomlString(rewritten, "sandbox_namespace")).toBe("default"); + expect(parseTomlString(rewritten, "sandbox_label")).toBe("default"); } finally { fs.rmSync(stateDir, { recursive: true, force: true }); } @@ -687,8 +797,8 @@ describe("docker-driver-gateway config TOML", () => { const misleadingToml = fs .readFileSync(configPath, "utf-8") .replace( - /^sandbox_namespace = .*\n/m, - `operator_note = """\nsandbox_namespace = "${namespace}"\n"""\n`, + /^sandbox_label = .*\n/m, + `operator_note = """\nsandbox_label = "${namespace}"\n"""\n`, ); fs.writeFileSync(configPath, misleadingToml, { encoding: "utf-8", mode: 0o600 }); @@ -821,7 +931,7 @@ describe("docker-driver-gateway config TOML", () => { const config = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf-8"); expect(parseTomlString(config, "gateway_id")).toBe(gatewayIdForStateDir(stateDir)); - expect(parseTomlString(config, "sandbox_namespace")).toBe(gatewayIdForStateDir(stateDir)); + expect(parseTomlString(config, "sandbox_label")).toBe(gatewayIdForStateDir(stateDir)); expect(fs.readFileSync(path.join(stateDir, "openshell.db"), "utf-8")).toBe("legacy-database"); } finally { fs.rmSync(stateDir, { recursive: true, force: true }); @@ -959,9 +1069,9 @@ describe("docker-driver-gateway config TOML", () => { }); const rewritten = fs.readFileSync(configPath, "utf-8"); - expect(rewritten).toContain('compute_drivers = ["podman"]'); + expect(rewritten).toContain('compute_driver = "podman"'); expect(rewritten).toContain("[openshell.drivers.podman]"); - expect(rewritten).not.toContain("sandbox_namespace"); + expect(rewritten).not.toContain("sandbox_label"); expect(parseTomlString(rewritten, "gateway_id")).toBe(gatewayId); expect(parseTomlString(rewritten, "socket_path")).toBe( path.join(stateDir, "new-podman.sock"), @@ -983,3 +1093,48 @@ describe("docker-driver-gateway config TOML", () => { } }); }); + +describe("provider-projected OpenShell driver layouts", () => { + it.each(["split-supervisor", "inline-supervisor"] as const)( + "renders and reuses the projected %s layout without matching a provider identity", + (driverConfigLayout) => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-layout-")); + try { + fs.chmodSync(stateDir, 0o700); + const env = { ...baseGatewayEnv(stateDir), OPENSHELL_DRIVERS: "example" }; + const sourceRuntime = prepareNativePodmanGatewayHostRuntime({ + environment: env, + platform: "linux", + }); + const gatewayRuntime = { + ...sourceRuntime, + providerId: "example-provider", + openShellDriver: "example", + gatewayConfig: { ...sourceRuntime.gatewayConfig, driverConfigLayout }, + }; + prepareDockerDriverGatewayConfigEnv(env, stateDir, "/usr/bin/openshell-sandbox", { + gatewayRuntime, + }); + const configPath = path.join(stateDir, "openshell-gateway.toml"); + const before = fs.readFileSync(configPath, "utf8"); + const parsed = parseToml(before) as { + openshell: { gateway: { compute_driver: string }; drivers: Record }; + }; + expect(parsed.openshell.gateway.compute_driver).toBe("example"); + const driver = parsed.openshell.drivers.example!; + expect(Object.hasOwn(driver, "network_name")).toBe( + driverConfigLayout === "inline-supervisor", + ); + expect(Object.hasOwn(driver, "host_gateway_ip")).toBe( + driverConfigLayout === "inline-supervisor", + ); + prepareDockerDriverGatewayConfigEnv(env, stateDir, "/usr/bin/openshell-sandbox", { + gatewayRuntime, + }); + expect(fs.readFileSync(configPath, "utf8")).toBe(before); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }, + ); +}); diff --git a/src/lib/onboard/docker-driver-gateway-config.ts b/src/lib/onboard/docker-driver-gateway-config.ts index 46e0672551f..348fea92bf7 100644 --- a/src/lib/onboard/docker-driver-gateway-config.ts +++ b/src/lib/onboard/docker-driver-gateway-config.ts @@ -11,6 +11,13 @@ import { } from "node:crypto"; import fs from "node:fs"; import { prepareNativeGatewayCreation } from "./gateway/native-creation"; +import type { ResolvedCorporateCa } from "./corporate-ca-types"; +import { + type GatewayNetworkTrust, + NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV, + readGatewayNetworkTrust, + stageGatewayNetworkTrust, +} from "./gateway/network-trust"; import { DOCKER_DRIVER_GATEWAY_CONFIG_NAME } from "./gateway/config-name"; import path from "node:path"; import { isDeepStrictEqual } from "node:util"; @@ -138,6 +145,7 @@ interface LegacyGatewayIdentity { externalComponent: ExternalComponentGatewayConfiguration | null; gatewayId: string; jwtProof: LegacyJwtBundleProof; + networkTrust: GatewayNetworkTrust | null; kind: "legacy"; sandboxNamespace: "default"; } @@ -147,6 +155,7 @@ type DockerDriverGatewayIdentity = | { configProof: ExistingConfigProof | null; externalComponent: ExternalComponentGatewayConfiguration | null; + networkTrust: GatewayNetworkTrust | null; kind: "scoped"; gatewayId: string; sandboxNamespace: string; @@ -693,6 +702,11 @@ function existingGatewayIdentityFromConfig( if (!parsed || !openshell || !gateway || !gatewayJwt || !drivers || !driverConfig) { throw ambiguousGatewayConfig(configPath, "the config does not match NemoClaw's schema"); } + const supervisor = asTomlTable(openshell.supervisor); + const networkTrust = readGatewayNetworkTrust( + stateDir, + asTomlTable(supervisor?.network)?.additional_ca_bundle, + ); const schemaVersion = openshell.version; if (schemaVersion !== 1 && schemaVersion !== 2) { throw ambiguousGatewayConfig(configPath, "unsupported gateway schema version"); @@ -784,6 +798,7 @@ function existingGatewayIdentityFromConfig( externalComponent, schemaVersion, legacyGuestCredentials, + networkTrust?.path, ); if (originalToml !== canonicalToml) { throw ambiguousGatewayConfig( @@ -796,6 +811,7 @@ function existingGatewayIdentityFromConfig( const identity: LegacyGatewayIdentity = { configProof, externalComponent, + networkTrust, gatewayId: legacyGatewayId, jwtProof: asGatewayStateConflict(() => openOwnedLegacyJwtBundle(stateDir, state.uid)), kind: "legacy", @@ -809,6 +825,7 @@ function existingGatewayIdentityFromConfig( const identity: DockerDriverGatewayIdentity = { configProof, externalComponent, + networkTrust, gatewayId: scopedGatewayId, kind: "scoped", sandboxNamespace: scopedGatewayId, @@ -840,6 +857,7 @@ function resolveDockerDriverGatewayIdentity( return { configProof: null, externalComponent: null, + networkTrust: null, kind: "scoped", gatewayId, sandboxNamespace: gatewayId, @@ -898,6 +916,7 @@ function buildDockerDriverGatewayConfigTomlForIdentity( ? 2 : 1, legacyGuestCredentials = schemaVersion === 1, + additionalCaBundle?: string, ): string { const runtime = resolveGatewayRuntimeProjection(gatewayEnv, projectedRuntime); const driver = runtime.openShellDriver; @@ -1038,6 +1057,13 @@ function buildDockerDriverGatewayConfigTomlForIdentity( ); } + if (additionalCaBundle) { + sections.push( + "[openshell.supervisor.network]", + `additional_ca_bundle = ${tomlString(additionalCaBundle)}`, + "", + ); + } sections.push(`[openshell.drivers.${driver}]`); if (dockerConfig) sections.push(dockerConfig); if (schemaVersion === 1) sections.push(...guestTlsConfig); @@ -1073,61 +1099,65 @@ function writeDockerDriverGatewayConfigWithIdentity( identity: DockerDriverGatewayIdentity, runtime: RuntimeProviderGatewayHostRuntime, requestedExternalComponent?: ExternalComponentGatewayConfiguration | null, + corporateCa?: ResolvedCorporateCa | null, ): string { const configPath = path.join(stateDir, DOCKER_DRIVER_GATEWAY_CONFIG_NAME); const externalComponent = requestedExternalComponent === undefined ? identity.externalComponent : requestedExternalComponent; - if (identity.kind === "legacy") { - try { - assertExistingConfigProof(identity.configProof); - assertLegacyJwtBundleProof(identity.jwtProof); - writeRestrictedFileAtomic( - configPath, - buildDockerDriverGatewayConfigTomlForIdentity( - gatewayEnv, - sandboxBin, - identity.jwtProof.bundle, - identity.gatewayId, - identity.sandboxNamespace, - DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS, - runtime, - externalComponent, - ), - 0o600, - () => { - assertExistingConfigProof(identity.configProof); - assertLegacyJwtBundleProof(identity.jwtProof); - }, - ); - // Existing containers and sandbox JWTs embed this namespace and - // issuer. Other generated settings still follow the current runtime. - return configPath; - } finally { - closeRegularFileProof(identity.configProof); - closeLegacyJwtBundleProof(identity.jwtProof); - } - } try { - if (identity.configProof) assertExistingConfigProof(identity.configProof); - const jwtBundle = ensureDockerDriverGatewayJwtBundle(stateDir); + const assertIdentity = () => { + if (identity.configProof) assertExistingConfigProof(identity.configProof); + if (identity.kind === "legacy") assertLegacyJwtBundleProof(identity.jwtProof); + if (identity.networkTrust) readGatewayNetworkTrust(stateDir, identity.networkTrust.path); + }; + assertIdentity(); + const expectedTrust = gatewayEnv[NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV]; + if ( + corporateCa === undefined && + expectedTrust !== undefined && + expectedTrust !== (identity.networkTrust?.sha256 ?? "none") + ) { + throw ambiguousGatewayConfig(configPath, "the network trust configuration changed"); + } + const networkTrust = + corporateCa === undefined + ? identity.networkTrust + : corporateCa === null + ? null + : stageGatewayNetworkTrust(stateDir, corporateCa.pem); + const jwtBundle = + identity.kind === "legacy" + ? identity.jwtProof.bundle + : ensureDockerDriverGatewayJwtBundle(stateDir); writeRestrictedFileAtomic( configPath, - buildDockerDriverGatewayConfigToml( + buildDockerDriverGatewayConfigTomlForIdentity( gatewayEnv, sandboxBin, jwtBundle, identity.gatewayId, + identity.sandboxNamespace, + DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS, runtime, externalComponent, + undefined, + undefined, + networkTrust?.path, ), 0o600, - identity.configProof ? () => assertExistingConfigProof(identity.configProof!) : undefined, + () => { + assertIdentity(); + if (networkTrust) readGatewayNetworkTrust(stateDir, networkTrust.path); + }, ); + // Include absence so runtime comparison also detects removal of destination trust. + gatewayEnv[NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV] = networkTrust?.sha256 ?? "none"; return configPath; } finally { if (identity.configProof) closeRegularFileProof(identity.configProof); + if (identity.kind === "legacy") closeLegacyJwtBundleProof(identity.jwtProof); } } @@ -1216,6 +1246,7 @@ export function prepareDockerDriverGatewayConfigEnv( stateDir: string, sandboxBin?: string | null, options: { + corporateCa?: ResolvedCorporateCa | null; allowOpenShell0044PreAuthDatabase?: boolean; gatewayRuntime?: RuntimeProviderGatewayHostRuntime; externalComponent?: ExternalComponentGatewayConfiguration | null; @@ -1231,6 +1262,7 @@ function prepareGatewayConfig( stateDir: string, sandboxBin: string | null | undefined, options: { + corporateCa?: ResolvedCorporateCa | null; allowOpenShell0044PreAuthDatabase?: boolean; gatewayRuntime?: RuntimeProviderGatewayHostRuntime; externalComponent?: ExternalComponentGatewayConfiguration | null; @@ -1270,6 +1302,7 @@ function prepareGatewayConfig( identity, runtime, externalComponent, + options.corporateCa, ); // An explicit absence lets the existing runtime comparison detect removal. gatewayEnv[NEMOCLAW_EXTERNAL_COMPONENT_GATEWAY_IDENTITY_ENV] = observedComponentIdentity; diff --git a/src/lib/onboard/docker-driver-gateway-env-config-validation.test.ts b/src/lib/onboard/docker-driver-gateway-env-config-validation.test.ts index a7d591026a0..349fd42eadc 100644 --- a/src/lib/onboard/docker-driver-gateway-env-config-validation.test.ts +++ b/src/lib/onboard/docker-driver-gateway-env-config-validation.test.ts @@ -7,7 +7,6 @@ import path from "node:path"; import { describe, expect, it } from "vitest"; -import { DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS } from "./docker-driver-gateway-config"; import { assertDockerDriverGatewayAuthConfigSafe, assertDockerDriverGatewayBindAddressSafe, @@ -16,6 +15,101 @@ import { prepareNativePodmanGatewayHostRuntime } from "./runtime-provider/podman import { writeSafeGatewayAuthConfig } from "../../../test/support/docker-driver-gateway-env-test-support"; describe("Docker-driver gateway env config validation", () => { + it.each(["cert_path", "key_path", "client_ca_path"])("rejects missing TLS input %s", (key) => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-auth-")); + try { + const configPath = writeSafeGatewayAuthConfig(stateDir); + fs.writeFileSync( + configPath, + fs.readFileSync(configPath, "utf-8").replace(new RegExp("^" + key + " = .+\\n", "m"), ""), + ); + expect(() => + assertDockerDriverGatewayAuthConfigSafe( + { + OPENSHELL_BIND_ADDRESS: "127.0.0.1", + OPENSHELL_GATEWAY_CONFIG: configPath, + }, + {}, + ), + ).toThrow("openshell.gateway.tls." + key); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }); + + it.each([ + [ + "relative CA path", + /^client_ca_path = .+$/m, + 'client_ca_path = "relative.crt"', + "absolute TLS path", + ], + ["schema 1", /version = 2/, "version = 1", "openshell.version=2"], + [ + "unquoted CA path", + /^client_ca_path = .+$/m, + "client_ca_path = /invalid.crt", + "Invalid TOML document", + ], + ["invalid TLS boolean", /disable_tls = false/, "disable_tls = []", "disable_tls=false"], + [ + "retired TLS switch", + /\[openshell.gateway.tls\]/, + "[openshell.gateway.tls]\nrequire_client_auth = true", + "client-CA authentication", + ], + [ + "OIDC config", + /$/, + '\n[openshell.gateway.oidc]\nissuer = "https://issuer.invalid"\n', + "client-CA authentication", + ], + ] as const)( + "rejects %s without changing the config", + (_kind, pattern, replacement, expectedError) => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-auth-")); + try { + const configPath = writeSafeGatewayAuthConfig(stateDir); + const gatewayEnv: Record = { + OPENSHELL_BIND_ADDRESS: "127.0.0.1", + OPENSHELL_GATEWAY_CONFIG: configPath, + }; + const config = fs.readFileSync(configPath, "utf-8").replace(pattern, replacement); + fs.writeFileSync(configPath, config); + expect(() => assertDockerDriverGatewayAuthConfigSafe(gatewayEnv, {})).toThrow( + expectedError, + ); + expect(fs.readFileSync(configPath, "utf-8")).toBe(config); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }, + ); + + it.each([ + ["OIDC environment", { OPENSHELL_OIDC_ISSUER: "https://issuer.invalid" }, {}], + ["empty OIDC environment", { OPENSHELL_OIDC_ISSUER: "" }, {}], + ["OIDC gateway environment", {}, { OPENSHELL_OIDC_ISSUER: "https://issuer.invalid" }], + ] as const)("rejects %s without changing the config", (_kind, environment, overlay) => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-auth-")); + try { + const configPath = writeSafeGatewayAuthConfig(stateDir); + const before = fs.readFileSync(configPath, "utf-8"); + expect(() => + assertDockerDriverGatewayAuthConfigSafe( + { + OPENSHELL_BIND_ADDRESS: "127.0.0.1", + OPENSHELL_GATEWAY_CONFIG: configPath, + ...overlay, + }, + environment, + ), + ).toThrow("client-CA authentication"); + expect(fs.readFileSync(configPath, "utf-8")).toBe(before); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } + }); it("rejects wildcard gateway binds while gateway JWT auth is active", () => { expect(() => assertDockerDriverGatewayBindAddressSafe({ @@ -76,7 +170,7 @@ describe("Docker-driver gateway env config validation", () => { } }); - it.each(["signing_key_path", "public_key_path", "kid_path", "gateway_id", "ttl_secs"])( + it.each(["signing_key_path", "public_key_path", "kid_path", "gateway_id"])( "rejects a config missing gateway_jwt.$key", (key) => { const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-env-")); @@ -99,7 +193,7 @@ describe("Docker-driver gateway env config validation", () => { }, ); - it("rejects a gateway JWT TTL outside NemoClaw's bounded value", () => { + it.each(["0", "7200", "-1", "[]", '"0"'])("rejects explicit gateway JWT TTL %s", (ttl) => { const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-env-")); try { const configPath = writeSafeGatewayAuthConfig(stateDir); @@ -107,7 +201,7 @@ describe("Docker-driver gateway env config validation", () => { configPath, fs .readFileSync(configPath, "utf-8") - .replace(`ttl_secs = ${DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS}`, "ttl_secs = 7200"), + .replace(/^(gateway_id = .+)$/m, "$1\nttl_secs = " + ttl), ); expect(() => @@ -115,7 +209,7 @@ describe("Docker-driver gateway env config validation", () => { OPENSHELL_BIND_ADDRESS: "127.0.0.1", OPENSHELL_GATEWAY_CONFIG: configPath, }), - ).toThrow(`gateway_jwt.ttl_secs=${DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS}`); + ).toThrow("must omit gateway_jwt.ttl_secs"); } finally { fs.rmSync(stateDir, { recursive: true, force: true }); } diff --git a/src/lib/onboard/docker-driver-gateway-env-service.test.ts b/src/lib/onboard/docker-driver-gateway-env-service.test.ts index 88cd8da347a..4d37788ff71 100644 --- a/src/lib/onboard/docker-driver-gateway-env-service.test.ts +++ b/src/lib/onboard/docker-driver-gateway-env-service.test.ts @@ -257,7 +257,7 @@ describe("package-managed Docker-driver gateway env service", () => { ).toThrow(/not supported for the OpenShell Docker-driver gateway/); }); - it.each(["signing_key_path", "public_key_path", "kid_path", "gateway_id", "ttl_secs"])( + it.each(["signing_key_path", "public_key_path", "kid_path", "gateway_id"])( "rejects incomplete gateway JWT config before writing env or starting the service [%s] (#6903)", (key) => { const tempHome = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-env-")); diff --git a/src/lib/onboard/docker-driver-gateway-env.test.ts b/src/lib/onboard/docker-driver-gateway-env.test.ts index de9979e9c53..2cf241301ff 100644 --- a/src/lib/onboard/docker-driver-gateway-env.test.ts +++ b/src/lib/onboard/docker-driver-gateway-env.test.ts @@ -191,7 +191,7 @@ describe("buildDockerDriverGatewayEnv", () => { OPENSHELL_PODMAN_SOCKET: "/run/user/1001/podman/podman.sock", }); const toml = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf-8"); - expect(toml).toContain('compute_drivers = ["podman"]'); + expect(toml).toContain('compute_driver = "podman"'); expect(toml).toContain("[openshell.drivers.podman]"); expect(toml).toContain(`host_gateway_ip = "${PORTABLE_HOST_GATEWAY_IP}"`); expect(toml).toContain('socket_path = "/run/user/1001/podman/podman.sock"'); @@ -223,7 +223,7 @@ describe("buildDockerDriverGatewayEnv", () => { expect(env.CONTAINERS_CONF).toBeUndefined(); expect(env.NETAVARK_FW).toBeUndefined(); const toml = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf-8"); - expect(toml).toContain('compute_drivers = ["podman"]'); + expect(toml).toContain('compute_driver = "podman"'); expect(toml).toContain(`socket_path = "${env.OPENSHELL_PODMAN_SOCKET}"`); expect(toml).not.toContain("supervisor_bin"); } finally { diff --git a/src/lib/onboard/docker-driver-gateway-env.ts b/src/lib/onboard/docker-driver-gateway-env.ts index 358b8d42e03..675d6bc31bf 100644 --- a/src/lib/onboard/docker-driver-gateway-env.ts +++ b/src/lib/onboard/docker-driver-gateway-env.ts @@ -2,6 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 import fs from "node:fs"; +import { resolveCorporateCa } from "./corporate-ca"; +import { isCorporateCaImportDisabled } from "./corporate-ca-policy"; +import { NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV } from "./gateway/network-trust"; import os from "node:os"; import path from "node:path"; import { parse as parseToml } from "smol-toml"; @@ -85,6 +88,7 @@ export const DOCKER_DRIVER_GATEWAY_RUNTIME_ENV_KEYS = [ "NEMOCLAW_DOCKER_ENABLE_BIND_MOUNTS", NEMOCLAW_EXTERNAL_COMPONENT_GATEWAY_IDENTITY_ENV, NEMOCLAW_OPENSHELL_SANDBOX_NAMESPACE_ENV, + NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV, "NETAVARK_FW", ] as const; @@ -526,6 +530,9 @@ export function buildDockerDriverGatewayEnv({ allowOpenShell0044PreAuthDatabase: process.env.NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE === "1", gatewayRuntime: runtime, + corporateCa: isCorporateCaImportDisabled(process.env) + ? null + : (resolveCorporateCa() ?? undefined), }); return env; } diff --git a/src/lib/onboard/docker-driver-gateway-launch.test.ts b/src/lib/onboard/docker-driver-gateway-launch.test.ts index 7bfddd5a512..dd9cd30f3cc 100644 --- a/src/lib/onboard/docker-driver-gateway-launch.test.ts +++ b/src/lib/onboard/docker-driver-gateway-launch.test.ts @@ -128,10 +128,10 @@ describe("docker-driver-gateway-launch", () => { "/home/shadeform/.local/bin/openshell-sandbox", ); - expect(toml).toContain('compute_drivers = ["docker"]'); - expect(toml).toContain('sandbox_namespace = "nemoclaw"'); + expect(toml).toContain('compute_driver = "docker"'); + expect(toml).toContain('sandbox_label = "nemoclaw"'); expect(toml).toContain('grpc_endpoint = "https://127.0.0.1:8080"'); - expect(toml).toContain('network_name = "openshell-docker"'); + expect(toml).not.toContain("network_name ="); expect(toml).toContain('supervisor_image = "ghcr.io/nvidia/openshell/supervisor:0.0.44"'); expect(toml).toContain('supervisor_bin = "/home/shadeform/.local/bin/openshell-sandbox"'); }); @@ -182,7 +182,7 @@ describe("docker-driver-gateway-launch", () => { expect(toml).toContain("[openshell.drivers.podman]"); expect(toml).toContain('socket_path = "/run/user/1001/podman/podman.sock"'); - expect(toml).not.toContain("sandbox_namespace"); + expect(toml).not.toContain("sandbox_label"); }); it("rejects wildcard binds for direct host gateway launches", () => { diff --git a/src/lib/onboard/docker-driver-gateway-network-trust.test.ts b/src/lib/onboard/docker-driver-gateway-network-trust.test.ts new file mode 100644 index 00000000000..d8ac08b06d9 --- /dev/null +++ b/src/lib/onboard/docker-driver-gateway-network-trust.test.ts @@ -0,0 +1,196 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import path from "node:path"; +import { parse } from "smol-toml"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { + baseGatewayEnv, + jwtBundlePaths, +} from "../../../test/support/openshell-gateway-config-helpers"; +import { PEM, PRIVATE_KEY, tmpDir, writeCa } from "./__test-helpers__/corporate-ca-fixtures"; +import { resolveCorporateCa } from "./corporate-ca"; +import { prepareDockerDriverGatewayConfigEnv } from "./docker-driver-gateway-config"; +import { + assertDockerDriverGatewayAuthConfigSafe, + buildDockerDriverGatewayEnv, + buildDockerGatewayDebEnvFile, + configureDockerDriverGatewayExternalComponent, +} from "./docker-driver-gateway-env"; +import { hashDockerDriverGatewayEnv } from "./docker-driver-gateway-runtime-marker"; +import { NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV } from "./gateway/network-trust"; +import { prepareNativePodmanGatewayHostRuntime } from "./runtime-provider/podman-runtime-surfaces"; + +function resolved(pem = PEM) { + return { + pem, + sourcePath: "/not-needed-after-validation/ca.pem", + sourceEnv: "NEMOCLAW_CORPORATE_CA_BUNDLE", + }; +} + +function trustPath(env: Record): string { + const parsed = parse(fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8")) as unknown as { + openshell: { supervisor: { network: { additional_ca_bundle: string } } }; + }; + return parsed.openshell.supervisor.network.additional_ca_bundle; +} + +describe("Gateway private destination trust", () => { + beforeEach(() => { + for (const key of [ + "NEMOCLAW_CORPORATE_CA_BUNDLE", + "REQUESTS_CA_BUNDLE", + "CURL_CA_BUNDLE", + "SSL_CERT_FILE", + "NEMOCLAW_CORPORATE_CA_IMPORT", + ]) { + vi.stubEnv(key, undefined); + } + vi.stubEnv("NEMOCLAW_CORPORATE_CA_ANCHOR_DIRS", ""); + }); + + it.each(["docker", "podman"] as const)( + "preserves %s destination trust and authentication on restart without the original source", + (driver) => { + const root = tmpDir(); + const env = baseGatewayEnv(root); + const runtime = + driver === "podman" + ? prepareNativePodmanGatewayHostRuntime({ + environment: { OPENSHELL_DRIVERS: "podman" }, + platform: "linux", + socketPath: path.join(root, "podman.sock"), + }) + : undefined; + prepareDockerDriverGatewayConfigEnv(env, root, null, { + gatewayRuntime: runtime, + corporateCa: resolved(), + }); + const original = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8"); + const savedCa = trustPath(env); + const jwt = Object.values(jwtBundlePaths(root)).map((file) => fs.readFileSync(file, "utf8")); + const restarted = baseGatewayEnv(root); + prepareDockerDriverGatewayConfigEnv(restarted, root, null, { gatewayRuntime: runtime }); + expect(fs.readFileSync(restarted.OPENSHELL_GATEWAY_CONFIG, "utf8")).toBe(original); + expect(trustPath(restarted)).toBe(savedCa); + expect(hashDockerDriverGatewayEnv(restarted)).toBe(hashDockerDriverGatewayEnv(env)); + expect( + Object.values(jwtBundlePaths(root)).map((file) => fs.readFileSync(file, "utf8")), + ).toEqual(jwt); + const parsed = parse(original) as unknown as { + openshell: { gateway: Record }; + }; + expect(parsed.openshell.gateway).toMatchObject({ + guest_tls_ca: path.join(root, "tls", "ca.crt"), + tls: { client_ca_path: path.join(root, "tls", "ca.crt") }, + mtls_auth: { enabled: true }, + auth: { allow_unauthenticated_users: false }, + }); + expect(() => + assertDockerDriverGatewayAuthConfigSafe(restarted, {}, "linux", runtime), + ).not.toThrow(); + }, + ); + + it("changes the runtime identity for rotation and removal while retaining old snapshots and JWT keys", () => { + const root = tmpDir(); + const env = baseGatewayEnv(root); + prepareDockerDriverGatewayConfigEnv(env, root, null, { corporateCa: resolved() }); + const first = { ...env }; + const oldPath = trustPath(env); + const jwt = fs.readFileSync(jwtBundlePaths(root).signingKeyPath, "utf8"); + prepareDockerDriverGatewayConfigEnv(env, root, null, { corporateCa: resolved(PEM + PEM) }); + const rotated = { ...env }; + expect(hashDockerDriverGatewayEnv(rotated)).not.toBe(hashDockerDriverGatewayEnv(first)); + expect(trustPath(env)).not.toBe(oldPath); + prepareDockerDriverGatewayConfigEnv(env, root, null, { corporateCa: null }); + expect(hashDockerDriverGatewayEnv(env)).not.toBe(hashDockerDriverGatewayEnv(rotated)); + expect(env[NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV]).toBe("none"); + expect(fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8")).not.toContain( + "additional_ca_bundle", + ); + expect(fs.readFileSync(oldPath, "utf8")).toBe(PEM); + expect(fs.readFileSync(jwtBundlePaths(root).signingKeyPath, "utf8")).toBe(jwt); + expect(() => prepareDockerDriverGatewayConfigEnv(first, root)).toThrow( + /network trust configuration changed/, + ); + }); + + it("keeps the old config and identity after invalid CA material or snapshot tampering", () => { + const root = tmpDir(); + const env = baseGatewayEnv(root); + prepareDockerDriverGatewayConfigEnv(env, root, null, { corporateCa: resolved() }); + const original = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8"); + const jwt = fs.readFileSync(jwtBundlePaths(root).signingKeyPath, "utf8"); + expect(() => + prepareDockerDriverGatewayConfigEnv(env, root, null, { + corporateCa: resolved(PEM + PRIVATE_KEY), + }), + ).toThrow(/only normalized PEM/); + const saved = trustPath(env); + fs.chmodSync(saved, 0o644); + expect(() => prepareDockerDriverGatewayConfigEnv(env, root)).toThrow( + /read-only certificate snapshot/, + ); + expect(fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8")).toBe(original); + expect(fs.readFileSync(jwtBundlePaths(root).signingKeyPath, "utf8")).toBe(jwt); + }); + + it("preserves a retained legacy gateway issuer and namespace when adding destination trust", () => { + const root = tmpDir(); + const env = baseGatewayEnv(root); + prepareDockerDriverGatewayConfigEnv(env, root); + const legacyId = `nemoclaw-${path.basename(root)}`; + const config = fs + .readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8") + .replace(/^gateway_id = .*$/mu, `gateway_id = "${legacyId}"`) + .replace(/^sandbox_label = .*$/mu, 'sandbox_label = "default"'); + fs.writeFileSync(env.OPENSHELL_GATEWAY_CONFIG, config); + const jwt = fs.readFileSync(jwtBundlePaths(root).signingKeyPath, "utf8"); + prepareDockerDriverGatewayConfigEnv(env, root, null, { corporateCa: resolved() }); + const next = fs.readFileSync(env.OPENSHELL_GATEWAY_CONFIG, "utf8"); + expect(next).toContain(`gateway_id = "${legacyId}"`); + expect(next).toContain('sandbox_label = "default"'); + expect(next).toContain("additional_ca_bundle"); + expect(fs.readFileSync(jwtBundlePaths(root).signingKeyPath, "utf8")).toBe(jwt); + }); + + it("uses validated source certificates, preserves them during component changes, and honors explicit import removal", () => { + const root = tmpDir(); + const source = writeCa(tmpDir(), PEM + PRIVATE_KEY); + vi.stubEnv("NEMOCLAW_CORPORATE_CA_BUNDLE", source); + const build = () => + buildDockerDriverGatewayEnv({ + stateDir: root, + platform: "linux", + getDockerSupervisorImage: () => "supervisor:test", + resolveSandboxBin: () => null, + }); + const env = build(); + const saved = trustPath(env); + expect(fs.readFileSync(saved, "utf8")).toBe(resolveCorporateCa()!.pem); + expect(fs.readFileSync(saved, "utf8")).not.toContain("PRIVATE"); + configureDockerDriverGatewayExternalComponent(env, { + componentId: "policy-governance", + interceptorSocketPath: "/run/user/1000/component/interceptor.sock", + }); + expect(trustPath(env)).toBe(saved); + const serviceEnv = buildDockerGatewayDebEnvFile( + "NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256=stale\n", + env, + ); + expect(serviceEnv).toContain( + `${NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV}=${env[NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV]}\n`, + ); + expect(serviceEnv).not.toContain("stale"); + fs.unlinkSync(source); + vi.stubEnv("NEMOCLAW_CORPORATE_CA_BUNDLE", undefined); + expect(trustPath(build())).toBe(saved); + vi.stubEnv("NEMOCLAW_CORPORATE_CA_IMPORT", "false"); + const removed = build(); + expect(removed[NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV]).toBe("none"); + expect(fs.readFileSync(saved, "utf8")).toBe(PEM); + }); +}); diff --git a/src/lib/onboard/docker-driver-gateway-runtime.test.ts b/src/lib/onboard/docker-driver-gateway-runtime.test.ts index feb31353a8b..b564da34ed6 100644 --- a/src/lib/onboard/docker-driver-gateway-runtime.test.ts +++ b/src/lib/onboard/docker-driver-gateway-runtime.test.ts @@ -44,7 +44,7 @@ function makeHelpers(overrides: Partial = {}): { loadDockerDriverGatewayEnv: createGatewayEnvLoader(dockerDriverGatewayEnv), runCapture, shouldUseOpenshellDevChannel: () => false, - supportedOpenshellFallbackVersion: "0.0.116", + supportedOpenshellFallbackVersion: "0.1.7", ...overrides, }; return { @@ -105,7 +105,7 @@ describe("docker-driver gateway runtime helpers", () => { }, () => { const { helpers } = makeHelpers({ - supportedOpenshellFallbackVersion: "0.0.116", + supportedOpenshellFallbackVersion: "0.1.7", }); expect(helpers.getDockerDriverGatewayStateDir()).toBe(path.resolve(stateDir)); @@ -119,7 +119,7 @@ describe("docker-driver gateway runtime helpers", () => { expect(env.OPENSHELL_DOCKER_NETWORK_NAME).toBe("custom-openshell-docker"); expect(env.OPENSHELL_DOCKER_SUPERVISOR_BIN).toBe(path.resolve(sandboxBin)); expect(env.OPENSHELL_DOCKER_SUPERVISOR_IMAGE).toBe( - "ghcr.io/nvidia/openshell/supervisor@sha256:c8c42aef16c200063e32cbf72e553e4ead027085427b555efafd95063ecead42", + "ghcr.io/nvidia/openshell/supervisor@sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06", ); expect(env.OPENSHELL_GATEWAY_CONFIG).toBe( path.join(path.resolve(stateDir), "openshell-gateway.toml"), @@ -209,7 +209,7 @@ describe("docker-driver gateway runtime helpers", () => { withTemporaryGatewayState(() => { const explicit = makeHelpers({ shouldUseOpenshellDevChannel: () => true }); expect(() => explicit.helpers.getDockerDriverGatewayEnv("openshell 0.0.72", "linux")).toThrow( - "exact stable OpenShell 0.0.116", + "exact stable OpenShell 0.1.7", ); const detected = makeHelpers({ @@ -217,25 +217,25 @@ describe("docker-driver gateway runtime helpers", () => { }); expect(() => detected.helpers.getDockerDriverGatewayEnv("openshell 0.0.72-dev.8+g7bce1223", "linux"), - ).toThrow("exact stable OpenShell 0.0.116"); + ).toThrow("exact stable OpenShell 0.1.7"); }); }); - it("pins the stable 0.0.116 supervisor and rejects a foreign override", () => { + it("pins the stable 0.1.7 supervisor and rejects a foreign override", () => { const image = (fallback: string) => makeHelpers({ - getBlueprintMaxOpenshellVersion: () => "0.0.116", + getBlueprintMaxOpenshellVersion: () => "0.1.7", supportedOpenshellFallbackVersion: fallback, }).helpers.getDockerDriverGatewayEnv(null, "linux").OPENSHELL_DOCKER_SUPERVISOR_IMAGE; const stable = withTemporaryGatewayState(() => - withEnv({ OPENSHELL_DOCKER_SUPERVISOR_IMAGE: undefined }, () => image("0.0.116")), + withEnv({ OPENSHELL_DOCKER_SUPERVISOR_IMAGE: undefined }, () => image("0.1.7")), ); const qualified = - "ghcr.io/nvidia/openshell/supervisor@sha256:c8c42aef16c200063e32cbf72e553e4ead027085427b555efafd95063ecead42"; + "ghcr.io/nvidia/openshell/supervisor@sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06"; expect(stable).toBe(qualified); expect( withTemporaryGatewayState(() => - withEnv({ OPENSHELL_DOCKER_SUPERVISOR_IMAGE: qualified }, () => image("0.0.116")), + withEnv({ OPENSHELL_DOCKER_SUPERVISOR_IMAGE: qualified }, () => image("0.1.7")), ), ).toBe(qualified); expect(() => @@ -244,7 +244,7 @@ describe("docker-driver gateway runtime helpers", () => { { OPENSHELL_DOCKER_SUPERVISOR_IMAGE: "registry.example.test/supervisor@sha256:override", }, - () => image("0.0.116"), + () => image("0.1.7"), ), ), ).toThrow("requires the reviewed Docker supervisor image"); @@ -254,7 +254,7 @@ describe("docker-driver gateway runtime helpers", () => { const { helpers } = makeHelpers(); expect(() => helpers.getDockerDriverGatewayEnv("openshell 0.0.106", "linux")).toThrow( - "requires exact stable OpenShell 0.0.116; found 0.0.106", + "requires exact stable OpenShell 0.1.7; found 0.0.106", ); }); diff --git a/src/lib/onboard/docker-driver-gateway-runtime.ts b/src/lib/onboard/docker-driver-gateway-runtime.ts index 9aa42b6d243..bb4eb886ca5 100644 --- a/src/lib/onboard/docker-driver-gateway-runtime.ts +++ b/src/lib/onboard/docker-driver-gateway-runtime.ts @@ -49,8 +49,12 @@ const OPENSHELL_SUPERVISOR_MANIFEST_DIGESTS: Readonly> = "0.0.99": "sha256:ea3632b6e9528e2309103af5b6949606fcdc83ca1f69e8db81482a25bea84bb6", "0.0.101": "sha256:b58be5e40c788977ffa0e8305a8cad9c656efdf1a3fe182582a00ca870bb0edb", "0.0.116": "sha256:c8c42aef16c200063e32cbf72e553e4ead027085427b555efafd95063ecead42", + "0.1.2": "sha256:d7b5264bb6bc56f4796e6fa3617b8e4a8d785be0b7293542efd8cc250b0fb67a", + "0.1.3": "sha256:4ab2b970cdb02c7518ba4ccbbadb4861ae9a8d0b8770e5b160697f0453b0a8e4", + "0.1.5": "sha256:fabcd01f656e9d23df4386f82e80b4f48b90f3452d7d5c58889007643a069161", + "0.1.7": "sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06", }; -const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.0.116"; +const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.1.7"; /** Resolve the canonical gateway name without bypassing the binding owner. */ export function resolveDockerDriverGatewayName(gatewayPort: number): string { diff --git a/src/lib/onboard/docker-driver-gateway-service-version-gate.test.ts b/src/lib/onboard/docker-driver-gateway-service-version-gate.test.ts index 9a0d2fa96e0..70ce0c17c53 100644 --- a/src/lib/onboard/docker-driver-gateway-service-version-gate.test.ts +++ b/src/lib/onboard/docker-driver-gateway-service-version-gate.test.ts @@ -252,7 +252,7 @@ describe("package-managed gateway version gate (#8094)", () => { message: expect.stringContaining("development build"), }); expect(result).toMatchObject({ - message: expect.stringContaining("exact stable OpenShell 0.0.116"), + message: expect.stringContaining("exact stable OpenShell 0.1.7"), }); }); diff --git a/src/lib/onboard/docker-driver-gateway-service.ts b/src/lib/onboard/docker-driver-gateway-service.ts index beea35033f4..3cef63cd338 100644 --- a/src/lib/onboard/docker-driver-gateway-service.ts +++ b/src/lib/onboard/docker-driver-gateway-service.ts @@ -27,7 +27,7 @@ export const NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE = "nemoclaw-openshell-gatew export const OPENSHELL_GATEWAY_HOMEBREW_SERVICE = "openshell"; export const OPENSHELL_GATEWAY_HOMEBREW_TAP = "nvidia/openshell"; export const OPENSHELL_GATEWAY_HOMEBREW_FORMULA_SHA256 = - "cf00a9441589702ffe006720fd6a9dffc0f0745b337036aad26dc53eb94c1558"; + "daa268c73d627cc72fb5c36b041e0e8625b1125cc83706f76772d52b5da081d7"; export const NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE_MARKER = "NEMOCLAW_MANAGED_OPENSHELL_GATEWAY=1"; export const NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE_MARKER_LINE = `# ${NEMOCLAW_OPENSHELL_GATEWAY_USER_SERVICE_MARKER}`; @@ -289,7 +289,7 @@ export function checkUpstreamGatewayVersion( version, message: ` Refusing the system OpenShell gateway service: ${binaryPath} is a development build. ` + - "Install exact stable OpenShell 0.0.116 before retrying NemoClaw.", + "Install exact stable OpenShell 0.1.7 before retrying NemoClaw.", }; } const bounds = (opts.getUpstreamGatewayVersionBounds ?? defaultUpstreamGatewayVersionBounds)(); diff --git a/src/lib/onboard/docker-gpu-diagnostic-redaction.test.ts b/src/lib/onboard/docker-gpu-diagnostic-redaction.test.ts index 984a8fd360c..3571b8b643e 100644 --- a/src/lib/onboard/docker-gpu-diagnostic-redaction.test.ts +++ b/src/lib/onboard/docker-gpu-diagnostic-redaction.test.ts @@ -7,6 +7,8 @@ import path from "node:path"; import { describe, expect, it, vi } from "vitest"; +import { createDockerGpuDiagnosticRedactor } from "./docker-gpu-diagnostic-redaction"; +import { openshellMainProcessSpecEnvValue } from "./docker-startup-command-env"; import type { OpenShellGpuDiagnostics } from "../adapters/openshell/gpu-diagnostics"; import { buildDockerGpuMode, @@ -238,3 +240,32 @@ describe("Docker GPU diagnostic redaction", () => { } }); }); + +describe("native main-process diagnostic credentials", () => { + it("redacts native argv assignments and the serialized spec from log sinks", () => { + const spec = openshellMainProcessSpecEnvValue( + [ + "env", + "NEMOCLAW_EXTRA_PLACEHOLDER_KEYS=CUSTOM_VALUE", + "CUSTOM_VALUE=opaque-custom value", + "API_KEY=opaque-key-value", + "nemoclaw-start", + ], + false, + ); + const redactor = createDockerGpuDiagnosticRedactor(); + redactor.rememberInspect({ Config: { Env: [`OPENSHELL_MAIN_PROCESS_SPEC=${spec}`] } }); + const output = redactor.redactText(`${spec} opaque-custom value opaque-key-value`); + expect(output).not.toContain(spec); + expect(output).not.toContain("opaque-custom value"); + expect(output).not.toContain("opaque-key-value"); + }); + it("refuses malformed main-process metadata before trusting log redaction", () => { + const redactor = createDockerGpuDiagnosticRedactor(); + expect(() => + redactor.rememberInspect({ + Config: { Env: ["OPENSHELL_MAIN_PROCESS_SPEC=invalid"] }, + }), + ).toThrow(/not valid JSON/); + }); +}); diff --git a/src/lib/onboard/docker-gpu-diagnostic-redaction.ts b/src/lib/onboard/docker-gpu-diagnostic-redaction.ts index cf3a1af5ae0..39029c23260 100644 --- a/src/lib/onboard/docker-gpu-diagnostic-redaction.ts +++ b/src/lib/onboard/docker-gpu-diagnostic-redaction.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { parseOpenShellMainProcessSpecEnvValue } from "./docker-startup-command-env"; import { redact, redactFull } from "../security/redact"; import type { DockerContainerInspect } from "./docker-gpu-patch-types"; @@ -20,8 +21,12 @@ function inspectEnv(inspect: DockerContainerInspect): Map { function startupCommandEnv(inspect: DockerContainerInspect): Map { const assignments = new Map(); - const command = inspectEnv(inspect).get("OPENSHELL_SANDBOX_COMMAND") ?? ""; - const tokens = command.trim().split(/\s+/u); + const env = inspectEnv(inspect); + const spec = env.get("OPENSHELL_MAIN_PROCESS_SPEC"); + const tokens = + spec === undefined + ? (env.get("OPENSHELL_SANDBOX_COMMAND") ?? "").trim().split(/\s+/u) + : [...parseOpenShellMainProcessSpecEnvValue(spec).command]; if (tokens.shift() !== "env") return assignments; for (const token of tokens) { const separator = token.indexOf("="); @@ -63,7 +68,10 @@ export function discoverDockerGpuDiagnosticSensitiveValues( return [...env, ...startupEnv] .filter( ([key, value]) => - (SENSITIVE_ENV_KEY.test(key) || extraPlaceholderKeys.has(key)) && value.length > 0, + (SENSITIVE_ENV_KEY.test(key) || + key === "OPENSHELL_MAIN_PROCESS_SPEC" || + extraPlaceholderKeys.has(key)) && + value.length > 0, ) .map(([, value]) => value); } diff --git a/src/lib/onboard/docker-gpu-local-inference.ts b/src/lib/onboard/docker-gpu-local-inference.ts index 9d41dcd0d89..eff434bf0f8 100644 --- a/src/lib/onboard/docker-gpu-local-inference.ts +++ b/src/lib/onboard/docker-gpu-local-inference.ts @@ -208,7 +208,7 @@ export function resolveGpuSandboxInferenceRoute( const nativeExpected = usesNativeLocalInferenceForAgent(agentName, provider, selection.endpointUrl, localNim) && !isPersistedPiSharedLocalInferenceRoute(entry, selection); - if (localNim || (!nativeExpected && entry.nativeLocalProviderAttachment === undefined)) { + if (!nativeExpected && entry.nativeLocalProviderAttachment === undefined) { return { kind: "shared", gatewayName }; } const attachment = normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment); diff --git a/src/lib/onboard/docker-gpu-patch-clone.test.ts b/src/lib/onboard/docker-gpu-patch-clone.test.ts index eb8fd06a786..6e3af4f66e4 100644 --- a/src/lib/onboard/docker-gpu-patch-clone.test.ts +++ b/src/lib/onboard/docker-gpu-patch-clone.test.ts @@ -84,7 +84,7 @@ describe("Docker GPU clone envelope", () => { expect(args.some((arg) => arg.startsWith("OPENSHELL_SANDBOX_COMMAND="))).toBe(false); }); - it("rejects a malformed OpenShell 0.0.116 main-process transport before recreation", () => { + it("rejects malformed OpenShell main-process metadata before recreation", () => { const inspect = inspectFixture(); inspect.Config!.Env = ['OPENSHELL_MAIN_PROCESS_SPEC={"version":2}']; @@ -92,7 +92,7 @@ describe("Docker GPU clone envelope", () => { buildDockerGpuCloneRunArgs(inspect, buildDockerGpuMode("startup-command"), { openshellSandboxCommand: NEMOCLAW_STARTUP_ARGV, }), - ).toThrow(/does not match the 0\.0\.116 Docker contract/u); + ).toThrow(/does not match the supported Docker contract/u); }); it("omits only OpenShell's OCI-user marker at the exact NemoClaw workspace boundary (#8662)", () => { diff --git a/src/lib/onboard/docker-startup-command-env.test.ts b/src/lib/onboard/docker-startup-command-env.test.ts new file mode 100644 index 00000000000..fbd67f3f5d7 --- /dev/null +++ b/src/lib/onboard/docker-startup-command-env.test.ts @@ -0,0 +1,45 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import { + parseOpenShellMainProcessSpecEnvValue, + replaceOpenShellMainProcessSpecCommand, +} from "./docker-startup-command-env"; + +describe("OpenShell main-process attachment metadata", () => { + it.each([{}, { await_main_process_attachment: false }, { await_main_process_attachment: true }])( + "preserves attachment metadata when replacing the command: %j", + (attachment) => { + const original = { + version: 1, + command: ["env", "TOKEN=opaque-value", "nemoclaw-start"], + tty: false, + ...attachment, + }; + expect(parseOpenShellMainProcessSpecEnvValue(JSON.stringify(original))).toEqual(original); + expect( + JSON.parse( + replaceOpenShellMainProcessSpecCommand(JSON.stringify(original), [ + "replacement", + "argument with spaces", + ]), + ), + ).toEqual({ ...original, command: ["replacement", "argument with spaces"] }); + }, + ); + + it.each([ + { await_main_process_attachment: "false" }, + { await_main_process_attachment: null }, + { await_main_process_attachment: 1 }, + { unexpected: false }, + { await_main_process_attachment: false, unexpected: false }, + ])("rejects unsupported metadata: %j", (extra) => { + expect(() => + parseOpenShellMainProcessSpecEnvValue( + JSON.stringify({ version: 1, command: ["nemoclaw-start"], tty: false, ...extra }), + ), + ).toThrow(/contract/); + }); +}); diff --git a/src/lib/onboard/docker-startup-command-env.ts b/src/lib/onboard/docker-startup-command-env.ts index 1637117e645..997252e831d 100644 --- a/src/lib/onboard/docker-startup-command-env.ts +++ b/src/lib/onboard/docker-startup-command-env.ts @@ -199,6 +199,7 @@ export type OpenShellMainProcessSpec = Readonly<{ version: 1; command: readonly string[]; tty: boolean; + await_main_process_attachment?: boolean; }>; function exactMainProcessCommand(command: unknown): readonly string[] { @@ -222,7 +223,7 @@ function exactMainProcessCommand(command: unknown): readonly string[] { return Object.freeze(result); } -/** Decode the exact Docker-driver main-process transport introduced in OpenShell 0.0.116. */ +/** Decode the Docker main-process transport, including the 0.1.3 attachment flag. */ export function parseOpenShellMainProcessSpecEnvValue(value: string): OpenShellMainProcessSpec { let parsed: unknown; try { @@ -234,21 +235,28 @@ export function parseOpenShellMainProcessSpecEnvValue(value: string): OpenShellM throw new Error("OpenShell main-process spec is not an object."); } const record = parsed as Record; - const keys = Object.keys(record).sort(); + const keys = Object.keys(record) + .filter((key) => key !== "await_main_process_attachment") + .sort(); if ( keys.length !== 3 || keys[0] !== "command" || keys[1] !== "tty" || keys[2] !== "version" || record.version !== OPENSHELL_MAIN_PROCESS_SPEC_VERSION || - typeof record.tty !== "boolean" + typeof record.tty !== "boolean" || + (Object.hasOwn(record, "await_main_process_attachment") && + typeof record.await_main_process_attachment !== "boolean") ) { - throw new Error("OpenShell main-process spec does not match the 0.0.116 Docker contract."); + throw new Error("OpenShell main-process spec does not match the supported Docker contract."); } return Object.freeze({ version: OPENSHELL_MAIN_PROCESS_SPEC_VERSION, command: exactMainProcessCommand(record.command), tty: record.tty, + ...(typeof record.await_main_process_attachment === "boolean" + ? { await_main_process_attachment: record.await_main_process_attachment } + : {}), }); } @@ -265,5 +273,5 @@ export function replaceOpenShellMainProcessSpecCommand( command: readonly string[], ): string { const spec = parseOpenShellMainProcessSpecEnvValue(value); - return openshellMainProcessSpecEnvValue(command, spec.tty); + return JSON.stringify({ ...spec, command: exactMainProcessCommand(command) }); } diff --git a/src/lib/onboard/dockerfile-patch-native-hosted.test.ts b/src/lib/onboard/dockerfile-patch-native-hosted.test.ts index 559668e0afe..120539da91c 100644 --- a/src/lib/onboard/dockerfile-patch-native-hosted.test.ts +++ b/src/lib/onboard/dockerfile-patch-native-hosted.test.ts @@ -25,7 +25,7 @@ describe("Hermes Dockerfile route selection", () => { { nativeProvider: undefined, upstreamEndpointUrl: "https://custom.example/v1", - expected: "https://inference.local/v1", + expected: "https://custom.example/v1", }, { nativeProvider: true, diff --git a/src/lib/onboard/dockerfile-remote-dashboard-bind-contract.ts b/src/lib/onboard/dockerfile-remote-dashboard-bind-contract.ts index 9ea7ff373b2..6c9083b1e25 100644 --- a/src/lib/onboard/dockerfile-remote-dashboard-bind-contract.ts +++ b/src/lib/onboard/dockerfile-remote-dashboard-bind-contract.ts @@ -28,6 +28,11 @@ const EXACT_CUSTOM_POST_GENERATOR_RUN_RE = [CONFIG_MODE_RE, MESSAGING_BUILD_APPL // openclaw.json>` to an otherwise safe command. A lifecycle test verifies // these digests against the checked-in Dockerfile. const CANONICAL_POST_GENERATOR_INSTRUCTION_SHA256 = new Set([ + // Native mediation removes the obsolete proxy config and clears the gateway + // token; this instruction preserves the generated dashboard binding. + "18f3a380f3ea017b820cf07dda4e74b27c1eab8d83c8258b31b0cc7f8f73b98e", + // Tool-search validation generates configs only under its temporary HOME. + "13e59f553ca6e1b01b57c9e38804a49aaeb137cf2c1b4b6c522ebf7dc3a5bc7d", "d0ed2268f42964e8a150deb39e68e31313e520503fa2c131a4d7cc8b21191da0", "6f457f365f5c0d128e5e3b549a630b5bd9ebd223919f2c2c8e6a31235d763781", "dca7d3dbc030e4efa77c850b9d21a826358c69c7d2062f3eee2f5a57eeb07aa2", diff --git a/src/lib/onboard/experimental/hermes-authenticated-health.test.ts b/src/lib/onboard/experimental/hermes-authenticated-health.test.ts index aa71f6140fd..9a7f7e99697 100644 --- a/src/lib/onboard/experimental/hermes-authenticated-health.test.ts +++ b/src/lib/onboard/experimental/hermes-authenticated-health.test.ts @@ -19,7 +19,7 @@ class Health(http.server.BaseHTTPRequestHandler): def do_GET(self): authorized = self.headers.get("Authorization") == "Bearer " + key requests.append({"authorized": authorized, "path": self.path}) - self.send_response(case["status"] if authorized else 401) + self.send_response(404 if self.path != "/health/detailed" else case["status"] if authorized else 401) if case["redirect"]: self.send_header("Location", "http://127.0.0.1:1/stolen") self.end_headers() @@ -94,7 +94,7 @@ describe.each(PROBES)("Hermes authenticated health $name", ({ name, program }) = const output = runProbe(program, `export API_SERVER_KEY=${key}\n`); expect(output.code).toBe(0); expect(output.stdout).toContain(name === "waiter" ? "result=ready" : "200"); - expect(output.requests).toEqual([{ authorized: true, path: "/health" }]); + expect(output.requests).toEqual([{ authorized: true, path: "/health/detailed" }]); expect(output.connections).toEqual([{ host: "127.0.0.1", port: expect.any(Number) }]); }); @@ -102,7 +102,7 @@ describe.each(PROBES)("Hermes authenticated health $name", ({ name, program }) = const output = runProbe(program, `API_SERVER_KEY=${KEY}\n`, 200, true); expect(output.code).toBe(0); expect(output.stdout).toContain(name === "waiter" ? "result=ready" : "200"); - expect(output.requests).toEqual([{ authorized: true, path: "/health" }]); + expect(output.requests).toEqual([{ authorized: true, path: "/health/detailed" }]); expect(output.connections).toHaveLength(1); }); @@ -130,7 +130,7 @@ describe.each(PROBES)("Hermes authenticated health $name", ({ name, program }) = name === "waiter" ? "result=not-ready" : key !== KEY ? "401" : String(status), ); expect(output.requests.length).toBeGreaterThan(0); - expect(output.requests.every((request) => request.path === "/health")).toBe(true); + expect(output.requests.every((request) => request.path === "/health/detailed")).toBe(true); }); }); @@ -144,5 +144,5 @@ it("honors a non-default configured success status", () => { ); expect(output.code).toBe(0); expect(output.stdout).toContain("result=ready"); - expect(output.requests).toEqual([{ authorized: true, path: "/health" }]); + expect(output.requests).toEqual([{ authorized: true, path: "/health/detailed" }]); }); diff --git a/src/lib/onboard/experimental/hermes-portable-build-context-files.ts b/src/lib/onboard/experimental/hermes-portable-build-context-files.ts index 1df4b6f6c72..5423241244d 100644 --- a/src/lib/onboard/experimental/hermes-portable-build-context-files.ts +++ b/src/lib/onboard/experimental/hermes-portable-build-context-files.ts @@ -49,6 +49,7 @@ export const HERMES_PORTABLE_BUILD_CONTEXT_FILES = [ { path: "agents/hermes/runtime-config-guard.py", mode: "100755" }, { path: "agents/hermes/sharp-security-probes.mts", mode: "100644" }, { path: "agents/hermes/start.sh", mode: "100755" }, + { path: "agents/hermes/startup-completion.py", mode: "100755" }, { path: "agents/hermes/validate-cli-adapter.py", mode: "100755" }, { path: "agents/hermes/validate-env-secret-boundary.py", mode: "100755" }, { path: "ci/reviewed-npm-audit.json", mode: "100644" }, @@ -121,12 +122,12 @@ export const HERMES_PORTABLE_BUILD_CONTEXT_FILES = [ { path: "nemoclaw-blueprint/provider-profiles/entra-runtime-v1.yaml", mode: "100644" }, { path: "nemoclaw-blueprint/provider-profiles/langfuse-hermes-v1.yaml", mode: "100644" }, { path: "nemoclaw-blueprint/provider-profiles/nemoclaw-mcp-v1.yaml", mode: "100644" }, + { path: "nemoclaw-blueprint/provider-profiles/nemoclaw-messaging-v1.yaml", mode: "100644" }, { path: "nemoclaw-blueprint/provider-profiles/okta-runtime-v1.yaml", mode: "100644" }, { path: "nemoclaw-blueprint/provider-profiles/tavily-hermes-v1.yaml", mode: "100644" }, { path: "nemoclaw-blueprint/provider-profiles/tavily.yaml", mode: "100644" }, { path: "nemoclaw-blueprint/router/llm-router", mode: "160000" }, { path: "nemoclaw-blueprint/router/pool-config.yaml", mode: "100644" }, - { path: "nemoclaw-blueprint/scripts/http-proxy-fix.js", mode: "100644" }, { path: "nemoclaw-blueprint/scripts/nemotron-inference-fix.js", mode: "100644" }, { path: "nemoclaw-blueprint/scripts/sandbox-safety-net.js", mode: "100644" }, { path: "nemoclaw-blueprint/tsconfig.json", mode: "100644" }, @@ -147,7 +148,7 @@ export const HERMES_PORTABLE_BUILD_CONTEXT_FILES = [ { path: "scripts/security/standalone-security-package-inventory.mts", mode: "100644" }, { path: "scripts/upgrade-bundled-npm.mts", mode: "100755" }, { - path: "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.116.json", + path: "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json", mode: "100644", }, { path: "src/lib/hermes-managed-route.ts", mode: "100644" }, @@ -156,10 +157,10 @@ export const HERMES_PORTABLE_BUILD_CONTEXT_FILES = [ { path: "src/lib/core/endpoint-url-safety.ts", mode: "100644" }, { path: "src/lib/inference/native-nvidia/contract.ts", mode: "100644" }, { path: "src/lib/inference/native-provider/agent-config.ts", mode: "100644" }, - { path: "src/lib/inference/native-provider/contract.ts", mode: "100644" }, - { path: "src/lib/inference/native-provider/hosted.ts", mode: "100644" }, + { path: "src/lib/inference/native-provider/catalog.ts", mode: "100644" }, { path: "src/lib/inference-credential.ts", mode: "100644" }, { path: "src/lib/messaging/AGENTS.md", mode: "100644" }, + { path: "src/lib/messaging/applier/__test-helpers__/provider-fixtures.ts", mode: "100644" }, { path: "src/lib/messaging/applier/agent-config-remove.test.ts", mode: "100644" }, { path: "src/lib/messaging/applier/agent-config.ts", mode: "100644" }, { path: "src/lib/messaging/applier/build/messaging-build-applier.mts", mode: "100755" }, @@ -176,6 +177,8 @@ export const HERMES_PORTABLE_BUILD_CONTEXT_FILES = [ { path: "src/lib/messaging/applier/conflict-detection/slack-socket-mode.ts", mode: "100644" }, { path: "src/lib/messaging/applier/conflict-detection/types.ts", mode: "100644" }, { path: "src/lib/messaging/applier/credential-env-cleanup.ts", mode: "100644" }, + { path: "src/lib/messaging/applier/credential-family-profile.test.ts", mode: "100644" }, + { path: "src/lib/messaging/applier/credential-family-profile.ts", mode: "100644" }, { path: "src/lib/messaging/applier/hook-phases.test.ts", mode: "100644" }, { path: "src/lib/messaging/applier/hook-phases.ts", mode: "100644" }, { path: "src/lib/messaging/applier/host-state-applier.test.ts", mode: "100644" }, @@ -242,14 +245,6 @@ export const HERMES_PORTABLE_BUILD_CONTEXT_FILES = [ path: "src/lib/messaging/channels/googlechat/runtime/googlechat-outbound-auth.ts", mode: "100644", }, - { - path: "src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.test.ts", - mode: "100644", - }, - { - path: "src/lib/messaging/channels/googlechat/runtime/googlechat-trusted-proxy-fetch.ts", - mode: "100644", - }, { path: "src/lib/messaging/channels/googlechat/runtime/hermes-adapter.py", mode: "100644" }, { path: "src/lib/messaging/channels/googlechat/runtime/hermes-adapter.test.ts", mode: "100644" }, { path: "src/lib/messaging/channels/googlechat/template-resolver.test.ts", mode: "100644" }, diff --git a/src/lib/onboard/experimental/hermes-portable-build-context.test.ts b/src/lib/onboard/experimental/hermes-portable-build-context.test.ts index dad20ba8639..4a959b612b2 100644 --- a/src/lib/onboard/experimental/hermes-portable-build-context.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-build-context.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { spawnSync } from "node:child_process"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -126,6 +127,27 @@ describe("Hermes portable staged build context", testTimeoutOptions(30_000), () expect( fs.statSync(path.join(inferredContext, "agents/hermes/check-native-credentials.ts")).isFile(), ).toBe(true); + fs.mkdirSync(path.join(stateDir, ".hermes")); + const generated = spawnSync( + process.execPath, + [path.join(inferredContext, "agents/hermes/generate-config.ts")], + { + cwd: inferredContext, + env: { + HOME: stateDir, + PATH: process.env.PATH, + NEMOCLAW_MODEL: "synthetic-model", + NEMOCLAW_UPSTREAM_PROVIDER: "openai-api", + NEMOCLAW_INFERENCE_BASE_URL: "https://api.openai.com/v1", + NEMOCLAW_INFERENCE_API: "openai-completions", + }, + encoding: "utf8", + }, + ); + expect(generated.status, generated.stderr).toBe(0); + expect(fs.readFileSync(path.join(stateDir, ".hermes/config.yaml"), "utf8")).toContain( + "https://api.openai.com/v1", + ); expect(fs.existsSync(path.join(inferredContext, "agents/hermes/Dockerfile"))).toBe(false); expect(plan.authority.sourceRevision).toMatch(/^[a-f0-9]{40,64}$/u); expect(plan.authority.contextManifestSha256).toMatch(/^[a-f0-9]{64}$/u); diff --git a/src/lib/onboard/experimental/hermes-portable-build-context.ts b/src/lib/onboard/experimental/hermes-portable-build-context.ts index 20914050033..91b0dbb78dd 100644 --- a/src/lib/onboard/experimental/hermes-portable-build-context.ts +++ b/src/lib/onboard/experimental/hermes-portable-build-context.ts @@ -80,6 +80,7 @@ const LOCAL_COPY_SOURCES = [ "agents/hermes/runtime-config-guard.py", "agents/hermes/sharp-security-probes.mts", "agents/hermes/start.sh", + "agents/hermes/startup-completion.py", "agents/hermes/validate-cli-adapter.py", "agents/hermes/validate-env-secret-boundary.py", "nemoclaw-blueprint/", @@ -101,15 +102,14 @@ const LOCAL_COPY_SOURCES = [ "scripts/security/standalone-security-package-inventory.mts", "scripts/upgrade-bundled-npm.mts", "ci/reviewed-npm-audit.json", - "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.116.json", + "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json", "src/lib/hermes-managed-route.ts", "src/lib/hermes-switchyard-routing.ts", "src/lib/inference/native-local/agent-config.ts", "src/lib/core/endpoint-url-safety.ts", "src/lib/inference/native-nvidia/contract.ts", "src/lib/inference/native-provider/agent-config.ts", - "src/lib/inference/native-provider/contract.ts", - "src/lib/inference/native-provider/hosted.ts", + "src/lib/inference/native-provider/catalog.ts", "src/lib/inference-credential.ts", "src/lib/messaging/", "src/lib/messaging/channels/googlechat/runtime/hermes-adapter.py", diff --git a/src/lib/onboard/experimental/hermes-portable-container.test.ts b/src/lib/onboard/experimental/hermes-portable-container.test.ts index 136af5bf313..d0b1d7775a9 100644 --- a/src/lib/onboard/experimental/hermes-portable-container.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-container.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { randomUUID } from "node:crypto"; +import { spawnSync } from "node:child_process"; import { describe, expect, it, vi } from "vitest"; @@ -113,7 +114,7 @@ function receipt(): HermesPortablePendingReceipt { function inspect( restartPolicy = "no", - labels = LABELS, + labels: Record = LABELS, running = true, status = running ? "running" : "exited", paused = false, @@ -155,6 +156,96 @@ function activeReceipt(running = true): HermesPortableConfiguredReceipt { } describe("Hermes portable container authority", () => { + it.each([ + ["200", "matching", "a"], + ["401", "incorrect", "b"], + ])( + "returns %s for the %s API key at the protected health endpoint", + (status, _case, serverKey) => { + const result = spawnSync( + "python3", + [ + "-I", + "-c", + String.raw` +import http.client, json, pathlib, sys +from unittest.mock import patch + +inputs = json.load(sys.stdin) + +class Connection: + def __init__(self, host, port, timeout): + assert host == "127.0.0.1" and port == 8642 and timeout == 5 + def request(self, method, target, headers): + assert method == "GET" + # The pinned Hermes API keeps /health public and protects /health/detailed. + if target == "/health": + self.status = 200 + elif target == "/health/detailed": + self.status = 200 if headers.get("Authorization") == "Bearer " + inputs["serverKey"] else 401 + else: + self.status = 404 + def getresponse(self): + return self + def close(self): + pass + +with patch.object(pathlib.Path, "read_text", return_value="API_SERVER_KEY=" + "a" * 64): + with patch.object(http.client, "HTTPConnection", Connection): + exec(inputs["program"], {}) +`, + ], + { + encoding: "utf8", + input: JSON.stringify({ + program: hermesPortableContainerInternals.authenticatedHealthScript, + serverKey: serverKey.repeat(64), + }), + timeout: 10_000, + }, + ); + expect({ + status: result.status, + stdout: result.stdout.trim(), + stderr: result.stderr, + }).toEqual({ + status: 0, + stdout: status, + stderr: "", + }); + }, + ); + + it("enrolls a labelled workload while excluding its supervisor", () => { + const workloadLabels = { ...LABELS, "openshell.ai/isolation-role": "sandbox" }; + const podman = vi + .fn() + .mockReturnValueOnce({ status: 0, stdout: ID + "\n", stderr: "" }) + .mockReturnValueOnce(inspect("no", workloadLabels)); + const enrolled = enrollHermesPortableContainer(receipt(), SANDBOX_ID, { + podman, + assertSocketAuthority: vi.fn(), + }); + expect(enrolled.authority.containerId).toBe(ID); + expect(enrolled.labels["openshell.ai/isolation-role"]).toBe("sandbox"); + expect(podman.mock.calls[0]?.[0]).toContain("label!=openshell.ai/isolation-role=supervisor"); + expect(podman).toHaveBeenCalledTimes(2); + }); + + it.each(["supervisor", "unknown", ""])("rejects the non-workload role %s", (role) => { + const podman = vi + .fn() + .mockReturnValueOnce({ status: 0, stdout: ID + "\n", stderr: "" }) + .mockReturnValueOnce(inspect("no", { ...LABELS, "openshell.ai/isolation-role": role })); + expect(() => + enrollHermesPortableContainer(receipt(), SANDBOX_ID, { + podman, + assertSocketAuthority: vi.fn(), + }), + ).toThrow("isolation role does not identify a sandbox workload"); + expect(podman).toHaveBeenCalledTimes(2); + }); + it("enrolls exactly one running full-ID container with exact OpenShell labels (#9203)", () => { const podman = vi .fn() @@ -178,6 +269,7 @@ describe("Hermes portable container authority", () => { hermesPortableContainerInternals.labelsDigest(LABELS), ); expect(assertSocketAuthority).toHaveBeenCalledTimes(4); + expect(podman.mock.calls[0]?.[0]).toContain("label!=openshell.ai/isolation-role=supervisor"); }); it.each([ diff --git a/src/lib/onboard/experimental/hermes-portable-container.ts b/src/lib/onboard/experimental/hermes-portable-container.ts index f15a69caa28..eda6475f7d9 100644 --- a/src/lib/onboard/experimental/hermes-portable-container.ts +++ b/src/lib/onboard/experimental/hermes-portable-container.ts @@ -6,6 +6,7 @@ import { isDeepStrictEqual } from "node:util"; import { assertPodmanSocketAuthority, type PodmanSocketAuthorityDeps } from "../../adapters/podman"; import { + PODMAN_ISOLATION_ROLE_LABEL, PODMAN_MANAGED_LABEL, PODMAN_SANDBOX_CONTAINER_PREFIX, PODMAN_SANDBOX_ID_LABEL, @@ -64,7 +65,7 @@ if re.fullmatch(r"[0-9a-f]{64}", key) is None: def authenticated_health_status(port, timeout): connection = http.client.HTTPConnection("127.0.0.1", port, timeout=timeout) try: - connection.request("GET", "/health", headers={"Authorization": "Bearer " + key}) + connection.request("GET", "/health/detailed", headers={"Authorization": "Bearer " + key}) response = connection.getresponse() try: return response.status @@ -308,6 +309,7 @@ function parseInspection( readonly sandboxName: string; readonly sandboxId: string; readonly containerId: string; + readonly role?: "sandbox" | "supervisor"; }, ): HermesPortableContainerInspection { let decoded: unknown; @@ -323,6 +325,13 @@ function parseInspection( if (containerId !== expected.containerId) fail("inspect returned another container ID"); const config = record(row.Config, "inspect Config"); const containerLabels = labels(config.Labels); + const isolationRole = containerLabels[PODMAN_ISOLATION_ROLE_LABEL]; + const role = expected.role ?? "sandbox"; + if (isolationRole !== role && !(role === "sandbox" && isolationRole === undefined)) { + fail( + `inspect isolation role does not identify a ${role === "sandbox" ? "sandbox workload" : "supervisor"}`, + ); + } const required = { [PODMAN_MANAGED_LABEL]: "true", [PODMAN_SANDBOX_ID_LABEL]: expected.sandboxId, @@ -333,7 +342,10 @@ function parseInspection( for (const [key, value] of Object.entries(required)) { if (containerLabels[key] !== value) fail(`inspect label '${key}' disagrees with OpenShell`); } - const expectedName = `${PODMAN_SANDBOX_CONTAINER_PREFIX}${expected.sandboxName}-${expected.sandboxId}`; + const expectedName = + role === "supervisor" + ? `openshell-supervisor-${expected.sandboxId}` + : `${PODMAN_SANDBOX_CONTAINER_PREFIX}${expected.sandboxName}-${expected.sandboxId}`; const name = text(row.Name, "container name"); if (name !== expectedName) fail("inspect container name disagrees with OpenShell identity"); const state = record(row.State, "inspect State"); @@ -376,6 +388,7 @@ function inspectExact( sandboxId: string, containerId: string, deps: HermesPortableContainerDeps, + role: "sandbox" | "supervisor" = "sandbox", ): HermesPortableContainerInspection { const measure = deps.inspectionTiming?.measure ?? ((_stage, operation) => operation()); measure("preGuard", () => assertSocket(receipt, deps)); @@ -391,10 +404,61 @@ function inspectExact( sandboxName: receipt.sandboxName, sandboxId, containerId, + role, }), ); } +/** Pin the companion before stop; never accept workload-only exit for an isolated pair. */ +export function prepareHermesPortableSupervisorStopCheck( + receipt: HermesPortableConfiguredReceipt, + deps: HermesPortableContainerDeps, + workload: HermesPortableContainerInspection, +): () => boolean { + if (workload.labels[PODMAN_ISOLATION_ROLE_LABEL] !== "sandbox") return () => true; + assertSocket(receipt, deps); + const output = requireCommand( + deps.podman( + [ + "ps", + "--all", + "--no-trunc", + "--filter", + `label=${PODMAN_MANAGED_LABEL}=true`, + "--filter", + `label=${PODMAN_SANDBOX_NAME_LABEL}=${receipt.sandboxName}`, + "--filter", + `label=${PODMAN_ISOLATION_ROLE_LABEL}=supervisor`, + "--format", + "{{.ID}}", + ], + INSPECT_TIMEOUT_MS, + ), + "supervisor lookup", + ); + assertSocket(receipt, deps); + const ids = output + .split(/\r?\n/u) + .map((line) => line.trim()) + .filter(Boolean); + if (ids.length !== 1 || !FULL_ID.test(ids[0]!)) { + fail("stop requires exactly one full supervisor container ID"); + } + const inspect = () => + inspectExact(receipt, receipt.container.sandboxId, ids[0]!, deps, "supervisor"); + const initial = inspect(); + return () => { + const current = inspect(); + if ( + current.authority.imageId !== initial.authority.imageId || + current.authority.labelsSha256 !== initial.authority.labelsSha256 + ) { + fail("supervisor immutable identity changed during stop"); + } + return !current.authority.running && !current.paused && current.status === "exited"; + }; +} + /** Enroll exactly one live OpenShell-managed container after Ready. */ export function enrollHermesPortableContainer( receipt: HermesPortableLifecycleReceipt, @@ -414,6 +478,8 @@ export function enrollHermesPortableContainer( `label=${PODMAN_SANDBOX_NAME_LABEL}=${receipt.sandboxName}`, "--filter", `label=${PODMAN_SANDBOX_WORKSPACE_LABEL}=${PODMAN_SANDBOX_WORKSPACE}`, + "--filter", + `label!=${PODMAN_ISOLATION_ROLE_LABEL}=supervisor`, "--format", "{{.ID}}", ], diff --git a/src/lib/onboard/experimental/hermes-portable-lifecycle-error.test-fixture.ts b/src/lib/onboard/experimental/hermes-portable-lifecycle-error.test-fixture.ts new file mode 100644 index 00000000000..ab10176f326 --- /dev/null +++ b/src/lib/onboard/experimental/hermes-portable-lifecycle-error.test-fixture.ts @@ -0,0 +1,25 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { vi } from "vitest"; +import type { createHermesPortableLifecycleTestDeps } from "./hermes-portable-lifecycle.test-fixture"; + +/** Add the pinned server's Error details without changing later lifecycle observations. */ +export function withOpenShellErrorFields( + capture: ReturnType["captureOpenShell"], + fields: Record, +) { + return vi.fn((args: readonly string[]) => { + const result = capture(args); + if (args[0] !== "sandbox" || args[1] !== "list") return result; + const rows = JSON.parse(result.stdout) as Record[]; + return { + ...result, + stdout: JSON.stringify( + rows.map((row) => + row.phase === "Error" ? { ...row, workspace: "default", ...fields } : row, + ), + ), + }; + }); +} diff --git a/src/lib/onboard/experimental/hermes-portable-lifecycle-error.test.ts b/src/lib/onboard/experimental/hermes-portable-lifecycle-error.test.ts new file mode 100644 index 00000000000..18602999009 --- /dev/null +++ b/src/lib/onboard/experimental/hermes-portable-lifecycle-error.test.ts @@ -0,0 +1,174 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { loadAgent } from "../../agent/defs"; +import { withMcpLifecycleLock } from "../../state/mcp-lifecycle-lock"; +import { + recoverHermesPortableSandboxLifecycle, + type HermesPortableLifecycleDeps, +} from "./hermes-portable-lifecycle"; +import { + createHermesPortableLifecycleTestReceipt, + createHermesPortableLifecycleTestDeps, + SANDBOX, + GATEWAY, + GENERATION, + CONTAINER_ID, + IMAGE, + SANDBOX_ID, + POLICY, + LABELS, +} from "./hermes-portable-lifecycle.test-fixture"; +import { openshellMutationCalls } from "./hermes-portable-lifecycle.test-fixtures"; +import { withOpenShellErrorFields } from "./hermes-portable-lifecycle-error.test-fixture"; + +let stateDir: string; +let policyPath: string; +beforeEach(() => { + stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-error-")); + policyPath = path.join(stateDir, "policy.yaml"); + fs.writeFileSync(policyPath, POLICY, { mode: 0o600 }); +}); +afterEach(() => { + vi.restoreAllMocks(); + fs.rmSync(stateDir, { recursive: true, force: true }); +}); +function fixture(running: boolean, startStatus = 0) { + const receipt = createHermesPortableLifecycleTestReceipt({ + agent: loadAgent("hermes"), + stateDir, + policyPath, + homeDir: "/home/test", + sandboxName: SANDBOX, + gatewayName: GATEWAY, + lifecycleGeneration: GENERATION, + containerId: CONTAINER_ID, + imageDigest: IMAGE, + sandboxId: SANDBOX_ID, + labels: LABELS, + }); + return createHermesPortableLifecycleTestDeps(stateDir, receipt, running, { + initialPhase: "Error", + startStatus, + }); +} +function recover(deps: HermesPortableLifecycleDeps) { + return withMcpLifecycleLock( + SANDBOX, + () => + recoverHermesPortableSandboxLifecycle( + SANDBOX, + { + agent: "hermes", + gatewayName: GATEWAY, + lifecycleGeneration: GENERATION, + openshellDriver: "docker", + provider: "ollama", + }, + deps, + ), + { stateDir: path.join(stateDir, "state") }, + ); +} +const restartable = [ + { + name: "failed main process", + fields: { + exit_code: 7, + conditions: [{ type: "Ready", status: "False", reason: "MainProcessFailed" }], + }, + }, + { + name: "reclaimed provisioning timeout", + fields: { + provisioning: { + timeout_time: "2026-10-09T01:00:00Z", + cleanup_completed_time: "2026-10-09T01:00:01Z", + }, + }, + }, +].flatMap((scenario) => [false, true].map((running) => ({ ...scenario, running }))); + +describe("Hermes OpenShell error recovery", () => { + it.each(restartable)( + "recovers $name with running=$running through native start", + async ({ fields, running }) => { + const f = fixture(running); + const command = withOpenShellErrorFields(f.captureOpenShell, fields); + await expect(recover({ ...f.deps, captureOpenShell: command })).resolves.toEqual({ + kind: "recovered", + }); + expect(openshellMutationCalls(command, "start")).toHaveLength(1); + expect(openshellMutationCalls(command, "stop")).toHaveLength(0); + expect(f.deps.container.podman(["container", "inspect"])).toEqual( + expect.objectContaining({ stdout: expect.stringContaining('"Running":true') }), + ); + expect(f.podman.mock.calls.filter(([args]) => args[1] === "start")).toHaveLength(0); + }, + ); + it.each(restartable)( + "rolls back only a new start after $name fails with running=$running", + async ({ fields, running }) => { + const f = fixture(running, 1); + const command = withOpenShellErrorFields(f.captureOpenShell, fields); + await expect(recover({ ...f.deps, captureOpenShell: command })).rejects.toThrow( + "OpenShell start failed with status 1", + ); + expect(f.launchOpenShell).not.toHaveBeenCalled(); + expect(openshellMutationCalls(command, "start")).toHaveLength(1); + expect(openshellMutationCalls(command, "stop")).toHaveLength(running ? 0 : 1); + expect(f.deps.container.podman(["container", "inspect"])).toEqual( + expect.objectContaining({ stdout: expect.stringContaining(`"Running":${running}`) }), + ); + expect(f.podman.mock.calls.filter(([args]) => args[1] === "start")).toHaveLength(0); + }, + ); + it.each([ + { + name: "missing exit code", + fields: { conditions: [{ type: "Ready", status: "False", reason: "MainProcessFailed" }] }, + }, + { + name: "infrastructure failure", + fields: { + exit_code: 7, + conditions: [{ type: "Ready", status: "False", reason: "InfrastructureFailed" }], + }, + }, + { + name: "pending cleanup", + fields: { + exit_code: 7, + conditions: [{ type: "Ready", status: "False", reason: "MainProcessFailed" }], + provisioning: { timeout_time: "2026-10-09T01:00:00Z", cleanup_completed_time: null }, + }, + }, + { + name: "invalid cleanup timestamp", + fields: { + provisioning: { timeout_time: "2026-10-09T01:00:00Z", cleanup_completed_time: "invalid" }, + }, + }, + { + name: "another workspace", + fields: { + workspace: "another", + exit_code: 7, + conditions: [{ type: "Ready", status: "False", reason: "MainProcessFailed" }], + }, + }, + ])("preserves an Error workload with $name without mutation", async ({ fields }) => { + const f = fixture(false); + const command = withOpenShellErrorFields(f.captureOpenShell, fields); + await expect(recover({ ...f.deps, captureOpenShell: command })).rejects.toThrow( + "saved OpenShell phase Error", + ); + expect(openshellMutationCalls(command, "start")).toHaveLength(0); + expect(openshellMutationCalls(command, "stop")).toHaveLength(0); + expect(f.launchOpenShell).not.toHaveBeenCalled(); + }); +}); diff --git a/src/lib/onboard/experimental/hermes-portable-lifecycle.test-fixture.ts b/src/lib/onboard/experimental/hermes-portable-lifecycle.test-fixture.ts index a1f491ca7b2..632c777ea80 100644 --- a/src/lib/onboard/experimental/hermes-portable-lifecycle.test-fixture.ts +++ b/src/lib/onboard/experimental/hermes-portable-lifecycle.test-fixture.ts @@ -32,7 +32,7 @@ const PODMAN_BYTES = Buffer.from("podman-5.7.0-test", "utf8"); export function testOpenShellExecutableAuthority(): HermesPortableOpenShellExecutableAuthority { return { - version: "0.0.116", + version: "0.1.7", executable: { executablePath: "/usr/bin/openshell", device: "1", @@ -247,6 +247,7 @@ export function createHermesPortableLifecycleTestDeps( readonly startStatus?: number; readonly initialPhase?: "Ready" | "Error" | "Stopped"; readonly nonRunningStatus?: string; + readonly labels?: Readonly>; } = {}, ) { let running = initiallyRunning, @@ -270,7 +271,7 @@ export function createHermesPortableLifecycleTestDeps( Id: CONTAINER_ID, Image: IMAGE, Name: receipt.container.name, - Config: { Labels: LABELS }, + Config: { Labels: options.labels ?? LABELS }, State: { Running: running, Paused: false, @@ -373,7 +374,7 @@ export function createHermesPortableLifecycleTestDeps( gatewayName: GATEWAY, lifecycleGeneration: GENERATION, lifecycleLiveIdentityFingerprint: liveIdentityFingerprint, - openshellVersion: "0.0.116", + openshellVersion: "0.1.7", ...options.registry, }) as SandboxEntry, captureOpenShell, diff --git a/src/lib/onboard/experimental/hermes-portable-lifecycle.ts b/src/lib/onboard/experimental/hermes-portable-lifecycle.ts index e035565445b..618547777d8 100644 --- a/src/lib/onboard/experimental/hermes-portable-lifecycle.ts +++ b/src/lib/onboard/experimental/hermes-portable-lifecycle.ts @@ -12,6 +12,7 @@ import { fingerprintOpenShellSandboxId, fingerprintOpenShellSandboxLiveIdentity, observeOpenShellSandboxId, + parseStrictOpenShellSandboxListJson, } from "../../adapters/openshell/sandbox-identity"; import { classifyOpenShellSandboxPresence, @@ -30,6 +31,7 @@ import type { SandboxEntry } from "../../state/registry/types"; import { SANITIZED_PRIVILEGED_ENV } from "../runtime-provider/privileged-sandbox-environment"; import { PinnedSandboxResourceIdentityChangedError } from "../runtime-provider/privileged-sandbox-control-errors"; import { + PODMAN_ISOLATION_ROLE_LABEL, PODMAN_MANAGED_LABEL, PODMAN_SANDBOX_NAME_LABEL, PODMAN_SANDBOX_WORKSPACE, @@ -41,6 +43,7 @@ import { assertCurrentHermesPortableContainer, createHermesPortableContainerInspectionTiming, observeHermesPortableAuthenticatedHealth, + prepareHermesPortableSupervisorStopCheck, type HermesPortableContainerDeps, type HermesPortableContainerInspection, type HermesPortablePodmanResult, @@ -624,6 +627,7 @@ interface QualifiedHermesPortableLifecycle { readonly capturePolicy: HermesPortablePolicyCapture; readonly rawCapture: NonNullable; readonly openShellPhase: string; + readonly openShellErrorRestartable: boolean; readonly commandBudget?: (maximumMs: number) => number; readonly hasTransactionAuthority: boolean; readonly assertTransactionCurrent: () => void; @@ -882,6 +886,7 @@ function observeOpenShellIdentity( readonly sandboxId: string; readonly liveIdentityFingerprint: string; readonly phase: string; + readonly errorRestartable: boolean; } { const gateway = capture( ["sandbox", "list", "-g", receipt.gatewayName, "-o", "json"], @@ -922,7 +927,51 @@ function observeOpenShellIdentity( ) { fail("OpenShell sandbox identity disagrees with the receipt container"); } - return { sandboxId, liveIdentityFingerprint, phase: listed.phase }; + const row = parseStrictOpenShellSandboxListJson(String(gateway.stdout))?.find( + (candidate) => candidate.id === sandboxId && candidate.name === receipt.sandboxName, + ); + return { + sandboxId, + liveIdentityFingerprint, + phase: listed.phase, + errorRestartable: isRestartableOpenShellError(row), + }; +} + +function isRestartableOpenShellError(value: unknown): boolean { + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const row = value as Record; + if (row.phase !== "Error" || row.workspace !== PODMAN_SANDBOX_WORKSPACE) return false; + const provisioning = row.provisioning; + if (provisioning && typeof provisioning === "object" && !Array.isArray(provisioning)) { + const record = provisioning as Record; + if (record.timeout_time != null) { + return ( + typeof record.timeout_time === "string" && + Number.isFinite(Date.parse(record.timeout_time)) && + typeof record.cleanup_completed_time === "string" && + Number.isFinite(Date.parse(record.cleanup_completed_time)) + ); + } + } + // Match the pinned gateway's failed-main-process predicate; it rechecks under its lifecycle lock. + return ( + typeof row.exit_code === "number" && + Number.isInteger(row.exit_code) && + Array.isArray(row.conditions) && + row.conditions.some( + (condition: unknown) => + !!condition && + typeof condition === "object" && + "type" in condition && + condition.type === "Ready" && + "status" in condition && + typeof condition.status === "string" && + condition.status.toLowerCase() === "false" && + "reason" in condition && + condition.reason === "MainProcessFailed", + ) + ); } function policyCapture( @@ -1119,6 +1168,7 @@ async function qualify( rawCapture, capturePolicy, openShellPhase: liveIdentity.phase, + openShellErrorRestartable: liveIdentity.errorRestartable, commandBudget: options.commandBudget, hasTransactionAuthority, assertTransactionCurrent, @@ -1306,7 +1356,15 @@ async function rollbackStartedHermesPortableRecovery( qualified.assertTransactionCurrent(); } failureClass = "container-stop-settlement"; - if (qualified.openShellPhase === "Stopped") { + const supervisorStopped = prepareHermesPortableSupervisorStopCheck( + qualified.receipt, + qualified.containerDeps, + qualified.container, + ); + if ( + qualified.openShellPhase === "Stopped" && + qualified.container.labels[PODMAN_ISOLATION_ROLE_LABEL] !== "sandbox" + ) { armOpenShellV0116StopAssist(qualified); } captureRetainedLifecycleCommand( @@ -1316,7 +1374,14 @@ async function rollbackStartedHermesPortableRecovery( OPENSHELL_LIFECYCLE_MUTATION_TIMEOUT_MS, ); failureClass = "openshell-terminal-settlement"; - await settleStoppedHermesPortableLifecycle(sandboxName, context, deps, qualified, timing); + await settleStoppedHermesPortableLifecycle( + sandboxName, + context, + deps, + qualified, + timing, + supervisorStopped, + ); } catch (error) { throw new HermesPortableRollbackAttemptError(failureClass, error); } @@ -1327,7 +1392,8 @@ async function settleStoppedHermesPortableLifecycle( context: PortableDemoLifecycleContext, deps: HermesPortableLifecycleDeps, authority: QualifiedHermesPortableLifecycle, - timing?: HermesPortableLifecycleTimingRecorder, + timing: HermesPortableLifecycleTimingRecorder | undefined, + supervisorStopped: () => boolean, ): Promise { let stopped: QualifiedHermesPortableLifecycle | null = null; const settled = await waitFor(STOP_SETTLEMENT_TIMEOUT_MS, deps, async () => { @@ -1350,7 +1416,13 @@ async function settleStoppedHermesPortableLifecycle( if (container.authority.running || container.status !== "exited") return false; return false; } - if (current.container.authority.running || current.container.status !== "exited") return false; + const companionExited = supervisorStopped(); + if ( + current.container.authority.running || + current.container.status !== "exited" || + !companionExited + ) + return false; if (current.openShellPhase === "Error" || current.openShellPhase === "Stopped") { stopped = current; return true; @@ -1644,18 +1716,19 @@ export async function recoverHermesPortableSandboxLifecycle( } primaryFailureClass = "container-start"; const wasRunning = qualified.container.authority.running; - const needsStart = !wasRunning || qualified.openShellPhase === "Stopped"; + const needsStart = + !wasRunning || qualified.openShellPhase === "Stopped" || qualified.openShellErrorRestartable; timing.setContainerAction(wasRunning ? "reused" : "unknown"); let rollbackAuthority = qualified; let startedByRecovery = false; try { if ( - qualified.openShellPhase === "Error" || + (qualified.openShellPhase === "Error" && !qualified.openShellErrorRestartable) || (!wasRunning && qualified.openShellPhase === "Ready") ) { fail( `cannot recover saved OpenShell phase ${qualified.openShellPhase} with receipt-owned container ${qualified.container.status}. ` + - "The current OpenShell start API requires Stopped and cannot repair this mismatch. " + + "No supported OpenShell start transition was established for this state. " + "Preserve startup diagnostics and resolve the saved state through OpenShell before retrying launch.", ); } @@ -2356,6 +2429,11 @@ export async function stopHermesPortableSandboxLifecycle( "Error", "Stopped", ]); + const supervisorStopped = prepareHermesPortableSupervisorStopCheck( + qualified.receipt, + qualified.containerDeps, + qualified.container, + ); if (!qualified.container.authority.running && qualified.container.status === "exited") { if (qualified.openShellPhase === "Ready") { qualified.capture( @@ -2363,13 +2441,21 @@ export async function stopHermesPortableSandboxLifecycle( OPENSHELL_LIFECYCLE_MUTATION_TIMEOUT_MS, ); } - await settleStoppedHermesPortableLifecycle(sandboxName, context, deps, qualified); + await settleStoppedHermesPortableLifecycle( + sandboxName, + context, + deps, + qualified, + undefined, + supervisorStopped, + ); return { kind: "already-stopped" }; } if (qualified.container.authority.running && qualified.openShellPhase === "Stopped") { fail("OpenShell Stopped phase disagrees with the running receipt container"); } if (qualified.container.authority.running) beforeStop(); + supervisorStopped(); qualified = await qualify(sandboxName, context, deps, qualified.snapshot, [ "Ready", "Error", @@ -2378,7 +2464,10 @@ export async function stopHermesPortableSandboxLifecycle( if (qualified.container.authority.running && qualified.openShellPhase === "Stopped") { fail("OpenShell Stopped phase disagrees with the running receipt container"); } - if (qualified.openShellPhase === "Ready") { + if ( + qualified.openShellPhase === "Ready" && + qualified.container.labels[PODMAN_ISOLATION_ROLE_LABEL] !== "sandbox" + ) { // OpenShell 0.0.116's rootless Podman supervisor drops CAP_KILL, so its // SIGTERM cannot reach the sandbox-UID workload (OpenShell #3036). Arm one // same-UID, exact-entrypoint signal before the authenticated stop request; @@ -2389,7 +2478,14 @@ export async function stopHermesPortableSandboxLifecycle( openshellLifecycleMutationArgs(qualified.receipt, "stop"), OPENSHELL_LIFECYCLE_MUTATION_TIMEOUT_MS, ); - await settleStoppedHermesPortableLifecycle(sandboxName, context, deps, qualified); + await settleStoppedHermesPortableLifecycle( + sandboxName, + context, + deps, + qualified, + undefined, + supervisorStopped, + ); return { kind: "stopped" }; } diff --git a/src/lib/onboard/experimental/hermes-portable-ollama-gateway-transaction.ts b/src/lib/onboard/experimental/hermes-portable-ollama-gateway-transaction.ts index 756cbece162..e1151b14c2f 100644 --- a/src/lib/onboard/experimental/hermes-portable-ollama-gateway-transaction.ts +++ b/src/lib/onboard/experimental/hermes-portable-ollama-gateway-transaction.ts @@ -4,6 +4,18 @@ import { createHash, randomUUID } from "node:crypto"; import fs from "node:fs"; import path from "node:path"; +import os from "node:os"; +import { + prepareNativeLocalProfile, + type NativeLocalProviderAttachment, +} from "../../inference/native-local/profile"; +import { NATIVE_LOCAL_CREDENTIAL_ENV } from "../../inference/native-local/agent-config"; +import { ensureNativeProvider } from "../../inference/native-provider/lifecycle"; +import { requireSandboxPolicyAuthority } from "../../adapters/openshell/provider-policy"; +import { + parseCheckedInProviderProfileContract, + exportedProviderProfileMatchesContract, +} from "../../adapters/openshell/provider-profile"; import { isDeepStrictEqual } from "node:util"; import { createManagedProviderAdapter } from "../../adapters/openshell/managed-provider-adapter"; @@ -279,7 +291,7 @@ type GatewayProviderJournalIntent = Readonly<{ sandboxName: string; provider: "ollama-local"; model: string; - type: "openai"; + type: string; credentialEnv: string; providerCredentialEnv: string; baseUrl: "http://host.openshell.internal:11434/v1"; @@ -298,6 +310,76 @@ type GatewayProviderJournalRecoveryScope = Omit< "transactionId" | "providerCredentialEnv" >; +function portableNativeBinding( + intent: Pick< + GatewayProviderJournalIntent, + "gatewayName" | "sandboxName" | "transactionId" | "baseUrl" + >, +): Parameters[0] { + return { + provider: "ollama-local", + agent: "hermes", + endpointUrl: intent.baseUrl, + credentialEnv: NATIVE_LOCAL_CREDENTIAL_ENV, + authMode: "sentinel", + gatewayName: intent.gatewayName, + sandboxName: intent.sandboxName, + transactionId: intent.transactionId, + }; +} + +function journalIntent( + scope: GatewayProviderJournalRecoveryScope & + Partial>, + transactionId: string, + native: boolean, +): GatewayProviderJournalIntent { + const binding = portableNativeBinding({ ...scope, transactionId }); + return Object.freeze({ + transactionId, + targetSha256: scope.targetSha256, + gatewayName: scope.gatewayName, + sandboxName: scope.sandboxName, + provider: scope.provider, + model: scope.model, + type: native ? prepareNativeLocalProfile(binding).profileId : "openai", + credentialEnv: scope.credentialEnv, + providerCredentialEnv: native + ? NATIVE_LOCAL_CREDENTIAL_ENV + : `${scope.credentialEnv}_${transactionId.toUpperCase()}`, + baseUrl: scope.baseUrl, + }); +} + +/** Read only the format discriminator; the journal parser proves every field before use. */ +function retainedJournalIsNative(directory: string): boolean { + const serialized = readPortableGatewayState(() => + openPrivateStateFile( + directory, + GATEWAY_PROVIDER_JOURNAL_FILE, + "gateway provider journal", + ).readExact(), + ); + if (serialized === null) return true; + return readPortableGatewayState(() => { + let parsed: unknown; + try { + parsed = JSON.parse(serialized); + } catch { + throw new Error("Hermes Portable inference gateway provider journal is malformed."); + } + const record = requireRecord(parsed, "Hermes Portable gateway journal"); + const intent = requireRecord(record.intent, "Hermes Portable gateway journal intent"); + return intent.type !== "openai"; + }); +} + +function providerNameForIntent(intent: GatewayProviderJournalIntent): string { + return intent.type === "openai" + ? intent.provider + : prepareNativeLocalProfile(portableNativeBinding(intent)).providerName; +} + function recoverGatewayProviderJournalTransactionId( directory: string, scope: GatewayProviderJournalRecoveryScope, @@ -335,19 +417,8 @@ function recoverGatewayProviderJournalTransactionId( "Hermes Portable inference gateway provider journal identity is malformed.", ); } - const providerCredentialEnv = `${scope.credentialEnv}_${transactionId.toUpperCase()}`; - const intent: GatewayProviderJournalIntent = Object.freeze({ - transactionId, - targetSha256: scope.targetSha256, - gatewayName: scope.gatewayName, - sandboxName: scope.sandboxName, - provider: scope.provider, - model: scope.model, - type: scope.type, - credentialEnv: scope.credentialEnv, - providerCredentialEnv, - baseUrl: scope.baseUrl, - }); + const intent = journalIntent(scope, transactionId, true); + const providerCredentialEnv = intent.providerCredentialEnv; if ( providerCredentialEnv.length > 128 || !SAFE_CREDENTIAL_ENV.test(providerCredentialEnv) || @@ -566,9 +637,10 @@ function gatewayReportsProviderAbsent(output: string, provider: string): boolean function observeExactGatewayProvider( runGatewayOpenshell: HermesPortableOllamaGatewayRunner, - provider: string, - expectedProviderCredentialEnv: string, + intent: GatewayProviderJournalIntent, ): GatewayProviderObservation { + const provider = providerNameForIntent(intent); + const expectedProviderCredentialEnv = intent.providerCredentialEnv; const result = runGatewayOpenshell(["provider", "get", provider], { ignoreError: true, suppressOutput: true, @@ -589,11 +661,12 @@ function observeExactGatewayProvider( if ( !metadata || metadata.name !== provider || - metadata.type !== "openai" || + metadata.type !== intent.type || metadata.credentialKeys.length !== 1 || metadata.credentialKeys[0] !== expectedProviderCredentialEnv || - metadata.configKeys.length !== 1 || - metadata.configKeys[0] !== "OPENAI_BASE_URL" || + (intent.type !== "openai" + ? metadata.configKeys.length !== 0 + : metadata.configKeys.length !== 1 || metadata.configKeys[0] !== "OPENAI_BASE_URL") || !GATEWAY_PROVIDER_ID.test(id) || !Number.isSafeInteger(resourceVersion) || resourceVersion < 1 @@ -602,6 +675,27 @@ function observeExactGatewayProvider( "Hermes Portable inference found ambiguous gateway provider authority.", ); } + if (intent.type !== "openai") { + const profile = prepareNativeLocalProfile(portableNativeBinding(intent)); + const expected = parseCheckedInProviderProfileContract(JSON.stringify(profile.document)); + const exported = runGatewayOpenshell( + ["provider", "profile", "export", profile.profileId, "--output", "json"], + { + ignoreError: true, + suppressOutput: true, + stdio: ["ignore", "pipe", "pipe"], + timeout: GATEWAY_PROVIDER_PROBE_TIMEOUT_MS, + }, + ); + if ( + !expected || + exported.status !== 0 || + !exportedProviderProfileMatchesContract(String(exported.stdout ?? ""), expected) + ) + throw portableGatewayStateConflict( + "Hermes Portable native provider profile authority changed.", + ); + } return { kind: "present", id, resourceVersion }; } @@ -610,16 +704,16 @@ function exactGatewayMutation( expectedGatewayName: string, expectedModel: string, expectedSandboxName: string, - expectedCredentialEnv: string, - expectedProviderCredentialEnv: string, + intent: GatewayProviderJournalIntent, journalStore: ReturnType, receiptPublished: boolean, ): Readonly<{ prepareGatewayMutation: HostLocalInferenceStartupSelection["prepareGatewayMutation"]; recoverUnpublishedRoute: boolean; }> { - const readExact = (provider: string): GatewayProviderObservation => - observeExactGatewayProvider(runGatewayOpenshell, provider, expectedProviderCredentialEnv); + const providerName = providerNameForIntent(intent); + const readExact = (): GatewayProviderObservation => + observeExactGatewayProvider(runGatewayOpenshell, intent); const matchesAuthority = ( observation: GatewayProviderObservation, authority: GatewayProviderAuthority, @@ -634,7 +728,6 @@ function exactGatewayMutation( ? Object.freeze({ id: observation.id, resourceVersion: observation.resourceVersion }) : null; const deleteRecordedProvider = async ( - provider: string, journal: GatewayProviderJournal, ): Promise => { const authority = journal.providerAuthority; @@ -643,7 +736,7 @@ function exactGatewayMutation( "Hermes Portable inference gateway provider rollback journal is incomplete.", ); } - const current = readExact(provider); + const current = readExact(); if (current.kind === "absent") { return journalStore.transition(journal, "rolled-back", authority); } @@ -662,10 +755,10 @@ function exactGatewayMutation( }), ).deleteProvider({ target: { kind: "selected" }, - providerName: provider, + providerName, timeoutMs: GATEWAY_PROVIDER_MUTATION_TIMEOUT_MS, }); - const after = readExact(provider); + const after = readExact(); if (after.kind === "absent") { return journalStore.transition(journal, "rolled-back", authority); } @@ -680,7 +773,7 @@ function exactGatewayMutation( throw new Error("Hermes Portable inference gateway provider remained after recorded rollback."); }; const journalAtEntry = journalStore.load(); - const providerAtEntry = readExact("ollama-local"); + const providerAtEntry = readExact(); const recoverUnpublishedRoute = !receiptPublished && journalAtEntry?.phase === "created" && @@ -734,10 +827,10 @@ function exactGatewayMutation( throw new Error("Hermes Portable inference gateway mutation authority changed."); } let journal = journalStore.load(); - let current = readExact(input.provider); + let current = readExact(); if (journal?.phase === "rolling-back") { - journal = await deleteRecordedProvider(input.provider, journal); - current = readExact(input.provider); + journal = await deleteRecordedProvider(journal); + current = readExact(); } if (receiptPublished) { if (journal?.phase !== "created" && journal?.phase !== "committed") { @@ -782,37 +875,40 @@ function exactGatewayMutation( } } return Object.freeze({ - upsertProvider( - name: string, - type: string, - credentialEnv: string, - baseUrl: string, - env: NodeJS.ProcessEnv = {}, - ) { - if ( - name !== input.provider || - type !== "openai" || - credentialEnv !== expectedCredentialEnv || - baseUrl !== input.providerBaseUrl || - Object.keys(env).length !== 1 || - env[expectedCredentialEnv] !== "ollama" - ) { - throw new Error("Hermes Portable inference provider mutation authority changed."); - } + async registerNativeProvider(): Promise { + const binding = portableNativeBinding(intent); + const profile = prepareNativeLocalProfile(binding); + const attachment = ( + authority: GatewayProviderAuthority, + ): NativeLocalProviderAttachment => ({ + ...binding, + schemaVersion: 1, + profileId: profile.profileId, + providerName, + providerId: authority.id, + }); + await requireSandboxPolicyAuthority(async (args) => + runGatewayOpenshell(args, { + ignoreError: true, + suppressOutput: true, + stdio: ["ignore", "pipe", "pipe"], + timeout: GATEWAY_PROVIDER_PROBE_TIMEOUT_MS, + }), + ); let active = journalStore.load(); if (!active) { throw portableGatewayStateConflict( "Hermes Portable inference gateway provider intent disappeared.", ); } - let before = readExact(input.provider); + let before = readExact(); if (active.phase === "created" || active.phase === "committed") { if (!active.providerAuthority || !matchesAuthority(before, active.providerAuthority)) { throw portableGatewayStateConflict( "Hermes Portable inference recorded gateway provider authority changed.", ); } - return { ok: true }; + return attachment(active.providerAuthority!); } if (active.phase === "prepared") { if (before.kind !== "absent") { @@ -821,7 +917,7 @@ function exactGatewayMutation( ); } active = journalStore.transition(active, "creating", null); - before = readExact(input.provider); + before = readExact(); } if (active.phase !== "creating") { throw portableGatewayStateConflict( @@ -836,33 +932,36 @@ function exactGatewayMutation( ); } journalStore.transition(active, "created", authority); - return { ok: true }; + return attachment(authority); } - const result = runGatewayOpenshell( - [ - "provider", - "create", - "--name", - input.provider, - "--type", - "openai", - "--credential", - expectedProviderCredentialEnv, - "--config", - `OPENAI_BASE_URL=${input.providerBaseUrl}`, - ], - { - ignoreError: true, - suppressOutput: true, - stdio: ["ignore", "pipe", "pipe"], - env: { [expectedProviderCredentialEnv]: "ollama" }, - timeout: GATEWAY_PROVIDER_MUTATION_TIMEOUT_MS, - }, - ); - const after = readExact(input.provider); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-portable-profile-")); + let createError: unknown; + try { + const profilePath = path.join(directory, "profile.yaml"); + fs.writeFileSync(profilePath, JSON.stringify(profile.document), { mode: 0o600 }); + await ensureNativeProvider({ + adapter: createManagedProviderAdapter((args, opts) => + runGatewayOpenshell(args, { + ...opts, + ignoreError: true, + suppressOutput: true, + stdio: ["ignore", "pipe", "pipe"], + timeout: GATEWAY_PROVIDER_MUTATION_TIMEOUT_MS, + }), + ), + target: { kind: "selected" }, + profile: { ...profile, profilePath, label: "Portable Ollama" }, + credentialValue: "ollama", + }); + } catch (error) { + createError = error; + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + const after = readExact(); const authority = createdAuthority(after); if (!authority) { - if (after.kind === "absent" && result.status !== 0) { + if (after.kind === "absent" && createError) { throw new Error("Hermes Portable inference could not create its gateway provider."); } throw portableGatewayStateConflict( @@ -870,10 +969,10 @@ function exactGatewayMutation( ); } journalStore.transition(active, "created", authority); - return { ok: true }; + return attachment(authority); }, commit() { - const current = readExact(input.provider); + const current = readExact(); const active = journalStore.load(); if ( (active?.phase !== "created" && active?.phase !== "committed") || @@ -897,7 +996,7 @@ function exactGatewayMutation( "Hermes Portable inference refused rollback of a published provider.", ); } - const observed = readExact(input.provider); + const observed = readExact(); if (active.phase === "rolled-back") { if (observed.kind !== "absent") { throw portableGatewayStateConflict( @@ -930,7 +1029,7 @@ function exactGatewayMutation( } active = journalStore.transition(active, "rolling-back", active.providerAuthority); } - await deleteRecordedProvider(input.provider, active); + await deleteRecordedProvider(active); }, }); }; @@ -981,25 +1080,12 @@ export function createHermesPortableOllamaGatewayTransaction(options: { recoveredReceipt?.publication?.transactionId ?? recoveredJournalTransactionId ?? options.transactionId; - const providerCredentialEnv = `${options.credentialEnv}_${transactionId.toUpperCase()}`; - if (providerCredentialEnv.length > 128 || !SAFE_CREDENTIAL_ENV.test(providerCredentialEnv)) { - throw new Error("Hermes Portable Ollama transaction credential authority is invalid."); - } - const gatewayProviderJournal = createGatewayProviderJournalStore( - options.directory, - Object.freeze({ - transactionId, - targetSha256: options.targetSha256, - gatewayName: options.gatewayName, - sandboxName: options.sandboxName, - provider: "ollama-local", - model: options.model, - type: "openai", - credentialEnv: options.credentialEnv, - providerCredentialEnv, - baseUrl: "http://host.openshell.internal:11434/v1", - }), - ); + if (!retainedJournalIsNative(options.directory)) + throw portableGatewayStateConflict( + "Retained shared Ollama routes require explicit retirement before native onboarding.", + ); + const intent = journalIntent(recoveryScope, transactionId, true); + const gatewayProviderJournal = createGatewayProviderJournalStore(options.directory, intent); const receiptWriter = createReceiptWriter( options.directory, transactionId, @@ -1023,8 +1109,7 @@ export function createHermesPortableOllamaGatewayTransaction(options: { options.gatewayName, options.model, options.sandboxName, - options.credentialEnv, - providerCredentialEnv, + intent, gatewayProviderJournal, publishedReceipt !== null, ); @@ -1085,18 +1170,22 @@ export function prepareHermesPortableOllamaPublishedReceiptAuthority(options: { } const journalStore = createGatewayProviderJournalStore( options.directory, - Object.freeze({ + journalIntent( + { + transactionId, + targetSha256: receipt.publication.targetSha256, + gatewayName: options.gatewayName, + sandboxName: options.sandboxName, + provider: "ollama-local" as const, + model: receipt.inference.model, + type: "openai" as const, + credentialEnv: options.credentialEnv, + providerCredentialEnv, + baseUrl: "http://host.openshell.internal:11434/v1" as const, + }, transactionId, - targetSha256: receipt.publication.targetSha256, - gatewayName: options.gatewayName, - sandboxName: options.sandboxName, - provider: "ollama-local" as const, - model: receipt.inference.model, - type: "openai" as const, - credentialEnv: options.credentialEnv, - providerCredentialEnv, - baseUrl: "http://host.openshell.internal:11434/v1" as const, - }), + retainedJournalIsNative(options.directory), + ), "open-existing", ); const journal = journalStore.load(); @@ -1150,18 +1239,22 @@ export function prepareHermesPortableOllamaPublishedInferenceAuthority(options: if (providerCredentialEnv.length > 128 || !SAFE_CREDENTIAL_ENV.test(providerCredentialEnv)) { throw new Error("Hermes Portable Ollama transaction credential authority is invalid."); } - const intent = Object.freeze({ + const intent = journalIntent( + { + transactionId, + targetSha256, + gatewayName: options.gatewayName, + sandboxName: options.sandboxName, + provider: "ollama-local" as const, + model: receipt.inference.model, + type: "openai" as const, + credentialEnv: options.credentialEnv, + providerCredentialEnv, + baseUrl: "http://host.openshell.internal:11434/v1" as const, + }, transactionId, - targetSha256, - gatewayName: options.gatewayName, - sandboxName: options.sandboxName, - provider: "ollama-local" as const, - model: receipt.inference.model, - type: "openai" as const, - credentialEnv: options.credentialEnv, - providerCredentialEnv, - baseUrl: "http://host.openshell.internal:11434/v1" as const, - }); + retainedJournalIsNative(options.directory), + ); const journalStore = createGatewayProviderJournalStore( options.directory, intent, @@ -1171,11 +1264,7 @@ export function prepareHermesPortableOllamaPublishedInferenceAuthority(options: if (journal?.phase !== "committed" || journal.providerAuthority === null) { throw new Error("Hermes Portable Ollama gateway publication is not committed."); } - const provider = observeExactGatewayProvider( - options.runGatewayOpenshell, - "ollama-local", - providerCredentialEnv, - ); + const provider = observeExactGatewayProvider(options.runGatewayOpenshell, journal.intent); if ( provider.kind !== "present" || provider.id !== journal.providerAuthority.id || @@ -1196,8 +1285,7 @@ export function prepareHermesPortableOllamaPublishedInferenceAuthority(options: assertTransactionCurrent(); const currentProvider = observeExactGatewayProvider( options.runGatewayOpenshell, - "ollama-local", - providerCredentialEnv, + journal.intent, ); if (!isDeepStrictEqual(currentProvider, provider)) { throw new Error("Hermes Portable Ollama gateway provider authority changed."); @@ -1246,24 +1334,29 @@ export function prepareHermesPortableOllamaProviderRetirement(options: { readonly runGatewayOpenshell: HermesPortableOllamaGatewayRunner; readonly allowAbsent?: boolean; }): PreparedHermesPortableOllamaProviderRetirement { + const transactionId = options.transactionId; const providerCredentialEnv = `${options.credentialEnv}_${options.transactionId.toUpperCase()}`; if (providerCredentialEnv.length > 128 || !SAFE_CREDENTIAL_ENV.test(providerCredentialEnv)) { throw new Error("Hermes Portable Ollama transaction credential authority is invalid."); } const store = createGatewayProviderJournalStore( options.directory, - Object.freeze({ - transactionId: options.transactionId, - targetSha256: options.targetSha256, - gatewayName: options.gatewayName, - sandboxName: options.sandboxName, - provider: "ollama-local", - model: options.model, - type: "openai", - credentialEnv: options.credentialEnv, - providerCredentialEnv, - baseUrl: "http://host.openshell.internal:11434/v1", - }), + journalIntent( + { + transactionId: options.transactionId, + targetSha256: options.targetSha256, + gatewayName: options.gatewayName, + sandboxName: options.sandboxName, + provider: "ollama-local", + model: options.model, + type: "openai", + credentialEnv: options.credentialEnv, + providerCredentialEnv, + baseUrl: "http://host.openshell.internal:11434/v1", + }, + transactionId, + retainedJournalIsNative(options.directory), + ), "open-existing", ); const journal = store.load(); @@ -1272,8 +1365,7 @@ export function prepareHermesPortableOllamaProviderRetirement(options: { } const expectedJournal = `${JSON.stringify(journal)}\n`; const expected = journal.providerAuthority; - const observe = () => - observeExactGatewayProvider(options.runGatewayOpenshell, "ollama-local", providerCredentialEnv); + const observe = () => observeExactGatewayProvider(options.runGatewayOpenshell, journal.intent); const matches = ( observation: GatewayProviderObservation, ): observation is Extract => @@ -1328,7 +1420,7 @@ export function prepareHermesPortableOllamaProviderRetirement(options: { }), ).deleteProvider({ target: { kind: "selected" }, - providerName: "ollama-local", + providerName: providerNameForIntent(journal.intent), timeoutMs: GATEWAY_PROVIDER_MUTATION_TIMEOUT_MS, }); const after = observe(); diff --git a/src/lib/onboard/experimental/hermes-portable-ollama-inference.test.ts b/src/lib/onboard/experimental/hermes-portable-ollama-inference.test.ts index de1ae19ff22..801fc19e1cd 100644 --- a/src/lib/onboard/experimental/hermes-portable-ollama-inference.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-ollama-inference.test.ts @@ -119,17 +119,8 @@ const gatewayMutationInput = { providerBaseUrl: "http://host.openshell.internal:11434/v1", } as const; -function createExactGatewayProvider( - mutation: HostLocalInferenceGatewayMutation, - baseUrl: string = gatewayMutationInput.providerBaseUrl, -) { - return mutation.upsertProvider!( - gatewayMutationInput.provider, - "openai", - "NEMOCLAW_OLLAMA_PROXY_TOKEN", - baseUrl, - { NEMOCLAW_OLLAMA_PROXY_TOKEN: "ollama" }, - ); +function createExactGatewayProvider(mutation: HostLocalInferenceGatewayMutation) { + return mutation.registerNativeProvider!(); } function gatewayJournalPath(fixture: ReturnType): string { @@ -164,7 +155,7 @@ async function publishPortableInference(fixture: ReturnType { const journalBefore = snapshotExactTestFile(journalPath); const mutationsBefore = fixture.gatewayProvider .calls() - .filter(({ args }) => args[0] === "provider" && args[1] !== "get").length; + .filter( + ({ args }) => + args[0] === "provider" && + args[1] !== "get" && + !(args[1] === "profile" && args[2] === "export"), + ).length; const authority = prepareHermesPortableOllamaPublishedInferenceAuthority({ directory: path.dirname(receiptPath), @@ -407,7 +403,12 @@ describe("Hermes Portable Ollama inference activation", () => { expect( fixture.gatewayProvider .calls() - .filter(({ args }) => args[0] === "provider" && args[1] !== "get").length, + .filter( + ({ args }) => + args[0] === "provider" && + args[1] !== "get" && + !(args[1] === "profile" && args[2] === "export"), + ).length, ).toBe(mutationsBefore); }); @@ -614,7 +615,7 @@ describe("Hermes Portable Ollama inference activation", () => { const route = prepareManagedRoute(fixture, selection); route.prepared.validateBeforeCommit(); const gatewayMutation = await selection.prepareGatewayMutation(gatewayMutationInput); - createExactGatewayProvider(gatewayMutation); + await createExactGatewayProvider(gatewayMutation); await gatewayMutation.commit(); route.prepared.commit(); fixture.events.push("provider-operation"); @@ -669,10 +670,8 @@ describe("Hermes Portable Ollama inference activation", () => { expect( fixture.gatewayProvider .calls() - .every(({ args, timeout }) => - args[1] === "get" - ? timeout === OPENSHELL_PROBE_TIMEOUT_MS - : timeout === OPENSHELL_OPERATION_TIMEOUT_MS, + .every(({ timeout }) => + [OPENSHELL_PROBE_TIMEOUT_MS, OPENSHELL_OPERATION_TIMEOUT_MS].includes(timeout), ), ).toBe(true); }); @@ -694,7 +693,7 @@ describe("Hermes Portable Ollama inference activation", () => { const route = prepareManagedRoute(fixture, selection); route.prepared.validateBeforeCommit(); const mutation = await selection.prepareGatewayMutation({ ...gatewayMutationInput, model }); - createExactGatewayProvider(mutation); + await createExactGatewayProvider(mutation); await mutation.commit(); route.prepared.commit(); expect(route.receipt).toMatchObject({ inference: { model } }); @@ -760,7 +759,7 @@ describe("Hermes Portable Ollama inference activation", () => { ...gatewayMutationInput, gatewayName, }); - createExactGatewayProvider(mutation); + await createExactGatewayProvider(mutation); await mutation.commit(); route.prepared.commit(); const journal = gatewayJournal(fixture); @@ -806,7 +805,7 @@ describe("Hermes Portable Ollama inference activation", () => { expect(gatewayJournal(fixture)).toEqual(durableJournal); await expect(resumed.prepareGatewayMutation(gatewayMutationInput)).resolves.toBeDefined(); expect( - fixture.events.filter((event) => event.includes("provider create --name ollama-local")), + fixture.events.filter((event) => event.includes("provider create --name nemoclaw-local-v1-")), ).toHaveLength(1); }); @@ -824,7 +823,7 @@ describe("Hermes Portable Ollama inference activation", () => { expect(healed.request).not.toHaveProperty("recover", true); expect(gatewayJournal(fixture)).toMatchObject({ phase: "committed" }); expect( - fixture.events.filter((event) => event.includes("provider create --name ollama-local")), + fixture.events.filter((event) => event.includes("provider create --name nemoclaw-local-v1-")), ).toHaveLength(1); }); @@ -1026,7 +1025,7 @@ describe("Hermes Portable Ollama inference activation", () => { const recovered = prepareManagedRoute(fixture, interrupted); recovered.prepared.validateBeforeCommit(); const gatewayMutation = await interrupted.prepareGatewayMutation(gatewayMutationInput); - createExactGatewayProvider(gatewayMutation); + await createExactGatewayProvider(gatewayMutation); await gatewayMutation.commit(); recovered.prepared.commit(); const published = fixture.resolve({ @@ -1054,23 +1053,29 @@ describe("Hermes Portable Ollama inference activation", () => { ); fixture.gatewayProvider.setMalformed(false); const mutation = await selection.prepareGatewayMutation(gatewayMutationInput); - expect(createExactGatewayProvider(mutation)).toEqual({ ok: true }); + expect(await createExactGatewayProvider(mutation)).toMatchObject({ + schemaVersion: 1, + providerId: "portable-ollama-provider", + }); await mutation.commit(); await mutation.rollback(); expect(fixture.gatewayProvider.isPresent()).toBe(false); }); - it("creates the Portable OpenAI provider without a compatibility-profile mutation (#11229)", async () => { + it("creates the Portable native provider with its endpoint-bound profile (#11229)", async () => { const fixture = createRuntimeFixture(); const mutation = await fixture.resolve()!.prepareGatewayMutation(gatewayMutationInput); - expect(createExactGatewayProvider(mutation)).toEqual({ ok: true }); + expect(await createExactGatewayProvider(mutation)).toMatchObject({ + schemaVersion: 1, + providerId: "portable-ollama-provider", + }); const providerCreate = fixture.events.findIndex((event) => - event.includes("provider create --name ollama-local"), + event.includes("provider create --name nemoclaw-local-v1-"), ); expect(providerCreate).toBeGreaterThanOrEqual(0); - expect(fixture.events.some((event) => event.includes("provider profile"))).toBe(false); + expect(fixture.events.some((event) => event.includes("provider profile import"))).toBe(true); }); it("resumes the journaled provider-create crash window and publishes exact ownership (#9596)", async () => { @@ -1092,7 +1097,7 @@ describe("Hermes Portable Ollama inference activation", () => { .mockImplementationOnce(() => { throw new Error("injected death before provider identity persistence"); }); - expect(() => createExactGatewayProvider(mutation)).toThrow( + await expect(createExactGatewayProvider(mutation)).rejects.toThrow( "injected death before provider identity persistence", ); rename.mockRestore(); @@ -1109,7 +1114,10 @@ describe("Hermes Portable Ollama inference activation", () => { const recoveredRoute = prepareManagedRoute(fixture, restarted); recoveredRoute.prepared.validateBeforeCommit(); const resumedMutation = await restarted.prepareGatewayMutation(gatewayMutationInput); - expect(createExactGatewayProvider(resumedMutation)).toEqual({ ok: true }); + expect(await createExactGatewayProvider(resumedMutation)).toMatchObject({ + schemaVersion: 1, + providerId: "portable-ollama-provider", + }); await resumedMutation.commit(); recoveredRoute.prepared.commit(); const published = fixture.resolve({ @@ -1124,13 +1132,11 @@ describe("Hermes Portable Ollama inference activation", () => { providerAuthority: { id: "portable-ollama-provider", resourceVersion: 1 }, }); expect(gatewayJournal(fixture).intent.providerCredentialEnv).toMatch( - /^NEMOCLAW_OLLAMA_PROXY_TOKEN_[A-F0-9]{64}$/u, + /^NEMOCLAW_LOCAL_INFERENCE_TOKEN$/u, ); expect( fixture.events.filter((event) => - event.startsWith( - "openshell:provider create --name ollama-local --type openai --credential NEMOCLAW_OLLAMA_PROXY_TOKEN_", - ), + event.startsWith("openshell:provider create --name nemoclaw-local-v1-"), ), ).toHaveLength(1); expect(fixture.gatewayProvider.isPresent()).toBe(true); @@ -1146,7 +1152,7 @@ describe("Hermes Portable Ollama inference activation", () => { route.prepared.validateBeforeCommit(); const input = { ...gatewayMutationInput, gatewayName }; const mutation = await selection.prepareGatewayMutation(input); - createExactGatewayProvider(mutation); + await createExactGatewayProvider(mutation); await mutation.commit(); const interruptedTransactionId = gatewayJournal(fixture).intent.transactionId; @@ -1160,12 +1166,17 @@ describe("Hermes Portable Ollama inference activation", () => { const recoveredRoute = prepareManagedRoute(fixture, restarted); recoveredRoute.prepared.validateBeforeCommit(); const resumedMutation = await restarted.prepareGatewayMutation(input); - expect(createExactGatewayProvider(resumedMutation)).toEqual({ ok: true }); + expect(await createExactGatewayProvider(resumedMutation)).toMatchObject({ + schemaVersion: 1, + providerId: "portable-ollama-provider", + }); await resumedMutation.commit(); recoveredRoute.prepared.commit(); expect( - fixture.events.filter((event) => event.includes("provider create --name ollama-local")), + fixture.events.filter((event) => + event.includes("provider create --name nemoclaw-local-v1-"), + ), ).toHaveLength(1); expect(gatewayJournal(fixture)).toMatchObject({ phase: "committed", @@ -1226,10 +1237,13 @@ describe("Hermes Portable Ollama inference activation", () => { ); fixture.gatewayProvider.setLookupFailure(false); const mutation = await selection.prepareGatewayMutation(gatewayMutationInput); - expect(() => createExactGatewayProvider(mutation, "http://192.0.2.2:11434/v1")).toThrow( - "provider mutation authority changed", - ); - createExactGatewayProvider(mutation); + await expect( + selection.prepareGatewayMutation({ + ...gatewayMutationInput, + providerBaseUrl: "http://192.0.2.2:11434/v1", + }), + ).rejects.toThrow("gateway mutation authority changed"); + await createExactGatewayProvider(mutation); fixture.gatewayProvider.bumpResourceVersion(); expect(() => mutation.commit()).toThrow("gateway provider authority changed"); await expect(mutation.rollback()).rejects.toThrow( @@ -1246,10 +1260,10 @@ describe("Hermes Portable Ollama inference activation", () => { "NEMOCLAW_OLLAMA_PROXY_TOKEN_FOREIGN_TRANSACTION", ); - expect(() => createExactGatewayProvider(mutation)).toThrow( + await expect(createExactGatewayProvider(mutation)).rejects.toThrow( "ambiguous gateway provider authority", ); - expect(() => createExactGatewayProvider(mutation)).toThrow(GatewayStateConflictError); + await expect(createExactGatewayProvider(mutation)).rejects.toThrow(GatewayStateConflictError); expect(gatewayJournal(fixture)).toMatchObject({ phase: "creating", @@ -1285,7 +1299,7 @@ describe("Hermes Portable Ollama inference activation", () => { const fixture = createRuntimeFixture(); const selection = fixture.resolve()!; const mutation = await selection.prepareGatewayMutation(gatewayMutationInput); - createExactGatewayProvider(mutation); + await createExactGatewayProvider(mutation); const journalPath = gatewayJournalPath(fixture); const serialized = serialize(gatewayJournal(fixture)); fs.writeFileSync(journalPath, serialized, { mode: 0o600 }); @@ -1320,7 +1334,10 @@ describe("Hermes Portable Ollama inference activation", () => { const mutation = await selection.prepareGatewayMutation(gatewayMutationInput); fixture.gatewayProvider.setCreateTransportAmbiguity(true); - expect(createExactGatewayProvider(mutation)).toEqual({ ok: true }); + expect(await createExactGatewayProvider(mutation)).toMatchObject({ + schemaVersion: 1, + providerId: "portable-ollama-provider", + }); expect(gatewayJournal(fixture)).toMatchObject({ phase: "created", providerAuthority: { id: "portable-ollama-provider", resourceVersion: 1 }, @@ -1328,9 +1345,7 @@ describe("Hermes Portable Ollama inference activation", () => { expect(fixture.gatewayProvider.credentialEnv()).toBe( gatewayJournal(fixture).intent.providerCredentialEnv, ); - expect(fixture.gatewayProvider.credentialEnv()).toMatch( - /^NEMOCLAW_OLLAMA_PROXY_TOKEN_[A-F0-9]{64}$/u, - ); + expect(fixture.gatewayProvider.credentialEnv()).toMatch(/^NEMOCLAW_LOCAL_INFERENCE_TOKEN$/u); }); it.each([ @@ -1409,7 +1424,7 @@ describe("Hermes Portable Ollama inference activation", () => { const fixture = createRuntimeFixture(); const selection = fixture.resolve()!; const mutation = await selection.prepareGatewayMutation(gatewayMutationInput); - createExactGatewayProvider(mutation); + await createExactGatewayProvider(mutation); mutate(fixture, selection); const mutationsBefore = fixture.events.filter( (event) => event.includes("provider create") || event.includes("provider delete"), @@ -1451,7 +1466,7 @@ describe("Hermes Portable Ollama inference activation", () => { const fixture = createRuntimeFixture(); const selection = fixture.resolve()!; const mutation = await selection.prepareGatewayMutation(gatewayMutationInput); - createExactGatewayProvider(mutation); + await createExactGatewayProvider(mutation); await mutation.commit(); fixture.gatewayProvider.setDeleteFailure(true); await expect(mutation.rollback()).rejects.toThrow( @@ -1465,7 +1480,10 @@ describe("Hermes Portable Ollama inference activation", () => { const resumed = await restarted.prepareGatewayMutation(gatewayMutationInput); expect(fixture.gatewayProvider.isPresent()).toBe(false); expect(gatewayJournal(fixture)).toMatchObject({ phase: "prepared" }); - expect(createExactGatewayProvider(resumed)).toEqual({ ok: true }); + expect(await createExactGatewayProvider(resumed)).toMatchObject({ + schemaVersion: 1, + providerId: "portable-ollama-provider", + }); expect(fixture.gatewayProvider.isPresent()).toBe(true); }); }); diff --git a/src/lib/onboard/experimental/hermes-portable-ollama-inference.ts b/src/lib/onboard/experimental/hermes-portable-ollama-inference.ts index 99b3cfde902..d8d952f8bb0 100644 --- a/src/lib/onboard/experimental/hermes-portable-ollama-inference.ts +++ b/src/lib/onboard/experimental/hermes-portable-ollama-inference.ts @@ -1693,7 +1693,6 @@ export function createHermesPortableOllamaInferenceResolver( redactOnboardCommandDiagnosticText, ); const selection: HostLocalInferenceStartupSelection = Object.freeze({ - sharedInferenceRoute: true, runtimeProviderId: "podman", request: Object.freeze({ application: "hermes" as const, diff --git a/src/lib/onboard/experimental/hermes-portable-onboarding-created-identity.test.ts b/src/lib/onboard/experimental/hermes-portable-onboarding-created-identity.test.ts index efd9b80726f..1bea82dfc00 100644 --- a/src/lib/onboard/experimental/hermes-portable-onboarding-created-identity.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-onboarding-created-identity.test.ts @@ -23,7 +23,7 @@ const CREATED_IDENTITY_SELECTOR_ARGS = [ `${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${CREATE_ATTEMPT_NONCE}`, "--output", "json", - "--limit", + "--page-size", "2", ]; @@ -172,7 +172,7 @@ describe("Hermes Portable created-identity capture", () => { "json", "--selector", `${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${CREATE_ATTEMPT_NONCE}`, - "--limit", + "--page-size", "2", ], ], diff --git a/src/lib/onboard/experimental/hermes-portable-onboarding-gateway-port-recovery.test.ts b/src/lib/onboard/experimental/hermes-portable-onboarding-gateway-port-recovery.test.ts index 529b15cc19c..960d17b6fee 100644 --- a/src/lib/onboard/experimental/hermes-portable-onboarding-gateway-port-recovery.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-onboarding-gateway-port-recovery.test.ts @@ -120,15 +120,16 @@ describe("Hermes portable onboarding gateway-port recovery", () => { model: freshPortableInput.model, providerBaseUrl: baseUrl, }); - expect( - mutation.upsertProvider!( - freshPortableInput.provider, - "openai", - "NEMOCLAW_OLLAMA_PROXY_TOKEN", - baseUrl, - { NEMOCLAW_OLLAMA_PROXY_TOKEN: "ollama" }, - ), - ).toMatchObject({ ok: true }); + const interruptedAttachment = await mutation.registerNativeProvider!(); + expect(interruptedAttachment).toMatchObject({ + provider: "ollama-local", + agent: "hermes", + gatewayName, + sandboxName: freshPortableInput.sandboxName, + endpointUrl: baseUrl, + authMode: "sentinel", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }); await mutation.commit(); preparedStartup.prepared.commit(); const transactionRoot = path.join(fixture.resolverOptions.stateDir, "portable-inference"); @@ -215,7 +216,7 @@ describe("Hermes portable onboarding gateway-port recovery", () => { const runner = require("../../runner") as typeof import("../../runner"); vi.spyOn(runner, "run").mockImplementation((command, options) => { const args = Array.isArray(command) ? command.map(String) : [String(command)]; - const providerIndex = args.indexOf("provider"); + const providerIndex = args.findIndex((arg) => arg === "provider" || arg === "policy"); expect( providerIndex, `Unexpected onboarding command: ${args.join(" ")}`, diff --git a/src/lib/onboard/experimental/hermes-portable-onboarding.test.ts b/src/lib/onboard/experimental/hermes-portable-onboarding.test.ts index 2edd17f3698..e628678c532 100644 --- a/src/lib/onboard/experimental/hermes-portable-onboarding.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-onboarding.test.ts @@ -1491,7 +1491,7 @@ network_policies: gatewayPort: 8080, lifecycleGeneration: "generation-1", openshellDriver: "docker", - openshellVersion: "0.0.116", + openshellVersion: "0.1.7", }; const entry = override === null ? null : { ...matching, ...override }; diff --git a/src/lib/onboard/experimental/hermes-portable-onboarding.ts b/src/lib/onboard/experimental/hermes-portable-onboarding.ts index 9902cb0ef8d..9a5745c6a4c 100644 --- a/src/lib/onboard/experimental/hermes-portable-onboarding.ts +++ b/src/lib/onboard/experimental/hermes-portable-onboarding.ts @@ -302,7 +302,7 @@ function scopeHermesPortableCreatedIdentityArgs( args[4] !== "--selector" || args[6] !== "--output" || args[7] !== "json" || - args[8] !== "--limit" || + args[8] !== "--page-size" || args[9] !== "2" ) { return null; @@ -322,7 +322,7 @@ function scopeHermesPortableCreatedIdentityArgs( `${CREATE_ATTEMPT_SELECTOR_PREFIX}${nonce}`, "--output", "json", - "--limit", + "--page-size", "2", ]; } diff --git a/src/lib/onboard/experimental/hermes-portable-operating-authority.test.ts b/src/lib/onboard/experimental/hermes-portable-operating-authority.test.ts index c673bb9ec72..b43ed01b2a9 100644 --- a/src/lib/onboard/experimental/hermes-portable-operating-authority.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-operating-authority.test.ts @@ -83,7 +83,7 @@ function snapshot(withSuccessor = true): HermesPortableReceiptSnapshot & { readonly receipt: HermesPortableConfiguredReceipt; } { const openshellExecutableAuthority: HermesPortableOpenShellExecutableAuthority = { - version: "0.0.116", + version: "0.1.7", executable: executable("/usr/bin/openshell", "b".repeat(64)), }; const podmanExecutableAuthority: HermesPortablePodmanExecutableAuthority = { @@ -197,7 +197,7 @@ describe("Hermes Portable schema-8 operation authority", () => { it("captures operation-local authority for an explicitly admitted schema-7 receipt (#10423)", () => { const captureSocketAuthority = vi.fn(() => socket("99")); const captureOpenShellExecutableAuthority = vi.fn(() => ({ - version: "0.0.116" as const, + version: "0.1.7" as const, executable: executable("/usr/bin/openshell", "b".repeat(64)), })); const capturePodmanExecutableAuthority = vi.fn(() => ({ @@ -227,7 +227,7 @@ describe("Hermes Portable schema-8 operation authority", () => { env: environment(), captureSocketAuthority: () => socket("99"), captureOpenShellExecutableAuthority: () => ({ - version: "0.0.116", + version: "0.1.7", executable: executable("/usr/bin/openshell", "b".repeat(64)), }), capturePodmanExecutableAuthority: () => ({ @@ -245,7 +245,7 @@ describe("Hermes Portable schema-8 operation authority", () => { env: environment(), captureSocketAuthority: () => socket("99", 0o40710), captureOpenShellExecutableAuthority: () => ({ - version: "0.0.116", + version: "0.1.7", executable: executable("/usr/bin/openshell", "b".repeat(64)), }), capturePodmanExecutableAuthority: () => ({ @@ -285,7 +285,7 @@ describe("Hermes Portable schema-8 operation authority", () => { env: environment(), captureSocketAuthority: capture, captureOpenShellExecutableAuthority: () => ({ - version: "0.0.116", + version: "0.1.7", executable: executable("/usr/bin/openshell", "b".repeat(64)), }), capturePodmanExecutableAuthority: () => ({ @@ -302,7 +302,7 @@ describe("Hermes Portable schema-8 operation authority", () => { env: environment(), captureSocketAuthority: () => socket(String(100 + captures++)), captureOpenShellExecutableAuthority: () => ({ - version: "0.0.116", + version: "0.1.7", executable: executable("/usr/bin/openshell", "b".repeat(64)), }), capturePodmanExecutableAuthority: () => ({ @@ -321,7 +321,7 @@ describe("Hermes Portable schema-8 operation authority", () => { env: environment(), captureSocketAuthority: () => socket("99"), captureOpenShellExecutableAuthority: () => ({ - version: "0.0.116", + version: "0.1.7", executable: executable("/usr/bin/openshell", "b".repeat(64)), }), capturePodmanExecutableAuthority: () => ({ @@ -338,7 +338,7 @@ describe("Hermes Portable schema-8 operation authority", () => { it("checks transaction identity without repeating executable behavior probes (#10423)", () => { const openshell = { - version: "0.0.116" as const, + version: "0.1.7" as const, executable: executable("/usr/bin/openshell", "b".repeat(64)), }; const podman = { @@ -374,7 +374,7 @@ describe("Hermes Portable schema-8 operation authority", () => { env: environment(), captureSocketAuthority: () => socket("99"), captureOpenShellExecutableAuthority: () => ({ - version: "0.0.116", + version: "0.1.7", executable: executable( "/usr/bin/openshell", (owner === "openshell" ? "0" : "b").repeat(64), diff --git a/src/lib/onboard/experimental/hermes-portable-pair.test-fixture.ts b/src/lib/onboard/experimental/hermes-portable-pair.test-fixture.ts new file mode 100644 index 00000000000..6606fa87af4 --- /dev/null +++ b/src/lib/onboard/experimental/hermes-portable-pair.test-fixture.ts @@ -0,0 +1,84 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { + createHermesPortableLifecycleTestReceipt, + createHermesPortableLifecycleTestDeps, + SANDBOX, + GATEWAY, + GENERATION, + CONTAINER_ID, + IMAGE, + SANDBOX_ID, + LABELS, +} from "./hermes-portable-lifecycle.test-fixture"; + +/** Model the 0.1.2 companion independently of the workload's stop state. */ +export function createHermesPortablePairFixture( + stateDir: string, + policyPath: string, + initiallyRunning: boolean, + agent: Parameters[0]["agent"], +) { + const labels = { ...LABELS, "openshell.ai/isolation-role": "sandbox" }; + const receipt = createHermesPortableLifecycleTestReceipt({ + agent, + stateDir, + policyPath, + homeDir: "/home/test", + sandboxName: SANDBOX, + gatewayName: GATEWAY, + lifecycleGeneration: GENERATION, + containerId: CONTAINER_ID, + imageDigest: IMAGE, + sandboxId: SANDBOX_ID, + labels, + }); + const fixture = createHermesPortableLifecycleTestDeps(stateDir, receipt, initiallyRunning, { + labels, + }); + const capture = fixture.podman.getMockImplementation()!; + const supervisorId = "c".repeat(64); + const supervisor = { + Id: supervisorId, + Image: IMAGE, + Name: `openshell-supervisor-${SANDBOX_ID}`, + Config: { Labels: { ...LABELS, "openshell.ai/isolation-role": "supervisor" } }, + State: { Running: true, Paused: false, Status: "running" }, + HostConfig: { RestartPolicy: { Name: "no" } }, + }; + fixture.podman.mockImplementation((args) => { + if (args[0] === "ps") return { status: 0, stdout: supervisorId, stderr: "" }; + if (args[1] === "inspect" && args[2] === supervisorId) { + return { status: 0, stdout: JSON.stringify([supervisor]), stderr: "" }; + } + return capture(args); + }); + return { + ...fixture, + supervisor, + failStartupAndSetSupervisorStopState(supervisorRemainsRunning: boolean) { + const captureOpenShell = fixture.captureOpenShell.getMockImplementation()!; + fixture.captureOpenShell.mockImplementation((args) => { + if (args.includes("python3")) { + return { status: 0, stdout: "unavailable\n", stderr: "" }; + } + if (args[0] === "sandbox" && args[1] === "stop") { + supervisor.State.Running = supervisorRemainsRunning; + supervisor.State.Status = supervisorRemainsRunning ? "running" : "exited"; + } + return captureOpenShell(args); + }); + }, + exitSupervisorAfter(milliseconds: number) { + const sleep = fixture.deps.sleep.getMockImplementation()!; + fixture.deps.sleep.mockImplementation((elapsed) => { + sleep(elapsed); + if (fixture.deps.now() >= milliseconds) { + supervisor.State.Running = false; + supervisor.State.Status = "exited"; + } + }); + }, + }; +} diff --git a/src/lib/onboard/experimental/hermes-portable-pair.test.ts b/src/lib/onboard/experimental/hermes-portable-pair.test.ts new file mode 100644 index 00000000000..f2cf9d295f6 --- /dev/null +++ b/src/lib/onboard/experimental/hermes-portable-pair.test.ts @@ -0,0 +1,183 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import { loadAgent } from "../../agent/defs"; +import { withMcpLifecycleLock } from "../../state/mcp-lifecycle-lock"; +import { SANDBOX, GATEWAY, GENERATION, POLICY } from "./hermes-portable-lifecycle.test-fixture"; +import { createHermesPortablePairFixture } from "./hermes-portable-pair.test-fixture"; +import { openshellMutationCalls } from "./hermes-portable-lifecycle.test-fixtures"; +import { + hermesPortableLifecycleInternals, + recoverHermesPortableSandboxLifecycle, + stopHermesPortableSandboxLifecycle, + type HermesPortableLifecycleDeps, +} from "./hermes-portable-lifecycle"; +import { publishHermesPortableSuccessorReceipt } from "./hermes-portable-receipt"; + +let stateDir: string; +let policyPath: string; +function pairedStopFixture(initiallyRunning: boolean) { + return createHermesPortablePairFixture( + stateDir, + policyPath, + initiallyRunning, + loadAgent("hermes"), + ); +} +async function publishSuccessor(): Promise { + await withMcpLifecycleLock( + SANDBOX, + () => publishHermesPortableSuccessorReceipt(SANDBOX, stateDir), + { stateDir: path.join(stateDir, "state") }, + ); +} +function lifecycleContext() { + return { + agent: "hermes", + gatewayName: GATEWAY, + lifecycleGeneration: GENERATION, + openshellDriver: "docker", + provider: "ollama", + }; +} + +beforeEach(() => { + stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-lifecycle-")); + policyPath = path.join(stateDir, "policy.yaml"); + fs.writeFileSync(policyPath, POLICY, { mode: 0o600 }); +}); + +afterEach(() => { + vi.unstubAllEnvs(); + vi.restoreAllMocks(); + fs.rmSync(stateDir, { recursive: true, force: true }); +}); + +function recoverWithLifecycleLock(deps: HermesPortableLifecycleDeps) { + return withMcpLifecycleLock( + SANDBOX, + () => recoverHermesPortableSandboxLifecycle(SANDBOX, lifecycleContext(), deps), + { stateDir: path.join(stateDir, "state") }, + ); +} + +describe("Hermes portable paired-container stop", () => { + it.each([true, false])( + "verifies supervisor exit during Hermes rollback (supervisor remains running=%s)", + async (supervisorRemainsRunning) => { + const { deps, captureOpenShell, failStartupAndSetSupervisorStopState } = + pairedStopFixture(false); + await publishSuccessor(); + failStartupAndSetSupervisorStopState(supervisorRemainsRunning); + await expect(recoverWithLifecycleLock(deps)).rejects.toThrow( + supervisorRemainsRunning + ? "rollback=openshell-terminal-settlement-unproved" + : "managed startup did not pass authenticated health", + ); + expect(openshellMutationCalls(captureOpenShell, "stop")).toHaveLength(1); + }, + ); + + it.each([true, false])( + "rejects stopped Hermes success when its supervisor is running (workload initially running=%s)", + async (initiallyRunning) => { + const { deps, podman } = pairedStopFixture(initiallyRunning); + await expect( + withMcpLifecycleLock( + SANDBOX, + () => stopHermesPortableSandboxLifecycle(SANDBOX, lifecycleContext(), vi.fn(), deps), + { stateDir: path.join(stateDir, "state") }, + ), + ).rejects.toThrow("did not settle"); + expect(podman.mock.calls.some(([args]) => args[1] === "stop" || args[1] === "rm")).toBe( + false, + ); + }, + ); + + it.each([true, false])( + "waits for both Hermes containers to exit (workload initially running=%s)", + async (initiallyRunning) => { + const { deps, captureOpenShell, exitSupervisorAfter } = pairedStopFixture(initiallyRunning); + exitSupervisorAfter(2_000); + const result = await withMcpLifecycleLock( + SANDBOX, + () => stopHermesPortableSandboxLifecycle(SANDBOX, lifecycleContext(), vi.fn(), deps), + { stateDir: path.join(stateDir, "state") }, + ); + expect(result).toEqual({ kind: initiallyRunning ? "stopped" : "already-stopped" }); + expect(deps.now()).toBe(2_000); + expect( + captureOpenShell.mock.calls.some(([args]) => + args.includes(hermesPortableLifecycleInternals.openShellV0116StopAssistProgram), + ), + ).toBe(false); + }, + ); + + it.each([ + "openshell.ai/isolation-role", + "openshell.ai/sandbox-id", + "openshell.ai/sandbox-workspace", + "openshell.ai/sandbox-namespace", + ] as const)("rejects a Hermes supervisor with changed %s before stop", async (label) => { + const { deps, supervisor, captureOpenShell } = pairedStopFixture(true); + supervisor.Config.Labels[label] = "wrong-identity"; + const beforeStop = vi.fn(); + await expect( + withMcpLifecycleLock( + SANDBOX, + () => stopHermesPortableSandboxLifecycle(SANDBOX, lifecycleContext(), beforeStop, deps), + { stateDir: path.join(stateDir, "state") }, + ), + ).rejects.toThrow(/inspect (isolation role|label)/); + expect(beforeStop).not.toHaveBeenCalled(); + expect(openshellMutationCalls(captureOpenShell, "stop")).toHaveLength(0); + }); + + it("rejects a replaced Hermes supervisor after the stop callback without mutation", async () => { + const { deps, supervisor, captureOpenShell } = pairedStopFixture(true); + await expect( + withMcpLifecycleLock( + SANDBOX, + () => + stopHermesPortableSandboxLifecycle( + SANDBOX, + lifecycleContext(), + () => { + supervisor.Id = "d".repeat(64); + }, + deps, + ), + { stateDir: path.join(stateDir, "state") }, + ), + ).rejects.toThrow("inspect returned another container ID"); + expect(openshellMutationCalls(captureOpenShell, "stop")).toHaveLength(0); + }); + + it.each(["", `${"c".repeat(64)}\n${"d".repeat(64)}`])( + "rejects a missing or ambiguous Hermes supervisor before stop (%j)", + async (inventory) => { + const { deps, podman, captureOpenShell } = pairedStopFixture(true); + const capture = podman.getMockImplementation()!; + podman.mockImplementation((args) => + args[0] === "ps" ? { status: 0, stdout: inventory, stderr: "" } : capture(args), + ); + const beforeStop = vi.fn(); + await expect( + withMcpLifecycleLock( + SANDBOX, + () => stopHermesPortableSandboxLifecycle(SANDBOX, lifecycleContext(), beforeStop, deps), + { stateDir: path.join(stateDir, "state") }, + ), + ).rejects.toThrow("requires exactly one full supervisor container ID"); + expect(beforeStop).not.toHaveBeenCalled(); + expect(openshellMutationCalls(captureOpenShell, "stop")).toHaveLength(0); + }, + ); +}); diff --git a/src/lib/onboard/experimental/hermes-portable-receipt.test.ts b/src/lib/onboard/experimental/hermes-portable-receipt.test.ts index 1e1f75ac5c7..5686f7a1973 100644 --- a/src/lib/onboard/experimental/hermes-portable-receipt.test.ts +++ b/src/lib/onboard/experimental/hermes-portable-receipt.test.ts @@ -152,7 +152,7 @@ function socketAuthority(): PodmanSocketAuthority { function openshellExecutableAuthority(): HermesPortableOpenShellExecutableAuthority { return { - version: "0.0.116", + version: "0.1.7", executable: { executablePath: "/usr/bin/openshell", device: "1", @@ -422,20 +422,57 @@ describe("Hermes portable receipt identity", () => { expect(fs.existsSync(path.join(directory, "pending.json"))).toBe(false); }); - it("rejects an OpenShell 0.0.101 receipt before writing a stage (#9211)", () => { - const receipt = pending(); - const staleVersion = { - ...receipt, - openshellExecutableAuthority: { - ...receipt.openshellExecutableAuthority, - version: "0.0.101", - }, - }; - const directory = hermesPortableReceiptDirectory(SANDBOX, stateDir); + it.each(["0.0.101", "0.1.5", "0.1.8"])( + "rejects an OpenShell %s receipt before writing a stage (#9211)", + (version) => { + const receipt = pending(); + const staleVersion = { + ...receipt, + openshellExecutableAuthority: { + ...receipt.openshellExecutableAuthority, + version, + }, + }; + const directory = hermesPortableReceiptDirectory(SANDBOX, stateDir); - expect(() => publish(staleVersion as never)).toThrow("invalid OpenShell executable authority"); - expect(fs.existsSync(path.join(directory, "pending.json"))).toBe(false); - }); + expect(() => publish(staleVersion as never)).toThrow( + "invalid OpenShell executable authority", + ); + expect(fs.existsSync(path.join(directory, "pending.json"))).toBe(false); + }, + ); + + it.each([ + { + schema: 5, + prepare: () => publish(pending()), + target: (directory: string) => hermesPortableReceiptInternals.phasePath(directory, "pending"), + error: "invalid OpenShell executable authority", + }, + { + schema: 8, + prepare: () => { + publishActiveReceipt(); + publishSuccessor(); + }, + target: (directory: string) => hermesPortableReceiptInternals.successorPath(directory), + error: "invalid schema-8 successor authority", + }, + ])( + "rejects a retained 0.1.5 schema-$schema receipt without rewriting it", + ({ prepare, target: receiptPath, error }) => { + const directory = hermesPortableReceiptDirectory(SANDBOX, stateDir); + prepare(); + const target = receiptPath(directory); + const stored = JSON.parse(fs.readFileSync(target, "utf8")); + stored.openshellExecutableAuthority.version = "0.1.5"; + const previousBytes = Buffer.from(`${JSON.stringify(stored)}\n`); + fs.chmodSync(target, 0o600); + fs.writeFileSync(target, previousBytes); + expect(() => readHermesPortableLifecycleReceipt(SANDBOX, stateDir)).toThrow(error); + expect(fs.readFileSync(target)).toEqual(previousBytes); + }, + ); it("keeps the receipt root usable beyond eight independent Hermes sandboxes (#9203)", () => { const root = hermesPortableReceiptRoot(stateDir); diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts b/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts index bd80898a2e5..7a4ef2a4c65 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle-authority.test.ts @@ -345,6 +345,8 @@ describe("portable demo lifecycle authority", () => { "label=openshell.ai/sandbox-name=alpha", "--filter", "label=openshell.ai/sandbox-workspace=default", + "--filter", + "label!=openshell.ai/isolation-role=supervisor", "--format", "{{.ID}}", ], diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts b/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts index 91d85afc2ec..e045d8cd865 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle-identity.test.ts @@ -10,6 +10,10 @@ import type { PodmanSocketAuthority, PodmanSocketAuthorityDeps } from "../../ada import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; import { installPortableDemoSandboxLifecycle, + listPortableDemoSandboxLifecycleReceipts, + preparePortableDemoSandboxRemoval, + portableDemoLifecycleInternals, + stopPortableDemoSandboxLifecycle, type PortableDemoLifecycleDeps, recoverPortableDemoSandboxLifecycle as recoverPortableDemoSandboxLifecycleUnchecked, } from "./portable-demo-lifecycle"; @@ -83,6 +87,9 @@ function temporaryStateDir(): string { function createPodman() { let sandboxId = SANDBOX_ID; + let isolationRole: string | undefined; + let companion = false; + let duplicateWorkload = false; let managedLabel = "true"; let sandboxNameLabel = "alpha"; let sandboxNamespaceLabel = ""; @@ -96,7 +103,16 @@ function createPodman() { case "version": return { status: 0, stdout: JSON.stringify({ Server: { Version: "5.6.1" } }) }; case "ps": - return { status: 0, stdout: `${CONTAINER_ID}\n` }; + return { + status: 0, + stdout: + CONTAINER_ID + + "\n" + + (duplicateWorkload || + (companion && !command.includes("label!=openshell.ai/isolation-role=supervisor")) + ? "b".repeat(64) + "\n" + : ""), + }; case "inspect": return { status: 0, @@ -106,6 +122,9 @@ function createPodman() { Name: containerName, Config: { Labels: { + ...(isolationRole === undefined + ? {} + : { "openshell.ai/isolation-role": isolationRole }), "openshell.managed": managedLabel, "openshell.ai/sandbox-id": sandboxId, "openshell.ai/sandbox-name": sandboxNameLabel, @@ -125,6 +144,15 @@ function createPodman() { }); return { podman, + setIsolationRole(value: string | undefined) { + isolationRole = value; + }, + addCompanion() { + companion = true; + }, + addDuplicateWorkload() { + duplicateWorkload = true; + }, setSandboxId(value: string) { sandboxId = value; containerName = `openshell-default--alpha-${value}`; @@ -268,3 +296,436 @@ describe("portable demo OpenShell container identity", () => { expectRecoveryIdentityRefusal(stateDir, runtime); }); }); + +describe("portable receipt workload discovery", () => { + it("installs the workload receipt when a supervisor shares its labels", () => { + const stateDir = temporaryStateDir(); + const runtime = createPodman(); + runtime.setIsolationRole("sandbox"); + runtime.addCompanion(); + expect(() => installReceipt(stateDir, runtime.podman)).not.toThrow(); + expect(runtime.podman).toHaveBeenCalledWith( + expect.arrayContaining(["update", "--restart=unless-stopped", CONTAINER_ID]), + expect.any(Object), + ); + }); + + it("retains installation for legacy workloads without an isolation role", () => { + const runtime = createPodman(); + expect(() => installReceipt(temporaryStateDir(), runtime.podman)).not.toThrow(); + }); + + it.each(["supervisor", "unknown", ""])("rejects workload role %j before mutation", (role) => { + const runtime = createPodman(); + runtime.setIsolationRole(role); + expect(() => installReceipt(temporaryStateDir(), runtime.podman)).toThrow( + "OpenShell identity does not match", + ); + expect(runtime.podman.mock.calls.some(([args]) => args.includes("update"))).toBe(false); + }); + + it("rejects duplicate workloads after excluding the supervisor", () => { + const runtime = createPodman(); + runtime.addDuplicateWorkload(); + expect(() => installReceipt(temporaryStateDir(), runtime.podman)).toThrow( + "requires one exact Podman container", + ); + expect(runtime.podman.mock.calls.some(([args]) => args.includes("update"))).toBe(false); + }); +}); + +function pairRemovalFixture() { + const stateDir = temporaryStateDir(); + installReceipt(stateDir, createPodman().podman); + const receipt = listPortableDemoSandboxLifecycleReceipts(stateDir)[0]!; + const supervisorId = "b".repeat(64); + const unrelatedId = "c".repeat(64); + const labels = { + "openshell.managed": "true", + "openshell.ai/sandbox-id": SANDBOX_ID, + "openshell.ai/sandbox-name": "alpha", + "openshell.ai/sandbox-namespace": "", + "openshell.ai/sandbox-workspace": "default", + }; + const record = (id: string, name: string, role: string) => ({ + Id: id, + Name: name, + Config: { + Labels: { ...labels, "openshell.ai/isolation-role": role } as Record, + }, + State: { Running: true, Status: "running" }, + }); + const workload = record(CONTAINER_ID, "openshell-default--alpha-" + SANDBOX_ID, "sandbox"); + const supervisor = record(supervisorId, "openshell-supervisor-" + SANDBOX_ID, "supervisor"); + const records = new Map([ + [CONTAINER_ID, workload], + [supervisorId, supervisor], + [unrelatedId, record(unrelatedId, "unrelated", "sandbox")], + ]); + const index = new Set([CONTAINER_ID, supervisorId]); + const handlers: Record< + string, + (args: readonly string[]) => { + status: number; + stdout?: string; + stderr?: string; + } + > = { + ps: () => ({ status: 0, stdout: [...index].filter((id) => records.has(id)).join("\n") }), + inspect: (args) => { + const found = records.get(args[1]!); + return found + ? { status: 0, stdout: JSON.stringify([found]) } + : { status: 125, stderr: "no such container" }; + }, + rm: (args) => { + records.delete(args[2]!); + return { status: 0 }; + }, + }; + const unexpected = (args: readonly string[]): never => { + throw new Error("Unexpected fixture command: " + args.join(" ")); + }; + const podman = vi.fn((args: readonly string[]) => (handlers[args[0]!] ?? unexpected)(args)); + const assertRuntimeAuthority = vi.fn(); + return { + stateDir, + receipt, + supervisorId, + unrelatedId, + workload, + supervisor, + records, + index, + podman, + assertRuntimeAuthority, + prepare: () => + preparePortableDemoSandboxRemoval( + receipt, + { + assertRuntimeAuthority, + dockerHost: "unix://" + SOCKET_PATH, + podman, + }, + stateDir, + ), + removals: () => podman.mock.calls.filter(([args]) => args[0] === "rm").map(([args]) => args), + }; +} + +describe("portable receipt paired removal", () => { + it("removes the validated supervisor then workload and preserves unrelated resources", () => { + const f = pairRemovalFixture(); + const prepared = f.prepare(); + expect(prepared.present).toBe(true); + prepared.removeAndVerify(); + expect(() => prepared.verifyAbsent()).not.toThrow(); + expect(f.removals()).toEqual([ + ["rm", "--force", f.supervisorId], + ["rm", "--force", CONTAINER_ID], + ]); + expect([...f.records.keys()]).toEqual([f.unrelatedId]); + }); + + it("retains removal of a legacy workload without a companion or role label", () => { + const f = pairRemovalFixture(); + f.records.delete(f.supervisorId); + delete f.workload.Config.Labels["openshell.ai/isolation-role"]; + f.prepare().removeAndVerify(); + expect(f.removals()).toEqual([["rm", "--force", CONTAINER_ID]]); + }); + + it("removes a receipt-bound companion after the workload was already removed", () => { + const f = pairRemovalFixture(); + f.records.delete(CONTAINER_ID); + f.prepare().removeAndVerify(); + expect(f.removals()).toEqual([["rm", "--force", f.supervisorId]]); + }); + + it.each([ + ["openshell.ai/isolation-role", "sandbox"], + ["openshell.ai/sandbox-id", "another-sandbox"], + ["openshell.ai/sandbox-name", "another-name"], + ["openshell.ai/sandbox-namespace", "another-namespace"], + ["openshell.ai/sandbox-workspace", "another-workspace"], + ["openshell.managed", "false"], + ])("rejects a companion with changed %s before removal", (label, value) => { + const f = pairRemovalFixture(); + f.supervisor.Config.Labels[label] = value; + expect(() => f.prepare()).toThrow("replaced or ambiguous container"); + expect(f.removals()).toEqual([]); + }); + + it("rejects a companion whose engine name does not match its sandbox ID", () => { + const f = pairRemovalFixture(); + f.supervisor.Name = "unrelated"; + expect(() => f.prepare()).toThrow("replaced or ambiguous container"); + expect(f.removals()).toEqual([]); + }); + + it("rechecks the companion ID before deleting either resource", () => { + const f = pairRemovalFixture(); + const prepared = f.prepare(); + const replacement = "d".repeat(64); + f.records.delete(f.supervisorId); + f.records.set(replacement, { ...f.supervisor, Id: replacement }); + f.index.add(replacement); + expect(() => prepared.removeAndVerify()).toThrow("container presence changed"); + expect(f.removals()).toEqual([]); + }); + + it("does not remove the workload when supervisor removal fails", () => { + const f = pairRemovalFixture(); + const prepared = f.prepare(); + const original = f.podman.getMockImplementation()!; + f.podman.mockImplementation((args) => + args[0] === "rm" ? { status: 125, stderr: "permission denied" } : original(args), + ); + expect(() => prepared.removeAndVerify()).toThrow("still has a recorded Podman container"); + expect(f.removals()).toEqual([["rm", "--force", f.supervisorId]]); + expect(f.records.has(CONTAINER_ID)).toBe(true); + }); + + it("rejects cleanup proof when an unindexed captured companion still exists", () => { + const f = pairRemovalFixture(); + const prepared = f.prepare(); + f.records.delete(CONTAINER_ID); + f.index.delete(f.supervisorId); + expect(() => prepared.verifyAbsent()).toThrow("still has a recorded Podman container"); + expect(f.removals()).toEqual([]); + }); + + it("does not mutate after runtime authority is revoked", () => { + const f = pairRemovalFixture(); + const prepared = f.prepare(); + f.assertRuntimeAuthority.mockImplementation(() => { + throw new Error("socket changed"); + }); + expect(() => prepared.removeAndVerify()).toThrow("socket changed"); + expect(f.removals()).toEqual([]); + }); +}); + +function pairLifecycleFixture() { + const f = pairRemovalFixture(); + let now = 0; + const sleep = vi.fn((milliseconds: number) => { + now += milliseconds; + }); + const detail = { id: SANDBOX_ID, name: "alpha", workspace: "default", phase: "Stopped" }; + const setState = (running: boolean) => { + for (const record of [f.workload, f.supervisor]) { + record.State.Running = running; + record.State.Status = running ? "running" : "exited"; + } + detail.phase = running ? "Ready" : "Stopped"; + }; + setState(false); + const transition = vi.fn((action: string): { status: number | null; error?: Error } => { + setState(action === "start"); + return { status: 0 }; + }); + const handlers: Record< + string, + (args: readonly string[]) => { + status: number | null; + stdout?: string; + error?: Error; + } + > = { + get: () => ({ status: 0, stdout: JSON.stringify(detail) }), + start: () => transition("start"), + stop: () => transition("stop"), + exec: (args) => ({ status: 0, stdout: args.includes("curl") ? "200" : "" }), + }; + const unexpected = (args: readonly string[]): never => { + throw new Error("Unexpected fixture OpenShell command: " + args.join(" ")); + }; + const capture = vi.fn((args: readonly string[], _timeoutMs: number) => + (handlers[args[1]!] ?? unexpected)(args), + ); + const context = { + agent: "openclaw", + gatewayName: "nemoclaw", + lifecycleGeneration: CONTAINER_ID, + openshellDriver: "docker", + }; + const deps: PortableDemoLifecycleDeps = { + platform: "linux", + now: () => now, + sleep, + stateDir: f.stateDir, + env: { HOME: f.stateDir }, + podman: (args) => f.podman(args[0] === "--url" ? args.slice(2) : args), + captureOpenshell: capture, + log: vi.fn(), + hardenSocketDirectory: vi.fn(), + runtimeReadiness: { + ...READINESS, + podmanCapture: () => ({ + status: 0, + stdout: JSON.stringify({ Server: { Version: "5.6.1" } }), + stderr: "", + }), + }, + }; + const beforeStop = vi.fn(); + return { + ...f, + detail, + setState, + transition, + capture, + beforeStop, + sleep, + recover: () => recoverPortableDemoSandboxLifecycleUnchecked("alpha", context, deps), + stop: () => stopPortableDemoSandboxLifecycle("alpha", context, beforeStop, deps), + rawMutations: () => + f.podman.mock.calls.filter(([args]) => ["start", "stop"].includes(args[0]!)), + }; +} + +describe("portable split-container gateway lifecycle", () => { + it("starts through the named gateway with receipt identity checks", () => { + const f = pairLifecycleFixture(); + expect(f.recover()).toEqual({ kind: "already-running" }); + expect(f.transition).toHaveBeenCalledExactlyOnceWith("start"); + expect(f.capture).toHaveBeenCalledWith( + ["sandbox", "start", "-g", "nemoclaw", "--workspace", "default", "--", "alpha"], + 90_000, + ); + expect(f.capture.mock.calls.filter(([args]) => args[1] === "get")).toHaveLength(2); + expect(f.workload.State.Running && f.supervisor.State.Running).toBe(true); + expect(f.rawMutations()).toEqual([]); + }); + + it("stops both containers through OpenShell before returning stopped", () => { + const f = pairLifecycleFixture(); + f.setState(true); + expect(f.stop()).toEqual({ kind: "stopped" }); + expect(f.transition).toHaveBeenCalledExactlyOnceWith("stop"); + expect(f.beforeStop).toHaveBeenCalledOnce(); + expect(f.workload.State.Running || f.supervisor.State.Running).toBe(false); + expect(f.rawMutations()).toEqual([]); + }); + + it("returns already-stopped only after checking the gateway identity and phase", () => { + const f = pairLifecycleFixture(); + expect(f.stop()).toEqual({ kind: "already-stopped" }); + expect(f.transition).not.toHaveBeenCalled(); + expect(f.beforeStop).not.toHaveBeenCalled(); + expect(f.capture.mock.calls.filter(([args]) => args[1] === "get")).toHaveLength(1); + }); + + it("rejects stop transition success when the supervisor remains running", () => { + const f = pairLifecycleFixture(); + f.setState(true); + f.transition.mockImplementation(() => { + f.setState(false); + f.supervisor.State.Running = true; + f.supervisor.State.Status = "running"; + return { status: 0 }; + }); + expect(() => f.stop()).toThrow("did not settle into the exited state"); + expect(f.sleep).toHaveBeenCalled(); + expect(f.rawMutations()).toEqual([]); + expect(f.records.has(f.supervisorId)).toBe(true); + }); + + it("rejects already-stopped success when the supervisor remains running", () => { + const f = pairLifecycleFixture(); + f.supervisor.State.Running = true; + f.supervisor.State.Status = "running"; + expect(() => f.stop()).toThrow("did not settle into the exited state"); + expect(f.sleep).toHaveBeenCalled(); + expect(f.rawMutations()).toEqual([]); + expect(f.records.has(f.supervisorId)).toBe(true); + }); + + it("waits for the supervisor to exit after the workload stops", () => { + const f = pairLifecycleFixture(); + f.supervisor.State.Running = true; + f.supervisor.State.Status = "stopping"; + const advance = f.sleep.getMockImplementation()!; + f.sleep.mockImplementation((milliseconds) => { + advance(milliseconds); + f.supervisor.State.Running = false; + f.supervisor.State.Status = "exited"; + }); + expect(f.stop()).toEqual({ kind: "already-stopped" }); + expect(f.sleep).toHaveBeenCalled(); + expect(f.rawMutations()).toEqual([]); + }); + + it.each(["openshell.ai/sandbox-id", "openshell.ai/isolation-role"])( + "rejects supervisor %s mismatch before stopping", + (label) => { + const f = pairLifecycleFixture(); + f.setState(true); + f.supervisor.Config.Labels[label] = "another-identity"; + expect(() => f.stop()).toThrow(); + expect(f.transition).not.toHaveBeenCalled(); + expect(f.beforeStop).not.toHaveBeenCalled(); + expect(f.rawMutations()).toEqual([]); + }, + ); + + it.each(["id", "name", "workspace"] as const)( + "rejects gateway %s mismatch before mutation", + (key) => { + const f = pairLifecycleFixture(); + f.detail[key] = "another-identity"; + expect(() => f.recover()).toThrow("does not match the portable receipt"); + expect(f.transition).not.toHaveBeenCalled(); + expect(f.rawMutations()).toEqual([]); + }, + ); + + it("rejects changed gateway identity after an accepted start", () => { + const f = pairLifecycleFixture(); + const original = f.transition.getMockImplementation()!; + f.transition.mockImplementation((action) => { + const result = original(action); + f.detail.id = "replacement-id"; + return result; + }); + expect(() => f.recover()).toThrow("does not match the portable receipt"); + expect(f.transition).toHaveBeenCalledOnce(); + expect(f.rawMutations()).toEqual([]); + }); + + it("preserves both containers when the gateway refuses the start", () => { + const f = pairLifecycleFixture(); + f.transition.mockReturnValue({ status: 1 }); + expect(() => f.recover()).toThrow("no raw Podman fallback"); + expect(f.transition).toHaveBeenCalledOnce(); + expect(f.rawMutations()).toEqual([]); + expect(f.records.has(CONTAINER_ID) && f.records.has(f.supervisorId)).toBe(true); + }); + + it("accepts a timed-out stop only when the same sandbox is proven stopped", () => { + const f = pairLifecycleFixture(); + f.setState(true); + f.transition.mockImplementation(() => { + f.setState(false); + return { status: null, error: Object.assign(new Error("timeout"), { code: "ETIMEDOUT" }) }; + }); + expect(f.stop()).toEqual({ kind: "stopped" }); + expect(f.transition).toHaveBeenCalledOnce(); + expect(f.rawMutations()).toEqual([]); + }); + + it("rechecks the receipt after the before-stop hook", () => { + const f = pairLifecycleFixture(); + f.setState(true); + f.beforeStop.mockImplementation(() => { + const file = portableDemoLifecycleInternals.receiptPath("alpha", f.stateDir); + const receipt = JSON.parse(fs.readFileSync(file, "utf8")); + receipt.dashboardPort += 1; + fs.writeFileSync(file, JSON.stringify(receipt)); + }); + expect(() => f.stop()).toThrow("receipt changed"); + expect(f.transition).not.toHaveBeenCalled(); + expect(f.rawMutations()).toEqual([]); + }); +}); diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts b/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts index e2f5bca98ae..be1598f230b 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle.test.ts @@ -487,6 +487,8 @@ describe("portable demo sandbox lifecycle", () => { "label=openshell.ai/sandbox-name=alpha", "--filter", "label=openshell.ai/sandbox-workspace=default", + "--filter", + "label!=openshell.ai/isolation-role=supervisor", "--format", "{{.ID}}", ], diff --git a/src/lib/onboard/experimental/portable-demo-lifecycle.ts b/src/lib/onboard/experimental/portable-demo-lifecycle.ts index 2177cfda238..0f5580af863 100644 --- a/src/lib/onboard/experimental/portable-demo-lifecycle.ts +++ b/src/lib/onboard/experimental/portable-demo-lifecycle.ts @@ -15,11 +15,12 @@ import { hardenPodmanSocketDirectory, type PodmanSocketAuthorityDeps, } from "../../adapters/podman"; -import { ensureConfigDir } from "../../state/config-io"; +import { writePrivateConfig } from "./portable-private-config"; import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; import { parsePortableRuntimeAuthority } from "../../state/onboard/portable-runtime-authority"; import { isPortableExperimentalProfile } from "./portable-profile"; import { + PODMAN_ISOLATION_ROLE_LABEL, PODMAN_MANAGED_LABEL, PODMAN_SANDBOX_CONTAINER_PREFIX, PODMAN_SANDBOX_ID_LABEL, @@ -226,6 +227,7 @@ export interface PreparedPortableDemoSandboxDestroyAuthority { } interface PodmanContainerInspection { + isolatedPair: boolean; containerId: string; sandboxId: string; running: boolean; @@ -396,20 +398,10 @@ const receiptPath = portableDemoReceiptPath; const defaultStateDir = defaultPortableDemoStateDir; function writeReceipt(receipt: PortableDemoLifecycleReceipt, stateDir: string): void { - const filePath = receiptPath(receipt.sandboxName, stateDir); - ensureConfigDir(path.dirname(filePath)); - let file; - try { - file = openRegularFileNoFollow(filePath, { writable: true }); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; - file = openRegularFileNoFollow(filePath, { create: true, mode: 0o600, writable: true }); - } - try { - file.replaceUtf8(`${JSON.stringify(receipt, null, 2)}\n`, 0o600); - } finally { - file.close(); - } + writePrivateConfig( + receiptPath(receipt.sandboxName, stateDir), + `${JSON.stringify(receipt, null, 2)}\n`, + ); } function parseReceipt(value: unknown, sandboxName: string): PortableDemoLifecycleReceipt { @@ -625,6 +617,7 @@ function inspectPodmanContainer( sandboxName: string, podman: NonNullable, result: CommandResult = podman(["inspect", containerId]), + role: "sandbox" | "supervisor" = "sandbox", ): PodmanContainerInspection { requireCommand(result, `Inspecting portable sandbox '${sandboxName}'`); let parsed: unknown; @@ -643,11 +636,15 @@ function inspectPodmanContainer( const sandboxId = labels?.[PODMAN_SANDBOX_ID_LABEL]; const expectedContainerName = typeof sandboxId === "string" - ? `${PODMAN_SANDBOX_CONTAINER_PREFIX}${sandboxName}-${sandboxId}` + ? role === "supervisor" + ? `openshell-supervisor-${sandboxId}` + : `${PODMAN_SANDBOX_CONTAINER_PREFIX}${sandboxName}-${sandboxId}` : null; if ( inspection.Id !== containerId || inspection.Name !== expectedContainerName || + (labels?.[PODMAN_ISOLATION_ROLE_LABEL] !== role && + !(role === "sandbox" && labels?.[PODMAN_ISOLATION_ROLE_LABEL] === undefined)) || labels?.[PODMAN_MANAGED_LABEL] !== "true" || labels?.[PODMAN_SANDBOX_NAME_LABEL] !== sandboxName || labels?.[PODMAN_SANDBOX_NAMESPACE_LABEL] !== PODMAN_SANDBOX_NAMESPACE || @@ -664,7 +661,13 @@ function inspectPodmanContainer( typeof state.Status === "string" && state.Status.trim().length > 0 ? state.Status.trim().toLowerCase() : null; - return { containerId, sandboxId, running: state.Running, status }; + return { + isolatedPair: labels[PODMAN_ISOLATION_ROLE_LABEL] === "sandbox", + containerId, + sandboxId, + running: state.Running, + status, + }; } function isMissingPodmanContainer(result: CommandResult): boolean { @@ -689,6 +692,8 @@ function discoverPodmanContainer( `label=${PODMAN_SANDBOX_NAME_LABEL}=${sandboxName}`, "--filter", `label=${PODMAN_SANDBOX_WORKSPACE_LABEL}=${PODMAN_SANDBOX_WORKSPACE}`, + "--filter", + `label!=${PODMAN_ISOLATION_ROLE_LABEL}=supervisor`, "--format", "{{.ID}}", ]); @@ -826,66 +831,82 @@ export function preparePortableDemoSandboxRemoval( transport.assertRuntimeAuthority(); return current; }; - const inspectPresence = (): boolean => { + const inspectPresence = (): string[] => { const loaded = assertReceiptAndAuthority(); const matches = matchingPortableSandboxContainerIds( receiptRecord.sandboxName, transport.podman, ); - if (matches.length > 1 || (matches.length === 1 && matches[0] !== receiptRecord.containerId)) { + const companions = matches.filter((id) => id !== receiptRecord.containerId); + const ambiguous = (): never => { throw new Error( `Portable demo lifecycle found a replaced or ambiguous container for sandbox '${receiptRecord.sandboxName}'`, ); - } + }; + if (new Set(matches).size !== matches.length || companions.length > 1) ambiguous(); const result = transport.podman(["inspect", receiptRecord.containerId]); - if (isMissingPodmanContainer(result)) { - if (matches.length === 0) return false; - throw new Error( - `Portable demo lifecycle found a replacement container for sandbox '${receiptRecord.sandboxName}'`, + const workloadPresent = !isMissingPodmanContainer(result); + if (workloadPresent) { + requireReceiptOwnedInspection( + loaded, + inspectPodmanContainer( + receiptRecord.containerId, + receiptRecord.sandboxName, + transport.podman, + result, + ), ); } - const inspection = inspectPodmanContainer( - receiptRecord.containerId, - receiptRecord.sandboxName, - transport.podman, - result, - ); - requireReceiptOwnedInspection(loaded, inspection); - if (matches.length !== 1) { + if (workloadPresent !== matches.includes(receiptRecord.containerId)) { throw new Error( `Portable demo lifecycle could not prove the label index for sandbox '${receiptRecord.sandboxName}'`, ); } + for (const id of companions) { + // The receipt binds the sandbox ID. Its 0.1.2 companion must have the + // pinned driver's supervisor name, role and all the same ownership labels. + const companion = (() => { + try { + return inspectPodmanContainer( + id, + receiptRecord.sandboxName, + transport.podman, + transport.podman(["inspect", id]), + "supervisor", + ); + } catch { + return ambiguous(); + } + })(); + if (companion.sandboxId !== receiptRecord.sandboxId) ambiguous(); + } transport.assertRuntimeAuthority(); - return true; + // Remove the supervisor before its workload so its control process stops first. + return [...companions, ...(workloadPresent ? [receiptRecord.containerId] : [])]; }; - const present = inspectPresence(); + const containerIds = inspectPresence(); const revalidate = (): void => { - if (inspectPresence() !== present) { + if (!isDeepStrictEqual(inspectPresence(), containerIds)) { throw new Error( `Portable demo lifecycle container presence changed for sandbox '${receiptRecord.sandboxName}'`, ); } }; + const requireAbsent = (id: string): void => { + const inspected = transport.podman(["inspect", id]); + if (isMissingPodmanContainer(inspected)) return; + requireCommand(inspected, `Verifying portable sandbox '${receiptRecord.sandboxName}' removal`); + throw new Error( + `Portable sandbox '${receiptRecord.sandboxName}' still has a recorded Podman container`, + ); + }; const verifyAbsent = (): void => { assertReceiptAndAuthority(); - const inspected = transport.podman(["inspect", receiptRecord.containerId]); - if (!isMissingPodmanContainer(inspected)) { - if (inspected.status !== 0 || inspected.error) { - requireCommand( - inspected, - `Verifying portable sandbox '${receiptRecord.sandboxName}' removal`, - ); - } - throw new Error( - `Portable sandbox '${receiptRecord.sandboxName}' still has its recorded Podman container`, - ); - } - const remaining = matchingPortableSandboxContainerIds( - receiptRecord.sandboxName, - transport.podman, - ); - if (remaining.length !== 0) { + for (const id of new Set([receiptRecord.containerId, ...containerIds])) requireAbsent(id); + // Never filter out a supervisor or replacement when proving cleanup. + if ( + matchingPortableSandboxContainerIds(receiptRecord.sandboxName, transport.podman).length !== 0 + ) { throw new Error( `Portable demo lifecycle found a replacement container for sandbox '${receiptRecord.sandboxName}'`, ); @@ -893,12 +914,23 @@ export function preparePortableDemoSandboxRemoval( transport.assertRuntimeAuthority(); }; return { - present, + present: containerIds.length !== 0, receipt: receiptRecord, revalidate, removeAndVerify: () => { - assertReceiptAndAuthority(); - if (present) transport.podman(["rm", "--force", receiptRecord.containerId]); + revalidate(); + for (let index = 0; index < containerIds.length; index += 1) { + if (!isDeepStrictEqual(inspectPresence(), containerIds.slice(index))) { + throw new Error( + `Portable demo lifecycle container presence changed for sandbox '${receiptRecord.sandboxName}'`, + ); + } + const id = containerIds[index]!; + transport.podman(["rm", "--force", id]); + // A timed-out removal may have completed. Prove absence before touching + // the next resource; do not retry the mutation or rely on its exit code. + requireAbsent(id); + } verifyAbsent(); }, verifyAbsent, @@ -1495,6 +1527,80 @@ export function removePortableDemoSandboxLifecycleReceipt( removeReceipt(sandboxName, stateDir); } +function transitionPortablePair( + action: "start" | "stop", + receipt: PortableDemoLifecycleReceipt, + context: PortableDemoLifecycleContext, + stateDir: string, + authority: PortablePodmanLifecycleTransport, + capture: NonNullable, + beforeMutation: () => void = () => undefined, +): boolean { + const assertCurrent = (): void => { + if (!isDeepStrictEqual(loadReceipt(receipt.sandboxName, stateDir), receipt)) { + throw new Error("Portable lifecycle receipt changed before the OpenShell transition"); + } + requireCurrentRegistryGeneration(receipt, context.lifecycleGeneration); + authority.assertRuntimeAuthority(); + const inspection = inspectPodmanContainer( + receipt.containerId, + receipt.sandboxName, + authority.podman, + ); + requireReceiptOwnedInspection(receipt, inspection); + if (!inspection.isolatedPair) throw new Error("Portable workload isolation role changed"); + authority.assertRuntimeAuthority(); + }; + const scope = ["-g", context.gatewayName, "--workspace", PODMAN_SANDBOX_WORKSPACE]; + const phase = (): string => { + assertCurrent(); + const result = capture( + ["sandbox", "get", ...scope, "-o", "json", "--", receipt.sandboxName], + COMMAND_TIMEOUT_MS, + ); + if (result.status !== 0 || result.error) + throw new Error("OpenShell sandbox identity query failed"); + let value: unknown; + try { + value = JSON.parse(String(result.stdout ?? "")); + } catch { + throw new Error("OpenShell sandbox identity query returned invalid JSON"); + } + if ( + !isRecord(value) || + value.id !== receipt.sandboxId || + value.name !== receipt.sandboxName || + value.workspace !== PODMAN_SANDBOX_WORKSPACE || + typeof value.phase !== "string" + ) { + throw new Error("OpenShell gateway sandbox identity does not match the portable receipt"); + } + assertCurrent(); + return value.phase.toLowerCase(); + }; + const target = action === "start" ? "ready" : "stopped"; + if (phase() === target) return false; + beforeMutation(); + assertCurrent(); + // The gateway refreshes the authenticated bootstrap and manages both containers. + // A raw workload start cannot perform that transition. + const result = capture( + ["sandbox", action, ...scope, "--", receipt.sandboxName], + action === "start" ? EXEC_READY_TIMEOUT_MS : COMMAND_TIMEOUT_MS, + ); + const observed = phase(); + if (observed !== target || ((result.status !== 0 || result.error) && !isCommandTimeout(result))) { + throw new Error( + "OpenShell sandbox " + + action + + " did not establish " + + target + + "; no raw Podman fallback was attempted", + ); + } + return true; +} + /** * Recover the hidden portable profile after its Podman container or startup session stops. * Remove this temporary recovery path after #8058 supplies the durable provider lifecycle contract. @@ -1542,13 +1648,29 @@ export function recoverPortableDemoSandboxLifecycle( `Portable demo lifecycle refused container '${receipt.containerId}' because its OpenShell sandbox ID changed`, ); } - if (!inspection.running) { + const openshellBinary = deps.openshellBinary ?? commandEnv.NEMOCLAW_OPENSHELL_BIN ?? "openshell"; + const capture = + deps.captureOpenshell ?? + ((args, timeoutMs) => defaultCaptureOpenshell(openshellBinary, args, timeoutMs, commandEnv)); + if (!inspection.running || inspection.isolatedPair) { lifecycleTiming.setContainerAction("started"); inspection = lifecycleTiming.measure("containerStart", () => { - requireCommand( - podman(["start", receipt.containerId]), - `Starting portable sandbox '${sandboxName}'`, - ); + if (inspection.isolatedPair) { + const changed = transitionPortablePair( + "start", + receipt, + context, + stateDir, + authority, + capture, + ); + lifecycleTiming.setContainerAction(changed ? "started" : "reused"); + } else { + requireCommand( + podman(["start", receipt.containerId]), + `Starting portable sandbox '${sandboxName}'`, + ); + } return inspectPodmanContainer(receipt.containerId, sandboxName, podman); }); if (!inspection.running) { @@ -1560,10 +1682,6 @@ export function recoverPortableDemoSandboxLifecycle( lifecycleTiming.setContainerAction("reused"); } - const openshellBinary = deps.openshellBinary ?? commandEnv.NEMOCLAW_OPENSHELL_BIN ?? "openshell"; - const capture = - deps.captureOpenshell ?? - ((args, timeoutMs) => defaultCaptureOpenshell(openshellBinary, args, timeoutMs, commandEnv)); const timing = { now: clock, sleep: deps.sleep ?? defaultSleep }; const gatewayName = context.gatewayName; const execReady = lifecycleTiming.measure("execReady", () => @@ -1773,6 +1891,56 @@ export function stopPortableDemoSandboxLifecycle( return !stopped.running && stopped.status === "exited"; }; + if (inspection.isolatedPair) { + const companions = matchingPortableSandboxContainerIds(sandboxName, authority.podman).filter( + (id) => id !== receipt.containerId, + ); + if (companions.length !== 1) { + throw new Error(`Portable sandbox '${sandboxName}' requires one receipt-bound supervisor`); + } + const supervisorId = companions[0]!; + const inspectSupervisor = (): PodmanContainerInspection => { + const supervisor = inspectPodmanContainer( + supervisorId, + sandboxName, + authority.podman, + authority.podman(["inspect", supervisorId]), + "supervisor", + ); + if (supervisor.sandboxId !== receipt.sandboxId) { + throw new Error(`Portable sandbox '${sandboxName}' supervisor identity changed`); + } + return supervisor; + }; + inspectSupervisor(); + const openshellBinary = + deps.openshellBinary ?? commandEnv.NEMOCLAW_OPENSHELL_BIN ?? "openshell"; + const capture = + deps.captureOpenshell ?? + ((args, timeoutMs) => defaultCaptureOpenshell(openshellBinary, args, timeoutMs, commandEnv)); + const changed = transitionPortablePair( + "stop", + receipt, + context, + stateDir, + authority, + capture, + () => { + beforeStop(); + inspectSupervisor(); + }, + ); + const inspectPairExitedState = (): boolean => { + const workloadExited = inspectExitedState(); + const supervisor = inspectSupervisor(); + return workloadExited && !supervisor.running && supervisor.status === "exited"; + }; + if (!waitFor(STOP_SETTLEMENT_TIMEOUT_MS, timing, inspectPairExitedState)) { + throw new Error(`Portable sandbox '${sandboxName}' did not settle into the exited state`); + } + return { kind: changed ? "stopped" : "already-stopped" }; + } + if (!inspection.running) { if (inspection.status === "exited") return { kind: "already-stopped" }; if (inspection.status !== "stopping") { diff --git a/src/lib/onboard/experimental/portable-host-preparation.ts b/src/lib/onboard/experimental/portable-host-preparation.ts index 0daa06b85c3..349f1e6a2ef 100644 --- a/src/lib/onboard/experimental/portable-host-preparation.ts +++ b/src/lib/onboard/experimental/portable-host-preparation.ts @@ -8,14 +8,13 @@ import os from "node:os"; import path from "node:path"; import { dockerSpawnSync } from "../../adapters/docker/exec"; -import { openRegularFileNoFollow } from "../../adapters/fs/regular-file"; import { assertPodmanSocketAuthority, capturePodmanSocketAuthority, hardenPodmanSocketDirectory, type PodmanSocketAuthority, } from "../../adapters/podman"; -import { ensureConfigDir } from "../../state/config-io"; +import { writePrivateConfig } from "./portable-private-config"; import type { CheckpointPortableRuntimeAuthority } from "../../state/onboard-checkpoint-types"; import { DOCKER_NETWORK_IPAM_INSPECT_FORMAT, @@ -417,26 +416,6 @@ function assertSocketInsideRuntime(runtimeDir: string, socketPath: string): void } } -function writePrivateConfig(filePath: string, value: string): void { - ensureConfigDir(path.dirname(filePath)); - let file; - try { - file = openRegularFileNoFollow(filePath, { writable: true }); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; - file = openRegularFileNoFollow(filePath, { - create: true, - mode: 0o600, - writable: true, - }); - } - try { - file.replaceUtf8(value, 0o600); - } finally { - file.close(); - } -} - function writePortableRuntimeConfig(configHome: string): string { writePrivateConfig( path.join(configHome, "containers", "registries.conf.d", "99-nemoclaw-portable.conf"), diff --git a/src/lib/onboard/experimental/portable-private-config.ts b/src/lib/onboard/experimental/portable-private-config.ts new file mode 100644 index 00000000000..7a57bb2bf8a --- /dev/null +++ b/src/lib/onboard/experimental/portable-private-config.ts @@ -0,0 +1,23 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import path from "node:path"; +import { openRegularFileNoFollow } from "../../adapters/fs/regular-file"; +import { ensureConfigDir } from "../../state/config-io"; + +/** Write portable host configuration through the retained regular-file descriptor. */ +export function writePrivateConfig(filePath: string, value: string): void { + ensureConfigDir(path.dirname(filePath)); + let file; + try { + file = openRegularFileNoFollow(filePath, { writable: true }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + file = openRegularFileNoFollow(filePath, { create: true, mode: 0o600, writable: true }); + } + try { + file.replaceUtf8(value, 0o600); + } finally { + file.close(); + } +} diff --git a/src/lib/onboard/external-component/README.md b/src/lib/onboard/external-component/README.md index 29533d78a7d..b0bc7155ba5 100644 --- a/src/lib/onboard/external-component/README.md +++ b/src/lib/onboard/external-component/README.md @@ -12,7 +12,7 @@ The component supplies policy. OpenShell validates effective policy and enforces This v2 implementation requires authenticated interceptor and middleware APIs, and interceptor-backed provider profiles. -The shared runtime pin is OpenShell 0.0.116. +The shared runtime pin is OpenShell 0.1.7. Qualify the accepted behavior against that release before claiming integration support. Keep runtime qualification and shared version pins in their existing owner. Dynamic registration is the accepted v2 contract. Do not add an insecure transport fallback. diff --git a/src/lib/onboard/external-component/connections.test.ts b/src/lib/onboard/external-component/connections.test.ts index 6a0ed4e58e4..2887bf579b9 100644 --- a/src/lib/onboard/external-component/connections.test.ts +++ b/src/lib/onboard/external-component/connections.test.ts @@ -522,6 +522,10 @@ describe("managed gateway connection configuration", () => { f.write(); const config = fs.readFileSync(f.env.OPENSHELL_GATEWAY_CONFIG!, "utf-8"); const parsed = parseToml(config) as any; + expect(parsed.openshell.drivers.docker).not.toHaveProperty("network_name"); + expect(config).toContain( + `# nemoclaw-external-component-network = "${f.env.OPENSHELL_DOCKER_NETWORK_NAME}"`, + ); const gateway = parsed.openshell.gateway; expect(gateway.provider_profile_sources).toEqual([ { type: "interceptor", name: f.component.componentId }, @@ -557,6 +561,7 @@ describe("managed gateway connection configuration", () => { "allow_insecure_transport = false", "172.30.115.1", "ca-sha256", + 'nemoclaw-external-component-network = "openshell-docker"', ])("refuses to replace changed configuration at %s (#11507)", (setting) => { const f = fixture(); f.write(); diff --git a/src/lib/onboard/extra-placeholder-keys.test.ts b/src/lib/onboard/extra-placeholder-keys.test.ts index f58ab63fbfe..3facd78fc84 100644 --- a/src/lib/onboard/extra-placeholder-keys.test.ts +++ b/src/lib/onboard/extra-placeholder-keys.test.ts @@ -161,13 +161,13 @@ describe("registerExtraPlaceholderProviders", () => { name: "my-sandbox-telegram-bridge", envKey: "TELEGRAM_BOT_TOKEN", token: "telegram-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, { name: "my-sandbox-slack-bridge", envKey: "SLACK_BOT_TOKEN", token: "slack-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, ]; const warnings: string[] = []; @@ -181,7 +181,7 @@ describe("registerExtraPlaceholderProviders", () => { name: "my-sandbox-telegram-bridge", envKey: "TELEGRAM_BOT_TOKEN", token: "telegram-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", additionalCredentials: [ { envKey: "TELEGRAM_BOT_TOKEN_AGENT_A", token: "telegram-token-A" }, ], @@ -190,7 +190,7 @@ describe("registerExtraPlaceholderProviders", () => { name: "my-sandbox-slack-bridge", envKey: "SLACK_BOT_TOKEN", token: "slack-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", additionalCredentials: [{ envKey: "SLACK_BOT_TOKEN_AGENT_B", token: "slack-token-B" }], }, ]); @@ -210,7 +210,7 @@ describe("registerExtraPlaceholderProviders", () => { name: "my-sandbox-telegram-bridge", envKey: "TELEGRAM_BOT_TOKEN", token: "telegram-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, ]; const extraKeys = registerExtraPlaceholderProviders(messagingTokenDefs); @@ -220,7 +220,7 @@ describe("registerExtraPlaceholderProviders", () => { name: "my-sandbox-telegram-bridge", envKey: "TELEGRAM_BOT_TOKEN", token: "telegram-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", additionalCredentials: [{ envKey: "TELEGRAM_BOT_TOKEN_AGENT_MISSING", token: null }], }, ]); @@ -241,7 +241,7 @@ describe("registerExtraPlaceholderProviders", () => { name: "my-sandbox-telegram-bridge", envKey: "TELEGRAM_BOT_TOKEN", token: "telegram-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, ]; const warnings: string[] = []; @@ -254,7 +254,7 @@ describe("registerExtraPlaceholderProviders", () => { name: "my-sandbox-telegram-bridge", envKey: "TELEGRAM_BOT_TOKEN", token: "telegram-token", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", additionalCredentials: [ { envKey: "TELEGRAM_BOT_TOKEN_AGENT_A", token: "telegram-token-A" }, ], diff --git a/src/lib/onboard/gateway-provider-metadata.test.ts b/src/lib/onboard/gateway-provider-metadata.test.ts index 68f1e4e865d..163b0b7f35a 100644 --- a/src/lib/onboard/gateway-provider-metadata.test.ts +++ b/src/lib/onboard/gateway-provider-metadata.test.ts @@ -95,11 +95,11 @@ describe("gateway provider metadata", () => { it("distinguishes exact, missing, incompatible, and indeterminate credential providers", () => { const expected = { name: "alpha-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialKey: "TELEGRAM_BOT_TOKEN", }; const exact = - "Name: alpha-telegram-bridge\nType: nemoclaw-mcp-v1\nCredential keys: TELEGRAM_BOT_TOKEN\nConfig keys: \n"; + "Name: alpha-telegram-bridge\nType: nemoclaw-messaging-v1\nCredential keys: TELEGRAM_BOT_TOKEN\nConfig keys: \n"; expect( inspectGatewayCredentialFamilyProviderBinding(expected, () => ({ status: 0, stdout: exact })), @@ -107,7 +107,7 @@ describe("gateway provider metadata", () => { expect( inspectGatewayCredentialFamilyProviderBinding(expected, () => ({ status: 0, - stdout: exact.replace("Type: nemoclaw-mcp-v1", "Type: generic"), + stdout: exact.replace("Type: nemoclaw-messaging-v1", "Type: generic"), })), ).toEqual({ kind: "collision" }); expect( @@ -282,7 +282,7 @@ describe("gateway provider metadata", () => { inspectGatewayCredentialFamilyProviderBinding( { name: "alpha-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialKey: "TELEGRAM_BOT_TOKEN", }, () => result, diff --git a/src/lib/onboard/gateway-provider-metadata.ts b/src/lib/onboard/gateway-provider-metadata.ts index ba255dad8cf..1f0e00d369f 100644 --- a/src/lib/onboard/gateway-provider-metadata.ts +++ b/src/lib/onboard/gateway-provider-metadata.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { reportsExactProviderNotFound } from "../adapters/openshell/provider-diagnostic-cli"; +import { matchesCredentialFamilyProfileType } from "../messaging/applier/credential-family-profile"; import type { OpenShellProviderMetadata } from "../adapters/openshell/provider-adapter"; import { isValidCliOpenShellProviderIdentifier, @@ -76,7 +77,12 @@ export function matchesGatewayCredentialFamilyProviderBinding( return Boolean( metadata && metadata.name === expected.name && - metadata.type === expected.type && + matchesCredentialFamilyProfileType( + metadata.type, + expected.type, + expected.credentialKey, + metadata.credentialKeys, + ) && metadata.configKeys.length === 0 && metadata.credentialKeys.includes(expected.credentialKey) && metadata.credentialKeys.every( diff --git a/src/lib/onboard/gateway-reuse.test.ts b/src/lib/onboard/gateway-reuse.test.ts index ac012c0d792..9d6fe56755c 100644 --- a/src/lib/onboard/gateway-reuse.test.ts +++ b/src/lib/onboard/gateway-reuse.test.ts @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { afterEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { OPENSHELL_PROBE_TIMEOUT_MS } from "../adapters/openshell/timeouts"; import * as gatewayEnv from "./docker-driver-gateway-env"; @@ -177,6 +177,42 @@ function createDockerDriverReuseApplication( } describe("Docker-driver gateway reuse application", () => { + beforeEach(() => { + const runtime = gatewayEnv.observeConfiguredGatewayHostRuntime({ + platform: "linux", + architecture: "x64", + environment: {}, + }); + vi.spyOn(gatewayEnv, "observeConfiguredGatewayHostRuntime").mockReturnValue({ + ...runtime, + gatewayConfig: { ...runtime.gatewayConfig, driverConfigLayout: "inline-supervisor" }, + }); + }); + + it.each([false, true])( + "checks split-supervisor process drift without requiring a bridge (drift=%s)", + async (drift) => { + const runtime = gatewayEnv.observeConfiguredGatewayHostRuntime(); + vi.mocked(gatewayEnv.observeConfiguredGatewayHostRuntime).mockReturnValue({ + ...runtime, + gatewayConfig: { ...runtime.gatewayConfig, driverConfigLayout: "split-supervisor" }, + }); + const inspectDockerDriverNetwork = vi.fn(() => ({ kind: "absent" as const })); + const getDockerDriverGatewayReuseDrift = vi.fn(() => + drift ? { reason: "runtime environment changed" } : null, + ); + const application = createDockerDriverReuseApplication({ + inspectDockerDriverNetwork, + getDockerDriverGatewayReuseDrift, + }); + await expect(application.refreshDockerDriverGatewayReuseState("healthy")).resolves.toBe( + drift ? "stale" : "healthy", + ); + expect(getDockerDriverGatewayReuseDrift).toHaveBeenCalledOnce(); + expect(inspectDockerDriverNetwork).not.toHaveBeenCalled(); + }, + ); + it("preserves provider-owned readiness without inspecting Docker", async () => { vi.spyOn(gatewayEnv, "configuredRuntimeProviderOwnsHostReadiness").mockReturnValue(true); const resolveOpenShellGatewayBinary = vi.fn(() => "/opt/openshell-gateway"); diff --git a/src/lib/onboard/gateway-reuse.ts b/src/lib/onboard/gateway-reuse.ts index 0fea258429a..38f31b4c711 100644 --- a/src/lib/onboard/gateway-reuse.ts +++ b/src/lib/onboard/gateway-reuse.ts @@ -10,7 +10,10 @@ import type { import { type OpenShellRuntimeSelection } from "../adapters/openshell/command-argv"; import { type GatewayReuseState } from "../state/gateway"; import * as dockerDriverGatewayLaunch from "./docker-driver-gateway-launch"; -import { configuredRuntimeProviderOwnsHostReadiness } from "./docker-driver-gateway-env"; +import { + configuredRuntimeProviderOwnsHostReadiness, + observeConfiguredGatewayHostRuntime, +} from "./docker-driver-gateway-env"; import * as gatewayService from "./docker-driver-gateway-service"; import type { PortProbeResult } from "./preflight"; @@ -160,14 +163,17 @@ export function createDockerDriverGatewayReuseApplication( const desiredEnv = runtimeIdentity?.desiredEnv ?? baseDesiredEnv; const driftBin = resolveDriftGatewayBin(runtimeIdentity, gatewayBin); const identityBin = runtimeIdentity?.identityGatewayBin ?? gatewayBin; + const requiresBridge = + observeConfiguredGatewayHostRuntime().gatewayConfig.driverConfigLayout !== "split-supervisor"; const networkName = desiredEnv.OPENSHELL_DOCKER_NETWORK_NAME; - if (!networkName) { + if (requiresBridge && !networkName) { throw new Error( "NemoClaw cannot verify the OpenShell Docker network because the NemoClaw-managed OpenShell gateway configuration has no network name.", ); } function reconcileManagedGatewayNetwork(): GatewayReuseState { + if (!requiresBridge) return state; const network = inspectNetwork(networkName); if (network.kind === "present") return state; if (network.kind === "absent") { @@ -182,6 +188,7 @@ export function createDockerDriverGatewayReuseApplication( } function verifyNetworkWithoutLifecycleAuthority(): GatewayReuseState { + if (!requiresBridge) return state; const network = inspectNetwork(networkName); if (network.kind === "present") return state; if (network.kind === "absent") { diff --git a/src/lib/onboard/gateway-sandbox-reachability-severity.test.ts b/src/lib/onboard/gateway-sandbox-reachability-severity.test.ts index 9f2a5be4baf..681edb4aca7 100644 --- a/src/lib/onboard/gateway-sandbox-reachability-severity.test.ts +++ b/src/lib/onboard/gateway-sandbox-reachability-severity.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import { prepareNativePodmanGatewayHostRuntime } from "./runtime-provider/podman-runtime-surfaces"; import { formatSandboxBridgeUnreachableMessage, @@ -60,6 +61,11 @@ describe("sandbox bridge reachability severity (#6004)", () => { try { await expect( verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime: prepareNativePodmanGatewayHostRuntime({ + environment: {}, + platform: "linux", + socketPath: "/run/user/1000/podman/podman.sock", + }), autoApplyImpl: () => ({ applied: false, reason: "sudo_unavailable", diff --git a/src/lib/onboard/gateway-sandbox-reachability.test.ts b/src/lib/onboard/gateway-sandbox-reachability.test.ts index 72d8b7d04c4..f37a8251bc3 100644 --- a/src/lib/onboard/gateway-sandbox-reachability.test.ts +++ b/src/lib/onboard/gateway-sandbox-reachability.test.ts @@ -15,6 +15,7 @@ import { PORTABLE_DOCKER_NETWORK_SUBNET, PORTABLE_HOST_GATEWAY_IP, } from "./experimental/portable-profile"; +import { createDockerRuntimeProviderBundle } from "./runtime-provider/docker"; import { prepareNativePodmanGatewayHostRuntime } from "./runtime-provider/podman-runtime-surfaces"; describe("gateway sandbox reachability route modeling", () => { @@ -827,6 +828,36 @@ describe("tryAutoApplyUfwRule (#4265)", () => { }); }); +const podmanGatewayRuntime = prepareNativePodmanGatewayHostRuntime({ + environment: {}, + platform: "linux", + socketPath: "/run/user/1000/podman/podman.sock", +}); + +describe("split Docker supervisor gateway reachability", () => { + it("does not probe or change a retained legacy bridge", async () => { + const gatewayRuntime = createDockerRuntimeProviderBundle().gateway.prepareHostRuntime({ + environment: {}, + platform: "linux", + }); + const reachabilityImpl = vi.fn().mockResolvedValue({ + ok: false, + reason: "tcp_failed", + routeKind: "bridge_gateway", + gatewayIp: "172.18.0.1", + }); + const autoApplyImpl = vi.fn(); + await verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime, + reachabilityImpl, + autoApplyImpl, + autoApplyOptedInImpl: () => true, + }); + expect(reachabilityImpl).not.toHaveBeenCalled(); + expect(autoApplyImpl).not.toHaveBeenCalled(); + }); +}); + describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => { const hostGatewayTcpFailure = { ok: false as const, @@ -849,6 +880,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => { const log = vi.spyOn(console, "log").mockImplementation(() => undefined); try { await verifySandboxBridgeGatewayReachableOrExit(true, { + gatewayRuntime: podmanGatewayRuntime, port: 19080, reachabilityImpl, retryAttempts: 3, @@ -882,6 +914,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => { vi.spyOn(console, "log").mockImplementation(() => undefined); await verifySandboxBridgeGatewayReachableOrExit(true, { + gatewayRuntime: podmanGatewayRuntime, reachabilityImpl, retryAttempts: 3, retryDelayMs: 25, @@ -901,6 +934,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => { try { await expect( verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime: podmanGatewayRuntime, reachabilityImpl, retryAttempts: 3, retryDelayMs: 25, @@ -934,6 +968,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => { try { await expect( verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime: podmanGatewayRuntime, reachabilityImpl, sleepMsImpl, }), @@ -963,6 +998,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit host-gateway retry", () => { try { await expect( verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime: podmanGatewayRuntime, autoApplyOptedInImpl: () => false, reachabilityImpl, retryAttempts: 3, @@ -993,6 +1029,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit UFW auto-apply (#4265)", () const autoApplyImpl = vi.fn(); const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); await verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime: podmanGatewayRuntime, autoApplyImpl, autoApplyOptedInImpl: () => true, reachabilityImpl: () => ({ @@ -1014,6 +1051,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit UFW auto-apply (#4265)", () const autoApplyImpl = vi.fn().mockReturnValue({ applied: true, reason: "applied" }); const log = vi.spyOn(console, "log").mockImplementation(() => undefined); await verifySandboxBridgeGatewayReachableOrExit(true, { + gatewayRuntime: podmanGatewayRuntime, autoApplyImpl, autoApplyOptedInImpl: () => true, reachabilityImpl, @@ -1031,6 +1069,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit UFW auto-apply (#4265)", () const error = vi.spyOn(console, "error").mockImplementation(() => undefined); await expect( verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime: podmanGatewayRuntime, autoApplyImpl, autoApplyOptedInImpl: () => true, reachabilityImpl, @@ -1047,6 +1086,7 @@ describe("verifySandboxBridgeGatewayReachableOrExit UFW auto-apply (#4265)", () const error = vi.spyOn(console, "error").mockImplementation(() => undefined); await expect( verifySandboxBridgeGatewayReachableOrExit(false, { + gatewayRuntime: podmanGatewayRuntime, autoApplyImpl: () => ({ applied: false, reason: "ufw_inactive" }), autoApplyOptedInImpl: () => true, reachabilityImpl: () => tcpFailure, diff --git a/src/lib/onboard/gateway-sandbox-reachability.ts b/src/lib/onboard/gateway-sandbox-reachability.ts index 29198c256c7..fda61be4b0f 100644 --- a/src/lib/onboard/gateway-sandbox-reachability.ts +++ b/src/lib/onboard/gateway-sandbox-reachability.ts @@ -559,6 +559,7 @@ export function formatSandboxBridgeUnreachableMessage( } interface SandboxBridgeVerifierOptions { + gatewayRuntime?: RuntimeProviderGatewayHostRuntime; output?: Pick; skip?: boolean; port?: number; @@ -604,8 +605,16 @@ export async function verifySandboxBridgeGatewayReachableOrExit( ); return; } + const gatewayRuntime = + options.gatewayRuntime ?? observeConfiguredGatewayHostRuntime({ environment: process.env }); + // The split Docker supervisor uses host networking; its workload has no + // network interface. A retained legacy bridge cannot verify this route. + if (gatewayRuntime.gatewayConfig.driverConfigLayout === "split-supervisor") return; const port = options.port ?? GATEWAY_PORT; - const reachability = options.reachabilityImpl ?? isSandboxBridgeGatewayReachable; + const reachability = + options.reachabilityImpl ?? + ((probeOptions: { port: number }) => + isSandboxBridgeGatewayReachable({ ...probeOptions, gatewayRuntime })); const autoApplyOptedIn = options.autoApplyOptedInImpl ?? isUfwAutoApplyOptedIn; const autoApply = options.autoApplyImpl ?? diff --git a/src/lib/onboard/gateway/docker-driver-start.ts b/src/lib/onboard/gateway/docker-driver-start.ts index 1669453c7e4..d266cd7f619 100644 --- a/src/lib/onboard/gateway/docker-driver-start.ts +++ b/src/lib/onboard/gateway/docker-driver-start.ts @@ -317,29 +317,18 @@ export function createDockerDriverGatewayStart( !creation() ) throw new Error("Fresh gateway configuration changed before policy initialization."); - const policy = await ( - deps.initializeNativeProviderPolicy ?? initializeNativeProviderPolicy - )(creationGatewayName, (args) => - captureSanitizedResolvedOpenshellAsync(args, { - ...runtimeOptions, - ignoreError: true, - includeStreams: true, - includeStderr: true, - timeout: 30_000, - outputLimitBytes: 65_536, - }), + await (deps.initializeNativeProviderPolicy ?? initializeNativeProviderPolicy)( + creationGatewayName, + (args) => + captureSanitizedResolvedOpenshellAsync(args, { + ...runtimeOptions, + ignoreError: true, + includeStreams: true, + includeStderr: true, + timeout: 30_000, + outputLimitBytes: 65_536, + }), ); - if (policy === "disabled") { - if ( - deps.gatewayName() !== creationGatewayName || - deps.gatewayPort() !== creationGatewayPort || - !creation() - ) - throw new Error("Fresh gateway configuration changed during policy initialization."); - (output?.warn ?? console.warn)( - "Native local inference is unavailable because provider composition remains disabled. Hosted provider onboarding can continue.", - ); - } }; const servicePortOwnership = deps.createGatewayServicePortOwnership(initialPortCheck, { exitOnFailure, diff --git a/src/lib/onboard/gateway/late-binding.test.ts b/src/lib/onboard/gateway/late-binding.test.ts index 327edbb1820..481dc27e0b5 100644 --- a/src/lib/onboard/gateway/late-binding.test.ts +++ b/src/lib/onboard/gateway/late-binding.test.ts @@ -681,11 +681,11 @@ describe("gateway lifecycle late binding", () => { gatewayPort: 9777, getCachedOpenshellBinary: () => null, getBlueprintMaxOpenshellVersion: () => null, - getInstalledOpenshellVersion: () => "0.0.116", + getInstalledOpenshellVersion: () => "0.1.7", isOpenshellDevVersion: () => false, runCapture: () => "", shouldUseOpenshellDevChannel: () => false, - supportedOpenshellFallbackVersion: "0.0.116", + supportedOpenshellFallbackVersion: "0.1.7", loadDockerDriverGatewayEnv: gatewayBinding.createGatewayEnvLoader( await vi.importActual( "../docker-driver-gateway-env", diff --git a/src/lib/onboard/gateway/network-trust.test.ts b/src/lib/onboard/gateway/network-trust.test.ts new file mode 100644 index 00000000000..e93d5a05276 --- /dev/null +++ b/src/lib/onboard/gateway/network-trust.test.ts @@ -0,0 +1,136 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; + +import { + BAD_PEM, + LEAF_PEM, + PEM, + PRIVATE_KEY, + tmpDir, +} from "../__test-helpers__/corporate-ca-fixtures"; +import { MAX_CORPORATE_CA_BYTES, MAX_CORPORATE_CA_CERTS } from "../corporate-ca-policy"; +import { readGatewayNetworkTrust, stageGatewayNetworkTrust } from "./network-trust"; + +describe("Gateway destination CA snapshots", () => { + it("stores normalized public certificates once and retains previous generations", () => { + const root = tmpDir(); + const first = stageGatewayNetworkTrust(root, PEM); + const readSnapshot = (file: string) => { + const fd = fs.openSync(file, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); + try { + return { metadata: fs.fstatSync(fd), content: fs.readFileSync(fd, "utf8") }; + } finally { + fs.closeSync(fd); + } + }; + const initial = readSnapshot(first.path); + const inode = initial.metadata.ino; + expect(first.sha256).toBe(createHash("sha256").update(PEM).digest("hex")); + expect(initial.metadata.mode & 0o777).toBe(0o444); + expect(fs.statSync(path.dirname(first.path)).mode & 0o777).toBe(0o700); + expect(stageGatewayNetworkTrust(root, PEM)).toEqual(first); + expect(readSnapshot(first.path).metadata.ino).toBe(inode); + const rotated = stageGatewayNetworkTrust(root, PEM + PEM); + expect(rotated.sha256).not.toBe(first.sha256); + expect(readSnapshot(first.path).content).toBe(PEM); + expect(readSnapshot(rotated.path).content).toBe(PEM + PEM); + expect(fs.readdirSync(path.dirname(first.path))).toHaveLength(2); + }); + + it.each([ + ["empty", ""], + ["malformed", BAD_PEM], + ["leaf", LEAF_PEM], + ["key only", PRIVATE_KEY], + ["adjacent key", PEM + PRIVATE_KEY], + ["adjacent text", PEM + "unexpected"], + ["oversized", "x".repeat(MAX_CORPORATE_CA_BYTES + 1)], + ["too many certificates", PEM.repeat(MAX_CORPORATE_CA_CERTS + 1)], + ])("rejects %s material before creating gateway state", (_name, pem) => { + const root = path.join(tmpDir(), "gateway"); + expect(() => stageGatewayNetworkTrust(root, pem)).toThrow(/corporate CA/); + expect(fs.existsSync(root)).toBe(false); + }); + + it.each([0o644, 0o666, 0o400])( + "rejects an existing snapshot with mode %s without replacing it", + (mode) => { + const root = tmpDir(); + const saved = stageGatewayNetworkTrust(root, PEM); + fs.chmodSync(saved.path, mode); + const before = fs.statSync(saved.path); + expect(() => stageGatewayNetworkTrust(root, PEM)).toThrow(/read-only certificate snapshot/); + expect(fs.statSync(saved.path).ino).toBe(before.ino); + expect(fs.statSync(saved.path).mode).toBe(before.mode); + }, + ); + + it("rejects a changed or missing snapshot", () => { + const root = tmpDir(); + const saved = stageGatewayNetworkTrust(root, PEM); + fs.chmodSync(saved.path, 0o600); + fs.writeFileSync(saved.path, PEM + PEM); + fs.chmodSync(saved.path, 0o444); + expect(() => readGatewayNetworkTrust(root, saved.path)).toThrow(/snapshot changed/); + fs.unlinkSync(saved.path); + expect(() => readGatewayNetworkTrust(root, saved.path)).toThrow(/ENOENT/); + }); + + it.each([ + ["symbolic", fs.symlinkSync], + ["hard", fs.linkSync], + ] as const)("rejects a %s link to certificate material", (_kind, createLink) => { + const root = tmpDir(); + const saved = stageGatewayNetworkTrust(root, PEM); + const other = path.join(tmpDir(), "ca.pem"); + fs.renameSync(saved.path, other); + createLink(other, saved.path); + expect(() => readGatewayNetworkTrust(root, saved.path)).toThrow(); + expect(() => stageGatewayNetworkTrust(root, PEM)).toThrow(); + expect(fs.readFileSync(other, "utf8")).toBe(PEM); + }); + + it("rejects a certificate selected from another gateway", () => { + const saved = stageGatewayNetworkTrust(tmpDir(), PEM); + expect(() => readGatewayNetworkTrust(tmpDir(), saved.path)).toThrow( + /owned certificate snapshot/, + ); + }); + + it.each(["state", "trust"])("rejects a writable %s directory", (location) => { + const root = tmpDir(); + const saved = stageGatewayNetworkTrust(root, PEM); + const directory = location === "state" ? root : path.dirname(saved.path); + fs.chmodSync(directory, 0o770); + expect(() => readGatewayNetworkTrust(root, saved.path)).toThrow(/mode 0700/); + expect(() => stageGatewayNetworkTrust(root, PEM)).toThrow(/mode 0700/); + }); + + it("rejects a symlinked trust directory without touching its target", () => { + const root = tmpDir(); + const other = tmpDir(); + fs.symlinkSync(other, path.join(root, "network-trust")); + expect(() => stageGatewayNetworkTrust(root, PEM)).toThrow(/owned directory/); + expect(fs.readdirSync(other)).toEqual([]); + }); + + it("removes only its own temporary file when publication fails", () => { + const root = tmpDir(); + const saved = stageGatewayNetworkTrust(root, PEM); + const retained = path.join(path.dirname(saved.path), ".ca-other.tmp"); + fs.writeFileSync(retained, "another operation", { mode: 0o600 }); + vi.spyOn(fs, "linkSync").mockImplementation(() => { + throw new Error("publication interrupted"); + }); + expect(() => stageGatewayNetworkTrust(root, PEM + PEM)).toThrow("publication interrupted"); + expect(fs.readdirSync(path.dirname(saved.path)).sort()).toEqual( + [path.basename(saved.path), path.basename(retained)].sort(), + ); + expect(fs.readFileSync(saved.path, "utf8")).toBe(PEM); + }); +}); diff --git a/src/lib/onboard/gateway/network-trust.ts b/src/lib/onboard/gateway/network-trust.ts new file mode 100644 index 00000000000..8e472bf3a46 --- /dev/null +++ b/src/lib/onboard/gateway/network-trust.ts @@ -0,0 +1,126 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { createHash, randomBytes } from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; + +import { openRegularFileNoFollow } from "../../adapters/fs/regular-file"; +import { MAX_CORPORATE_CA_BYTES } from "../corporate-ca-policy"; +import { validateCorporateCaPem } from "../corporate-ca-validation"; + +export const NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256_ENV = "NEMOCLAW_GATEWAY_NETWORK_TRUST_SHA256"; +const DIRECTORY = "network-trust"; + +export interface GatewayNetworkTrust { + readonly path: string; + readonly sha256: string; +} + +function ownedDirectory(directory: string, expected?: fs.Stats): fs.Stats { + const current = fs.lstatSync(directory); + if ( + typeof process.getuid !== "function" || + !current.isDirectory() || + current.isSymbolicLink() || + current.uid !== process.getuid() || + (current.mode & 0o777) !== 0o700 || + (expected && (current.dev !== expected.dev || current.ino !== expected.ino)) + ) { + throw new Error( + `Gateway network trust requires an unchanged owned directory with mode 0700: ${directory}`, + ); + } + return current; +} + +/** Only accept a certificate snapshot owned by this gateway, never an arbitrary configured path. */ +export function readGatewayNetworkTrust( + stateDir: string, + configuredPath: unknown, +): GatewayNetworkTrust | null { + if (configuredPath === undefined || configuredPath === null) return null; + const directory = path.join(stateDir, DIRECTORY); + const name = typeof configuredPath === "string" ? path.basename(configuredPath) : ""; + const digest = /^additional-ca-([a-f0-9]{64})\.pem$/u.exec(name)?.[1]; + if ( + !digest || + configuredPath !== path.join(directory, name) || + !path.isAbsolute(configuredPath) + ) { + throw new Error("Gateway network trust must select its owned certificate snapshot"); + } + const state = ownedDirectory(stateDir); + const parent = ownedDirectory(directory); + const file = openRegularFileNoFollow(configuredPath); + try { + const stat = file.stat(); + if (stat.uid !== state.uid || (stat.mode & 0o777) !== 0o444) { + throw new Error("Gateway network trust requires an owned read-only certificate snapshot"); + } + const bytes = file.readBytes(MAX_CORPORATE_CA_BYTES); + const normalized = validateCorporateCaPem(bytes.toString("utf8")); + if ( + !bytes.equals(Buffer.from(normalized, "utf8")) || + createHash("sha256").update(bytes).digest("hex") !== digest + ) { + throw new Error("Gateway network trust certificate snapshot changed"); + } + ownedDirectory(stateDir, state); + ownedDirectory(directory, parent); + return { path: configuredPath, sha256: digest }; + } finally { + file.close(); + } +} + +/** Retain previous generations because a running supervisor can still use their certificate paths. */ +export function stageGatewayNetworkTrust(stateDir: string, pem: string): GatewayNetworkTrust { + const normalized = validateCorporateCaPem(pem); + const sha256 = createHash("sha256").update(normalized, "utf8").digest("hex"); + fs.mkdirSync(stateDir, { recursive: true, mode: 0o700 }); + const state = ownedDirectory(stateDir); + const directory = path.join(stateDir, DIRECTORY); + try { + fs.mkdirSync(directory, { mode: 0o700 }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + } + const parent = ownedDirectory(directory); + const target = path.join(directory, `additional-ca-${sha256}.pem`); + const temporary = path.join( + directory, + `.ca-${process.pid}-${randomBytes(8).toString("hex")}.tmp`, + ); + let descriptor: number | undefined; + let temporaryCreated = false; + try { + descriptor = fs.openSync( + temporary, + fs.constants.O_WRONLY | fs.constants.O_CREAT | fs.constants.O_EXCL | fs.constants.O_NOFOLLOW, + 0o600, + ); + temporaryCreated = true; + fs.writeFileSync(descriptor, normalized, "utf8"); + fs.fchmodSync(descriptor, 0o444); + fs.fsyncSync(descriptor); + fs.closeSync(descriptor); + descriptor = undefined; + ownedDirectory(stateDir, state); + ownedDirectory(directory, parent); + try { + // A hard link publishes complete bytes without replacing an existing generation. + fs.linkSync(temporary, target); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; + } + } finally { + if (descriptor !== undefined) fs.closeSync(descriptor); + if (temporaryCreated) { + ownedDirectory(stateDir, state); + ownedDirectory(directory, parent); + fs.unlinkSync(temporary); + } + } + return readGatewayNetworkTrust(stateDir, target)!; +} diff --git a/src/lib/onboard/hermes/initial-inference-route.test.ts b/src/lib/onboard/hermes/initial-inference-route.test.ts index 5b37910c640..cb8c11b6dfa 100644 --- a/src/lib/onboard/hermes/initial-inference-route.test.ts +++ b/src/lib/onboard/hermes/initial-inference-route.test.ts @@ -1,6 +1,12 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { inspectInferenceProviderProfile } from "../../adapters/openshell/provider-profile"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; +import { + readVerifiedRegisteredCredentialHandle, + verifyRegisteredInferenceProviderAttachment, +} from "../../actions/inference-set/registered"; import { describe, expect, it, vi } from "vitest"; import { patchHermesInferenceConfig } from "../../actions/inference-set"; import { applyHermesManagedRoute } from "../../hermes-managed-route"; @@ -103,6 +109,9 @@ function fixture( events.push("credential"); return issued("v7"); }), + readRegisteredCredentialHandle: vi.fn(async () => { + throw new Error("unexpected registered selection"); + }), resolveAgentConfig: vi.fn(() => target), readSandboxConfig: vi.fn(() => { events.push("read"); @@ -330,14 +339,6 @@ describe("external Hermes initial native-local route", () => { f.entry.endpointUrl = "https://inference.local/v1"; }, }, - { - name: "owned Local NIM", - setup(f: ReturnType) { - delete f.entry.nativeLocalProviderAttachment; - f.entry.provider = "vllm-local"; - f.entry.nimContainer = "nemoclaw-nim-owned"; - }, - }, { name: "hosted", setup(f: ReturnType) { @@ -355,6 +356,33 @@ describe("external Hermes initial native-local route", () => { expect(f.deps.writeSandboxConfig).not.toHaveBeenCalled(); }); + it("projects an attached native NIM route", async () => { + const f = fixture("vllm-local"); + const binding = { ...f.receipt, authMode: "sentinel" as const }; + const receipt = { ...binding, ...nativeLocalIdentity(binding) }; + f.entry.nimContainer = "nemoclaw-nim-owned"; + f.entry.nativeLocalProviderAttachment = receipt; + await f.run(); + expect(f.deps.readCredentialHandle).toHaveBeenCalledExactlyOnceWith( + "alpha", + "gateway-a", + receipt, + ); + expect(f.config.model).toMatchObject({ base_url: receipt.endpointUrl, api_key: issued("v7") }); + expect(f.config.providers["vllm-local"].api_key).toBe(issued("v7")); + expect(f.events).toEqual(["lock", "credential", "read", "write", "restart", "unlock"]); + }); + + it("rejects legacy NIM before reading credentials or changing config", async () => { + const f = fixture("vllm-local"); + f.entry.nimContainer = "nemoclaw-nim-owned"; + delete f.entry.nativeLocalProviderAttachment; + await expect(f.run()).rejects.toThrow("does not match"); + expect(f.events).toEqual([]); + expect(f.deps.readCredentialHandle).not.toHaveBeenCalled(); + expect(f.deps.writeSandboxConfig).not.toHaveBeenCalled(); + }); + it.each(["name", "agent", "gatewayName", "provider", "model", "endpointUrl"] as const)( "refuses a mismatched registered %s before reading credentials or config", async (field) => { @@ -610,6 +638,8 @@ describe("restored Hermes native route projection", () => { }); expect(f.events).toEqual(["lock", "read", "write", "unlock"]); expect(f.deps.writeSandboxConfig).toHaveBeenLastCalledWith("alpha", f.target, f.config, { + gatewayName: "gateway-a", + deferReload: true, restoreNativeLocalCredential: true, }); expect(f.deps.restartGateway).not.toHaveBeenCalled(); @@ -660,6 +690,147 @@ describe("restored Hermes native route projection", () => { }); }); +describe("registered Hermes native route restoration", () => { + function registeredFixture() { + const f = fixture(); + const native = registeredInferenceFixture(true); + Object.assign(f.entry, { + name: native.binding.sandboxName, + gatewayName: native.binding.gatewayName, + provider: native.binding.providerName, + endpointUrl: native.binding.endpointUrl, + preferredInferenceApi: native.binding.inferenceApi, + credentialEnv: native.binding.credentialEnv, + registeredInferenceProviderAttachment: native.binding, + nativeLocalProviderAttachment: undefined, + }); + const read = vi.fn(async () => native.handle); + vi.mocked(f.deps.readRegisteredCredentialHandle).mockImplementation(async (attachment) => { + await verifyRegisteredInferenceProviderAttachment({ + adapter: native.adapter, + expected: attachment, + }); + return readVerifiedRegisteredCredentialHandle({ + attachment, + adapter: native.adapter, + read, + sleep: async () => {}, + }); + }); + const run = (projection: "route" | "credential" = "route") => + createInitialHermesInferenceRoute(f.deps, { + activation: "deferred", + workload: "registered", + projection, + })( + f.entry.name, + f.entry.model!, + native.binding.providerName, + native.binding.gatewayName, + f.revalidate, + ); + return { ...f, native, read, run }; + } + + it.each(["openai-completions", "openai-responses", "anthropic-messages"] as const)( + "refreshes only registered credential slots while preserving %s configuration", + async (api) => { + const f = registeredFixture(); + // The fixture profile must match the selected API as well as its endpoint. + f.native.profile.credentials[0].auth_style = + api === "anthropic-messages" ? "header" : "bearer"; + f.native.profile.credentials[0].header_name = + api === "anthropic-messages" ? "x-api-key" : "authorization"; + f.native.binding = { + ...f.native.binding, + inferenceApi: api, + profileDigest: inspectInferenceProviderProfile(JSON.stringify(f.native.profile))!.digest, + }; + f.entry.registeredInferenceProviderAttachment = f.native.binding; + f.entry.preferredInferenceApi = api; + await f.run(); + f.config.model.context_length = 32768; + f.config.model.temperature = 0.25; + f.config.providers[f.native.binding.providerName].discover_models = false; + f.config.custom_providers.find((row) => row.name === f.native.binding.providerName)!.timeout = + 120; + const restored = structuredClone(f.config); + const fresh = `openshell:resolve:env:s${"b".repeat(64)}_VENDOR_API_KEY`; + f.read.mockResolvedValue(fresh); + await f.run("credential"); + restored.model.api_key = fresh; + restored.providers[f.native.binding.providerName].api_key = fresh; + restored.custom_providers.find((row) => row.name === f.native.binding.providerName)!.api_key = + fresh; + expect(f.config).toEqual(restored); + expect(f.deps.writeSandboxConfig).toHaveBeenLastCalledWith(f.entry.name, f.target, f.config, { + gatewayName: f.native.binding.gatewayName, + deferReload: true, + restoreNativeLocalCredential: true, + }); + expect(f.deps.readCredentialHandle).not.toHaveBeenCalled(); + expect(f.deps.restartGateway).not.toHaveBeenCalled(); + expect(f.native.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.native.adapter.deleteProvider).not.toHaveBeenCalled(); + }, + ); + + it.each([ + "replaced", + "detached", + "wrong-gateway", + "changed-selection", + "raw-handle", + "wrong-environment", + "missing-handle", + "changed-revision", + ])("refuses registered restore with %s before writing or restarting", async (state) => { + const f = registeredFixture(); + await f.run(); + const before = structuredClone(f.config); + vi.mocked(f.deps.writeSandboxConfig).mockClear(); + const changes: Record void> = { + replaced: () => { + f.native.state.id = "other"; + }, + detached: () => { + f.native.state.attached = false; + }, + "wrong-gateway": () => { + f.entry.gatewayName = "other"; + }, + "changed-selection": () => { + f.read.mockImplementation(async () => { + f.entry.model = "other"; + return f.native.handle; + }); + }, + "missing-handle": () => { + f.read.mockRejectedValue(new Error("issued handle unavailable")); + }, + "changed-revision": () => { + f.read.mockImplementation(async () => { + f.native.state.revision++; + return f.native.handle; + }); + }, + "raw-handle": () => { + f.read.mockResolvedValue("raw-secret"); + }, + "wrong-environment": () => { + f.read.mockResolvedValue(`openshell:resolve:env:s${"b".repeat(64)}_OTHER_KEY`); + }, + }; + changes[state](); + await expect(f.run("credential")).rejects.toThrow(); + expect(f.config).toEqual(before); + expect(f.deps.writeSandboxConfig).not.toHaveBeenCalled(); + expect(f.deps.restartGateway).not.toHaveBeenCalled(); + expect(f.native.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.native.adapter.deleteProvider).not.toHaveBeenCalled(); + }); +}); + describe("restored Hermes NVIDIA route", () => { it.each([false, true])( "rebinds NVIDIA credentials with Discord enrolled: %s", diff --git a/src/lib/onboard/hermes/initial-inference-route.ts b/src/lib/onboard/hermes/initial-inference-route.ts index 485c666c157..54377f8693e 100644 --- a/src/lib/onboard/hermes/initial-inference-route.ts +++ b/src/lib/onboard/hermes/initial-inference-route.ts @@ -1,6 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + recordedRegisteredInferenceProviderAttachment, + type RegisteredInferenceProviderAttachment, +} from "../../inference/native-provider/registered"; import { normalizeNativeNvidiaProviderAttachment, NVIDIA_HOSTED_NATIVE_ENDPOINT, @@ -24,6 +28,7 @@ export interface InitialHermesInferenceRouteDeps { gatewayName: string, expected: NativeLocalProviderAttachment, ): Promise; + readRegisteredCredentialHandle(expected: RegisteredInferenceProviderAttachment): Promise; readNvidiaCredentialHandle?( sandboxName: string, gatewayName: string, @@ -40,7 +45,7 @@ export interface InitialHermesInferenceRouteDeps { sandboxName: string, target: AgentConfigTarget, config: ConfigObject, - options?: { restoreNativeLocalCredential?: true }, + options?: { gatewayName: string; deferReload: true; restoreNativeLocalCredential?: true }, ): void; restartGateway(sandboxName: string, gatewayName: string): Promise<{ ok: boolean }>; } @@ -53,7 +58,7 @@ export type InitializeHermesInferenceRoute = ( revalidateSandboxIdentity?: (operation: string) => void, ) => Promise; -/** Project an already attached local route into a publisher-owned Hermes image. */ +/** Project an already attached native route into a publisher-owned Hermes image. */ export function createInitialHermesInferenceRoute( overrides: Pick & Partial, @@ -78,6 +83,7 @@ export function createInitialHermesInferenceRoute( provider === "nvidia-prod" && initial.nativeNvidiaProviderAttachment !== undefined; if ( + initial.registeredInferenceProviderAttachment === undefined && !nvidia && initial.nativeLocalProviderAttachment === undefined && !usesNativeLocalInferenceForAgent( @@ -94,6 +100,7 @@ export function createInitialHermesInferenceRoute( const mismatch = () => new Error("Initial Hermes inference route does not match its registered selection."); const selection = (entry: SandboxEntry | null | undefined) => { + const registered = entry && recordedRegisteredInferenceProviderAttachment(entry); const receipt = nvidia ? normalizeNativeNvidiaProviderAttachment(entry?.nativeNvidiaProviderAttachment) : normalizeNativeLocalProviderAttachment(entry?.nativeLocalProviderAttachment); @@ -107,24 +114,28 @@ export function createInitialHermesInferenceRoute( entry.gatewayName !== gatewayName || entry.provider !== provider || entry.model !== model || - !receipt || - (nvidia - ? entry.endpointUrl != null && entry.endpointUrl !== NVIDIA_HOSTED_NATIVE_ENDPOINT - : !local || - local.agent !== "hermes" || - local.sandboxName !== sandboxName || - local.gatewayName !== gatewayName || - local.provider !== provider || - !nativeLocalAttachmentMatchesSelectedEndpoint( - entry.provider, - entry.endpointUrl, - local.endpointUrl, - entry.hostLocalInferenceReceipt, - )) + (!registered && + (!receipt || + (nvidia + ? entry.endpointUrl != null && entry.endpointUrl !== NVIDIA_HOSTED_NATIVE_ENDPOINT + : !local || + local.agent !== "hermes" || + local.sandboxName !== sandboxName || + local.gatewayName !== gatewayName || + local.provider !== provider || + !nativeLocalAttachmentMatchesSelectedEndpoint( + entry.provider, + entry.endpointUrl, + local.endpointUrl, + entry.hostLocalInferenceReceipt, + )))) ) throw mismatch(); return { receipt, + registered: registered ?? undefined, + endpointUrl: + registered?.endpointUrl ?? (nvidia ? NVIDIA_HOSTED_NATIVE_ENDPOINT : local!.endpointUrl), hostLocalInferenceReceipt: entry.hostLocalInferenceReceipt ?? null, preferredInferenceApi: entry.preferredInferenceApi ?? null, workload: entry.workload, @@ -143,17 +154,19 @@ export function createInitialHermesInferenceRoute( assertCurrent(`read initial Hermes inference route in sandbox '${sandboxName}'`); const target = deps.resolveAgentConfig(sandboxName); if (target.agentName !== "hermes") throw mismatch(); - const handle = nvidia - ? await deps.readNvidiaCredentialHandle!( - sandboxName, - gatewayName, - selected.receipt as NativeNvidiaProviderAttachment, - ) - : await deps.readCredentialHandle( - sandboxName, - gatewayName, - selected.receipt as NativeLocalProviderAttachment, - ); + const handle = selected.registered + ? await deps.readRegisteredCredentialHandle(selected.registered) + : nvidia + ? await deps.readNvidiaCredentialHandle!( + sandboxName, + gatewayName, + selected.receipt as NativeNvidiaProviderAttachment, + ) + : await deps.readCredentialHandle( + sandboxName, + gatewayName, + selected.receipt as NativeLocalProviderAttachment, + ); assertCurrent(`read Hermes config in sandbox '${sandboxName}'`); const config = deps.readSandboxConfig(sandboxName, target, gatewayName); // Keep the same object: the guarded writer requires the non-enumerable @@ -166,11 +179,10 @@ export function createInitialHermesInferenceRoute( model, selected.preferredInferenceApi, undefined, - nvidia - ? NVIDIA_HOSTED_NATIVE_ENDPOINT - : (selected.receipt as NativeLocalProviderAttachment).endpointUrl, + selected.endpointUrl, true, handle, + selected.registered, ); if (options.projection === "credential") { refreshHermesNativeLocalCredential( @@ -181,19 +193,22 @@ export function createInitialHermesInferenceRoute( baseUrl: route.inferenceBaseUrl, inferenceApi: route.inferenceApi, nativeCredentialHandle: handle, + registeredCredentialEnv: selected.registered?.credentialEnv, }, { recoverStrippedNvidia: nvidia && options.workload === "registered" }, ); } assertCurrent(`apply initial Hermes inference route in sandbox '${sandboxName}'`); if (options.projection === "credential") { - // Restored state belongs to the archived generation. The guard permits - // only this verified credential replacement while sealing those inputs. + // Restored state belongs to the archived generation. The native guard + // validates only this credential replacement before the owning restart. deps.writeSandboxConfig(sandboxName, target, config, { + gatewayName, + deferReload: true, restoreNativeLocalCredential: true, }); } else { - deps.writeSandboxConfig(sandboxName, target, config); + deps.writeSandboxConfig(sandboxName, target, config, { gatewayName, deferReload: true }); } assertCurrent(`restart initial Hermes gateway in sandbox '${sandboxName}'`); // Restart even on a repeated projection: a prior write may have committed @@ -227,6 +242,19 @@ const defaultDeps: Omit = { throw new Error("Initial Hermes inference route requires an issued credential handle."); return handle; }, + readRegisteredCredentialHandle: async (attachment) => { + const owner = + require("../../actions/inference-set/registered") as typeof import("../../actions/inference-set/registered"); + const { createDefaultInferenceSetProviderAdapter } = + require("../../actions/inference-set-provider") as typeof import("../../actions/inference-set-provider"); + const adapter = createDefaultInferenceSetProviderAdapter(); + await owner.verifyRegisteredInferenceProviderAttachment({ adapter, expected: attachment }); + return owner.readVerifiedRegisteredCredentialHandle({ + attachment, + adapter, + sleep: (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)), + }); + }, readNvidiaCredentialHandle: async (sandboxName, gatewayName, expected) => { const handle = await ( require("../../actions/inference-set-provider") as typeof import("../../actions/inference-set-provider") diff --git a/src/lib/onboard/host-service-reachability.test.ts b/src/lib/onboard/host-service-reachability.test.ts index ec98b8dc0c3..c50e2f8a8cf 100644 --- a/src/lib/onboard/host-service-reachability.test.ts +++ b/src/lib/onboard/host-service-reachability.test.ts @@ -9,6 +9,7 @@ import { describe, expect, it, vi } from "vitest"; import { PORTABLE_HOST_GATEWAY_IP } from "./experimental/portable-profile"; +import { observeConfiguredGatewayHostRuntime } from "./docker-driver-gateway-env"; import { prepareNativePodmanGatewayHostRuntime } from "./runtime-provider/podman-runtime-surfaces"; // Mock the docker adapter so the test never loads runner.ts (which requires @@ -32,6 +33,82 @@ function makeNetwork(partial: { subnet?: string; gatewayIp?: string } = {}): { } describe("probeHostServiceSandboxReachability", () => { + it.each([undefined, makeNetwork()])( + "probes native llama.cpp loopback without depending on a retained bridge (%j)", + async (legacyNetwork) => { + const inspectNetworkImpl = vi.fn(() => legacyNetwork); + const runImpl = vi.fn((args: readonly string[]) => ({ + status: args.includes("host.openshell.internal:127.0.0.1") ? 0 : 1, + })); + const result = await probeHostServiceSandboxReachability({ + port: 8081, + nativeSupervisorNetwork: true, + platform: "linux", + inspectNetworkImpl, + usesHostGatewayRouteImpl: () => false, + runImpl, + }); + expect(result).toMatchObject({ ok: true, networkName: "host", gatewayIp: "127.0.0.1" }); + expect(inspectNetworkImpl).not.toHaveBeenCalled(); + expect(runImpl).toHaveBeenCalledWith( + expect.arrayContaining([ + "--network", + "host", + "--cap-drop=ALL", + "--security-opt=no-new-privileges", + "--read-only", + "host.openshell.internal:127.0.0.1", + "nc", + "-zw5", + "8081", + ]), + 15_000, + ); + }, + ); + + it.each([ + { status: 1, stderr: "nc: connection refused", reason: "tcp_failed" }, + { status: 125, stderr: "Docker daemon unavailable", reason: "probe_unavailable" }, + { status: null, stderr: "process timeout", reason: "probe_unavailable" }, + { status: 1, stderr: "nc: bad address", reason: "probe_unavailable" }, + ])("classifies native llama.cpp probe failure $stderr", async ({ status, stderr, reason }) => { + const result = await probeHostServiceSandboxReachability({ + port: 8081, + nativeSupervisorNetwork: true, + platform: "linux", + usesHostGatewayRouteImpl: () => false, + runImpl: () => ({ status, stderr }), + }); + expect(result).toMatchObject({ ok: false, reason, networkName: "host" }); + }); + + it.each(["linux", "darwin"] as const)( + "retains the Desktop host-gateway probe on %s for attached llama.cpp", + async (platform) => { + const inspectNetworkImpl = vi.fn(() => makeNetwork()); + const runImpl = vi.fn((_args: readonly string[]) => ({ status: 0 })); + const result = await probeHostServiceSandboxReachability({ + port: 8081, + nativeSupervisorNetwork: true, + platform, + gatewayRuntime: observeConfiguredGatewayHostRuntime({ + platform, + architecture: "arm64", + environment: {}, + }), + inspectNetworkImpl, + usesHostGatewayRouteImpl: () => true, + runImpl, + }); + expect(result).toMatchObject({ ok: true, networkName: DEFAULT_PROBE_NETWORK }); + expect(inspectNetworkImpl).toHaveBeenCalledWith(DEFAULT_PROBE_NETWORK); + expect(runImpl.mock.calls[0]?.[0]).toEqual( + expect.arrayContaining([DEFAULT_PROBE_NETWORK, "host.openshell.internal:host-gateway"]), + ); + }, + ); + it("returns ok and echoes the probed port when nc connects", async () => { const result = await probeHostServiceSandboxReachability({ port: 4000, @@ -119,6 +196,7 @@ describe("probeHostServiceSandboxReachability", () => { const result = await probeHostServiceSandboxReachability({ port: 11435, + nativeSupervisorNetwork: true, inspectNetworkImpl: () => ({ subnet: "10.89.0.0/24" }), runImpl: (args) => { capturedArgs = args; @@ -158,6 +236,7 @@ describe("probeHostServiceSandboxReachability", () => { gatewayRuntime, platform: "linux", port: 11435, + nativeSupervisorNetwork: true, }); expect(result).toMatchObject({ ok: true, reason: "ok" }); diff --git a/src/lib/onboard/host-service-reachability.ts b/src/lib/onboard/host-service-reachability.ts index dc0567860e4..c55faae47f3 100644 --- a/src/lib/onboard/host-service-reachability.ts +++ b/src/lib/onboard/host-service-reachability.ts @@ -2,19 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 /** - * Sandbox-side reachability probe for a host service exposed on a TCP port. - * - * Issue #3340 / #4564: On Brev VMs (and any Linux host with UFW default-deny), - * a host service — the Ollama auth proxy on 11435, the Model Router on 4000 — - * is unreachable from the sandbox's Docker bridge network even though - * host-side `curl 127.0.0.1:` succeeds. Host-side validation cannot - * detect this because `host.openshell.internal` only resolves inside the - * sandbox network. This probe runs a short-lived container on the same Docker - * network OpenShell uses for sandboxes and performs a TCP connect to the host - * service port, mirroring the exact route the real sandbox takes. A - * `tcp_failed` result means a host firewall is blocking the port; the caller - * can then surface an actionable `ufw allow` remediation before declaring - * onboard successful. + * Check host-service TCP reachability from the caller's runtime network. + * Attached llama.cpp selects native supervisor networking on Linux Docker. + * Existing bridge callers retain their network and firewall diagnostics. */ import { @@ -63,6 +53,8 @@ interface ProbeRunResult { export interface HostServiceReachabilityOptions { port: number; + /** Attached llama.cpp uses the native Docker supervisor's host network. */ + nativeSupervisorNetwork?: boolean; networkName?: string; timeoutSec?: number; probeImage?: string; @@ -96,7 +88,6 @@ function isNameResolutionFailure(detail: string): boolean { export async function probeHostServiceSandboxReachability( opts: HostServiceReachabilityOptions, ): Promise { - const networkName = opts.networkName ?? resolveDockerDriverNetworkName(); const port = opts.port; const timeoutSec = opts.timeoutSec ?? PROBE_TIMEOUT_SEC; const probeImage = opts.probeImage ?? PROBE_IMAGE; @@ -110,7 +101,18 @@ export async function probeHostServiceSandboxReachability( const usesHostGatewayRoute = opts.usesHostGatewayRouteImpl ?? managedGatewayRuntime.network.usesHostGatewayRoute; const runImpl = opts.runImpl ?? managedGatewayRuntime.network.run; - const network = inspectNetwork(networkName); + const supervisorNetwork = + opts.nativeSupervisorNetwork === true && + !managedGatewayRuntime.usesHostGatewayRoute && + !usesHostGatewayRoute() + ? managedGatewayRuntime.network.supervisorNetwork + : undefined; + const networkName = supervisorNetwork + ? supervisorNetwork.name + : (opts.networkName ?? resolveDockerDriverNetworkName()); + const network = supervisorNetwork + ? { gatewayIp: supervisorNetwork.hostAddress, subnet: undefined } + : inspectNetwork(networkName); if (!network) { return { ok: false, @@ -151,6 +153,9 @@ export async function probeHostServiceSandboxReachability( "--pull=missing", "--network", networkName, + ...(supervisorNetwork + ? ["--cap-drop=ALL", "--security-opt=no-new-privileges", "--read-only"] + : []), "--add-host", `${HOST_INTERNAL_NAME}:${hostInternalTarget}`, probeImage, diff --git a/src/lib/onboard/inference-providers/hermes.test.ts b/src/lib/onboard/inference-providers/hermes.test.ts index a6afe2b6092..94316355f18 100644 --- a/src/lib/onboard/inference-providers/hermes.test.ts +++ b/src/lib/onboard/inference-providers/hermes.test.ts @@ -459,7 +459,7 @@ describe("native Hermes Provider setup", () => { ok: true, nativeInference: { endpointUrl: endpoint, credentialEnv: "OPENAI_API_KEY" }, }); - expect(prepareNativeProvider).toHaveBeenCalledExactlyOnceWith("host-secret", endpoint); + expect(prepareNativeProvider).toHaveBeenCalledExactlyOnceWith("host-secret", endpoint, false); expect(deps.inferenceRouteMutator.setInferenceRoute).not.toHaveBeenCalled(); expect(deps.verifyInferenceRoute).not.toHaveBeenCalled(); expect(deps.verifyOnboardInferenceSmoke).not.toHaveBeenCalled(); @@ -473,27 +473,47 @@ describe("native Hermes Provider setup", () => { }, ); - it("reuses exact registered authority noninteractively without another login", async () => { - const prepareNativeProvider = vi.fn(async () => receipt); - const deps = makeDeps({ - getNativeProviderAuthority: () => receipt, - prepareNativeProvider, - isNonInteractive: () => true, - }); - await expect( - setupHermesProviderInference( - { ...makeArgs(endpoint), provider: "hermes-provider" }, - deps as never, - ), - ).resolves.toEqual({ - ok: true, - nativeInference: { endpointUrl: endpoint, credentialEnv: "OPENAI_API_KEY" }, - }); - expect(prepareNativeProvider).toHaveBeenCalledExactlyOnceWith(null, endpoint); - expect(deps.hermesProviderAuth.ensureHermesProviderOAuthCredentials).not.toHaveBeenCalled(); - expect(deps.hermesProviderAuth.ensureHermesProviderApiKeyCredentials).not.toHaveBeenCalled(); - expect(deps.inferenceRouteMutator.setInferenceRoute).not.toHaveBeenCalled(); - }); + it.each([ + { endpoint, omit: false, allowedIps: ["8.8.8.8"] }, + { + endpoint: "http://host.openshell.internal:42103/team/v1", + omit: false, + allowedIps: undefined, + }, + { endpoint: "http://host.openshell.internal:42103/team/v1", omit: true, allowedIps: undefined }, + ])( + "reuses registered $endpoint with omitted URL $omit without another login", + async ({ endpoint, omit, allowedIps }) => { + const definition = hostedNativeProvider("hermes-provider", endpoint)!; + const receipt = { + schemaVersion: 1 as const, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "retained-id", + endpointUrl: endpoint, + ...(allowedIps ? { allowedIps } : {}), + }; + const prepareNativeProvider = vi.fn(async () => receipt); + const deps = makeDeps({ + getNativeProviderAuthority: () => receipt, + prepareNativeProvider, + isNonInteractive: () => true, + }); + await expect( + setupHermesProviderInference( + { ...makeArgs(omit ? null : endpoint), provider: "hermes-provider" }, + deps as never, + ), + ).resolves.toEqual({ + ok: true, + nativeInference: { endpointUrl: endpoint, credentialEnv: "OPENAI_API_KEY" }, + }); + expect(prepareNativeProvider).toHaveBeenCalledExactlyOnceWith(null, endpoint, true); + expect(deps.hermesProviderAuth.ensureHermesProviderOAuthCredentials).not.toHaveBeenCalled(); + expect(deps.hermesProviderAuth.ensureHermesProviderApiKeyCredentials).not.toHaveBeenCalled(); + expect(deps.inferenceRouteMutator.setInferenceRoute).not.toHaveBeenCalled(); + }, + ); it("does not publish success or use the shared route when native registration fails", async () => { const deps = makeDeps({ @@ -554,3 +574,92 @@ describe("Hermes managed-tool profile reuse", () => { expect(deps.hermesProviderAuth.ensureHermesProviderOAuthCredentials).not.toHaveBeenCalled(); }); }); + +it.each([ + { endpointUrl: "http://retained.example/team/v1", registered: "http://8.8.8.8/team/v1" }, + { endpointUrl: "https://8.8.8.8/team/v1", registered: "https://8.8.8.8/team/v1" }, + { + endpointUrl: "http://host.openshell.internal:42103/team/v1", + registered: "http://host.openshell.internal:42103/team/v1", + }, +])( + "restores retained Hermes $endpointUrl through native credential registration", + async ({ endpointUrl, registered }) => { + const definition = hostedNativeProvider("hermes-provider", registered)!; + const receipt = { + schemaVersion: 1 as const, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "retained-id", + endpointUrl: registered, + }; + const prepareNativeProvider = vi.fn(async () => receipt); + const register = vi.fn(async (_sandbox, options) => { + await options.registerInferenceCredential({ + apiKey: "retained-host-key", + credentialEnv: "NOUS_API_KEY", + baseUrl: options.baseUrl, + }); + return { inference_base_url: options.baseUrl, credential_env: "OPENAI_API_KEY" }; + }); + const deps = makeDeps({ + lookup: publicLookup(), + getNativeProviderAuthority: () => undefined, + prepareNativeProvider, + resolveHermesNousApiKey: () => "retained-host-key", + hermesProviderAuth: { + isHermesProviderRegistered: vi.fn(), + ensureHermesProviderApiKeyCredentials: register, + ensureHermesProviderOAuthCredentials: vi.fn(), + }, + }); + const result = await setupHermesProviderInference( + { ...makeArgs(endpointUrl), provider: "hermes-provider" }, + deps as never, + ); + expect(prepareNativeProvider).toHaveBeenCalledExactlyOnceWith( + "retained-host-key", + registered, + true, + ); + expect(result).toEqual({ + ok: true, + nativeInference: { endpointUrl: registered, credentialEnv: "OPENAI_API_KEY" }, + }); + expect(deps.inferenceRouteMutator.setInferenceRoute).not.toHaveBeenCalled(); + expect( + JSON.stringify([result, deps.registry.updateSandbox.mock.calls, deps.log.mock.calls]), + ).not.toContain("retained-host-key"); + }, +); + +it("rejects a login-returned HTTP replacement for a retained Hermes endpoint", async () => { + const prepareNativeProvider = vi.fn(); + const deps = makeDeps({ + getNativeProviderAuthority: () => undefined, + prepareNativeProvider, + normalizeHermesAuthMethod: () => "oauth", + hermesProviderAuth: { + isHermesProviderRegistered: vi.fn(), + ensureHermesProviderApiKeyCredentials: vi.fn(), + ensureHermesProviderOAuthCredentials: async ( + _sandbox: unknown, + options: { registerInferenceCredential: (value: unknown) => Promise }, + ) => + options.registerInferenceCredential({ + apiKey: "host-secret", + credentialEnv: "OPENAI_API_KEY", + baseUrl: "http://different.example/v1", + }), + }, + }); + await expect( + setupHermesProviderInference( + { ...makeArgs("http://host.openshell.internal:42103/v1"), provider: "hermes-provider" }, + deps as never, + ), + ).resolves.toEqual({ retry: "selection" }); + expect(prepareNativeProvider).not.toHaveBeenCalled(); + expect(deps.error).toHaveBeenCalledWith(expect.stringContaining("requires an HTTPS endpoint")); + expect(deps.registry.updateSandbox).not.toHaveBeenCalled(); +}); diff --git a/src/lib/onboard/inference-providers/hermes.ts b/src/lib/onboard/inference-providers/hermes.ts index 57c3d6444cc..e938b9c6687 100644 --- a/src/lib/onboard/inference-providers/hermes.ts +++ b/src/lib/onboard/inference-providers/hermes.ts @@ -7,6 +7,7 @@ import { hostedNativeProvider, canonicalHermesNativeEndpoint, + canonicalRecordedHermesEndpoint, usesNativeHermesEndpoint, } from "../../inference/native-provider/hosted"; import type { HermesInferenceCredentialRegistrar } from "../../hermes-provider-auth"; @@ -42,13 +43,20 @@ export async function setupHermesProviderInference( const authority = deps.getNativeProviderAuthority?.(); const defaultEndpoint = hostedNativeProvider("hermes-provider")!.endpoint; const recordedEndpoint = authority?.endpointUrl ?? (authority ? defaultEndpoint : undefined); + const selectedEndpoint = endpointUrl ?? recordedEndpoint; + const retainedEndpoint = Boolean( + selectedEndpoint && selectedEndpoint.replace(/\/+$/, "") !== defaultEndpoint, + ); const usesNative = provider === "hermes-provider" && usesNativeHermesEndpoint(endpointUrl, authority); if (usesNative && (!deps.prepareNativeProvider || !deps.getNativeProviderAuthority)) { throw new Error("Native Hermes setup is missing its ownership and registration dependencies"); } let resolvedEndpointUrl = endpointUrl; - if (endpointUrl && !usesNative) { + if ( + endpointUrl && + (!usesNative || (retainedEndpoint && endpointUrl.replace(/\/+$/, "") !== recordedEndpoint)) + ) { let parsedEndpoint: URL; try { parsedEndpoint = new URL(endpointUrl); @@ -131,9 +139,15 @@ export async function setupHermesProviderInference( let nativePrepared = false; const registerInferenceCredential: HermesInferenceCredentialRegistrar | undefined = usesNative ? async ({ apiKey, baseUrl }) => { + // A login-returned replacement remains HTTPS-only. Only the exact retained + // endpoint already validated above may keep its HTTP or runtime alias route. + const retainsEndpoint = retainedEndpoint && baseUrl === resolvedEndpointUrl; const receipt = await deps.prepareNativeProvider!( apiKey, - canonicalHermesNativeEndpoint(baseUrl), + retainsEndpoint + ? canonicalRecordedHermesEndpoint(baseUrl) + : canonicalHermesNativeEndpoint(baseUrl), + retainsEndpoint, ); resolvedEndpointUrl = receipt.endpointUrl ?? defaultEndpoint; nativePrepared = true; @@ -193,7 +207,11 @@ export async function setupHermesProviderInference( if (!nativePrepared) { if (shouldPrepareHermesCredentials) throw new Error("Hermes login did not publish a native provider credential"); - const receipt = await deps.prepareNativeProvider!(null, resolvedEndpointUrl!); + const receipt = await deps.prepareNativeProvider!( + null, + resolvedEndpointUrl!, + retainedEndpoint, + ); resolvedEndpointUrl = receipt.endpointUrl ?? defaultEndpoint; } // Gateway-held credentials are verified from the new sandbox after attachment. diff --git a/src/lib/onboard/inference-providers/types.ts b/src/lib/onboard/inference-providers/types.ts index 9b65524e6ec..c2dcb1d7333 100644 --- a/src/lib/onboard/inference-providers/types.ts +++ b/src/lib/onboard/inference-providers/types.ts @@ -212,6 +212,7 @@ export type HermesDeps = CommonDeps & prepareNativeProvider?( apiKey: string | null, endpointUrl: string, + retainedHermesEndpoint?: boolean, ): Promise; hermesProviderAuth: { HERMES_PROVIDER_NAME: string; diff --git a/src/lib/onboard/inference-providers/vllm-local.test.ts b/src/lib/onboard/inference-providers/vllm-local.test.ts index 91b514d6778..057d2c7af06 100644 --- a/src/lib/onboard/inference-providers/vllm-local.test.ts +++ b/src/lib/onboard/inference-providers/vllm-local.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { afterEach, describe, expect, it, vi } from "vitest"; +import * as nim from "../../inference/nim"; import { ManagedVllmInspectionError, UnsafeLegacyVllmBindingsError, @@ -194,3 +195,56 @@ describe("vLLM host fallback after local-provider validation fails", () => { }, ); }); + +describe("Local NIM provider selection", () => { + it("uses its sentinel and owned port without reading managed vLLM or host credentials", async () => { + vi.spyOn(nim, "getServedModelId").mockReturnValue("local/nim-model"); + vi.stubEnv("NGC_API_KEY", "ngc-download-secret"); + vi.stubEnv("OPENAI_API_KEY", "host-openai-secret"); + const getManagedVllmProviderBinding = vi.fn(() => ({ + baseUrl: "http://host.openshell.internal:18000/v1", + apiKey: "managed-vllm-secret", + })); + const validateLocalProvider = vi.fn(() => ({ ok: true as const })); + const getLocalProviderBaseUrl = vi.fn(() => "http://host.openshell.internal:18000/v1"); + const selected = deps({ + getManagedVllmProviderBinding, + validateLocalProvider, + getLocalProviderBaseUrl, + }); + await expect( + setupVllmLocalInference( + { model: "local/nim-model", provider: "vllm-local", localNimSelected: true }, + selected, + ), + ).resolves.toEqual({ done: false }); + expect(selected.upsertProvider).toHaveBeenCalledWith( + "vllm-local", + "openai", + CREDENTIAL_ENV, + nim.getNimInferenceEndpoints().baseUrl, + { [CREDENTIAL_ENV]: "dummy" }, + ); + expect(getManagedVllmProviderBinding).not.toHaveBeenCalled(); + expect(validateLocalProvider).not.toHaveBeenCalled(); + expect(getLocalProviderBaseUrl).not.toHaveBeenCalled(); + }); + + it.each([null, "different-model"])( + "stops before provider mutation when NIM serves %s", + async (observed) => { + vi.spyOn(nim, "getServedModelId").mockReturnValue(observed); + const selected = deps(); + await expect( + setupVllmLocalInference( + { model: "local/nim-model", provider: "vllm-local", localNimSelected: true }, + selected, + ), + ).rejects.toThrow("exit 1"); + expect(selected.upsertProvider).not.toHaveBeenCalled(); + expect(selected.error).toHaveBeenCalledWith( + " Local NVIDIA NIM does not serve the selected model. Onboarding stopped.", + ); + }, + ); +}); diff --git a/src/lib/onboard/inference-providers/vllm-local.ts b/src/lib/onboard/inference-providers/vllm-local.ts index 21f9db3f1ca..16342b5e773 100644 --- a/src/lib/onboard/inference-providers/vllm-local.ts +++ b/src/lib/onboard/inference-providers/vllm-local.ts @@ -6,13 +6,14 @@ import { isLocalProviderProbeOutputHealthy } from "../../inference/serving/model-inventory"; import type { SetupInferenceResult, VllmDeps } from "./types"; +import { getNimInferenceEndpoints, getServedModelId } from "../../inference/nim"; import { ManagedVllmInspectionError, UnsafeLegacyVllmBindingsError, } from "../../inference/serving/vllm-host-local-lifecycle"; export async function setupVllmLocalInference( - args: { model: string; provider: string }, + args: { model: string; provider: string; localNimSelected?: boolean }, deps: VllmDeps, ): Promise<{ done: true; result: SetupInferenceResult } | { done: false }> { const { model, provider } = args; @@ -29,9 +30,13 @@ export async function setupVllmLocalInference( error, } = deps; + if (args.localNimSelected && getServedModelId() !== model) { + error(" Local NVIDIA NIM does not serve the selected model. Onboarding stopped."); + return exitProcess(1); + } let managedBinding: ReturnType; try { - managedBinding = getManagedVllmProviderBinding(); + managedBinding = args.localNimSelected ? null : getManagedVllmProviderBinding(); } catch (bindingError) { error( bindingError instanceof UnsafeLegacyVllmBindingsError || @@ -42,7 +47,9 @@ export async function setupVllmLocalInference( return exitProcess(1); } - const validation = validateLocalProvider(provider); + const validation = args.localNimSelected + ? { ok: true as const } + : validateLocalProvider(provider); if (!validation.ok) { const hostCheck = getLocalProviderHealthCheck(provider); const hostResult = hostCheck @@ -70,7 +77,9 @@ export async function setupVllmLocalInference( return exitProcess(1); } } - const baseUrl = managedBinding?.baseUrl ?? getLocalProviderBaseUrl(provider); + const baseUrl = args.localNimSelected + ? getNimInferenceEndpoints().baseUrl + : (managedBinding?.baseUrl ?? getLocalProviderBaseUrl(provider)); const providerToken = managedBinding?.apiKey ?? "dummy"; // Use a dedicated internal credential env so the gateway does not pick // up the user's host OPENAI_API_KEY for local vLLM. vLLM does not enforce diff --git a/src/lib/onboard/inference-selection-validation.ts b/src/lib/onboard/inference-selection-validation.ts index 48c4b55f18a..ebe9f41d927 100644 --- a/src/lib/onboard/inference-selection-validation.ts +++ b/src/lib/onboard/inference-selection-validation.ts @@ -74,7 +74,7 @@ export interface OpenAiSelectionValidationOptions { requireChatCompletionsToolCalling?: boolean; retryChatCompletionsToolReadiness?: boolean; useNvidiaEndpointProbePayload?: boolean; - /** Provider identity used only for safe, provider-specific diagnostics. */ + /** Provider identity used for bounded HTTP retries and provider-specific diagnostics. */ provider?: string; /** Provider-owned default model used only for safe, provider-specific diagnostics. */ providerDefaultModel?: string; diff --git a/src/lib/onboard/initial-policy-landlock.test.ts b/src/lib/onboard/initial-policy-landlock.test.ts index a935fcbbf11..684a4332f25 100644 --- a/src/lib/onboard/initial-policy-landlock.test.ts +++ b/src/lib/onboard/initial-policy-landlock.test.ts @@ -17,7 +17,7 @@ vi.mock("./openshell-version", async (importOriginal) => ({ import { prepareInitialSandboxCreatePolicy } from "./initial-policy"; it.each( - ["0.0.116", "0.1.5"].flatMap((version) => + ["0.0.116", "0.1.5", "0.1.7"].flatMap((version) => [false, true].map((directGpu) => ({ version, directGpu })), ), )( @@ -52,12 +52,14 @@ it.each( ); it.each([ - { agent: "langchain-deepagents-code", version: "0.0.116", mode: "strict" }, + { agent: "langchain-deepagents-code", version: "0.0.116", mode: "unchanged" }, { agent: "langchain-deepagents-code", version: "0.1.3", mode: "hard_requirement" }, { agent: "langchain-deepagents-code", version: "0.1.5", mode: "hard_requirement" }, - { agent: "pi", version: "0.0.116", mode: "strict" }, + { agent: "langchain-deepagents-code", version: "0.1.7", mode: "hard_requirement" }, + { agent: "pi", version: "0.0.116", mode: "unchanged" }, { agent: "pi", version: "0.1.3", mode: "hard_requirement" }, { agent: "pi", version: "0.1.5", mode: "hard_requirement" }, + { agent: "pi", version: "0.1.7", mode: "hard_requirement" }, ])("preserves required Landlock enforcement for $agent on $version", ({ agent, version, mode }) => { runtime.version = version; const source = path.resolve( @@ -71,7 +73,10 @@ it.each([ try { const effective = YAML.parse(fs.readFileSync(policy.policyPath, "utf8")); const baseline = YAML.parse(original); - expect(effective.landlock).toEqual({ ...baseline.landlock, compatibility: mode }); + expect(effective.landlock).toEqual({ + ...baseline.landlock, + compatibility: mode === "unchanged" ? baseline.landlock.compatibility : mode, + }); expect(effective.filesystem_policy).toEqual(baseline.filesystem_policy); expect(fs.readFileSync(source, "utf8")).toBe(original); } finally { diff --git a/src/lib/onboard/initial-policy-real-policy.test.ts b/src/lib/onboard/initial-policy-real-policy.test.ts index bf305bc9445..db51552dd05 100644 --- a/src/lib/onboard/initial-policy-real-policy.test.ts +++ b/src/lib/onboard/initial-policy-real-policy.test.ts @@ -171,8 +171,18 @@ describe("initial sandbox policy real preset merge", () => { expect.objectContaining({ host: "integrate.api.nvidia.com", allowed_ips: ["8.8.8.8"], - tls: "terminate", + port: 443, + protocol: "rest", enforcement: "enforce", + rules: [ + { allow: { method: "GET", path: "/v1/models" } }, + { + allow: { + method: "POST", + path: api === "openai-responses" ? "/v1/responses" : "/v1/chat/completions", + }, + }, + ], }), ]); expect(policy.network_policies).not.toHaveProperty("nvidia"); @@ -231,6 +241,7 @@ describe("initial sandbox policy real preset merge", () => { { path: "/usr/bin/node" }, { path: "/opt/hermes/.venv/bin/python" }, { path: "/opt/hermes/.venv/bin/python3" }, + { path: "/usr/bin/python3.13" }, { path: "/opt/venv/bin/python3" }, { path: "/usr/local/bin/curl" }, { path: "/usr/bin/curl" }, @@ -502,6 +513,19 @@ describe("initial sandbox policy real preset merge", () => { expect(endpoints.every((endpoint) => endpoint.credential_binding === undefined)).toBe(true); }); + it("grants Hermes DRM name discovery without GPU device access", () => { + const policy = readPreparedPolicy( + prepareInitialSandboxCreatePolicy(repoPath("agents", "hermes", "policy-additions.yaml"), [], { + agentName: "hermes", + }), + ); + const readOnly = policy.filesystem_policy?.read_only ?? []; + const readWrite = policy.filesystem_policy?.read_write ?? []; + expect(readOnly).toContain("/sys/class/drm"); + expect([...readOnly, ...readWrite]).not.toEqual(expect.arrayContaining(["/dev/dri"])); + expect(readWrite).not.toContain("/sys/class/drm"); + }); + it.each(shippingPolicyCases)( "prepares $agent policy $path with writable PTY devices but not their symlink", (policyCase) => { diff --git a/src/lib/onboard/lifecycle-contracts.md b/src/lib/onboard/lifecycle-contracts.md index 08045910305..e8b3cc16010 100644 --- a/src/lib/onboard/lifecycle-contracts.md +++ b/src/lib/onboard/lifecycle-contracts.md @@ -12,13 +12,15 @@ Related guides: [`README.md`](README.md) describes package placement, [`machine/ ## Native Model Router resume -Ready OpenClaw Model Router reuse restores the recorded native provider attachment through -`ensureNativeLocalProviderAttached` before recording sandbox success. The sandbox handler holds -its existing sandbox and gateway mutation locks, requires matching endpoint and provider authority, -and revalidates the registered live sandbox identity before provider commands and completion. +Ready Model Router reuse restores the recorded native provider attachment through `ensureNativeLocalProviderAttached` before recording sandbox success. +This applies to OpenClaw, Hermes, Deep Agents Code, and Pi with `nvidia-router` or the provider name configured by the routed blueprint. +`nativeModelRouterEndpoint` verifies the recorded endpoint against that blueprint. +The sandbox handler holds its existing sandbox and gateway mutation locks and requires matching provider authority. +It revalidates the registered live sandbox identity before provider commands and completion. A changed profile, provider identity, sandbox identity, or attachment receipt stops resume. -An already-attached provider is verified without another attach command. Fresh creation retains -its post-create attachment owner. `sandbox-native-router-resume.test.ts` covers this recovery seam. +An already-attached provider is verified without another attach command. +Fresh creation retains its post-create attachment owner. +`sandbox-native-router-resume.test.ts` covers this recovery boundary, including a configured provider name for all four agents. ## Shared vocabulary diff --git a/src/lib/onboard/machine/core-flow-phases.test.ts b/src/lib/onboard/machine/core-flow-phases.test.ts index fd089df6ee8..6dd25fac0fa 100644 --- a/src/lib/onboard/machine/core-flow-phases.test.ts +++ b/src/lib/onboard/machine/core-flow-phases.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it, vi } from "vitest"; +import { customProviderRebuildEntry } from "../../actions/sandbox/rebuild-flow-test-fixtures"; import * as credentialStore from "../../credentials/store"; import { type InferenceEndpointSource, @@ -930,6 +931,9 @@ describe("core onboard flow phases", () => { const setupInference = vi.fn(async () => ({ ok: true as const })); const updateSandboxRegistry = vi.fn(); const getSandboxRegistryEntry = vi.fn((_sandboxName: string) => ({ + ...(registeredProvider === "compatible-endpoint" + ? customProviderRebuildEntry(registeredEndpointUrl) + : {}), name: "my-sandbox", provider: registeredProvider, model: "custom/model", diff --git a/src/lib/onboard/machine/final-flow-phases.test.ts b/src/lib/onboard/machine/final-flow-phases.test.ts index 79db798d036..8b17140d813 100644 --- a/src/lib/onboard/machine/final-flow-phases.test.ts +++ b/src/lib/onboard/machine/final-flow-phases.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import * as initialHermesRoute from "./handlers/initial-hermes-inference-route"; import { context, createPhases, @@ -15,6 +16,28 @@ import { import { advanceTo } from "./result"; describe("final onboard flow phases", () => { + it.each([ + { name: "external Hermes", agent: "hermes", fromImage: "image@sha256:abc", calls: 1 }, + { name: "managed Hermes", agent: "hermes", fromImage: null, calls: 0 }, + { + name: "external Deep Agents", + agent: "langchain-deepagents-code", + fromImage: "image@sha256:abc", + calls: 0, + }, + ])("initializes the Hermes route only for $name", async (selection) => { + const initialize = vi + .spyOn(initialHermesRoute, "initializeExternalHermesInferenceRoute") + .mockResolvedValue(); + const session = createSession(); + session.metadata.fromImage = selection.fromImage; + const [phase] = createPhases("agent_setup", []); + await phase.run( + context({ agent: { name: selection.agent }, session, revalidateSandboxIdentity: vi.fn() }), + ); + expect(initialize).toHaveBeenCalledTimes(selection.calls); + }); + it.each([ { name: "fresh custom image", diff --git a/src/lib/onboard/machine/final-flow-phases.ts b/src/lib/onboard/machine/final-flow-phases.ts index 38acb79f133..5d1d0d358a0 100644 --- a/src/lib/onboard/machine/final-flow-phases.ts +++ b/src/lib/onboard/machine/final-flow-phases.ts @@ -110,6 +110,12 @@ export function createFinalOnboardFlowPhases< hermesToolGateways: context.hermesToolGateways, managedOpenclawStartup: options.managedOpenclawStartup === true, initializeNativeInferenceRoute, + initializeExternalHermesInference: + typeof context.agent === "object" && + context.agent !== null && + "name" in context.agent && + context.agent.name === "hermes" && + Boolean(context.session?.metadata?.fromImage), settleOpenclawStartupBeforeConfiguration, revalidateSandboxIdentity: context.revalidateSandboxIdentity, deps: options.agentSetupDeps, diff --git a/src/lib/onboard/machine/handlers/agent-setup.test.ts b/src/lib/onboard/machine/handlers/agent-setup.test.ts index 9db25f09392..fa0a7430fdb 100644 --- a/src/lib/onboard/machine/handlers/agent-setup.test.ts +++ b/src/lib/onboard/machine/handlers/agent-setup.test.ts @@ -14,7 +14,11 @@ function createDeps(overrides: Partial["deps"]> = let session = createSession(); const calls = { handleAgentSetup: vi.fn(async () => undefined), - context: vi.fn(() => ({ ctx: true, gatewayName: "nemoclaw-19090" })), + context: vi.fn(() => ({ + ctx: true, + gatewayName: "nemoclaw-19090", + getSandbox: vi.fn(() => null), + })), ensureDashboard: vi.fn(() => 18789), persistDashboardPort: vi.fn(), skipped: vi.fn(async (stepName: string) => { @@ -83,6 +87,60 @@ function baseOptions( } describe("handleAgentSetupState", () => { + it("initializes external-image Hermes inference before advancing to policies", async () => { + const initialize = vi.fn(async () => undefined); + const { deps, calls } = createDeps({ initializeExternalHermesInferenceRoute: initialize }); + const identity = vi.fn(); + await handleAgentSetupState({ + ...baseOptions(deps, { name: "hermes", displayName: "Hermes" }), + initializeExternalHermesInference: true, + revalidateSandboxIdentity: identity, + }); + expect(initialize).toHaveBeenCalledWith({ + getSandbox: calls.context.mock.results[0]!.value.getSandbox, + sandboxName: "my-assistant", + gatewayName: "nemoclaw-19090", + model: "model", + provider: "provider", + preferredInferenceApi: "openai-completions", + revalidateSandboxIdentity: identity, + }); + expect(calls.handleAgentSetup.mock.invocationCallOrder[0]).toBeLessThan( + initialize.mock.invocationCallOrder[0], + ); + expect(initialize.mock.invocationCallOrder[0]).toBeLessThan( + calls.ensureDashboard.mock.invocationCallOrder[0], + ); + }); + + it("stops external-image Hermes setup without verified sandbox identity", async () => { + const { deps, calls } = createDeps(); + await expect( + handleAgentSetupState({ + ...baseOptions(deps, { name: "hermes", displayName: "Hermes" }), + initializeExternalHermesInference: true, + }), + ).rejects.toThrow("verified sandbox identity"); + expect(calls.handleAgentSetup).not.toHaveBeenCalled(); + }); + + it("does not advance after external-image Hermes route initialization fails", async () => { + const { deps, calls } = createDeps({ + initializeExternalHermesInferenceRoute: vi.fn(async () => { + throw new Error("route failed"); + }), + }); + await expect( + handleAgentSetupState({ + ...baseOptions(deps, { name: "hermes", displayName: "Hermes" }), + initializeExternalHermesInference: true, + revalidateSandboxIdentity: vi.fn(), + }), + ).rejects.toThrow("route failed"); + expect(calls.ensureDashboard).not.toHaveBeenCalled(); + expect(calls.skipped).not.toHaveBeenCalled(); + }); + it("delegates non-OpenClaw agent setup and skips openclaw", async () => { const { deps, calls } = createDeps(); const agent = { name: "hermes", displayName: "Hermes" }; @@ -104,8 +162,7 @@ describe("handleAgentSetupState", () => { true, session, { - ctx: true, - gatewayName: "nemoclaw-19090", + ...calls.context.mock.results[0]!.value, revalidateSandboxIdentity, prepareHermesInferenceForward: expect.any(Function), }, diff --git a/src/lib/onboard/machine/handlers/agent-setup.ts b/src/lib/onboard/machine/handlers/agent-setup.ts index bcf515c48a1..80ecfcf8e9d 100644 --- a/src/lib/onboard/machine/handlers/agent-setup.ts +++ b/src/lib/onboard/machine/handlers/agent-setup.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { initializeExternalHermesInferenceRoute } from "./initial-hermes-inference-route"; import type { Session, SessionUpdates } from "../../../state/onboard-session"; import { initializeOpenclawInferenceRoute as initializeDefaultOpenclawInferenceRoute } from "../../openclaw/initial-inference-route"; import { advanceTo, type OnboardStateTransitionResult } from "../result"; @@ -17,9 +18,11 @@ export interface AgentSetupStateOptions { hermesToolGateways: string[]; managedOpenclawStartup?: boolean; initializeNativeInferenceRoute?: boolean; + initializeExternalHermesInference?: boolean; settleOpenclawStartupBeforeConfiguration?: boolean; revalidateSandboxIdentity?: (operation: string) => void; deps: { + initializeExternalHermesInferenceRoute?: typeof initializeExternalHermesInferenceRoute; handleAgentSetup( sandboxName: string, model: string, @@ -29,7 +32,10 @@ export interface AgentSetupStateOptions { session: Session | null, context: unknown, ): Promise; - agentSetupContext(): { gatewayName: string }; + agentSetupContext(): { + gatewayName: string; + getSandbox: Parameters[0]["getSandbox"]; + }; ensureAgentDashboardForward( sandboxName: string, agent: Agent | null, @@ -101,6 +107,7 @@ export async function handleAgentSetupState({ hermesToolGateways, managedOpenclawStartup = false, initializeNativeInferenceRoute = false, + initializeExternalHermesInference = false, settleOpenclawStartupBeforeConfiguration = false, revalidateSandboxIdentity, deps, @@ -127,6 +134,9 @@ export async function handleAgentSetupState({ }; if (agent) { + if (initializeExternalHermesInference && !revalidateSandboxIdentity) { + throw new Error("Initial Hermes inference route requires verified sandbox identity."); + } let forwardHandoff: Promise | undefined; const ensureForward = (): Promise => (forwardHandoff ??= (async () => { @@ -151,6 +161,19 @@ export async function handleAgentSetupState({ } }, }); + if (initializeExternalHermesInference) { + await (deps.initializeExternalHermesInferenceRoute ?? initializeExternalHermesInferenceRoute)( + { + sandboxName, + gatewayName: agentSetupContext.gatewayName, + getSandbox: agentSetupContext.getSandbox, + model, + provider, + preferredInferenceApi, + revalidateSandboxIdentity: revalidateSandboxIdentity!, + }, + ); + } await ensureForward(); session = await deps.recordStepSkipped("openclaw"); return { session, stateResult: advanceTo("policies", { metadata: { state: "agent_setup" } }) }; diff --git a/src/lib/onboard/machine/handlers/initial-hermes-inference-route.test.ts b/src/lib/onboard/machine/handlers/initial-hermes-inference-route.test.ts new file mode 100644 index 00000000000..b5955d6c134 --- /dev/null +++ b/src/lib/onboard/machine/handlers/initial-hermes-inference-route.test.ts @@ -0,0 +1,155 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import type { SandboxEntry } from "../../../state/registry"; +import type { ConfigObject } from "../../../security/credential-filter"; +import { patchHermesInferenceConfig } from "../../../actions/inference-set"; +import { initializeExternalHermesInferenceRoute } from "./initial-hermes-inference-route"; + +function fixture() { + const entry = { + agent: "hermes", + provider: "compatible-endpoint", + model: "selected-model", + gatewayName: "selected-gateway", + endpointUrl: "https://models.example.test/v1", + workload: { kind: "external-image" }, + nativeHostedProviderAttachment: { profileId: "native-custom" }, + } as SandboxEntry; + const config: ConfigObject = { + model: { default: "baked-model", base_url: "https://inference.local/v1" }, + }; + const target = { + agentName: "hermes", + configPath: "/sandbox/.hermes/config.yaml", + configDir: "/sandbox/.hermes", + configFile: "config.yaml", + format: "yaml", + }; + const operations: string[] = []; + const deps = { + getSandbox: vi.fn(() => entry), + readSandboxConfig: vi.fn(() => config), + resolveAgentConfig: vi.fn(() => target), + patchHermesInferenceConfig: vi.fn(patchHermesInferenceConfig), + readNativeNvidiaCredentialHandle: vi.fn(async () => "unused"), + writeSandboxConfig: vi.fn(() => { + operations.push("write"); + }), + }; + const options = { + getSandbox: deps.getSandbox, + sandboxName: "external-hermes", + gatewayName: entry.gatewayName!, + model: entry.model!, + provider: entry.provider!, + preferredInferenceApi: "openai-completions", + revalidateSandboxIdentity: vi.fn(() => { + operations.push("identity"); + }), + }; + return { entry, config, target, deps, options, operations }; +} + +describe("external-image Hermes initial inference route", () => { + it.each(["vllm-local", "ollama-local", "llama-cpp-local", "native-extra"])( + "preserves the route owned by the %s initializer", + async (provider) => { + const f = fixture(); + f.entry.provider = f.options.provider = provider; + f.entry.endpointUrl = "http://host.openshell.internal:8000/v1"; + f.entry.registeredInferenceProviderAttachment = + provider === "native-extra" ? ({ providerName: provider } as never) : undefined; + const before = structuredClone(f.config); + await initializeExternalHermesInferenceRoute(f.options, f.deps); + expect(f.deps.readSandboxConfig).not.toHaveBeenCalled(); + expect(f.deps.writeSandboxConfig).not.toHaveBeenCalled(); + expect(f.config).toEqual(before); + }, + ); + + it("uses the provider-bound NVIDIA handle from the selected gateway", async () => { + const f = fixture(); + f.entry.provider = f.options.provider = "nvidia-prod"; + const handle = "openshell:resolve:env:v42_NVIDIA_INFERENCE_API_KEY"; + f.deps.readNativeNvidiaCredentialHandle.mockResolvedValueOnce(handle); + await initializeExternalHermesInferenceRoute(f.options, f.deps); + expect(f.deps.readNativeNvidiaCredentialHandle).toHaveBeenCalledWith( + "external-hermes", + "selected-gateway", + ); + expect(f.config.model).toMatchObject({ + api_key: handle, + base_url: "https://integrate.api.nvidia.com/v1", + }); + }); + + it("does not write configuration when the NVIDIA handle is unavailable", async () => { + const f = fixture(); + f.entry.provider = f.options.provider = "nvidia-prod"; + f.deps.readNativeNvidiaCredentialHandle.mockRejectedValueOnce(new Error("handle unavailable")); + await expect(initializeExternalHermesInferenceRoute(f.options, f.deps)).rejects.toThrow( + "handle unavailable", + ); + expect(f.deps.writeSandboxConfig).not.toHaveBeenCalled(); + }); + + it("replaces baked inference.local configuration with the registered native selection", async () => { + const f = fixture(); + await initializeExternalHermesInferenceRoute(f.options, f.deps); + expect(f.config.model).toMatchObject({ + default: "selected-model", + base_url: "https://models.example.test/v1", + api_key: "${COMPATIBLE_API_KEY}", + }); + expect(f.deps.readSandboxConfig).toHaveBeenCalledWith( + "external-hermes", + f.target, + "selected-gateway", + ); + expect(f.operations).toEqual(["identity", "identity", "write"]); + }); + + it.each([ + { agent: "openclaw" }, + { provider: "openai-api" }, + { model: "other-model" }, + { gatewayName: "other-gateway" }, + { workload: { kind: "managed-image" } }, + ])("rejects a different registered selection %j before reading configuration", async (change) => { + const f = fixture(); + Object.assign(f.entry, change); + await expect(initializeExternalHermesInferenceRoute(f.options, f.deps)).rejects.toThrow( + "registered external-image selection", + ); + expect(f.deps.readSandboxConfig).not.toHaveBeenCalled(); + expect(f.deps.writeSandboxConfig).not.toHaveBeenCalled(); + }); + + it.each(["writeSandboxConfig"] as const)( + "reports a failed configuration transaction when %s fails", + async (operation) => { + const f = fixture(); + f.deps[operation].mockImplementation(() => { + throw new Error("transaction failed"); + }); + await expect(initializeExternalHermesInferenceRoute(f.options, f.deps)).rejects.toThrow( + "transaction failed", + ); + }, + ); + + it("stops before writing when sandbox identity changes after reading", async () => { + const f = fixture(); + f.options.revalidateSandboxIdentity + .mockImplementationOnce(() => undefined) + .mockImplementationOnce(() => { + throw new Error("identity changed"); + }); + await expect(initializeExternalHermesInferenceRoute(f.options, f.deps)).rejects.toThrow( + "identity changed", + ); + expect(f.deps.writeSandboxConfig).not.toHaveBeenCalled(); + }); +}); diff --git a/src/lib/onboard/machine/handlers/initial-hermes-inference-route.ts b/src/lib/onboard/machine/handlers/initial-hermes-inference-route.ts new file mode 100644 index 00000000000..615226f813e --- /dev/null +++ b/src/lib/onboard/machine/handlers/initial-hermes-inference-route.ts @@ -0,0 +1,98 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { usesNativeLocalInferenceForAgent } from "../../../inference/native-local/eligibility"; +import type { SandboxEntry } from "../../../state/registry"; +import { patchHermesInferenceConfig } from "../../../actions/inference-set"; +import { + readHermesNativeNvidiaCredentialHandleIfNeeded, + readNativeNvidiaCredentialHandle, +} from "../../../actions/inference-set-provider"; +import { readSandboxConfig, resolveAgentConfig, writeSandboxConfig } from "../../../sandbox/config"; + +const defaultDeps = { + patchHermesInferenceConfig, + readNativeNvidiaCredentialHandle, + readSandboxConfig, + resolveAgentConfig, + writeSandboxConfig, +}; + +export async function initializeExternalHermesInferenceRoute( + options: { + getSandbox: (name: string) => SandboxEntry | null; + sandboxName: string; + gatewayName: string; + model: string; + provider: string; + preferredInferenceApi: string | null; + revalidateSandboxIdentity: (operation: string) => void; + }, + deps = defaultDeps, +): Promise { + const { + sandboxName, + gatewayName, + model, + provider, + preferredInferenceApi, + revalidateSandboxIdentity, + } = options; + revalidateSandboxIdentity( + `read initial Hermes inference configuration in sandbox '${sandboxName}'`, + ); + const entry = options.getSandbox(sandboxName); + if ( + !entry || + entry.agent !== "hermes" || + entry.workload?.kind !== "external-image" || + entry.gatewayName !== gatewayName || + entry.provider !== provider || + entry.model !== model + ) { + throw new Error( + "Initial Hermes inference route does not match its registered external-image selection.", + ); + } + // The attached native route initializer owns projection and credential-only restore. + // Do not replace its provider-bound reference with a shared-route sentinel. + if ( + entry.registeredInferenceProviderAttachment || + entry.nativeLocalProviderAttachment || + usesNativeLocalInferenceForAgent( + "hermes", + provider, + entry.endpointUrl, + provider === "vllm-local" && Boolean(entry.nimContainer), + ) + ) + return; + const target = deps.resolveAgentConfig(sandboxName); + if (target.agentName !== "hermes") + throw new Error("Initial Hermes inference route requires Hermes configuration."); + const config = deps.readSandboxConfig(sandboxName, target, gatewayName); + const nativeProvider = Boolean( + entry.nativeHostedProviderAttachment || entry.nativeNvidiaProviderAttachment, + ); + const nativeCredentialHandle = await readHermesNativeNvidiaCredentialHandleIfNeeded({ + agentName: "hermes", + provider, + sandboxName, + gatewayName, + read: deps.readNativeNvidiaCredentialHandle, + }); + deps.patchHermesInferenceConfig( + config, + provider, + model, + preferredInferenceApi, + undefined, + entry.endpointUrl ?? undefined, + nativeProvider, + nativeCredentialHandle, + ); + revalidateSandboxIdentity( + `apply initial Hermes inference configuration in sandbox '${sandboxName}'`, + ); + deps.writeSandboxConfig(sandboxName, target, config, gatewayName); +} diff --git a/src/lib/onboard/machine/handlers/policies.test.ts b/src/lib/onboard/machine/handlers/policies.test.ts index ef611608973..8bace1f6abc 100644 --- a/src/lib/onboard/machine/handlers/policies.test.ts +++ b/src/lib/onboard/machine/handlers/policies.test.ts @@ -7,6 +7,17 @@ import { makeMessagingPlan } from "../../../../../test/helpers/messaging-plan-fi import { createSession } from "../../../state/onboard-session"; import { createPolicyHandlerDeps, basePolicyHandlerOptions } from "./policies-test-fixture"; import { handlePoliciesState } from "./policies"; +import { nativeLocalTestReceipt } from "../../../../../test/helpers/inference-set"; + +const nativeNimReceipt = nativeLocalTestReceipt( + "vllm-local", + "http://host.openshell.internal:8000/v1", + { + sandboxName: "my-assistant", + agent: "openclaw", + authMode: "sentinel", + }, +); describe("policy state handler", () => { it("verifies a resumed published Pi local route without a native attachment", async () => { @@ -32,19 +43,34 @@ describe("policy state handler", () => { ); }); - it("requires a native attachment for a fresh Pi local selection", async () => { - const { deps, calls } = createPolicyHandlerDeps(); - await expect( - handlePoliciesState({ - ...basePolicyHandlerOptions(deps), - agent: { name: "pi" }, - provider: "ollama-local", - model: "qwen3:8b", - endpointUrl: "http://host.openshell.internal:11435/v1", - }), - ).rejects.toThrow("requires its recorded provider attachment"); - expect(calls.setupPolicies).not.toHaveBeenCalled(); - }); + it.each([ + ["Pi", "pi", "ollama-local", "http://host.openshell.internal:11435/v1", null], + [ + "Local NIM", + "openclaw", + "vllm-local", + "http://host.openshell.internal:8000/v1", + "nemoclaw-nim-owned", + ], + ] as const)( + "requires a native attachment for a fresh %s selection", + async (_label, agent, provider, endpointUrl, nimContainer) => { + const { deps, calls } = createPolicyHandlerDeps({ + getActiveSandbox: vi.fn(() => ({ nimContainer })), + }); + await expect( + handlePoliciesState({ + ...basePolicyHandlerOptions(deps), + agent: { name: agent }, + provider, + model: "qwen3:8b", + endpointUrl, + }), + ).rejects.toThrow("requires its recorded provider attachment"); + expect(calls.smoke).not.toHaveBeenCalled(); + expect(calls.setupPolicies).not.toHaveBeenCalled(); + }, + ); it("resumes from the live OpenShell preset selection", async () => { const prepare = vi.fn(() => ({ @@ -218,15 +244,18 @@ describe("policy state handler", () => { expect(calls.complete).not.toHaveBeenCalled(); }); - it("verifies an owned Local NIM through the shared route before applying policies", async () => { + it("verifies an owned Local NIM after excluding the shared inference policy", async () => { const { deps, calls } = createPolicyHandlerDeps({ - getActiveSandbox: vi.fn(() => ({ nimContainer: "nemoclaw-nim-owned" })), + getActiveSandbox: vi.fn(() => ({ + nimContainer: "nemoclaw-nim-owned", + nativeLocalProviderAttachment: nativeNimReceipt, + })), }); await handlePoliciesState({ ...basePolicyHandlerOptions(deps), provider: "vllm-local", - endpointUrl: "http://127.0.0.1:8000/v1", + endpointUrl: nativeNimReceipt.endpointUrl, agent: { name: "openclaw" }, }); @@ -237,23 +266,29 @@ describe("policy state handler", () => { }), ); const smokeOptions = calls.smoke.mock.calls[0]?.[0]; - expect(smokeOptions).not.toHaveProperty("nativeLocalProviderAttachment"); + expect(smokeOptions).toHaveProperty("nativeLocalProviderAttachment", nativeNimReceipt); expect(smokeOptions).not.toHaveProperty("forceCanonicalRoute"); expect(smokeOptions).not.toHaveProperty("hostLocalInferenceProofAuthority"); + expect(calls.setupPolicies.mock.invocationCallOrder[0]).toBeLessThan( + calls.smoke.mock.invocationCallOrder[0]!, + ); expect(calls.smoke.mock.invocationCallOrder[0]).toBeLessThan( - calls.setupPolicies.mock.invocationCallOrder[0]!, + calls.complete.mock.invocationCallOrder[0]!, ); expect(calls.setupPolicies).toHaveBeenCalledWith( "my-assistant", - expect.objectContaining({ provider: "vllm-local" }), + expect.objectContaining({ provider: null, excludedPresets: ["local-inference"] }), ); expect(calls.complete).toHaveBeenCalledOnce(); }); - it("stops Local NIM policy application when shared-route verification fails", async () => { - const failure = new Error("Local NIM shared-route smoke failed"); + it("does not complete Local NIM policy setup when native route verification fails", async () => { + const failure = new Error("Local NIM native route smoke failed"); const { deps, calls } = createPolicyHandlerDeps({ - getActiveSandbox: vi.fn(() => ({ nimContainer: "nemoclaw-nim-owned" })), + getActiveSandbox: vi.fn(() => ({ + nimContainer: "nemoclaw-nim-owned", + nativeLocalProviderAttachment: nativeNimReceipt, + })), }); calls.smoke.mockImplementation(() => { throw failure; @@ -263,13 +298,16 @@ describe("policy state handler", () => { handlePoliciesState({ ...basePolicyHandlerOptions(deps), provider: "vllm-local", - endpointUrl: "http://127.0.0.1:8000/v1", + endpointUrl: nativeNimReceipt.endpointUrl, agent: { name: "openclaw" }, }), ).rejects.toBe(failure); expect(calls.smoke).toHaveBeenCalledOnce(); - expect(calls.setupPolicies).not.toHaveBeenCalled(); + expect(calls.setupPolicies).toHaveBeenCalledExactlyOnceWith( + "my-assistant", + expect.objectContaining({ provider: null, excludedPresets: ["local-inference"] }), + ); expect(calls.complete).not.toHaveBeenCalled(); }); diff --git a/src/lib/onboard/machine/handlers/provider-inference-legacy-local-route.test.ts b/src/lib/onboard/machine/handlers/provider-inference-legacy-local-route.test.ts index 493e63ccdfd..fc63e07c586 100644 --- a/src/lib/onboard/machine/handlers/provider-inference-legacy-local-route.test.ts +++ b/src/lib/onboard/machine/handlers/provider-inference-legacy-local-route.test.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import { getInstalledOpenshellVersion } from "../../openshell-version"; // The reservation helper must not inspect the operator's real registry. vi.mock("../../../state/registry", async (importOriginal) => ({ @@ -9,12 +10,85 @@ vi.mock("../../../state/registry", async (importOriginal) => ({ getSandbox: vi.fn(() => null), })); +import { cliOpenShellInstalledVersionObserver } from "../../../adapters/openshell/installed-version-cli"; + import { createSession } from "../../../state/onboard-session"; import type { SandboxEntry } from "../../../state/registry"; import { handleProviderInferenceState } from "./provider-inference"; import { baseOptions, createDeps } from "./provider-inference.test-support"; -describe("legacy local inference resume", () => { +function retainedPublicCustomResume(provider: string, version: string | null) { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue( + version === null + ? { ok: false, error: { kind: "unavailable", message: "unavailable" } } + : { ok: true, version }, + ); + const session = createSession({ + agent: "openclaw", + sandboxName: "my-assistant", + provider, + model: "served-model", + endpointUrl: "https://integrate.api.nvidia.com/v1", + credentialEnv: "COMPATIBLE_API_KEY", + preferredInferenceApi: "openai-completions", + }); + session.steps.provider_selection.status = "complete"; + const recorded = { + name: "my-assistant", + agent: "openclaw", + gatewayName: "nemoclaw", + createdAt: "2026-09-01T00:00:00Z", + provider: session.provider, + model: session.model, + endpointUrl: session.endpointUrl, + credentialEnv: session.credentialEnv, + preferredInferenceApi: session.preferredInferenceApi, + } as SandboxEntry; + const before = structuredClone({ session, recorded }); + const { deps, calls } = createDeps({ + getSandboxRegistryEntry: () => recorded, + getInstalledOpenshellVersion, + hydrateCredentialEnv: vi.fn(() => "host-key"), + isInferenceRouteReady: vi.fn(() => true), + }); + calls.complete.mockResolvedValue(session); + const result = handleProviderInferenceState({ + ...baseOptions(deps, session), + resume: true, + agent: { name: "openclaw" }, + sandboxName: "my-assistant", + selectedMessagingChannels: ["telegram"], + }); + return { result, calls, session, recorded, before }; +} + +describe("legacy shared inference resume", () => { + it.each([ + { provider: "compatible-endpoint", version: "0.1.3" }, + { provider: "compatible-anthropic-endpoint", version: "0.1.3" }, + { provider: "compatible-endpoint", version: null }, + ])( + "refuses retained public $provider resume before setup on OpenShell $version", + async ({ provider, version }) => { + const { result, calls, session, recorded, before } = retainedPublicCustomResume( + provider, + version, + ); + await expect(result).rejects.toThrow(/Back up|back up/); + expect(calls.setupInference).not.toHaveBeenCalled(); + expect(calls.startStep).not.toHaveBeenCalled(); + expect(calls.updateSandbox).not.toHaveBeenCalled(); + expect(calls.reserveRoute).not.toHaveBeenCalled(); + expect({ session, recorded }).toEqual(before); + }, + ); + + it("resumes a retained public compatible route on verified OpenShell 0.0.x", async () => { + const { result, calls } = retainedPublicCustomResume("compatible-endpoint", "0.0.116"); + await expect(result).resolves.toBeDefined(); + expect(calls.setupInference).toHaveBeenCalled(); + }); + it("keeps a fresh Pi local selection on the native provider path", async () => { const { deps, calls } = createDeps({ setupNim: vi.fn(async () => ({ @@ -56,6 +130,10 @@ describe("legacy local inference resume", () => { ])( "resumes an unchanged published Pi $provider selection through its shared route", async (selection) => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue({ + ok: true, + version: "0.0.116", + }); const session = createSession({ agent: "pi", sandboxName: "my-assistant", @@ -76,7 +154,10 @@ describe("legacy local inference resume", () => { credentialEnv: session.credentialEnv, preferredInferenceApi: session.preferredInferenceApi, } as SandboxEntry; - const { deps, calls } = createDeps({ getSandboxRegistryEntry: () => recorded }); + const { deps, calls } = createDeps({ + getSandboxRegistryEntry: () => recorded, + getInstalledOpenshellVersion, + }); await handleProviderInferenceState({ ...baseOptions(deps, session), @@ -102,3 +183,95 @@ describe("legacy local inference resume", () => { }, ); }); + +it("blocks resume of a legacy OpenClaw compatible route before inference setup", async () => { + const session = createSession({ + agent: "openclaw", + sandboxName: "my-assistant", + provider: "compatible-endpoint", + model: "served-model", + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN", + preferredInferenceApi: "openai-completions", + }); + session.steps.provider_selection.status = "complete"; + const recorded = { + name: "my-assistant", + agent: "openclaw", + gatewayName: "nemoclaw", + createdAt: "2026-09-01T00:00:00Z", + provider: session.provider, + model: session.model, + endpointUrl: session.endpointUrl, + credentialEnv: session.credentialEnv, + preferredInferenceApi: session.preferredInferenceApi, + } as SandboxEntry; + const { deps, calls } = createDeps({ + getSandboxRegistryEntry: () => recorded, + getInstalledOpenshellVersion, + }); + await expect( + handleProviderInferenceState({ + ...baseOptions(deps, session), + resume: true, + agent: { name: "openclaw" }, + sandboxName: "my-assistant", + }), + ).rejects.toThrow(/Back up|back up/); + expect(calls.setupInference).not.toHaveBeenCalled(); +}); + +it.each([ + { label: "0.1.3", observation: { ok: true, version: "0.1.3" } }, + { label: "0.1.0", observation: { ok: true, version: "0.1.0" } }, + { + label: "unknown", + observation: { ok: false, error: { kind: "unavailable", message: "unavailable" } }, + }, +] as const)( + "refuses retained Pi shared inference before mutation on OpenShell $label", + async ({ observation }) => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue( + observation, + ); + const session = createSession({ + agent: "pi", + sandboxName: "my-assistant", + provider: "ollama-local", + model: "qwen3:8b", + endpointUrl: "http://host.openshell.internal:11435/v1", + credentialEnv: "NEMOCLAW_OLLAMA_PROXY_TOKEN", + }); + session.steps.provider_selection.status = "complete"; + const recorded = { + name: "my-assistant", + agent: "pi", + gatewayName: "nemoclaw", + createdAt: "2026-09-01T00:00:00Z", + provider: session.provider, + model: session.model, + endpointUrl: session.endpointUrl, + credentialEnv: session.credentialEnv, + } as SandboxEntry; + const before = structuredClone({ session, recorded }); + const { deps, calls } = createDeps({ + getSandboxRegistryEntry: () => recorded, + getInstalledOpenshellVersion, + }); + await expect( + handleProviderInferenceState({ + ...baseOptions(deps, session), + resume: true, + agent: { name: "pi" }, + sandboxName: "my-assistant", + }), + ).rejects.toThrow("Back up"); + expect(calls.prepareLocalProviderForInference).not.toHaveBeenCalled(); + expect(calls.setupInference).not.toHaveBeenCalled(); + expect(calls.complete).not.toHaveBeenCalled(); + expect(calls.startStep).not.toHaveBeenCalled(); + expect(calls.updateSandbox).not.toHaveBeenCalled(); + expect(calls.reserveRoute).not.toHaveBeenCalled(); + expect({ session, recorded }).toEqual(before); + }, +); diff --git a/src/lib/onboard/machine/handlers/provider-inference-managed-llama-resume.test.ts b/src/lib/onboard/machine/handlers/provider-inference-managed-llama-resume.test.ts index 48c8792fdaf..f77480a9c77 100644 --- a/src/lib/onboard/machine/handlers/provider-inference-managed-llama-resume.test.ts +++ b/src/lib/onboard/machine/handlers/provider-inference-managed-llama-resume.test.ts @@ -192,12 +192,15 @@ describe("handleProviderInferenceState managed llama.cpp resume", () => { }); it.each([ - { resume: false, reachable: false, result: "exit 1" }, - { resume: true, reachable: false, result: "exit 1" }, - { resume: false, reachable: true, result: "complete" }, + { resume: false, reachable: false, result: "exit 1", networkName: "openshell" }, + { resume: true, reachable: false, result: "exit 1", networkName: "openshell" }, + { resume: false, reachable: true, result: "complete", networkName: "openshell" }, + { resume: false, reachable: false, result: "exit 1", networkName: "host" }, + { resume: true, reachable: false, result: "exit 1", networkName: "host" }, + { resume: false, reachable: true, result: "complete", networkName: "host" }, ])( - "requires a reachable operator llama.cpp route (resume=$resume, reachable=$reachable)", - async ({ resume, reachable, result }) => { + "requires a reachable operator llama.cpp route (resume=$resume, reachable=$reachable, network=$networkName)", + async ({ resume, reachable, result, networkName }) => { const route = { provider: "llama-cpp-local", model: "team/model-alias", @@ -219,8 +222,8 @@ describe("handleProviderInferenceState managed llama.cpp resume", () => { const probeLlamaCppSandboxReachability = vi.fn(async () => ({ ok: reachable, reason: reachable ? ("ok" as const) : ("tcp_failed" as const), - networkName: "openshell", - gatewayIp: "172.18.0.1", + networkName, + gatewayIp: networkName === "host" ? "127.0.0.1" : "172.18.0.1", })); const setupNim = vi.fn(async () => ({ ...baseSelection, ...route })); const { deps, calls } = createDeps({ @@ -245,6 +248,9 @@ describe("handleProviderInferenceState managed llama.cpp resume", () => { message.includes("host.openshell.internal:8081"), ), ).toBe(!reachable); + expect(calls.error.mock.calls.some(([message]) => message.includes("bind or publish"))).toBe( + !reachable && networkName !== "host", + ); expect(probeLlamaCppSandboxReachability).toHaveBeenCalledOnce(); expect(setupNim).toHaveBeenCalledTimes(resume ? 0 : 1); }, diff --git a/src/lib/onboard/machine/handlers/provider-inference-native-local-resume.test.ts b/src/lib/onboard/machine/handlers/provider-inference-native-local-resume.test.ts index f0a8ee684a4..9a9896fcab3 100644 --- a/src/lib/onboard/machine/handlers/provider-inference-native-local-resume.test.ts +++ b/src/lib/onboard/machine/handlers/provider-inference-native-local-resume.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import * as routerProfile from "../../../core/model-router-port"; import { afterEach, describe, expect, it, vi } from "vitest"; import { nativeLocalIdentity } from "../../../inference/native-local/contract"; @@ -70,40 +71,49 @@ describe("native local provider inference resume (#13540)", () => { for (const receipt of recorded.splice(0)) clearNativeLocalProviderAuthority(receipt); }); - it("reverifies the native Model Router through setup even when the shared route is ready (#13560)", async () => { - const session = createSession({ - agent: "openclaw", - sandboxName: "router-probe", - provider: "nvidia-router", - model: "nvidia-routed", - endpointUrl: "http://host.openshell.internal:4000/v1", - credentialEnv: "NVIDIA_INFERENCE_API_KEY", - }); - session.steps.provider_selection.status = "complete"; - const { deps, calls } = createDeps({ - isInferenceRouteReady: vi.fn(() => true), - hasNativeLocalProviderAuthority: () => true, - }); - calls.complete.mockResolvedValue(session); - await handleProviderInferenceState({ - ...baseOptions(deps, session), - resume: true, - sandboxName: "router-probe", - agent: { name: "openclaw" }, - }); - expect(calls.setupInference).toHaveBeenCalledWith( - "router-probe", - "nvidia-routed", - "nvidia-router", - "http://host.openshell.internal:4000/v1", - "NVIDIA_INFERENCE_API_KEY", - null, - [], - expect.objectContaining({ agentName: "openclaw", gatewayName: "nemoclaw" }), - ); - expect(calls.reupsertRoutedProvider).not.toHaveBeenCalled(); - expect(calls.skipped).not.toHaveBeenCalledWith("inference", "nvidia-router / nvidia-routed"); - }); + it.each(["openclaw", "hermes", "pi", "langchain-deepagents-code"] as const)( + "reverifies %s configured Model Router through setup when the shared route is ready (#13560)", + async (agent) => { + vi.spyOn(routerProfile, "loadBlueprintProfile").mockReturnValue({ + model: "nvidia-routed", + provider_name: "custom-router", + endpoint: "http://localhost:4000/v1", + router: { enabled: true, port: 4000 }, + }); + const session = createSession({ + agent, + sandboxName: "router-probe", + provider: "custom-router", + model: "nvidia-routed", + endpointUrl: "http://host.openshell.internal:4000/v1", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }); + session.steps.provider_selection.status = "complete"; + const { deps, calls } = createDeps({ + isInferenceRouteReady: vi.fn(() => true), + hasNativeLocalProviderAuthority: () => true, + }); + calls.complete.mockResolvedValue(session); + await handleProviderInferenceState({ + ...baseOptions(deps, session), + resume: true, + sandboxName: "router-probe", + agent: { name: agent }, + }); + expect(calls.setupInference).toHaveBeenCalledWith( + "router-probe", + "nvidia-routed", + "custom-router", + "http://host.openshell.internal:4000/v1", + "NVIDIA_INFERENCE_API_KEY", + null, + [], + expect.objectContaining({ agentName: agent, gatewayName: "nemoclaw" }), + ); + expect(calls.reupsertRoutedProvider).not.toHaveBeenCalled(); + expect(calls.skipped).not.toHaveBeenCalledWith("inference", "custom-router / nvidia-routed"); + }, + ); it("requires explicit recreation for an existing shared Model Router sandbox (#13560)", async () => { const session = createSession({ diff --git a/src/lib/onboard/machine/handlers/provider-inference-route-containment.test.ts b/src/lib/onboard/machine/handlers/provider-inference-route-containment.test.ts index 06bb418a657..12f17a46c86 100644 --- a/src/lib/onboard/machine/handlers/provider-inference-route-containment.test.ts +++ b/src/lib/onboard/machine/handlers/provider-inference-route-containment.test.ts @@ -119,7 +119,7 @@ function createDeps() { getOpenshellBinary: () => "/usr/bin/openshell", needsBedrockRuntimeAdapter: () => false, isInferenceRouteReady: () => true, - isRoutedInferenceProvider: (provider) => ["nvidia-router", "custom-router"].includes(provider), + isRoutedInferenceProvider: (provider) => provider === "nvidia-router", reconcileModelRouter: calls.reconcileRouter, reupsertRoutedProvider: calls.reupsertRoutedProvider, reserveSandboxInferenceRoute: calls.reserveRoute, @@ -354,139 +354,36 @@ describe("provider route containment", () => { expect(result).toMatchObject({ endpointSource: null, onboardEndpointUrl: null }); }); - it("allows routed-provider repair across a valid peer-route difference (#6315)", async () => { - const session = createSession({ provider: "custom-router", model: "router/model" }); - session.steps.provider_selection.status = "complete"; - const { calls, deps } = createDeps(); - reportDifferentRoute(calls, "custom-router", "router/model"); - - const options = resumeOptions(deps, session); - options.initial.endpointSource = "inference-set"; - await expect(handleProviderInferenceState(options)).resolves.toMatchObject({ - provider: "custom-router", - model: "router/model", - }); - - expect(calls.checkGatewayRouteCompatibility).toHaveBeenCalledWith({ - gatewayName: "nemoclaw-9090", - sandboxName: "target-sandbox", - route: { - provider: "custom-router", - model: "router/model", - endpointUrl: null, - credentialEnv: null, - preferredInferenceApi: null, - }, - }); - expect(calls.reconcileRouter).toHaveBeenCalledOnce(); - expect(calls.surfaceReady).toHaveBeenCalledOnce(); - expect(calls.reupsertRoutedProvider).toHaveBeenCalledOnce(); - expect(calls.reserveRoute).toHaveBeenCalledWith("target-sandbox", { - provider: "custom-router", + it("revalidates native Model Router setup on resume without reading the retired shared route", async () => { + const session = createSession({ + provider: "nvidia-router", model: "router/model", endpointUrl: "http://host.openshell.internal:4000/v1", - endpointSource: "inference-set", - credentialEnv: null, - preferredInferenceApi: null, - gatewayName: "nemoclaw-9090", - reservationSessionId: session.sessionId, - }); - expect(calls.updateSandbox).not.toHaveBeenCalled(); - expect(calls.setupInference).not.toHaveBeenCalled(); - }); - - it("rechecks routed repair after waiting for the gateway lock (#6315)", async () => { - const session = createSession({ provider: "custom-router", model: "router/model" }); - session.steps.provider_selection.status = "complete"; - const { calls, deps } = createDeps(); - let releaseLock!: () => void; - const lockReleased = new Promise((resolve) => { - releaseLock = resolve; - }); - let reportLockEntered!: () => void; - const lockEntered = new Promise((resolve) => { - reportLockEntered = resolve; + credentialEnv: "NVIDIA_INFERENCE_API_KEY", }); - deps.withGatewayRouteMutationLock = async (_gatewayName, operation) => { - reportLockEntered(); - await lockReleased; - return await operation(); - }; - - const repair = handleProviderInferenceState(resumeOptions(deps, session)); - await lockEntered; - reportDifferentRoute(calls, "custom-router", "router/model"); - releaseLock(); - - await expect(repair).resolves.toMatchObject({ provider: "custom-router" }); - expect(calls.reconcileRouter).toHaveBeenCalledOnce(); - expect(calls.reupsertRoutedProvider).toHaveBeenCalledOnce(); - expect(calls.updateSandbox).not.toHaveBeenCalled(); - }); - - it("waits for a cross-gateway teardown port lock before publishing routed resume repair (#9098)", async () => { - const session = createSession({ provider: "custom-router", model: "router/model" }); session.steps.provider_selection.status = "complete"; const { calls, deps } = createDeps(); - let insideGatewayLock = false; - let insidePortLock = false; - let reportPortLockRequested!: () => void; - const portLockRequested = new Promise((resolve) => { - reportPortLockRequested = resolve; - }); - let releaseTeardownPortLock!: () => void; - const teardownPortLockReleased = new Promise((resolve) => { - releaseTeardownPortLock = resolve; - }); - deps.withGatewayRouteMutationLock = async (gatewayName, operation) => { - expect(gatewayName).toBe("nemoclaw-9090"); - insideGatewayLock = true; - try { - return await operation(); - } finally { - insideGatewayLock = false; - } - }; - deps.withModelRouterPortLifecycleLock = async (port, operation) => { - expect(insideGatewayLock).toBe(true); - expect(port).toBe(4000); - reportPortLockRequested(); - await teardownPortLockReleased; - insidePortLock = true; - try { - return await operation(); - } finally { - insidePortLock = false; - } - }; - calls.reconcileRouter.mockImplementation(async () => { - expect(insideGatewayLock).toBe(true); - expect(insidePortLock).toBe(true); + const routeReady = vi.fn(() => { + throw new Error("retired shared route read"); }); - calls.reupsertRoutedProvider.mockImplementation(() => { - expect(insideGatewayLock).toBe(true); - expect(insidePortLock).toBe(true); - return { ok: true, endpointUrl: "http://host.openshell.internal:4000/v1" }; - }); - calls.reserveRoute.mockImplementation(() => { - expect(insideGatewayLock).toBe(true); - expect(insidePortLock).toBe(true); - return true; + deps.isInferenceRouteReady = routeReady; + const options = resumeOptions(deps, session); + options.agent = { name: "openclaw" } as never; + await expect(handleProviderInferenceState(options)).resolves.toMatchObject({ + provider: "nvidia-router", }); - - const repair = handleProviderInferenceState(resumeOptions(deps, session)); - await portLockRequested; - - expect(calls.reconcileRouter).not.toHaveBeenCalled(); + expect(calls.setupInference).toHaveBeenCalledWith( + "target-sandbox", + "router/model", + "nvidia-router", + "http://host.openshell.internal:4000/v1", + "NVIDIA_INFERENCE_API_KEY", + null, + [], + expect.objectContaining({ agentName: "openclaw", gatewayName: "nemoclaw-9090" }), + ); + expect(routeReady).not.toHaveBeenCalled(); expect(calls.reupsertRoutedProvider).not.toHaveBeenCalled(); - expect(calls.reserveRoute).not.toHaveBeenCalled(); - - releaseTeardownPortLock(); - await expect(repair).resolves.toMatchObject({ provider: "custom-router" }); - - expect(calls.reconcileRouter).toHaveBeenCalledOnce(); - expect(calls.reupsertRoutedProvider).toHaveBeenCalledOnce(); - expect(calls.reserveRoute).toHaveBeenCalledOnce(); }); it("allows compatible-endpoint refresh to reach the final setup boundary (#6315)", async () => { @@ -550,6 +447,51 @@ describe("provider route containment", () => { }); }); +it.each([ + { version: "0.0.14", pending: false, fresh: false }, + { version: "0.1.3", pending: true, fresh: false }, + { version: "0.1.3", pending: false, fresh: true }, +])( + "permits NVIDIA selection with runtime $version, pending $pending, fresh $fresh", + async ({ version, pending, fresh }) => { + const { deps } = createDeps(); + deps.getInstalledOpenshellVersion = () => version; + deps.getSandboxRegistryEntry = () => ({ + name: "target-sandbox", + provider: "nvidia-nim", + model: "nvidia/test", + gatewayName: "nemoclaw-9090", + ...(pending ? { pendingRouteReservation: true as const } : {}), + }); + await expect( + handleProviderInferenceState({ + ...resumeOptions(deps, createSession()), + fresh, + }), + ).resolves.toBeDefined(); + }, +); + +it.each(["0.1.3", undefined])( + "rejects retained legacy NVIDIA resume before mutation on runtime %s", + async (version) => { + const { deps, calls } = createDeps(); + deps.getInstalledOpenshellVersion = () => version ?? null; + deps.getSandboxRegistryEntry = () => ({ + name: "target-sandbox", + provider: "nvidia-nim", + model: "nvidia/test", + gatewayName: "nemoclaw-9090", + }); + await expect( + handleProviderInferenceState(resumeOptions(deps, createSession())), + ).rejects.toThrow("legacy shared NVIDIA inference route"); + expect(calls.setupInference).not.toHaveBeenCalled(); + expect(calls.reserveRoute).not.toHaveBeenCalled(); + expect(calls.updateSandbox).not.toHaveBeenCalled(); + }, +); + it.each([undefined, "2026-10-01T00:00:00Z"])( "blocks published legacy DCode resume with timestamp %s before mutation (#13197)", async (createdAt) => { @@ -574,7 +516,7 @@ it.each([undefined, "2026-10-01T00:00:00Z"])( ); it.each(["openclaw", "hermes", "langchain-deepagents-code"] as const)( - "resumes an owned Local NIM %s shared route without requiring a native attachment", + "preserves an owned legacy Local NIM %s route without a native attachment", async (agentName) => { const { deps, calls } = createDeps(); deps.getSandboxRegistryEntry = () => ({ @@ -596,22 +538,23 @@ it.each(["openclaw", "hermes", "langchain-deepagents-code"] as const)( const options = resumeOptions(deps, session); options.agent = { name: agentName } as never; options.forceInferenceSetup = true; - await expect(handleProviderInferenceState(options)).resolves.toMatchObject({ - provider: "vllm-local", - model: "fixture-model", - }); - expect(calls.error).not.toHaveBeenCalled(); + await expect(handleProviderInferenceState(options)).rejects.toThrow( + "legacy shared local inference route", + ); expect(calls.setupNim).not.toHaveBeenCalled(); - expect(calls.setupInference).toHaveBeenCalledOnce(); - expect(calls.setupInference.mock.calls[0]?.at(-1)).toMatchObject({ - agentName, - localNimSelected: true, - }); + expect(calls.setupInference).not.toHaveBeenCalled(); + expect(calls.reserveRoute).not.toHaveBeenCalled(); + expect(calls.updateSandbox).not.toHaveBeenCalled(); }, ); it("keeps a fresh OpenClaw native local endpoint independent of a shared peer", async () => { const { deps, calls } = createDeps(); + deps.probeLlamaCppSandboxReachability = async () => ({ + ok: true, + reason: "ok", + networkName: "host", + }); const selection = { ...fallbackSelection, provider: "llama-cpp-local", diff --git a/src/lib/onboard/machine/handlers/provider-inference-route-resume.test.ts b/src/lib/onboard/machine/handlers/provider-inference-route-resume.test.ts index 8b8ddf6985a..2a4cdd3b8b1 100644 --- a/src/lib/onboard/machine/handlers/provider-inference-route-resume.test.ts +++ b/src/lib/onboard/machine/handlers/provider-inference-route-resume.test.ts @@ -61,3 +61,43 @@ describe("published route repair on resume (#13018)", () => { expect(calls.setupInference.mock.calls[0]?.[7]?.deferPublishedRouteReservation).toBe(defer); }); }); + +it("reruns native local resume without exporting the retained gateway credential", async () => { + const session = createSession({ + provider: "compatible-endpoint", + model: "model", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "COMPATIBLE_API_KEY", + }); + session.steps.provider_selection.status = "complete"; + const { deps, calls } = createDeps({ + isInferenceRouteReady: vi.fn(() => true), + ensureResumeProviderReady: vi.fn(async () => ({ + forceInferenceSetup: true, + credentialEnv: "COMPATIBLE_API_KEY", + nativeCustomSetup: true, + nativeLocalSetup: true, + })), + }); + calls.hydrate.mockReturnValue(null); + calls.complete.mockResolvedValue(session); + await handleProviderInferenceState({ + ...baseOptions(deps, session), + resume: true, + sandboxName: "my-assistant", + }); + expect(calls.setupNim).not.toHaveBeenCalled(); + expect(calls.setupInference).toHaveBeenCalledWith( + "my-assistant", + "model", + "compatible-endpoint", + "http://host.openshell.internal:8000/v1", + "COMPATIBLE_API_KEY", + null, + [], + expect.objectContaining({ + reuseGatewayCredentialWithoutLocalKey: true, + skipHostInferenceSmoke: true, + }), + ); +}); diff --git a/src/lib/onboard/machine/handlers/provider-inference-selected-agent.test.ts b/src/lib/onboard/machine/handlers/provider-inference-selected-agent.test.ts index 97d033981c6..987631eb24f 100644 --- a/src/lib/onboard/machine/handlers/provider-inference-selected-agent.test.ts +++ b/src/lib/onboard/machine/handlers/provider-inference-selected-agent.test.ts @@ -3,6 +3,7 @@ import { expect, it, vi } from "vitest"; +import { createDirectSetupInferenceHarness } from "../../../../../test/helpers/onboard-split-context"; import { createSession } from "../../../state/onboard-session"; import { createSetupInference, type SetupInferenceDeps } from "../../setup-inference"; import { handleProviderInferenceState } from "./provider-inference"; @@ -56,7 +57,7 @@ it.each([ ); it.each(["openclaw", "hermes"])( - "keeps selected %s local inference on the existing shared route", + "passes the selected %s agent to local inference setup", async (name) => { const { deps, calls } = createDeps(); const session = createSession(); @@ -82,67 +83,29 @@ it.each(["openclaw", "hermes"])( ); it.each(["openclaw", "langchain-deepagents-code"])( - "keeps selected %s Local NIM on its owned shared route", + "registers selected %s Local NIM with its native sentinel credential", async (name) => { - let sharedRoute = { provider: "vllm-local", model: "prior-nim-model" }; - const inferenceRouteMutator = { - setInferenceRoute: vi.fn(async (request: { route: typeof sharedRoute }) => { - sharedRoute = request.route; - return { ok: true as const }; - }), - }; - const nativeProviderAdapter = { registerProvider: vi.fn(), attachProvider: vi.fn() }; - const requireNativeProviderPolicy = vi.fn(async () => { - throw new Error("native policy must not run for NIM"); + const nim = require("../../../inference/nim.ts") as typeof import("../../../inference/nim"); + vi.spyOn(nim, "getServedModelId").mockReturnValue("nim-model"); + const getManagedVllmProviderBinding = vi.fn(() => { + throw new Error("Local NIM must not read standalone vLLM credentials"); }); - const upsertProvider = vi.fn(async () => ({ ok: true as const })); - const setup = createSetupInference({ - checkGatewayRouteCompatibility: vi.fn(() => ({ ok: true as const })), - withSandboxMutationLock: async (_sandbox: string, operation: () => Promise | T) => - await operation(), - withGatewayRouteMutationLock: async (_gateway: string, operation: () => Promise | T) => - await operation(), - step: vi.fn(), - getGatewayName: () => "nemoclaw", - getSandbox: () => ({ - agent: name, - provider: "vllm-local", - model: "prior-nim-model", - endpointUrl: "http://host.openshell.internal:8000/v1", - nimContainer: "nemoclaw-nim-owned", - }), - localInference: { - loadPendingOllamaModelCleanup: vi.fn(() => []), - loadPersistedOllamaHost: vi.fn(() => null), + const harness = createDirectSetupInferenceHarness({ + overrides: { + applyLocalInferenceRoute: undefined, + getManagedVllmProviderBinding, + getSandbox: () => ({ + name: "nim-sandbox", + agent: name, + provider: "vllm-local", + model: "prior-nim-model", + endpointUrl: "http://host.openshell.internal:8000/v1", + nimContainer: "nemoclaw-nim-owned", + pendingRouteReservation: true, + }), }, - runOpenshell: vi.fn(() => ({ status: 0, stdout: "", stderr: "" })), - inferenceRouteMutator, - inferenceRouteObserver: { - observeInferenceRoute: vi.fn(async () => ({ - ok: true as const, - value: { state: "configured" as const, route: sharedRoute }, - })), - }, - upsertProvider, - updateSandbox: vi.fn(() => true), - verifyInferenceRoute: vi.fn(), - verifyOnboardInferenceSmoke: vi.fn(), - validateLocalProvider: vi.fn(() => ({ ok: true as const })), - getLocalProviderBaseUrl: () => "http://host.openshell.internal:8000/v1", - getManagedVllmProviderBinding: () => null, - vllmLocalCredentialEnv: "NEMOCLAW_VLLM_LOCAL_TOKEN", - localInferenceTimeoutSecs: 60, - isNonInteractive: () => true, - hermesProviderAuth: { HERMES_PROVIDER_NAME: "hermes-provider" }, - requireNativeProviderPolicy, - providerAdapter: nativeProviderAdapter, - log: vi.fn(), - error: vi.fn(), - exitProcess: (code: number): never => { - throw new Error(`exit ${code}`); - }, - } as unknown as SetupInferenceDeps); - const { deps, calls } = createDeps({ setupInference: setup }); + }); + const { deps, calls } = createDeps({ setupInference: harness.setupInference }); calls.setupNim.mockResolvedValue({ ...baseSelection, provider: "vllm-local", @@ -150,34 +113,29 @@ it.each(["openclaw", "langchain-deepagents-code"])( endpointUrl: "http://host.openshell.internal:8000/v1", nimContainer: "nemoclaw-nim-owned", }); - await handleProviderInferenceState({ ...baseOptions(deps, createSession()), sandboxName: "nim-sandbox", agent: { name }, }); - - expect(inferenceRouteMutator.setInferenceRoute).toHaveBeenCalledExactlyOnceWith( + const attachment = [...harness.native.authorities.values()][0]; + expect(attachment).toMatchObject({ + provider: "vllm-local", + authMode: "sentinel", + endpointUrl: "http://host.openshell.internal:8000/v1", + sandboxName: "nim-sandbox", + gatewayName: "nemoclaw", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }); + expect(attachment.agent).toBe(name === "openclaw" ? name : undefined); + expect(harness.native.adapter.createProvider).toHaveBeenCalledExactlyOnceWith( expect.objectContaining({ - target: { kind: "named", gatewayName: "nemoclaw" }, - route: { provider: "vllm-local", model: "nim-model" }, + type: attachment.profileId, + credentials: [{ name: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", value: "dummy" }], }), ); - expect(requireNativeProviderPolicy).not.toHaveBeenCalled(); - expect(nativeProviderAdapter.registerProvider).not.toHaveBeenCalled(); - expect(nativeProviderAdapter.attachProvider).not.toHaveBeenCalled(); - expect(upsertProvider).toHaveBeenCalledWith( - "vllm-local", - "openai", - "NEMOCLAW_VLLM_LOCAL_TOKEN", - "http://host.openshell.internal:8000/v1", - { NEMOCLAW_VLLM_LOCAL_TOKEN: "dummy" }, - "nemoclaw", - ); - const probeInferenceLocal = (url: string) => { - expect(url).toBe("http://inference.local/v1/models"); - return sharedRoute.model; - }; - expect(probeInferenceLocal("http://inference.local/v1/models")).toBe("nim-model"); + expect(harness.native.profiles).toHaveLength(1); + expect(getManagedVllmProviderBinding).not.toHaveBeenCalled(); + expect(harness.commands.some(({ command }) => command.startsWith("inference set"))).toBe(false); }, ); diff --git a/src/lib/onboard/machine/handlers/provider-inference.test.ts b/src/lib/onboard/machine/handlers/provider-inference.test.ts index 6f2620d726f..8c43a15c0fa 100644 --- a/src/lib/onboard/machine/handlers/provider-inference.test.ts +++ b/src/lib/onboard/machine/handlers/provider-inference.test.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../../test/helpers/registered-inference"; + import { describe, expect, it, vi } from "vitest"; import { decisionSelected, decisionUnset } from "../../../state/onboard-checkpoint-decision"; @@ -1205,188 +1207,52 @@ describe("handleProviderInferenceState", () => { ); }); - it("reconciles model router on resumed custom routed inference", async () => { + it("delegates routed resume to native setup without querying the retired shared route", async () => { const session = createSession({ - sandboxName: "router-sandbox", - provider: "custom-router", - model: "router/model", - sandboxPromptProgress: { - sandboxName: true, - webSearch: false, - messaging: false, - resourceProfile: false, - }, - }); - session.steps.provider_selection.status = "complete"; - const { deps, calls } = createDeps({ isInferenceRouteReady: vi.fn(() => true) }); - - await handleProviderInferenceState({ - ...baseOptions(deps, session), - resume: true, - sandboxName: "router-sandbox", - }); - - expect(calls.reconcileRouter).toHaveBeenCalledOnce(); - expect(calls.reserveRoute).toHaveBeenCalledWith("router-sandbox", { - provider: "custom-router", - model: "router/model", - endpointUrl: "http://host.openshell.internal:4000/v1", - endpointSource: null, - credentialEnv: null, - preferredInferenceApi: null, - gatewayName: "nemoclaw", - reservationSessionId: session.sessionId, - }); - }); - - // #5974 instance 5: the Model Router Python preflight (`prepareModelRouterVenv`) - // throws a plain Error (e.g. "above supported ceiling", with no `oclif.exit`) - // out of `reconcileModelRouter`. The routed branch must catch that throw and - // exit non-zero via `exitProcess(1)` so onboard reports the failure to `$?`, - // rather than the throw being swallowed or riding the oclif runner. The error - // reasons themselves are locked by `model-router-python.test.ts`. - it("exits non-zero when model router reconciliation throws (#5974)", async () => { - const session = createSession({ provider: "custom-router", model: "router/model" }); - session.steps.provider_selection.status = "complete"; - const { deps, calls } = createDeps({ - isInferenceRouteReady: vi.fn(() => true), - reconcileModelRouter: vi.fn(async () => { - throw new Error("version 3.14.0 above supported ceiling 3.14.0 (exclusive)"); - }), - }); - - await expect( - handleProviderInferenceState({ - ...baseOptions(deps, session), - resume: true, - sandboxName: "router-sandbox", - }), - ).rejects.toThrow("exit 1"); - - expect(calls.exit).toHaveBeenCalledWith(1); - expect(calls.error).toHaveBeenCalledWith( - expect.stringContaining("Failed to reconcile model router"), - ); - }); - - // Regression: #4564. On resume the routed provider was only reconciled, never - // re-upserted, so a stale localhost base URL recorded by an earlier run could - // survive in the gateway and break inference.local from the sandbox. - it("re-upserts the routed provider with the host alias on resume (#4564)", async () => { - const session = createSession({ - provider: "custom-router", + provider: "nvidia-router", model: "router/model", endpointUrl: "http://localhost:4000/v1", credentialEnv: "NVIDIA_INFERENCE_API_KEY", }); session.steps.provider_selection.status = "complete"; - const { deps, calls } = createDeps({ isInferenceRouteReady: vi.fn(() => true) }); - - const result = await handleProviderInferenceState({ + const isInferenceRouteReady = vi.fn(() => { + throw new Error("retired route"); + }); + const { deps, calls } = createDeps({ isInferenceRouteReady }); + await handleProviderInferenceState({ ...baseOptions(deps, session), resume: true, sandboxName: "router-sandbox", }); - - expect(calls.reconcileRouter).toHaveBeenCalledOnce(); - expect(calls.reupsertRoutedProvider).toHaveBeenCalledWith( - "nemoclaw", - "custom-router", + expect(calls.setupInference).toHaveBeenCalledWith( + "router-sandbox", + "router/model", + "nvidia-router", "http://localhost:4000/v1", "NVIDIA_INFERENCE_API_KEY", + null, + [], + expect.any(Object), ); - expect(calls.setupInference).not.toHaveBeenCalled(); - expect(result.endpointUrl).toBe("http://host.openshell.internal:4000/v1"); + expect(isInferenceRouteReady).not.toHaveBeenCalled(); + expect(calls.reupsertRoutedProvider).not.toHaveBeenCalled(); }); - it("reserves an authoritative routed repair inside the same gateway lock", async () => { - const session = createSession({ - provider: "custom-router", - model: "router/model", - endpointUrl: "http://localhost:4000/v1", - credentialEnv: "NVIDIA_INFERENCE_API_KEY", - }); + it("propagates native router setup failure during resume", async () => { + const session = createSession({ provider: "nvidia-router", model: "router/model" }); session.steps.provider_selection.status = "complete"; - let insideGatewayLock = false; - const gatewayLocks: string[] = []; - const withGatewayRouteMutationLock: ProviderInferenceStateOptions< - Gpu, - Agent, - Host - >["deps"]["withGatewayRouteMutationLock"] = async (gatewayName, operation) => { - gatewayLocks.push(gatewayName); - insideGatewayLock = true; - try { - return await operation(); - } finally { - insideGatewayLock = false; - } - }; - const { deps, calls } = createDeps({ - isInferenceRouteReady: vi.fn(() => true), - withGatewayRouteMutationLock, - }); - const rebuiltSession = createSession({ sessionId: "router-rebuild-session-id" }); - calls.complete.mockResolvedValueOnce(rebuiltSession); - calls.reconcileRouter.mockImplementation(async () => { - expect(insideGatewayLock).toBe(true); - }); - calls.reupsertRoutedProvider.mockImplementation(() => { - expect(insideGatewayLock).toBe(true); - return { ok: true, endpointUrl: "http://host.openshell.internal:4000/v1" }; - }); - calls.reserveRoute.mockImplementation(() => { - expect(insideGatewayLock).toBe(true); - return true; - }); - - await handleProviderInferenceState({ - ...baseOptions(deps, session), - resume: true, - authoritativeResumeConfig: true, - sandboxName: "router-rebuild", - }); - - expect(gatewayLocks).toEqual(["nemoclaw"]); - expect(calls.reserveRoute).toHaveBeenCalledWith("router-rebuild", { - provider: "custom-router", - model: "router/model", - endpointUrl: "http://host.openshell.internal:4000/v1", - endpointSource: null, - credentialEnv: "NVIDIA_INFERENCE_API_KEY", - preferredInferenceApi: null, - gatewayName: "nemoclaw", - reservationSessionId: rebuiltSession.sessionId, - }); - }); - - it("aborts resume when re-upserting the routed provider fails (#4564)", async () => { - const session = createSession({ - provider: "custom-router", - model: "router/model", - endpointUrl: "http://localhost:4000/v1", - }); - session.steps.provider_selection.status = "complete"; - const { deps, calls } = createDeps({ - isInferenceRouteReady: vi.fn(() => true), - reupsertRoutedProvider: vi.fn(() => ({ - ok: false, - endpointUrl: "http://host.openshell.internal:4000/v1", - message: "provider update failed", - status: 7, - })), + const { deps } = createDeps({ + setupInference: vi.fn(async () => { + throw new Error("router ownership changed"); + }), }); - await expect( handleProviderInferenceState({ ...baseOptions(deps, session), resume: true, sandboxName: "router-sandbox", }), - ).rejects.toThrow("exit 7"); - - expect(calls.error).toHaveBeenCalledWith(" provider update failed"); - expect(calls.exit).toHaveBeenCalledWith(7); + ).rejects.toThrow("router ownership changed"); }); it("returns to provider selection when inference setup requests a retry", async () => { @@ -1492,3 +1358,44 @@ describe("handleProviderInferenceState", () => { ); }); }); + +it("resumes a verified external selection without reading or rewriting the shared route", async () => { + const f = registeredInferenceFixture(); + const binding = { ...f.binding, gatewayName: "nemoclaw", sandboxName: "my-assistant" }; + const selection = { + provider: binding.providerName, + model: "model-a", + endpointUrl: binding.endpointUrl, + credentialEnv: binding.credentialEnv, + preferredInferenceApi: binding.inferenceApi, + }; + const session = createSession({ sandboxName: binding.sandboxName, ...selection }); + session.steps.provider_selection.status = "complete"; + const isInferenceRouteReady = vi.fn(() => { + throw new Error("retired shared route"); + }); + const { deps, calls } = createDeps({ + isInferenceRouteReady, + getSandboxRegistryEntry: () => ({ + name: binding.sandboxName, + gatewayName: binding.gatewayName, + ...selection, + registeredInferenceProviderAttachment: binding, + }), + ensureResumeProviderReady: vi.fn(async () => ({ + forceInferenceSetup: false, + credentialEnv: binding.credentialEnv, + registeredNativeSetup: true, + })), + }); + await handleProviderInferenceState({ + ...baseOptions(deps, session), + resume: true, + authoritativeResumeConfig: true, + sandboxName: binding.sandboxName, + }); + expect(isInferenceRouteReady).not.toHaveBeenCalled(); + expect(calls.setupInference).not.toHaveBeenCalled(); + expect(calls.hydrate).not.toHaveBeenCalled(); + expect(calls.reupsertRoutedProvider).not.toHaveBeenCalled(); +}); diff --git a/src/lib/onboard/machine/handlers/provider-inference.ts b/src/lib/onboard/machine/handlers/provider-inference.ts index 6dc36c8e378..ec9113fd447 100644 --- a/src/lib/onboard/machine/handlers/provider-inference.ts +++ b/src/lib/onboard/machine/handlers/provider-inference.ts @@ -1,10 +1,13 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { recordedRegisteredInferenceProviderAttachment } from "../../../inference/config"; + import type { ModelSelectionProvenance } from "../../../domain/telemetry/provenance"; import { clearAutoDetectedCompatibleContextWindow } from "../../../inference/compatible-endpoint-context"; import { hasLegacySharedLocalInferenceRoute, + hasLegacySharedCustomInferenceRoute, isPersistedPiSharedLocalInferenceRoute, resolveAgentProviderInferenceApi, usesNativeLocalInference, @@ -301,6 +304,8 @@ export interface ProviderInferenceStateOptions { forceInferenceSetup: boolean; credentialEnv: string | null; nativeCustomSetup?: boolean; + nativeLocalSetup?: boolean; + registeredNativeSetup?: boolean; }>; ensureManagedLlamaCppResumeReady( provider: string | null | undefined, @@ -341,6 +346,7 @@ export interface ProviderInferenceStateOptions { ): void; isNonInteractive(): boolean; getOpenshellBinary(): string; + getInstalledOpenshellVersion?(): string | null; needsBedrockRuntimeAdapter(provider: string, endpointUrl: string | null): boolean; isInferenceRouteReady(gatewayName: string, provider: string, model: string): boolean; hasNativeLocalProviderAuthority?( @@ -605,6 +611,7 @@ function canResumeInferenceRoute(input: { forceProviderSelection: boolean; forceInferenceSetup: boolean; effectiveResume: boolean; + registeredResumeVerified?: boolean; selectsNativeLocal: boolean; nativeLocalAttached(): boolean; routeReady(): boolean; @@ -613,14 +620,12 @@ function canResumeInferenceRoute(input: { // another sandbox on a reused gateway can own the same shared route. return ( !input.needsBedrockRuntimeAdapter && - // Reconcile the managed router and reverify its native profile on resume. - !(input.provider === "nvidia-router" && input.selectsNativeLocal) && input.provider !== OPENROUTER_PROVIDER_NAME && !input.hasHostLocalInference && !input.forceProviderSelection && !input.forceInferenceSetup && input.effectiveResume && - input.routeReady() && + (input.registeredResumeVerified || input.routeReady()) && (!input.selectsNativeLocal || input.nativeLocalAttached()) ); } @@ -875,13 +880,15 @@ async function ensureAttachedLlamaCppReachable( ): Promise { if (provider !== "llama-cpp-local" || managed) return; const result = await (deps.probeLlamaCppSandboxReachability?.() ?? - probeHostServiceSandboxReachability({ port: LLAMA_CPP_PORT })); + probeHostServiceSandboxReachability({ port: LLAMA_CPP_PORT, nativeSupervisorNetwork: true })); if (result.ok || result.reason !== "tcp_failed") return; deps.error( ` Sandbox containers cannot reach Local llama.cpp at host.openshell.internal:${LLAMA_CPP_PORT}.`, ); deps.error( - ` Keep host-loopback access and bind or publish port ${LLAMA_CPP_PORT} on ${result.gatewayIp ?? "the Docker gateway address"} for the sandbox network.`, + result.networkName === "host" + ? ` Check that the llama.cpp server listens on 127.0.0.1:${LLAMA_CPP_PORT} for the native supervisor.` + : ` Keep host-loopback access and bind or publish port ${LLAMA_CPP_PORT} on ${result.gatewayIp ?? "the Docker gateway address"} for the sandbox network.`, ); deps.error(" Retain API-key authentication, check the host firewall, then retry onboarding."); deps.exitProcess(1); @@ -1359,6 +1366,40 @@ async function resolveSelectionSandboxName( return deps.promptValidatedSandboxName(agent); } +function registeredResumeSelection( + getSandbox: ((name: string) => Partial | null) | undefined, + resume: boolean, + sandboxName: string | null, + selection: { + gatewayName: string; + provider: string | null; + endpointUrl: string | null; + credentialEnv: string | null; + preferredInferenceApi: string | null; + }, +) { + if (!resume || !sandboxName) return undefined; + const binding = recordedRegisteredInferenceProviderAttachment(getSandbox?.(sandboxName) ?? {}); + if ( + binding && + (binding.gatewayName !== selection.gatewayName || + binding.providerName !== selection.provider || + binding.endpointUrl !== selection.endpointUrl || + binding.credentialEnv !== selection.credentialEnv || + binding.inferenceApi !== selection.preferredInferenceApi) + ) + throw new Error("Registered provider resume selection does not match its recorded binding."); + return binding; +} + +function hydrateResumedProviderCredential( + registeredResumeVerified: boolean, + credentialEnv: string | null, + hydrate: (name: string | null) => string | null | undefined, +): string | null | undefined { + return registeredResumeVerified ? null : hydrate(credentialEnv); +} + function nativeLocalAgentOptions( agent: unknown, provider: string, @@ -1374,24 +1415,29 @@ function nativeLocalAgentOptions( : {}; } -function assertNoLegacySharedLocalResume( +function assertNoLegacySharedResume( deps: Pick< ProviderInferenceStateOptions["deps"], - "getSandboxRegistryEntry" + "getSandboxRegistryEntry" | "getInstalledOpenshellVersion" >, effectiveResume: boolean, sandboxName: string | null, ): void { const recorded = effectiveResume && sandboxName ? deps.getSandboxRegistryEntry?.(sandboxName) : null; + if (!recorded || recorded.pendingRouteReservation === true) return; + const local = hasLegacySharedLocalInferenceRoute(recorded); + const custom = hasLegacySharedCustomInferenceRoute(recorded); + const legacyNvidia = recorded.provider === "nvidia-nim"; + if (!local && !custom && !legacyNvidia) return; + const oldRuntime = /^0\.0\.\d+$/.test(deps.getInstalledOpenshellVersion?.() ?? ""); if ( - recorded && - recorded.pendingRouteReservation !== true && - hasLegacySharedLocalInferenceRoute(recorded) && - recorded.agent !== "pi" + (local && (recorded.agent !== "pi" || !oldRuntime)) || + ((custom || legacyNvidia) && !oldRuntime) ) { + const route = local ? "local" : legacyNvidia ? "NVIDIA" : "custom"; throw new Error( - `Sandbox ${sandboxName} uses a legacy shared local inference route. Resume is unavailable for this route. Back up the sandbox, then re-onboard it with --fresh --name ${sandboxName} --recreate-sandbox to attach a native provider before upgrading OpenShell.`, + `Sandbox ${sandboxName} uses a legacy shared ${route} inference route. Resume is unavailable for this route. Back up the sandbox, then re-onboard it with --fresh --name ${sandboxName} --recreate-sandbox to attach a native provider.`, ); } } @@ -1414,15 +1460,17 @@ function usesNativeLocalSelection( selection: { provider: string | null; endpointUrl: string | null }, localNimSelected: boolean, preserveSharedRoute: boolean, + registeredSelection = false, ): boolean { return ( - !preserveSharedRoute && - usesNativeLocalInferenceForAgent( - agent, - selection.provider, - selection.endpointUrl, - localNimSelected, - ) + registeredSelection || + (!preserveSharedRoute && + usesNativeLocalInferenceForAgent( + agent, + selection.provider, + selection.endpointUrl, + localNimSelected, + )) ); } @@ -1506,6 +1554,7 @@ export async function handleProviderInferenceState({ let inferenceCapabilityCache: OnboardInferenceCapabilityCache | undefined; let vllmModelIdentity: string | undefined; const effectiveResume = resume && !fresh; + assertNoLegacySharedResume(deps, effectiveResume, sandboxName); const preservesLegacyPiRoute = createPersistedPiRouteSelector( deps.getSandboxRegistryEntry, effectiveResume, @@ -1620,6 +1669,13 @@ export async function handleProviderInferenceState({ clearAutoDetectedCompatibleContextWindow(process.env); let forceInferenceSetup = initialForceInferenceSetup; let recoveredRecordedProvider = false; + let registeredResumeVerified = false; + const registeredResumeBinding = registeredResumeSelection( + deps.getSandboxRegistryEntry, + effectiveResume, + reusableResumeSandboxName, + { gatewayName, provider, endpointUrl, credentialEnv, preferredInferenceApi }, + ); const providerRecovery = createRecovery(fresh, sandboxName, session, deps, { recoveryReceipt: providerRecoveryReceipt, recoveryReceiptLedger: providerRecoveryReceiptLedger, @@ -1660,7 +1716,7 @@ export async function handleProviderInferenceState({ forceInferenceSetup ||= completeRecoveredReviewSelectionAfterInference || reviewRecoveredInteractively; let managedLlamaCppRecovered = false; - assertNoLegacySharedLocalResume(deps, effectiveResume, sandboxName); + assertNoLegacySharedResume(deps, effectiveResume, sandboxName); if (resumeProviderSelection) { assertOnboardReasoningEffortRoute(reasoningEffortRequest, provider, preferredInferenceApi); assertProviderInferenceRouteCompatible( @@ -1673,6 +1729,7 @@ export async function handleProviderInferenceState({ { provider: provider, endpointUrl }, Boolean(nimContainer), preservesLegacyPiRoute(sandboxName, { provider: provider, model: model, endpointUrl }), + Boolean(registeredResumeBinding), ), ); // A completed provider-selection checkpoint is not proof that a managed @@ -1693,6 +1750,9 @@ export async function handleProviderInferenceState({ undefined, { sandboxName: reusableResumeSandboxName, endpointUrl }, ); + registeredResumeVerified = recovery.registeredNativeSetup === true; + reuseGatewayCredentialWithoutLocalKey ||= recovery.nativeLocalSetup === true; + skipHostInferenceSmoke ||= recovery.nativeLocalSetup === true; forceInferenceSetup ||= recovery.forceInferenceSetup; credentialEnv = recovery.credentialEnv; // Rebuild may be resuming a legacy session whose step marker was never @@ -1722,7 +1782,11 @@ export async function handleProviderInferenceState({ " [resume] Refreshing the gateway provider to match the required inference surface.", ); } - const hydratedCredential = deps.hydrateCredentialEnv(credentialEnv); + const hydratedCredential = hydrateResumedProviderCredential( + registeredResumeVerified, + credentialEnv, + deps.hydrateCredentialEnv, + ); // A retained OpenRouter gateway provider still owns the credential, // while the host adapter needs only its persisted authorization hash // to recover a stopped process. Do not export the gateway credential @@ -2042,8 +2106,9 @@ export async function handleProviderInferenceState({ provider: selectedProvider, hasHostLocalInference: Boolean(resumeHostLocalInferenceSetupOptions.hostLocalInference), forceProviderSelection, - forceInferenceSetup, + forceInferenceSetup: forceInferenceSetup || deps.isRoutedInferenceProvider(selectedProvider), effectiveResume, + registeredResumeVerified, selectsNativeLocal: usesNativeLocalSelection( agentName(agent), { provider: selectedProvider, endpointUrl }, @@ -2189,6 +2254,7 @@ export async function handleProviderInferenceState({ model: selectedModel, endpointUrl, }), + registeredResumeVerified, ), ); try { @@ -2267,6 +2333,7 @@ export async function handleProviderInferenceState({ model: selectedModel, endpointUrl, }), + registeredResumeVerified, ), ); return reserveRecoveredSandboxInferenceRoute( diff --git a/src/lib/onboard/machine/handlers/sandbox-checkpoint-crash-recovery.test.ts b/src/lib/onboard/machine/handlers/sandbox-checkpoint-crash-recovery.test.ts index 805fc1f9381..96ae3e2f72f 100644 --- a/src/lib/onboard/machine/handlers/sandbox-checkpoint-crash-recovery.test.ts +++ b/src/lib/onboard/machine/handlers/sandbox-checkpoint-crash-recovery.test.ts @@ -2,6 +2,10 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import fs from "node:fs"; +import YAML from "yaml"; +import { REPOSITORY_ROOT } from "../../../core/repository-root"; +import { messagingCredentialProviderProfilePath } from "../../../messaging/provider-profile"; import { decisionSelected, decisionUnset } from "../../../state/onboard-checkpoint-decision"; import { @@ -92,13 +96,9 @@ type StubbedRunOpenshellResult = { status: number; stdout: string; stderr: strin const OK_RESULT: StubbedRunOpenshellResult = { status: 0, stdout: "", stderr: "" }; const EXACT_MESSAGING_PROFILE: StubbedRunOpenshellResult = { status: 0, - stdout: JSON.stringify({ - id: "nemoclaw-mcp-v1", - credentials: [], - endpoints: [], - binaries: [], - inference_capable: false, - }), + stdout: JSON.stringify( + YAML.parse(fs.readFileSync(messagingCredentialProviderProfilePath(REPOSITORY_ROOT), "utf8")), + ), stderr: "", }; const EXACT_BRAVE_PROFILE: StubbedRunOpenshellResult = { @@ -165,7 +165,10 @@ function fakeGatewayRunOpenshell() { const exactProfileExports = new Map([ ["provider profile -g nemoclaw export brave --output json", EXACT_BRAVE_PROFILE], - ["provider profile -g nemoclaw export nemoclaw-mcp-v1 --output json", EXACT_MESSAGING_PROFILE], + [ + "provider profile -g nemoclaw export nemoclaw-messaging-v1 --output json", + EXACT_MESSAGING_PROFILE, + ], ]); const rejectUnexpectedProfileCommand = (args: string[]): never => { throw new Error(`Unexpected provider profile command: ${args.join(" ")}`); @@ -951,7 +954,7 @@ describe("sandbox crash-recovery replay (#5961, #6228)", () => { name: "my-assistant-discord-bridge", envKey: "DISCORD_BOT_TOKEN", token: "discord-secret", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, ], true, @@ -991,7 +994,7 @@ describe("sandbox crash-recovery replay (#5961, #6228)", () => { expect(resumedSession.checkpoint?.bindings.registeredProviders).toEqual([ { name: "my-assistant-discord-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "DISCORD_BOT_TOKEN", }, ]); @@ -1032,7 +1035,7 @@ describe("sandbox crash-recovery replay (#5961, #6228)", () => { expect(stageSandboxCredentialProviders).toHaveBeenCalledTimes(2); expect(providerMatchesGatewayCredential).toHaveBeenCalledWith( "my-assistant-discord-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "DISCORD_BOT_TOKEN", ); expect(getSession().checkpoint?.effectGroups.messaging_providers).toBeUndefined(); diff --git a/src/lib/onboard/machine/handlers/sandbox-create-intent-boundary.test.ts b/src/lib/onboard/machine/handlers/sandbox-create-intent-boundary.test.ts index 5645ddb9184..37b3c140285 100644 --- a/src/lib/onboard/machine/handlers/sandbox-create-intent-boundary.test.ts +++ b/src/lib/onboard/machine/handlers/sandbox-create-intent-boundary.test.ts @@ -495,7 +495,7 @@ describe("sandbox create intent machine boundary", () => { (name: string, type: string, credentialEnvName: string) => (name === "tm-brave-search" && type === "brave" && credentialEnvName === "BRAVE_API_KEY") || (name === "tm-telegram-bridge" && - type === "nemoclaw-mcp-v1" && + type === "nemoclaw-messaging-v1" && credentialEnvName === "TELEGRAM_BOT_TOKEN"), ); const stageSandboxCredentialProviders = vi @@ -509,7 +509,7 @@ describe("sandbox create intent machine boundary", () => { return [ { name: "tm-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "TELEGRAM_BOT_TOKEN", }, ]; @@ -579,7 +579,7 @@ describe("sandbox create intent machine boundary", () => { requiredBindings: [ { name: "tm-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "TELEGRAM_BOT_TOKEN", }, ], @@ -610,7 +610,7 @@ describe("sandbox create intent machine boundary", () => { ); expect(providerMatchesGatewayCredential).toHaveBeenCalledWith( "tm-telegram-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "TELEGRAM_BOT_TOKEN", ); expect(calls.promptName).not.toHaveBeenCalled(); diff --git a/src/lib/onboard/machine/handlers/sandbox-messaging.test.ts b/src/lib/onboard/machine/handlers/sandbox-messaging.test.ts index f07e1d53909..cf6f7d3d2a7 100644 --- a/src/lib/onboard/machine/handlers/sandbox-messaging.test.ts +++ b/src/lib/onboard/machine/handlers/sandbox-messaging.test.ts @@ -1010,7 +1010,7 @@ describe("reconcileSandboxMessaging plan authority", () => { expect(deps.providerMatchesGatewayCredential).toHaveBeenCalledWith( "alpha-slack-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "SLACK_BOT_TOKEN", ); expect(deps.setupMessagingChannels).not.toHaveBeenCalled(); @@ -1368,7 +1368,7 @@ describe("reconcileSandboxMessaging completed checkpoint credentials", () => { expect(deps.providerMatchesGatewayCredential).toHaveBeenCalledWith( "alpha-telegram-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "TELEGRAM_BOT_TOKEN", ); expect(deps.setupMessagingChannels).not.toHaveBeenCalled(); @@ -1390,7 +1390,7 @@ describe("reconcileSandboxMessaging completed checkpoint credentials", () => { expect(deps.providerMatchesGatewayCredential).toHaveBeenCalledWith( "alpha-telegram-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "TELEGRAM_BOT_TOKEN", ); expect(deps.setupMessagingChannels).not.toHaveBeenCalled(); diff --git a/src/lib/onboard/machine/handlers/sandbox-native-router-resume.test.ts b/src/lib/onboard/machine/handlers/sandbox-native-router-resume.test.ts index 8887af6beb2..63a03e599b6 100644 --- a/src/lib/onboard/machine/handlers/sandbox-native-router-resume.test.ts +++ b/src/lib/onboard/machine/handlers/sandbox-native-router-resume.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import * as routerProfile from "../../../core/model-router-port"; import { afterEach, describe, expect, it, vi } from "vitest"; import * as managedProviderAdapter from "../../../adapters/openshell/managed-provider-adapter"; import * as providerCommand from "../../../adapters/openshell/provider-command"; @@ -19,10 +20,14 @@ vi.mock("../../messaging-channel-setup", () => ({ detectUnconfiguredMessagingChannels: vi.fn(() => []), })); -function routerResumeHarness(attached = false) { +function routerResumeHarness( + attached = false, + agent: "openclaw" | "hermes" | "pi" | "langchain-deepagents-code" = "openclaw", + provider = "nvidia-router", +) { const expected = nativeLocalSetupReceipt({ - agent: "openclaw", - provider: "nvidia-router", + ...(agent === "langchain-deepagents-code" ? {} : { agent }), + provider, sandboxName: "saved", gatewayName: "nemoclaw", endpointUrl: "http://host.openshell.internal:4000/v1", @@ -30,8 +35,8 @@ function routerResumeHarness(attached = false) { }); const entry: SandboxEntry = { name: "saved", - agent: "openclaw", - provider: "nvidia-router", + agent, + provider, model: "model", endpointUrl: expected.endpointUrl, credentialEnv: "NVIDIA_INFERENCE_API_KEY", @@ -41,8 +46,9 @@ function routerResumeHarness(attached = false) { lifecycleGeneration: "generation-1", lifecycleLiveIdentityFingerprint: "live-1", toolDisclosure: "progressive", + observabilityEnabled: false, }; - const session = createSession({ sandboxName: "saved" }); + const session = createSession({ sandboxName: "saved", agent }); session.steps.sandbox.status = "complete"; const native = createNativeLocalSetupHarness(); native.seed(expected); @@ -101,8 +107,8 @@ function routerResumeHarness(attached = false) { ...baseOptions(deps, session), resume: true, sandboxName: "saved", - agent: { name: "openclaw" }, - provider: "nvidia-router", + agent: { name: agent }, + provider, endpointUrl: expected.endpointUrl, credentialEnv: "NVIDIA_INFERENCE_API_KEY", }), @@ -124,21 +130,30 @@ function routerResumeHarness(attached = false) { describe("native Model Router sandbox resume", () => { afterEach(() => vi.restoreAllMocks()); - it("restores the recorded attachment before completing Ready sandbox reuse (#13560)", async () => { - const h = routerResumeHarness(); - await expect(h.run()).resolves.toMatchObject({ sandboxName: "saved" }); - expect(h.isAttached()).toBe(true); - expect(h.native.adapter.attachProvider).toHaveBeenCalledExactlyOnceWith({ - target: { kind: "named", gatewayName: "nemoclaw" }, - sandboxName: "saved", - providerName: h.expected.providerName, - }); - expect(h.calls.createSandbox).not.toHaveBeenCalled(); - expect(h.calls.recordSkip).toHaveBeenCalledOnce(); - expect(h.calls.recordSkip.mock.invocationCallOrder[0]).toBeGreaterThan( - h.native.adapter.attachProvider.mock.invocationCallOrder[0]!, - ); - }); + it.each(["openclaw", "hermes", "pi", "langchain-deepagents-code"] as const)( + "restores the recorded %s attachment before completing Ready sandbox reuse (#13560)", + async (agent) => { + vi.spyOn(routerProfile, "loadBlueprintProfile").mockReturnValue({ + model: "nvidia-routed", + provider_name: "team-router", + endpoint: "http://localhost:4000/v1", + router: { enabled: true, port: 4000 }, + }); + const h = routerResumeHarness(false, agent, "team-router"); + await expect(h.run()).resolves.toMatchObject({ sandboxName: "saved" }); + expect(h.isAttached()).toBe(true); + expect(h.native.adapter.attachProvider).toHaveBeenCalledExactlyOnceWith({ + target: { kind: "named", gatewayName: "nemoclaw" }, + sandboxName: "saved", + providerName: h.expected.providerName, + }); + expect(h.calls.createSandbox).not.toHaveBeenCalled(); + expect(h.calls.recordSkip).toHaveBeenCalledOnce(); + expect(h.calls.recordSkip.mock.invocationCallOrder[0]).toBeGreaterThan( + h.native.adapter.attachProvider.mock.invocationCallOrder[0]!, + ); + }, + ); it("verifies an already-attached provider without attaching it again (#13560)", async () => { const h = routerResumeHarness(true); @@ -161,11 +176,14 @@ describe("native Model Router sandbox resume", () => { it("refuses a changed native endpoint boundary before attachment (#13560)", async () => { const h = routerResumeHarness(); - h.entry.nativeLocalProviderAttachment = nativeLocalSetupReceipt({ + h.entry.nativeLocalProviderAttachment = { ...h.expected, endpointUrl: "http://host.openshell.internal:5000/v1", - }); - await expect(h.run()).rejects.toThrow("recorded sandbox attachment authority"); + }; + await expect(h.run()).rejects.toThrow("exit 1"); + expect(h.calls.error).toHaveBeenCalledWith( + expect.stringContaining("native local inference reservation"), + ); expect(h.native.adapter.attachProvider).not.toHaveBeenCalled(); expect(h.calls.recordSkip).not.toHaveBeenCalled(); }); diff --git a/src/lib/onboard/machine/handlers/sandbox-provider-effect-replay.test.ts b/src/lib/onboard/machine/handlers/sandbox-provider-effect-replay.test.ts index e19a3fcee95..fd06368b1e7 100644 --- a/src/lib/onboard/machine/handlers/sandbox-provider-effect-replay.test.ts +++ b/src/lib/onboard/machine/handlers/sandbox-provider-effect-replay.test.ts @@ -105,12 +105,12 @@ describe("handleSandboxState provider effect replay", () => { const slackAppToken = "xapp-current-token"; const slackBotBinding = { name: "my-assistant-slack-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "SLACK_BOT_TOKEN", }; const slackAppBinding = { name: "my-assistant-slack-app", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "SLACK_APP_TOKEN", }; const slackProviderBindings = [slackBotBinding, slackAppBinding]; @@ -141,7 +141,7 @@ describe("handleSandboxState provider effect replay", () => { }; const telegramBinding = { name: "my-assistant-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "TELEGRAM_BOT_TOKEN", }; const session = createSession({ @@ -318,7 +318,7 @@ describe("handleSandboxState provider effect replay", () => { }; const oldMessagingBinding = { name: "my-assistant-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "TELEGRAM_BOT_TOKEN", }; const session = createSession({ sandboxName: "my-assistant" }); @@ -465,7 +465,7 @@ describe("handleSandboxState provider effect replay", () => { }; const messagingBinding = { name: "my-assistant-telegram-bridge", - type: "nemoclaw-mcp-v1", + type: "nemoclaw-messaging-v1", credentialEnv: "TELEGRAM_BOT_TOKEN", }; const telegramToken = "telegram-current-token"; diff --git a/src/lib/onboard/machine/handlers/sandbox-ready-messaging.test.ts b/src/lib/onboard/machine/handlers/sandbox-ready-messaging.test.ts index 59a2dcd508d..295b23df682 100644 --- a/src/lib/onboard/machine/handlers/sandbox-ready-messaging.test.ts +++ b/src/lib/onboard/machine/handlers/sandbox-ready-messaging.test.ts @@ -224,7 +224,7 @@ describe("handleSandboxState Ready sandbox messaging", () => { expect(inspectGatewayCredential).toHaveBeenCalledWith( "saved-telegram-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "TELEGRAM_BOT_TOKEN", ); expect(calls.createSandbox).not.toHaveBeenCalled(); diff --git a/src/lib/onboard/machine/handlers/sandbox-resume.ts b/src/lib/onboard/machine/handlers/sandbox-resume.ts index 3877da48e2f..5f23573dfbd 100644 --- a/src/lib/onboard/machine/handlers/sandbox-resume.ts +++ b/src/lib/onboard/machine/handlers/sandbox-resume.ts @@ -1,6 +1,12 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { isDeepStrictEqual } from "node:util"; +import { + isModelRouterProvider, + nativeModelRouterEndpoint, +} from "../../../inference/native-local/model-router"; +import { createRegisteredSandboxIdentityRevalidation } from "../../sandbox-recreate-transaction"; import { createManagedProviderAdapter } from "../../../adapters/openshell/managed-provider-adapter"; import { runOpenshellProviderCommand } from "../../../adapters/openshell/provider-command"; import { ensureNativeLocalProviderAttached } from "../../../inference/native-local/profile"; @@ -472,3 +478,60 @@ export function reserveSandboxResumeRoute( ); } } + +/** Caller holds the sandbox and gateway locks until reuse is complete. */ +export async function restoreNativeRouterAttachment(input: { + sandboxName: string | null; + provider: string; + endpointUrl: string | null; + readRegistration: (name: string) => SandboxEntry | null; + observe: Parameters[1]["observe"]; + acceptsReservation(entry: SandboxEntry, sandboxName: string): boolean; + assertGatewayRouteCompatible(sandboxName: string): void; + adapter?: Parameters[0]["adapter"]; +}): Promise { + const { sandboxName } = input; + if (!isModelRouterProvider(input.provider)) return; + const entry = sandboxName ? input.readRegistration(sandboxName) : null; + const expected = entry?.nativeLocalProviderAttachment + ? structuredClone(entry.nativeLocalProviderAttachment) + : undefined; + if ( + !sandboxName || + !entry || + entry.name !== sandboxName || + !expected || + !input.acceptsReservation(entry, sandboxName) || + expected.endpointUrl !== input.endpointUrl || + !nativeModelRouterEndpoint(input.provider, expected.endpointUrl) + ) { + throw new Error( + "Native Model Router resume requires its recorded sandbox attachment authority.", + ); + } + const revalidateIdentity = createRegisteredSandboxIdentityRevalidation(entry, { + readRegistration: input.readRegistration, + observe: input.observe, + }); + if (!revalidateIdentity) { + throw new Error( + "Native Model Router resume requires a verified sandbox identity. Recreate this sandbox.", + ); + } + const revalidateSandboxIdentity = (operation: string): void => { + revalidateIdentity(operation); + input.assertGatewayRouteCompatible(sandboxName); + const current = input.readRegistration(sandboxName); + if (!isDeepStrictEqual(current?.nativeLocalProviderAttachment, expected)) { + throw new Error("Native Model Router attachment authority changed during resume."); + } + }; + revalidateSandboxIdentity("restore native Model Router attachment"); + await ensureResumedNativeLocalProviderAttached({ + adapter: input.adapter, + sandboxName, + expected, + revalidateSandboxIdentity, + }); + revalidateSandboxIdentity("complete native Model Router attachment recovery"); +} diff --git a/src/lib/onboard/machine/handlers/sandbox-route-publication.test.ts b/src/lib/onboard/machine/handlers/sandbox-route-publication.test.ts index 9b6285132c2..309f7a78628 100644 --- a/src/lib/onboard/machine/handlers/sandbox-route-publication.test.ts +++ b/src/lib/onboard/machine/handlers/sandbox-route-publication.test.ts @@ -598,7 +598,7 @@ describe.each(["langchain-deepagents-code", "openclaw", "hermes", "pi"] as const it("rejects a changed legacy source before starting recreation (#13197)", async () => { const { row, reserve, calls, run } = nativeRecreationHarness(agentName); row.model = "another-model"; - await expect(run()).rejects.toThrow("legacy route ownership changed"); + await expect(run()).rejects.toThrow("recorded route ownership changed"); expect(reserve).not.toHaveBeenCalled(); expect(calls.createSandbox).not.toHaveBeenCalled(); }); @@ -614,7 +614,7 @@ describe("Hermes staged recreation authority", () => { ])("rejects $label before recreation or reservation", async ({ receipt }) => { const { row, source, attachment, reserve, calls, run } = nativeRecreationHarness("hermes"); Object.assign(attachment, receipt); - await expect(run()).rejects.toThrow("legacy route ownership changed"); + await expect(run()).rejects.toThrow("recorded route ownership changed"); expect(reserve).not.toHaveBeenCalled(); expect(calls.createSandbox).not.toHaveBeenCalled(); expect(row).toEqual(source); @@ -626,7 +626,7 @@ describe("Hermes staged recreation authority", () => { { label: "missing session", options: { session: null } }, ])("retains the $label refusal", async ({ options }) => { const { row, source, reserve, calls, run } = nativeRecreationHarness("hermes"); - await expect(run(options)).rejects.toThrow("legacy route ownership changed"); + await expect(run(options)).rejects.toThrow("recorded route ownership changed"); expect(reserve).not.toHaveBeenCalled(); expect(calls.createSandbox).not.toHaveBeenCalled(); expect(row).toEqual(source); @@ -640,9 +640,53 @@ describe("Hermes staged recreation authority", () => { await intent.reservePublishedRouteAfterBackup(); throw new Error("must not reserve a changed source"); }; - await expect(run()).rejects.toThrow("legacy route ownership changed"); + await expect(run()).rejects.toThrow("recorded route ownership changed"); expect(reserve).not.toHaveBeenCalled(); expect(row.model).toBe("concurrent-selection"); expect(row.nativeLocalProviderAttachment).toBeUndefined(); }); }); + +describe("published native model replacement", () => { + it("reserves an implicit model replacement only after backup", async () => { + const { row, source, attachment, reserve, deps, run } = nativeRecreationHarness("openclaw"); + row.nativeLocalProviderAttachment = attachment; + source.nativeLocalProviderAttachment = structuredClone(attachment); + deps.createSandbox = async (...args: unknown[]) => { + expect(row).toEqual(source); + expect(reserve).not.toHaveBeenCalled(); + const intent = args.at(-2) as { reservePublishedRouteAfterBackup: () => Promise }; + await intent.reservePublishedRouteAfterBackup(); + expect(row.model).toBe("replacement-model"); + return "saved"; + }; + await run({ model: "replacement-model", recreateSandbox: () => false }); + expect(reserve).toHaveBeenCalledOnce(); + }); + + it("retains the full native source when backup refuses a model replacement", async () => { + const { row, source, attachment, reserve, deps, run } = nativeRecreationHarness("openclaw"); + row.nativeLocalProviderAttachment = attachment; + source.nativeLocalProviderAttachment = structuredClone(attachment); + deps.createSandbox = async () => { + throw new Error("backup refused"); + }; + await expect(run({ model: "replacement-model", recreateSandbox: () => false })).rejects.toThrow( + "backup refused", + ); + expect(row).toEqual(source); + expect(reserve).not.toHaveBeenCalled(); + }); + + it("rejects a replacement provider identity without explicit recreation", async () => { + const { row, source, attachment, reserve, calls, run } = nativeRecreationHarness("openclaw"); + row.nativeLocalProviderAttachment = { ...attachment, providerId: "previous-provider" }; + source.nativeLocalProviderAttachment = structuredClone(row.nativeLocalProviderAttachment); + await expect(run({ model: "replacement-model", recreateSandbox: () => false })).rejects.toThrow( + "route ownership changed", + ); + expect(row).toEqual(source); + expect(reserve).not.toHaveBeenCalled(); + expect(calls.createSandbox).not.toHaveBeenCalled(); + }); +}); diff --git a/src/lib/onboard/machine/handlers/sandbox.ts b/src/lib/onboard/machine/handlers/sandbox.ts index dce87a3ef84..b53f10fa4ad 100644 --- a/src/lib/onboard/machine/handlers/sandbox.ts +++ b/src/lib/onboard/machine/handlers/sandbox.ts @@ -103,6 +103,7 @@ import { import { nativeInferenceCreateIntentFields, nativeLocalReservationStatus, + retainsNativeLocalProviderAttachment, sandboxCreateInferenceSelection, withSandboxImageRegistrationFence, } from "../../sandbox-registration"; @@ -123,6 +124,7 @@ import { assertHermesSwitchyardRoutingReusable, decideSandboxResume, ensureResumedNativeLocalProviderAttached, + restoreNativeRouterAttachment, hasCompatibleEndpointReasoningDrift, hasHermesCompatibleAnthropicInferenceRouteDrift, hasHostMountConfigDrift, @@ -374,6 +376,7 @@ export interface SandboxStateOptions< resolveSandboxCreateIntent(input: { sandboxName: string; inferenceProvider?: string | null; + registeredInferenceProviderAttachment?: SandboxEntry["registeredInferenceProviderAttachment"]; nativeLocalProviderAttachment?: SandboxEntry["nativeLocalProviderAttachment"]; nativeNvidiaProviderAttachment?: SandboxEntry["nativeNvidiaProviderAttachment"]; nativeHostedProviderAttachment?: SandboxEntry["nativeHostedProviderAttachment"]; @@ -1348,10 +1351,14 @@ class SandboxStateFlow< ): SandboxEntry | null { const staged = this.options.stagedNativeLocalRecreation; if (!staged) return entry; + const retainsNativeBinding = retainsNativeLocalProviderAttachment( + staged.source, + staged.attachment, + ); const agentName = (this.options.agent as { name?: string } | null)?.name ?? "openclaw"; if ( this.options.resume || - !this.options.recreateSandbox(false) || + (!this.options.recreateSandbox(false) && !retainsNativeBinding) || !this.options.session?.sessionId || (agentName !== "langchain-deepagents-code" && agentName !== "openclaw" && @@ -1365,7 +1372,7 @@ class SandboxStateFlow< !isDeepStrictEqual(entry, staged.source) ) { throw new Error( - `Sandbox '${sandboxName}' legacy route ownership changed before recreation. Retry onboarding.`, + `Sandbox '${sandboxName}' recorded route ownership changed before recreation. Retry onboarding.`, ); } return { ...staged.source, nativeLocalProviderAttachment: staged.attachment }; @@ -1467,52 +1474,6 @@ class SandboxStateFlow< throw new Error("exitProcess returned after messaging channel state changed"); } - private async restoreNativeRouterAttachment(sandboxName: string | null): Promise { - const agentName = (this.options.agent as { name?: string } | null)?.name ?? "openclaw"; - if (agentName !== "openclaw" || this.options.provider !== "nvidia-router") return; - const entry = sandboxName ? this.deps.getSandboxRegistryEntry(sandboxName) : null; - const expected = entry?.nativeLocalProviderAttachment - ? structuredClone(entry.nativeLocalProviderAttachment) - : undefined; - if ( - !sandboxName || - !entry || - entry.name !== sandboxName || - !expected || - !this.acceptsNativeLocalReservation(entry, sandboxName) || - expected.endpointUrl !== this.options.endpointUrl - ) { - throw new Error( - "Native Model Router resume requires its recorded sandbox attachment authority.", - ); - } - const revalidateIdentity = createRegisteredSandboxIdentityRevalidation(entry, { - readRegistration: this.deps.getSandboxRegistryEntry, - observe: this.deps.getSandboxRecreateObservation, - }); - if (!revalidateIdentity) { - throw new Error( - "Native Model Router resume requires a verified sandbox identity. Recreate this sandbox.", - ); - } - const revalidateSandboxIdentity = (operation: string): void => { - revalidateIdentity(operation); - this.assertGatewayRouteCompatible(sandboxName); - const current = this.deps.getSandboxRegistryEntry(sandboxName); - if (!isDeepStrictEqual(current?.nativeLocalProviderAttachment, expected)) { - throw new Error("Native Model Router attachment authority changed during resume."); - } - }; - revalidateSandboxIdentity("restore native Model Router attachment"); - await ensureResumedNativeLocalProviderAttached({ - adapter: this.deps.nativeLocalProviderAdapter, - sandboxName, - expected, - revalidateSandboxIdentity, - }); - revalidateSandboxIdentity("complete native Model Router attachment recovery"); - } - private async reuseSandbox( state: SandboxStepState, note?: string, @@ -1520,7 +1481,16 @@ class SandboxStateFlow< return this.deps.withGatewayRouteMutationLock(this.options.gatewayName, async () => { await this.assertCheckpointBindingsStillLive(state); this.assertGatewayRouteCompatible(state.sandboxName); - await this.restoreNativeRouterAttachment(state.sandboxName); + await restoreNativeRouterAttachment({ + sandboxName: state.sandboxName, + provider: this.options.provider, + endpointUrl: this.options.endpointUrl, + readRegistration: this.deps.getSandboxRegistryEntry, + observe: this.deps.getSandboxRecreateObservation, + acceptsReservation: (entry, name) => this.acceptsNativeLocalReservation(entry, name), + assertGatewayRouteCompatible: (name) => this.assertGatewayRouteCompatible(name), + adapter: this.deps.nativeLocalProviderAdapter, + }); if (state.webSearchConfig) { const provider = webSearchProviderForConfig( state.webSearchConfig as unknown as SharedWebSearchConfig, diff --git a/src/lib/onboard/machine/resume-provider-shim.test.ts b/src/lib/onboard/machine/resume-provider-shim.test.ts index 39b428e82ad..5db7f4ab196 100644 --- a/src/lib/onboard/machine/resume-provider-shim.test.ts +++ b/src/lib/onboard/machine/resume-provider-shim.test.ts @@ -7,6 +7,8 @@ vi.mock("../../credentials/store", () => ({ resolveProviderCredential: vi.fn(() => null), })); +import { nativeLocalSetupReceipt } from "../../../../test/support/native-local-setup-harness"; +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; import { customNativeProvider } from "../../inference/native-provider/custom"; import { createResumeProviderShim } from "./resume-provider-shim"; @@ -15,7 +17,7 @@ describe("createResumeProviderShim", () => { vi.restoreAllMocks(); }); - it("binds the selected gateway and recovery dependencies", async () => { + it("sends routed recovery to native setup without querying a logical provider", async () => { vi.spyOn(console, "log").mockImplementation(() => undefined); const providerExistsInGateway = vi.fn(() => false); const isRoutedInferenceProvider = vi.fn(() => true); @@ -33,15 +35,9 @@ describe("createResumeProviderShim", () => { forceInferenceSetup: true, credentialEnv: "OPENAI_API_KEY", }); - expect(providerExistsInGateway).toHaveBeenCalledWith("routed-provider", "gateway-west"); + expect(providerExistsInGateway).not.toHaveBeenCalled(); expect(isRoutedInferenceProvider).toHaveBeenCalledWith("routed-provider"); - expect(replaceNamedCredential).toHaveBeenCalledWith( - "OPENAI_API_KEY", - expect.any(String), - null, - expect.any(Function), - undefined, - ); + expect(replaceNamedCredential).not.toHaveBeenCalled(); }); it.each([null, "NVIDIA_INFERENCE_API_KEY", "TEAM_NVIDIA_KEY"])( @@ -55,7 +51,6 @@ describe("createResumeProviderShim", () => { const providerExistsInGateway = vi.fn((name: string) => name === "nemoclaw-nvidia-prod-v1"); const replaceNamedCredential = vi.fn(async () => "unexpected"); const getSandbox = vi.fn(() => ({ - pendingRouteReservation: false, nativeNvidiaProviderAttachment: { schemaVersion: 1 as const, profileId: "nemoclaw-nvidia-inference-v1" as const, @@ -115,7 +110,7 @@ describe("createResumeProviderShim", () => { getSandbox: () => ({}), isNonInteractive: () => true, providerExistsInGateway, - isRoutedInferenceProvider: () => true, + isRoutedInferenceProvider: (provider) => provider === "nvidia-router", replaceNamedCredential: vi.fn(async () => "unexpected"), }); await expect( @@ -206,7 +201,7 @@ it.each([ custom: definition.custom, }, } - : { pendingRouteReservation: state === "reservation" }; + : { ...(state === "reservation" ? { pendingRouteReservation: true as const } : {}) }; const providerExistsInGateway = vi.fn(() => true); const replaceNamedCredential = vi.fn(async () => "unexpected"); const shim = createResumeProviderShim({ @@ -338,3 +333,173 @@ it("rejects an altered native attachment before recovering its credential bindin expect(h.providerExistsInGateway).not.toHaveBeenCalled(); expect(h.replaceNamedCredential).not.toHaveBeenCalled(); }); + +it.each(["retained", "recreated", "replaced", "wrong-endpoint"] as const)( + "resumes only the exact external binding: %s", + async (state) => { + const f = registeredInferenceFixture(state === "retained"); + const entry = { + name: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + provider: f.binding.providerName, + endpointUrl: f.binding.endpointUrl, + credentialEnv: f.binding.credentialEnv, + preferredInferenceApi: f.binding.inferenceApi, + registeredInferenceProviderAttachment: f.binding, + }; + const shim = createResumeProviderShim({ + getSandbox: () => entry, + providerAdapter: f.adapter, + isNonInteractive: () => true, + providerExistsInGateway: vi.fn(), + isRoutedInferenceProvider: () => false, + replaceNamedCredential: vi.fn(), + }); + f.state.id = state === "replaced" ? "replacement" : f.state.id; + const result = shim.ensureResumeProviderReady( + f.binding.gatewayName, + f.binding.providerName, + f.binding.credentialEnv, + undefined, + { + sandboxName: f.binding.sandboxName, + endpointUrl: + state === "wrong-endpoint" ? "https://other.example/v1" : f.binding.endpointUrl, + }, + ); + expect( + await result.then( + (value) => value, + () => "rejected", + ), + ).toEqual( + state === "retained" || state === "recreated" + ? { + forceInferenceSetup: false, + credentialEnv: f.binding.credentialEnv, + registeredNativeSetup: true, + } + : "rejected", + ); + expect(f.adapter.attachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + }, +); + +it.each(["openclaw", "hermes"] as const)( + "resumes %s native local credentials through the exact recorded provider", + async (agent) => { + const receipt = nativeLocalSetupReceipt({ + agent, + provider: "compatible-endpoint", + gatewayName: "gateway-west", + sandboxName: "alpha", + endpointUrl: "http://host.openshell.internal:8000/v1", + authMode: "authenticated", + }); + const providerExistsInGateway = vi.fn(() => false); + const replaceNamedCredential = vi.fn(); + const shim = createResumeProviderShim({ + getSandbox: () => ({ + agent, + provider: receipt.provider, + credentialEnv: "TEAM_KEY", + nativeLocalProviderAttachment: receipt, + }), + isNonInteractive: () => true, + providerExistsInGateway, + isRoutedInferenceProvider: () => false, + replaceNamedCredential, + }); + await expect( + shim.ensureResumeProviderReady("gateway-west", "compatible-endpoint", "TEAM_KEY", undefined, { + sandboxName: "alpha", + endpointUrl: "http://localhost:8000/v1", + }), + ).resolves.toEqual({ + forceInferenceSetup: true, + credentialEnv: "TEAM_KEY", + nativeCustomSetup: true, + nativeLocalSetup: true, + }); + expect(providerExistsInGateway).not.toHaveBeenCalled(); + expect(replaceNamedCredential).not.toHaveBeenCalled(); + }, +); + +it.each([ + { + gatewayName: "other", + sandboxName: "alpha", + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "TEAM_KEY", + }, + { + gatewayName: "gateway-west", + sandboxName: "other", + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "TEAM_KEY", + }, + { + gatewayName: "gateway-west", + sandboxName: "alpha", + provider: "compatible-endpoint", + recordedProvider: "vllm-local", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "TEAM_KEY", + }, + { + gatewayName: "gateway-west", + sandboxName: "alpha", + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8001/v1", + credentialEnv: "TEAM_KEY", + }, + { + gatewayName: "gateway-west", + sandboxName: "alpha", + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "OTHER_KEY", + }, +])( + "refuses changed native local resume selection before legacy recovery: %j", + async (selection) => { + const receipt = nativeLocalSetupReceipt({ + agent: "openclaw", + provider: "compatible-endpoint", + gatewayName: "gateway-west", + sandboxName: "alpha", + endpointUrl: "http://host.openshell.internal:8000/v1", + authMode: "authenticated", + }); + const providerExistsInGateway = vi.fn(() => true); + const replaceNamedCredential = vi.fn(); + const shim = createResumeProviderShim({ + getSandbox: () => ({ + agent: "openclaw", + provider: selection.recordedProvider ?? receipt.provider, + credentialEnv: "TEAM_KEY", + nativeLocalProviderAttachment: receipt, + }), + isNonInteractive: () => true, + providerExistsInGateway, + isRoutedInferenceProvider: () => false, + replaceNamedCredential, + }); + await expect( + shim.ensureResumeProviderReady( + selection.gatewayName, + selection.provider, + selection.credentialEnv, + undefined, + selection, + ), + ).rejects.toThrow("does not match its recorded binding"); + expect(providerExistsInGateway).not.toHaveBeenCalled(); + expect(replaceNamedCredential).not.toHaveBeenCalled(); + }, +); diff --git a/src/lib/onboard/machine/resume-provider-shim.ts b/src/lib/onboard/machine/resume-provider-shim.ts index f527de36fcd..25482f4d742 100644 --- a/src/lib/onboard/machine/resume-provider-shim.ts +++ b/src/lib/onboard/machine/resume-provider-shim.ts @@ -1,11 +1,18 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { createManagedProviderAdapter } from "../../adapters/openshell/managed-provider-adapter"; +import { + recordedRegisteredInferenceProviderAttachment, + registeredInferenceProviderRevision, +} from "../../inference/native-provider/registered"; +import { normalizeNativeLocalProviderAttachment } from "../../inference/native-local/profile"; +import { gatewayReachableCompatibleEndpointUrl } from "../../inference/native-local/eligibility"; +import type { SandboxEntry } from "../../state/registry"; import { runOpenshell } from "../../adapters/openshell/runtime"; import { normalizeNativeNvidiaProviderAttachment, NVIDIA_HOSTED_CREDENTIAL_ENV, NVIDIA_HOSTED_LOGICAL_PROVIDER, - type NativeNvidiaProviderAttachment, } from "../../inference/native-nvidia/contract"; import { recordedCustomCredentialEnv, @@ -33,12 +40,8 @@ const onboardProviders = require("../providers") as { }; export type ResumeProviderShimDeps = { - getSandbox?: (name: string) => - | (NonNullable[3]> & { - nativeNvidiaProviderAttachment?: NativeNvidiaProviderAttachment; - }) - | null - | undefined; + getSandbox?: (name: string) => Partial | null | undefined; + providerAdapter?: ReturnType; isNonInteractive: ResumeProviderRecoveryDeps["isNonInteractive"]; providerExistsInGateway(name: string, gatewayName: string): boolean | Promise; isRoutedInferenceProvider: ResumeProviderRecoveryDeps["isRoutedInferenceProvider"]; @@ -85,6 +88,61 @@ export function createResumeProviderShim(deps: ResumeProviderShimDeps) { credentialEnv: credentialEnv || NVIDIA_HOSTED_CREDENTIAL_ENV, }; } + if ( + provider === "compatible-endpoint" && + recorded?.nativeLocalProviderAttachment !== undefined + ) { + const expected = normalizeNativeLocalProviderAttachment( + recorded.nativeLocalProviderAttachment, + ); + if ( + !expected || + expected.gatewayName !== gatewayName || + expected.sandboxName !== selection?.sandboxName || + expected.provider !== provider || + expected.provider !== recorded.provider || + (expected.agent ?? "langchain-deepagents-code") !== recorded.agent || + expected.endpointUrl !== + gatewayReachableCompatibleEndpointUrl(provider, selection?.endpointUrl) || + (recorded.credentialEnv ?? null) !== (credentialEnv ?? null) + ) + throw new Error( + "Native local provider resume selection does not match its recorded binding.", + ); + // Native setup verifies the immutable gateway provider and retained credential. + // The logical provider name is not the sandbox-owned provider name. + return { + forceInferenceSetup: true, + credentialEnv: credentialEnv ?? null, + nativeCustomSetup: true, + nativeLocalSetup: true, + }; + } + if (recorded?.registeredInferenceProviderAttachment !== undefined) { + const expected = recordedRegisteredInferenceProviderAttachment(recorded); + if ( + !expected || + expected.gatewayName !== gatewayName || + expected.sandboxName !== selection?.sandboxName || + expected.providerName !== provider || + expected.endpointUrl !== selection?.endpointUrl || + expected.credentialEnv !== credentialEnv + ) + throw new Error( + "Registered provider resume selection does not match its recorded binding.", + ); + revalidateSandboxIdentity?.("verifying registered provider resume"); + await registeredInferenceProviderRevision({ + adapter: deps.providerAdapter ?? createManagedProviderAdapter(), + expected, + }); + revalidateSandboxIdentity?.("verified registered provider resume"); + return { + forceInferenceSetup: false, + credentialEnv: expected.credentialEnv, + registeredNativeSetup: true, + }; + } if ( provider && selection?.sandboxName && diff --git a/src/lib/onboard/managed-startup-agent-environment.test.ts b/src/lib/onboard/managed-startup-agent-environment.test.ts index febdfed8573..6a391e2913a 100644 --- a/src/lib/onboard/managed-startup-agent-environment.test.ts +++ b/src/lib/onboard/managed-startup-agent-environment.test.ts @@ -323,8 +323,6 @@ describe("managed startup agent environment", () => { NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE: "0.5", NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME: "openclaw-gateway", NEMOCLAW_PRIMARY_MODEL_REF: "inference/nvidia/nemotron-3-ultra-550b-a55b", - NEMOCLAW_PROXY_HOST: "10.200.0.1", - NEMOCLAW_PROXY_PORT: "3128", NEMOCLAW_REASONING: "true", NEMOCLAW_REASONING_EFFORT: "high", NEMOCLAW_SERVING_PRESET: "", @@ -532,8 +530,6 @@ describe("managed startup agent environment", () => { NEMOCLAW_MODEL: "claude-sonnet-4-5", HERMES_HOME: "/sandbox/.hermes", HERMES_LAZY_INSTALL_TARGET: "/sandbox/.hermes/lazy-packages", - NEMOCLAW_PROXY_HOST: "proxy_name", - NEMOCLAW_PROXY_PORT: "43128", NEMOCLAW_TOOL_DISCLOSURE: "direct", NEMOCLAW_UPSTREAM_PROVIDER: "anthropic-prod", NEMOCLAW_WEB_SEARCH_ENABLED: "1", @@ -695,24 +691,6 @@ describe("managed startup agent environment", () => { group: "root", mode: 0o444, }, - { - kind: "root-owned-file", - legacyInput: "NEMOCLAW_PROXY_HOST", - path: "/usr/local/share/nemoclaw/dcode-proxy-host", - contents: "10.200.0.1\n", - owner: "root", - group: "root", - mode: 0o444, - }, - { - kind: "root-owned-file", - legacyInput: "NEMOCLAW_PROXY_PORT", - path: "/usr/local/share/nemoclaw/dcode-proxy-port", - contents: "3128\n", - owner: "root", - group: "root", - mode: 0o444, - }, { kind: "root-owned-file", legacyInput: "NEMOCLAW_REASONING_EFFORT", @@ -736,7 +714,7 @@ describe("managed startup agent environment", () => { ]); }); - it("keeps the Pi managed route credential-free and confined to root-owned proxy files (#7930)", () => { + it("keeps Pi startup configuration credential-free without proxy files (#7930)", () => { const result = mapManagedStartupProfileToAgentEnvironment(piProfile()); expect(result.configurationEnvironment).toEqual({ @@ -769,24 +747,6 @@ describe("managed startup agent environment", () => { legacyInput: "NEMOCLAW_CORPORATE_CA_B64", expectedSha256: CA_SHA256, }, - { - kind: "root-owned-file", - legacyInput: "NEMOCLAW_PROXY_HOST", - path: "/usr/local/share/nemoclaw/pi-proxy-host", - contents: "10.200.0.1\n", - owner: "root", - group: "root", - mode: 0o444, - }, - { - kind: "root-owned-file", - legacyInput: "NEMOCLAW_PROXY_PORT", - path: "/usr/local/share/nemoclaw/pi-proxy-port", - contents: "3128\n", - owner: "root", - group: "root", - mode: 0o444, - }, ]); expect(result.actions).toEqual([ { kind: "generate-agent-config", agent: "pi", runAs: "sandbox" }, @@ -844,11 +804,7 @@ describe("managed startup agent environment", () => { NEMOCLAW_MODEL: "openai/gpt-5.4", NEMOCLAW_UPSTREAM_PROVIDER: "openrouter", }); - expect(after.materials).toEqual([ - before.materials[0], - { ...before.materials[1], contents: "10.200.0.9\n" }, - { ...before.materials[2], contents: "3129\n" }, - ]); + expect(after.materials).toEqual(before.materials); const serialized = JSON.stringify(after); expect(serialized).not.toContain("https://inference.local/v1"); expect(serialized).not.toContain("nvidia/nemotron-3-super-120b-a12b"); diff --git a/src/lib/onboard/managed-startup-image-action-plan.test.ts b/src/lib/onboard/managed-startup-image-action-plan.test.ts new file mode 100644 index 00000000000..bdd842fd7d4 --- /dev/null +++ b/src/lib/onboard/managed-startup-image-action-plan.test.ts @@ -0,0 +1,287 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import { + buildManagedStartupImageActionPlan, + type ManagedStartupImageActionPlanInput, +} from "./managed-startup/image-runtime"; +import type { ManagedStartupAgent, ManagedStartupDashboard } from "./managed-startup/profile"; + +function dashboard(agent: ManagedStartupAgent): ManagedStartupDashboard { + switch (agent) { + case "openclaw": + return { + agent, + mode: "loopback", + url: "http://127.0.0.1:18789", + port: 18_789, + bindAddress: "127.0.0.1", + wslExposure: false, + }; + case "hermes": + return { + agent, + mode: "disabled", + url: "http://127.0.0.1:18789", + publicPort: null, + internalPort: null, + tuiEnabled: false, + }; + case "langchain-deepagents-code": + return { agent, mode: "disabled" }; + case "pi": + return { agent, mode: "disabled" }; + } +} + +function actionInput( + agent: ManagedStartupAgent, + mode: "apply" | "clear" = "apply", +): ManagedStartupImageActionPlanInput { + const messagingActions = + agent === "langchain-deepagents-code" || agent === "pi" + ? [] + : [ + { + kind: "apply-messaging-plan" as const, + agent, + mode, + phase: "runtime-setup" as const, + runAs: "root" as const, + }, + { + kind: "apply-messaging-plan" as const, + agent, + mode, + phase: "post-agent-install" as const, + runAs: "sandbox" as const, + }, + ]; + return { + agent, + actions: [ + ...messagingActions.slice(0, 1), + { kind: "generate-agent-config", agent, runAs: "sandbox" }, + ...messagingActions.slice(1), + { kind: "configure-dashboard", dashboard: dashboard(agent) }, + ], + }; +} + +describe("buildManagedStartupImageActionPlan", () => { + it.each(["openclaw", "hermes"] as const)( + "constructs the complete offline %s messaging and config plan", + (agent) => { + const plan = buildManagedStartupImageActionPlan(actionInput(agent)); + + expect(plan.map(({ action, runAs }) => ({ action, runAs }))).toEqual([ + { action: "messaging-runtime-setup", runAs: "root" }, + { action: "generate-agent-config", runAs: "sandbox" }, + { action: "messaging-post-agent-install", runAs: "sandbox" }, + ]); + expect(plan[0]?.argv).toContain("runtime-setup"); + expect(plan[0]?.argv).toContain("apply"); + expect(plan[0]?.argv).not.toContain("--managed-startup-runtime"); + expect(plan[2]?.argv).toContain("post-agent-install"); + expect(plan[2]?.argv).toContain("apply"); + expect(plan[2]?.argv).toContain("--managed-startup-runtime"); + expect( + plan.some((command) => + command.argv.some((argument) => /^(?:npm|npx|pip|pip3|uv)$/u.test(argument)), + ), + ).toBe(false); + expect(Object.isFrozen(plan)).toBe(true); + expect( + plan.every((command) => Object.isFrozen(command) && Object.isFrozen(command.argv)), + ).toBe(true); + }, + ); + + it("constructs DCode's complete offline config plan without messaging actions", () => { + expect(buildManagedStartupImageActionPlan(actionInput("langchain-deepagents-code"))).toEqual([ + { + action: "generate-agent-config", + runAs: "sandbox", + argv: ["/usr/local/bin/node", "/opt/nemoclaw-deepagents-code/generate-config.ts"], + }, + ]); + }); + + it.each([ + ["openclaw", "/scripts/generate-openclaw-config.mts"], + ["hermes", "/opt/nemoclaw-hermes-config/generate-config.ts"], + ["langchain-deepagents-code", "/opt/nemoclaw-deepagents-code/generate-config.ts"], + ["pi", "/opt/nemoclaw-pi/generate-config.ts"], + ] as const)("selects the reviewed %s generator asset", (agent, generator) => { + const command = buildManagedStartupImageActionPlan(actionInput(agent)).find( + ({ action }) => action === "generate-agent-config", + ); + expect(command?.argv.at(-1)).toBe(generator); + }); + + it.each(["apply", "clear"] as const)( + "passes explicit %s intent to both messaging phases", + (mode) => { + const plan = buildManagedStartupImageActionPlan(actionInput("openclaw", mode)); + expect(plan[0]?.argv).toEqual(expect.arrayContaining(["--mode", mode])); + expect(plan[2]?.argv).toEqual(expect.arrayContaining(["--mode", mode])); + }, + ); + + it("keeps apply and clear as distinct reviewed commands", () => { + expect(buildManagedStartupImageActionPlan(actionInput("openclaw", "clear"))).not.toEqual( + buildManagedStartupImageActionPlan(actionInput("openclaw", "apply")), + ); + }); + + it.each([ + [ + "cross-agent action", + { + ...actionInput("openclaw"), + actions: [ + ...actionInput("openclaw").actions.slice(0, 1), + { kind: "generate-agent-config", agent: "hermes", runAs: "sandbox" }, + ...actionInput("openclaw").actions.slice(2), + ], + }, + /action for hermes cannot be used by openclaw/, + ], + [ + "partial messaging plan", + { + ...actionInput("hermes"), + actions: actionInput("hermes").actions.filter( + (action) => + action.kind !== "apply-messaging-plan" || action.phase !== "post-agent-install", + ), + }, + /requires 1 action for each messaging phase/, + ], + [ + "duplicate config action", + { + ...actionInput("langchain-deepagents-code"), + actions: [ + ...actionInput("langchain-deepagents-code").actions, + { + kind: "generate-agent-config", + agent: "langchain-deepagents-code", + runAs: "sandbox", + }, + ], + }, + /exactly one agent config/, + ], + [ + "out-of-order messaging", + { + ...actionInput("openclaw"), + actions: [ + ...actionInput("openclaw").actions.slice(1, 3), + actionInput("openclaw").actions[0], + actionInput("openclaw").actions[3], + ], + }, + /not in the required construction order/, + ], + [ + "root config generation", + { + ...actionInput("hermes"), + actions: actionInput("hermes").actions.map((action) => + action.kind === "generate-agent-config" ? { ...action, runAs: "root" } : action, + ), + }, + /configuration generation must run as sandbox/, + ], + [ + "sandbox messaging runtime setup", + { + ...actionInput("openclaw"), + actions: actionInput("openclaw").actions.map((action) => + action.kind === "apply-messaging-plan" && action.phase === "runtime-setup" + ? { ...action, runAs: "sandbox" } + : action, + ), + }, + /messaging runtime setup must run as root/, + ], + [ + "root messaging post-agent configuration", + { + ...actionInput("openclaw"), + actions: actionInput("openclaw").actions.map((action) => + action.kind === "apply-messaging-plan" && action.phase === "post-agent-install" + ? { ...action, runAs: "root" } + : action, + ), + }, + /messaging post-agent configuration must run as sandbox/, + ], + [ + "arbitrary command action", + { + ...actionInput("langchain-deepagents-code"), + actions: [ + ...actionInput("langchain-deepagents-code").actions, + { kind: "run-command", argv: ["npm", "install"] }, + ], + }, + /unsupported managed startup construction action/, + ], + [ + "missing dashboard", + { + ...actionInput("openclaw"), + actions: actionInput("openclaw").actions.filter( + (action) => action.kind !== "configure-dashboard", + ), + }, + /exactly one dashboard construction action/, + ], + [ + "duplicate dashboard", + { + ...actionInput("hermes"), + actions: [...actionInput("hermes").actions, actionInput("hermes").actions.at(-1)!], + }, + /exactly one dashboard construction action/, + ], + [ + "mismatched dashboard", + { + ...actionInput("openclaw"), + actions: actionInput("openclaw").actions.map((action) => + action.kind === "configure-dashboard" + ? { ...action, dashboard: dashboard("hermes") } + : action, + ), + }, + /dashboard for hermes cannot be used by openclaw/, + ], + [ + "unknown image agent", + { ...actionInput("openclaw"), agent: "unknown-agent" }, + /unsupported agent "unknown-agent"/, + ], + [ + "invalid messaging mode", + { + ...actionInput("openclaw"), + actions: actionInput("openclaw").actions.map((action) => + action.kind === "apply-messaging-plan" ? { ...action, mode: "replace" } : action, + ), + }, + /messaging intent must be apply or clear/, + ], + ])( + "fails closed for an incomplete or mismatched construction contract: %s", + (_name, input, message) => { + expect(() => + buildManagedStartupImageActionPlan(input as ManagedStartupImageActionPlanInput), + ).toThrow(message); + }, + ); +}); diff --git a/src/lib/onboard/managed-startup-image-runtime.test.ts b/src/lib/onboard/managed-startup-image-runtime.test.ts index 04a00ccb893..a28fbf21562 100644 --- a/src/lib/onboard/managed-startup-image-runtime.test.ts +++ b/src/lib/onboard/managed-startup-image-runtime.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import assert from "node:assert/strict"; import fs from "node:fs"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -20,7 +21,9 @@ import { mockRootReplayFilesystem } from "../../../test/helpers/managed-startup- import { applyManagedStartupImageProfile, applyManagedStartupRootRequest, - buildManagedStartupImageActionPlan, + applyManagedStartupOwnerRequest, + prepareManagedStartupDelegatedRequest, + finishManagedStartupDelegatedRequest, installHermesManagedPolicy, installHermesSwitchyardRoutingFiles, MANAGED_STARTUP_COMPLETION_FILE, @@ -28,7 +31,6 @@ import { MANAGED_STARTUP_RUNTIME_ENV_FILE, normalizeManagedStartupWorkspaceRoot, publishManagedStartupCompletionAfterCommit, - type ManagedStartupImageActionPlanInput, main as mainManagedStartupImageRuntime, verifyHermesSwitchyardRoutingFiles, } from "./managed-startup/image-runtime"; @@ -36,313 +38,563 @@ import { createHermesSwitchyardRoutingIntent, encodeHermesSwitchyardRoutingIntent, HERMES_SWITCHYARD_ROUTING_ENV, + HERMES_RELAY_PLUGINS_FILE, + HERMES_RELAY_PLUGIN_STATE_FILE, + HERMES_SWITCHYARD_DEPLOYMENT_FILE, serializeHermesSwitchyardRouting, } from "../hermes-switchyard-routing"; import { encodeManagedStartupProfile, + decodeManagedStartupProfile, fingerprintManagedStartupProfile, type ManagedStartupAgent, - type ManagedStartupDashboard, } from "./managed-startup/profile"; -import { createManagedStartupRootApplyRequest } from "./managed-startup/root-apply"; +import { + createManagedStartupRootApplyRequest, + serializeManagedStartupRootApplyRequest, +} from "./managed-startup/root-apply"; import * as sharedStateTransaction from "./managed-startup/shared-state-transaction"; +import * as application from "./managed-startup/application"; +import { managedImageRuntimeIdentity } from "./managed-image/agents"; -function dashboard(agent: ManagedStartupAgent): ManagedStartupDashboard { - switch (agent) { - case "openclaw": - return { - agent, - mode: "loopback", - url: "http://127.0.0.1:18789", - port: 18_789, - bindAddress: "127.0.0.1", - wslExposure: false, - }; - case "hermes": - return { - agent, - mode: "disabled", - url: "http://127.0.0.1:18789", - publicPort: null, - internalPort: null, - tuiEnabled: false, - }; - case "langchain-deepagents-code": - return { agent, mode: "disabled" }; - case "pi": - return { agent, mode: "disabled" }; +describe("managed startup image runtime", () => { + const policyTemporaryDirectories: string[] = []; + + function temporaryPolicyDirectory(): string { + const directory = fs.mkdtempSync(path.join(process.env.TMPDIR!, "nemoclaw-managed-policy-")); + policyTemporaryDirectories.push(directory); + return directory; } -} - -function actionInput( - agent: ManagedStartupAgent, - mode: "apply" | "clear" = "apply", -): ManagedStartupImageActionPlanInput { - const messagingActions = - agent === "langchain-deepagents-code" || agent === "pi" - ? [] - : [ - { - kind: "apply-messaging-plan" as const, - agent, - mode, - phase: "runtime-setup" as const, - runAs: "root" as const, - }, - { - kind: "apply-messaging-plan" as const, - agent, - mode, - phase: "post-agent-install" as const, - runAs: "sandbox" as const, - }, - ]; - return { - agent, - actions: [ - ...messagingActions.slice(0, 1), - { kind: "generate-agent-config", agent, runAs: "sandbox" }, - ...messagingActions.slice(1), - { kind: "configure-dashboard", dashboard: dashboard(agent) }, - ], - }; -} -describe("buildManagedStartupImageActionPlan", () => { - it.each(["openclaw", "hermes"] as const)( - "constructs the complete offline %s messaging and config plan", - (agent) => { - const plan = buildManagedStartupImageActionPlan(actionInput(agent)); + beforeEach(() => { + childProcessMock.spawnSync.mockReset().mockReturnValue({ error: undefined, status: 0 }); + coordinatorMock.coordinateManagedStartupApplication.mockReset(); + }); + afterEach(() => { + vi.restoreAllMocks(); + for (const directory of policyTemporaryDirectories.splice(0)) { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); - expect(plan.map(({ action, runAs }) => ({ action, runAs }))).toEqual([ - { action: "messaging-runtime-setup", runAs: "root" }, - { action: "generate-agent-config", runAs: "sandbox" }, - { action: "messaging-post-agent-install", runAs: "sandbox" }, - ]); - expect(plan[0]?.argv).toContain("runtime-setup"); - expect(plan[0]?.argv).toContain("apply"); - expect(plan[0]?.argv).not.toContain("--managed-startup-runtime"); - expect(plan[2]?.argv).toContain("post-agent-install"); - expect(plan[2]?.argv).toContain("apply"); - expect(plan[2]?.argv).toContain("--managed-startup-runtime"); + function stdin(text: string) { + const input = Buffer.from(text); + let offset = 0; + vi.spyOn(fs, "readSync").mockImplementation((( + fd: number, + destination: Buffer, + start: number, + length: number, + ) => { + expect(fd).toBe(0); + const count = Math.min(length, input.length - offset); + input.copy(destination, start, offset, offset + count); + offset += count; + return count; + }) as typeof fs.readSync); + } + + function delegatedRequest(agent: ManagedStartupAgent = "openclaw") { + return createManagedStartupRootApplyRequest({ + agent, + encodedProfile: encodeManagedStartupProfile(managedStartupE2eProfile(agent, false)), + }); + } + + function preparedApplication( + request = delegatedRequest(), + status: "prepared" | "already-committed" = "prepared", + ) { + return { + status, + stateDirectory: "/var/lib/nemoclaw/startup-profile", + generationDirectory: "/var/lib/nemoclaw/startup-profile/generation-test", + profilePath: "/var/lib/nemoclaw/startup-profile/generation-test/profile.json", + corporateCaPath: null, + expectedAgent: request.agent, + profile: decodeManagedStartupProfile(request.encodedProfile), + fingerprint: request.profileFingerprint, + } as const; + } + + it.each(["openclaw", "langchain-deepagents-code", "pi"] as const)( + "generates %s only through native-owner commands", + (agent) => { + const request = delegatedRequest(agent); + const identity = managedImageRuntimeIdentity(agent); + vi.spyOn(process, "geteuid").mockReturnValue(identity.uid); + vi.spyOn(process, "getegid").mockReturnValue(identity.gid); + vi.spyOn(fs, "lstatSync").mockReturnValue({ + isSymbolicLink: () => false, + isFile: () => true, + uid: 0, + gid: 0, + mode: 0o755, + } as fs.Stats); + childProcessMock.spawnSync.mockImplementation((command, args) => ({ + status: 0, + stdout: + command === "/usr/bin/id" + ? String(args[0] === "-u" ? identity.uid : identity.gid) + : command === "/usr/local/bin/openclaw" + ? '{"valid":true}' + : "private generation output", + stderr: "", + })); + const result = applyManagedStartupOwnerRequest(request, "a".repeat(64)); + expect(result).toEqual({ + agent, + profileFingerprint: request.profileFingerprint, + bootstrapIdentity: "a".repeat(64), + applied: true, + hermesPolicyB64: null, + hermesConfigHash: null, + }); + const commands = childProcessMock.spawnSync.mock.calls.filter( + ([command]) => command !== "/usr/bin/id", + ); expect( - plan.some((command) => - command.argv.some((argument) => /^(?:npm|npx|pip|pip3|uv)$/u.test(argument)), - ), - ).toBe(false); - expect(Object.isFrozen(plan)).toBe(true); + commands.some(([, args]) => args.some((arg: string) => /generate-.*config/u.test(arg))), + ).toBe(true); expect( - plan.every((command) => Object.isFrozen(command) && Object.isFrozen(command.argv)), + commands.every( + ([command, , options]) => command !== "/usr/bin/setpriv" && options.stdio === "pipe", + ), ).toBe(true); + expect(commands.flatMap(([, args]) => args)).not.toContain("runtime-setup"); }, ); - it("constructs DCode's complete offline config plan without messaging actions", () => { - expect(buildManagedStartupImageActionPlan(actionInput("langchain-deepagents-code"))).toEqual([ - { - action: "generate-agent-config", - runAs: "sandbox", - argv: ["/usr/local/bin/node", "/opt/nemoclaw-deepagents-code/generate-config.ts"], + it("does not regenerate mutable config during an owner replay", () => { + vi.spyOn(process, "geteuid").mockReturnValue(998); + vi.spyOn(process, "getegid").mockReturnValue(998); + expect(applyManagedStartupOwnerRequest(delegatedRequest(), "a".repeat(64), true).applied).toBe( + false, + ); + expect(childProcessMock.spawnSync).not.toHaveBeenCalled(); + }); + + it("prepares only root messaging setup and keeps generation pending", () => { + mockRootReplayFilesystem([]); + vi.spyOn(process, "geteuid").mockReturnValue(0); + const request = delegatedRequest(); + vi.spyOn( + sharedStateTransaction, + "getManagedStartupSharedStateTransactionStatus", + ).mockReturnValue("pending"); + vi.spyOn(application, "prepareManagedStartupApplication").mockReturnValue( + preparedApplication(request), + ); + const commit = vi.spyOn(application, "commitManagedStartupApplication"); + expect(prepareManagedStartupDelegatedRequest(request, "a".repeat(64))).toBe("prepared"); + expect(childProcessMock.spawnSync).toHaveBeenCalledTimes(1); + expect(childProcessMock.spawnSync.mock.calls[0]?.[1]).toContain("runtime-setup"); + expect(childProcessMock.spawnSync.mock.calls[0]?.[2].stdio).toBe("pipe"); + expect(commit).not.toHaveBeenCalled(); + }); + + it("rejects preparation without protected shared-state authority", () => { + vi.spyOn(process, "geteuid").mockReturnValue(0); + vi.spyOn( + sharedStateTransaction, + "getManagedStartupSharedStateTransactionStatus", + ).mockReturnValue("none"); + expect(() => prepareManagedStartupDelegatedRequest(delegatedRequest(), "a".repeat(64))).toThrow( + /protected shared-state transaction/u, + ); + expect(childProcessMock.spawnSync).not.toHaveBeenCalled(); + }); + + it("commits a matching owner result without rerunning generation or publishing completion", async () => { + const writes: string[] = []; + const filesystem = mockRootReplayFilesystem(writes); + vi.spyOn(process, "geteuid").mockReturnValue(0); + const request = delegatedRequest(); + const prepared = preparedApplication(request); + vi.spyOn( + sharedStateTransaction, + "getManagedStartupSharedStateTransactionStatus", + ).mockReturnValue("pending"); + vi.spyOn(application, "prepareManagedStartupApplication").mockReturnValue(prepared); + coordinatorMock.coordinateManagedStartupApplication.mockImplementation( + async (_input, adapters) => { + await adapters + .find((adapter: { agent: string }) => adapter.agent === request.agent) + .apply({ + agent: request.agent, + profile: prepared.profile, + fingerprint: prepared.fingerprint, + generationDirectory: prepared.generationDirectory, + profilePath: prepared.profilePath, + corporateCaPath: null, + }); + return { adapterApplied: true, application: { ...prepared, status: "committed" } }; }, + ); + const result = { + agent: request.agent, + profileFingerprint: request.profileFingerprint, + bootstrapIdentity: "a".repeat(64), + applied: true, + hermesPolicyB64: null, + hermesConfigHash: null, + }; + await expect( + finishManagedStartupDelegatedRequest(request, "a".repeat(64), JSON.stringify(result)), + ).resolves.toEqual( + expect.objectContaining({ + agent: "openclaw", + adapterApplied: true, + fingerprint: request.profileFingerprint, + }), + ); + expect(childProcessMock.spawnSync.mock.calls.map(([command]) => command)).toEqual([ + "/usr/sbin/update-ca-certificates", ]); + expect(filesystem.hasFile(MANAGED_STARTUP_RUNTIME_ENV_FILE)).toBe(true); + expect(filesystem.hasFile(MANAGED_STARTUP_COMPLETION_FILE)).toBe(false); }); - it.each([ - ["openclaw", "/scripts/generate-openclaw-config.mts"], - ["hermes", "/opt/nemoclaw-hermes-config/generate-config.ts"], - ["langchain-deepagents-code", "/opt/nemoclaw-deepagents-code/generate-config.ts"], - ["pi", "/opt/nemoclaw-pi/generate-config.ts"], - ] as const)("selects the reviewed %s generator asset", (agent, generator) => { - const command = buildManagedStartupImageActionPlan(actionInput(agent)).find( - ({ action }) => action === "generate-agent-config", + it("does not rerun root setup for a committed application", () => { + mockRootReplayFilesystem([]); + vi.spyOn(process, "geteuid").mockReturnValue(0); + const request = delegatedRequest(); + vi.spyOn( + sharedStateTransaction, + "getManagedStartupSharedStateTransactionStatus", + ).mockReturnValue("committed"); + vi.spyOn(application, "prepareManagedStartupApplication").mockReturnValue( + preparedApplication(request, "already-committed"), ); - expect(command?.argv.at(-1)).toBe(generator); + expect(prepareManagedStartupDelegatedRequest(request, "a".repeat(64))).toBe( + "already-committed", + ); + expect(childProcessMock.spawnSync).not.toHaveBeenCalled(); }); - it.each(["apply", "clear"] as const)( - "passes explicit %s intent to both messaging phases", - (mode) => { - const plan = buildManagedStartupImageActionPlan(actionInput("openclaw", mode)); - expect(plan[0]?.argv).toEqual(expect.arrayContaining(["--mode", mode])); - expect(plan[2]?.argv).toEqual(expect.arrayContaining(["--mode", mode])); + it.each(["policy", "hash"])( + "rejects malformed Hermes owner %s before root installation", + async (field) => { + mockRootReplayFilesystem([]); + vi.spyOn(process, "geteuid").mockReturnValue(0); + const request = delegatedRequest("hermes"); + vi.spyOn( + sharedStateTransaction, + "getManagedStartupSharedStateTransactionStatus", + ).mockReturnValue("pending"); + vi.spyOn(application, "prepareManagedStartupApplication").mockReturnValue( + preparedApplication(request), + ); + const result = { + agent: request.agent, + profileFingerprint: request.profileFingerprint, + bootstrapIdentity: "a".repeat(64), + applied: true, + hermesPolicyB64: field === "policy" ? "invalid!" : Buffer.from("{}").toString("base64"), + hermesConfigHash: + field === "hash" + ? "wrong-path" + : `${"b".repeat(64)} /sandbox/.hermes/config.yaml\n${"c".repeat(64)} /sandbox/.hermes/.env\n`, + }; + await expect( + finishManagedStartupDelegatedRequest(request, "a".repeat(64), JSON.stringify(result)), + ).rejects.toThrow(/invalid Hermes owner materials/u); + expect(coordinatorMock.coordinateManagedStartupApplication).not.toHaveBeenCalled(); + }, + ); + + it.each(["bootstrapIdentity", "profileFingerprint", "applied", "hermesPolicyB64"] as const)( + "does not commit after a mismatched owner %s", + async (field) => { + mockRootReplayFilesystem([]); + vi.spyOn(process, "geteuid").mockReturnValue(0); + const request = delegatedRequest(); + vi.spyOn( + sharedStateTransaction, + "getManagedStartupSharedStateTransactionStatus", + ).mockReturnValue("pending"); + vi.spyOn(application, "prepareManagedStartupApplication").mockReturnValue( + preparedApplication(request), + ); + const result = { + agent: request.agent, + profileFingerprint: request.profileFingerprint, + bootstrapIdentity: "a".repeat(64), + applied: true, + hermesPolicyB64: null, + hermesConfigHash: null, + [field]: field === "applied" ? false : "mismatch", + }; + await expect( + finishManagedStartupDelegatedRequest(request, "a".repeat(64), JSON.stringify(result)), + ).rejects.toThrow(/owner/u); + expect(coordinatorMock.coordinateManagedStartupApplication).not.toHaveBeenCalled(); }, ); - it("keeps apply and clear as distinct reviewed commands", () => { - expect(buildManagedStartupImageActionPlan(actionInput("openclaw", "clear"))).not.toEqual( - buildManagedStartupImageActionPlan(actionInput("openclaw", "apply")), + it("captures only as the exact agent owner and sends the snapshot on its private output pipe", async () => { + const profile = managedStartupE2eProfile("openclaw", false); + const encodedProfile = encodeManagedStartupProfile(profile); + stdin( + serializeManagedStartupRootApplyRequest( + createManagedStartupRootApplyRequest({ agent: "openclaw", encodedProfile }), + ), ); + vi.spyOn(process, "geteuid").mockReturnValue(998); + vi.spyOn(process, "getegid").mockReturnValue(998); + const capture = vi + .spyOn(sharedStateTransaction, "captureManagedStartupSharedState") + .mockReturnValue("synthetic capture only\n"); + const output = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + const begin = vi.spyOn(sharedStateTransaction, "beginManagedStartupSharedStateTransaction"); + await mainManagedStartupImageRuntime([ + "--capture-shared-state-stdin", + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]); + expect(capture).toHaveBeenCalledExactlyOnceWith(decodeManagedStartupProfile(encodedProfile), { + bootstrapIdentity: "a".repeat(64), + }); + expect(output).toHaveBeenCalledExactlyOnceWith("synthetic capture only\n"); + expect(begin).not.toHaveBeenCalled(); }); it.each([ - [ - "cross-agent action", - { - ...actionInput("openclaw"), - actions: [ - ...actionInput("openclaw").actions.slice(0, 1), - { kind: "generate-agent-config", agent: "hermes", runAs: "sandbox" }, - ...actionInput("openclaw").actions.slice(2), - ], - }, - /action for hermes cannot be used by openclaw/, - ], - [ - "partial messaging plan", - { - ...actionInput("hermes"), - actions: actionInput("hermes").actions.filter( - (action) => - action.kind !== "apply-messaging-plan" || action.phase !== "post-agent-install", - ), - }, - /requires 1 action for each messaging phase/, - ], - [ - "duplicate config action", - { - ...actionInput("langchain-deepagents-code"), - actions: [ - ...actionInput("langchain-deepagents-code").actions, - { - kind: "generate-agent-config", - agent: "langchain-deepagents-code", - runAs: "sandbox", - }, - ], - }, - /exactly one agent config/, - ], - [ - "out-of-order messaging", - { - ...actionInput("openclaw"), - actions: [ - ...actionInput("openclaw").actions.slice(1, 3), - actionInput("openclaw").actions[0], - actionInput("openclaw").actions[3], - ], - }, - /not in the required construction order/, - ], - [ - "root config generation", - { - ...actionInput("hermes"), - actions: actionInput("hermes").actions.map((action) => - action.kind === "generate-agent-config" ? { ...action, runAs: "root" } : action, - ), - }, - /configuration generation must run as sandbox/, - ], - [ - "sandbox messaging runtime setup", - { - ...actionInput("openclaw"), - actions: actionInput("openclaw").actions.map((action) => - action.kind === "apply-messaging-plan" && action.phase === "runtime-setup" - ? { ...action, runAs: "sandbox" } - : action, - ), - }, - /messaging runtime setup must run as root/, - ], - [ - "root messaging post-agent configuration", - { - ...actionInput("openclaw"), - actions: actionInput("openclaw").actions.map((action) => - action.kind === "apply-messaging-plan" && action.phase === "post-agent-install" - ? { ...action, runAs: "root" } - : action, - ), - }, - /messaging post-agent configuration must run as sandbox/, - ], - [ - "arbitrary command action", - { - ...actionInput("langchain-deepagents-code"), - actions: [ - ...actionInput("langchain-deepagents-code").actions, - { kind: "run-command", argv: ["npm", "install"] }, - ], - }, - /unsupported managed startup construction action/, - ], - [ - "missing dashboard", - { - ...actionInput("openclaw"), - actions: actionInput("openclaw").actions.filter( - (action) => action.kind !== "configure-dashboard", - ), - }, - /exactly one dashboard construction action/, - ], - [ - "duplicate dashboard", - { - ...actionInput("hermes"), - actions: [...actionInput("hermes").actions, actionInput("hermes").actions.at(-1)!], - }, - /exactly one dashboard construction action/, - ], - [ - "mismatched dashboard", + [0, 0], + [999, 998], + [998, 999], + ])("rejects capture identity %i:%i before reading private state", async (uid, gid) => { + vi.spyOn(process, "geteuid").mockReturnValue(uid); + vi.spyOn(process, "getegid").mockReturnValue(gid); + const capture = vi.spyOn(sharedStateTransaction, "captureManagedStartupSharedState"); + await expect( + mainManagedStartupImageRuntime([ + "--capture-shared-state-stdin", + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]), + ).rejects.toThrow(/exact native state owner/u); + expect(capture).not.toHaveBeenCalled(); + }); + + it("admits captured state only into the root-owned transaction for the requested bootstrap", async () => { + const profile = managedStartupE2eProfile("openclaw", false); + mockRootReplayFilesystem([]); + stdin( + JSON.stringify({ + encodedProfile: encodeManagedStartupProfile(profile), + capturedState: "synthetic transport", + }), + ); + vi.spyOn(process, "geteuid").mockReturnValue(0); + const begin = vi + .spyOn(sharedStateTransaction, "beginManagedStartupSharedStateTransaction") + .mockReturnValue(true); + const output = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + await mainManagedStartupImageRuntime([ + "--begin-shared-state-from-capture-stdin", + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]); + expect(begin).toHaveBeenCalledExactlyOnceWith( + decodeManagedStartupProfile(encodeManagedStartupProfile(profile)), { - ...actionInput("openclaw"), - actions: actionInput("openclaw").actions.map((action) => - action.kind === "configure-dashboard" - ? { ...action, dashboard: dashboard("hermes") } - : action, - ), + bootstrapIdentity: "a".repeat(64), + capturedState: "synthetic transport", }, - /dashboard for hermes cannot be used by openclaw/, - ], - [ - "unknown image agent", - { ...actionInput("openclaw"), agent: "unknown-agent" }, - /unsupported agent "unknown-agent"/, - ], + ); + expect(output).toHaveBeenCalledExactlyOnceWith("created\n"); + }); + + it("does not let the state owner admit its own trusted receipt", async () => { + vi.spyOn(process, "geteuid").mockReturnValue(998); + const begin = vi.spyOn(sharedStateTransaction, "beginManagedStartupSharedStateTransaction"); + await expect( + mainManagedStartupImageRuntime([ + "--begin-shared-state-from-capture-stdin", + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]), + ).rejects.toThrow(/requires container effective uid 0/u); + expect(begin).not.toHaveBeenCalled(); + }); + + it.each([ [ - "invalid messaging mode", - { - ...actionInput("openclaw"), - actions: actionInput("openclaw").actions.map((action) => - action.kind === "apply-messaging-plan" ? { ...action, mode: "replace" } : action, - ), - }, - /messaging intent must be apply or clear/, + "--restore-shared-state-stdin", + "restoreManagedStartupCapturedSharedState", + 998, + "observed private state", ], - ])( - "fails closed for an incomplete or mismatched construction contract: %s", - (_name, input, message) => { - expect(() => - buildManagedStartupImageActionPlan(input as ManagedStartupImageActionPlanInput), - ).toThrow(message); + ["--finish-shared-restoration-stdin", "finishManagedStartupSharedStateRestoration", 0, true], + ] as const)( + "routes %s through the required process identity", + async (action, operation, uid, result) => { + const encodedProfile = encodeManagedStartupProfile( + managedStartupE2eProfile("openclaw", false), + ); + vi.spyOn(process, "geteuid").mockReturnValue(uid); + vi.spyOn(process, "getegid").mockReturnValue(uid); + stdin(JSON.stringify({ encodedProfile, capturedState: "private transport" })); + const handler = vi.spyOn(sharedStateTransaction, operation).mockReturnValue(result); + const output = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + await mainManagedStartupImageRuntime([ + action, + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]); + expect(handler).toHaveBeenCalledExactlyOnceWith( + decodeManagedStartupProfile(encodedProfile), + "private transport", + { bootstrapIdentity: "a".repeat(64) }, + ); + expect(output).toHaveBeenCalledExactlyOnceWith(uid === 0 ? "restored\n" : result); }, ); -}); -describe("managed startup image runtime", () => { - const policyTemporaryDirectories: string[] = []; + it("exports the protected original only through a root process", async () => { + const encodedProfile = encodeManagedStartupProfile(managedStartupE2eProfile("openclaw", false)); + vi.spyOn(process, "geteuid").mockReturnValue(0); + stdin( + serializeManagedStartupRootApplyRequest( + createManagedStartupRootApplyRequest({ agent: "openclaw", encodedProfile }), + ), + ); + const handler = vi + .spyOn(sharedStateTransaction, "exportManagedStartupSharedStateRestoration") + .mockReturnValue("private original"); + const output = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + await mainManagedStartupImageRuntime([ + "--export-shared-restoration-stdin", + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]); + expect(handler).toHaveBeenCalledExactlyOnceWith(decodeManagedStartupProfile(encodedProfile), { + bootstrapIdentity: "a".repeat(64), + }); + expect(output).toHaveBeenCalledExactlyOnceWith("private original"); + }); - function temporaryPolicyDirectory(): string { - const directory = fs.mkdtempSync(path.join(process.env.TMPDIR!, "nemoclaw-managed-policy-")); - policyTemporaryDirectories.push(directory); - return directory; - } + it("reports delegated startup without consuming input or preparing state", async () => { + vi.spyOn(process, "geteuid").mockReturnValue(0); + const read = vi.spyOn(fs, "readSync"); + const begin = vi.spyOn(sharedStateTransaction, "beginManagedStartupSharedStateTransaction"); + const output = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + await mainManagedStartupImageRuntime(["--startup-protocol"]); + expect(output).toHaveBeenCalledExactlyOnceWith("delegated-owner-v1\n"); + expect(read).not.toHaveBeenCalled(); + expect(begin).not.toHaveBeenCalled(); + }); - beforeEach(() => { - childProcessMock.spawnSync.mockReset().mockReturnValue({ error: undefined, status: 0 }); - coordinatorMock.coordinateManagedStartupApplication.mockReset(); + it("prepares the protected parent before querying a new delegated transaction", async () => { + vi.spyOn(process, "geteuid").mockReturnValue(0); + mockRootReplayFilesystem([]); + const request = createManagedStartupRootApplyRequest({ + agent: "openclaw", + encodedProfile: encodeManagedStartupProfile(managedStartupE2eProfile("openclaw", false)), + }); + stdin(serializeManagedStartupRootApplyRequest(request)); + const status = vi + .spyOn(sharedStateTransaction, "getManagedStartupSharedStateTransactionStatus") + .mockImplementation(() => { + expect(fs.lstatSync("/var/lib/nemoclaw").isDirectory()).toBe(true); + return "none"; + }); + const begin = vi.spyOn(sharedStateTransaction, "beginManagedStartupSharedStateTransaction"); + const output = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + await mainManagedStartupImageRuntime([ + "--prepare-delegated-transaction-stdin", + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]); + expect(status).toHaveBeenCalledExactlyOnceWith({ + agent: "openclaw", + profileFingerprint: request.profileFingerprint, + bootstrapIdentity: "a".repeat(64), + }); + expect(output).toHaveBeenCalledExactlyOnceWith("none\n"); + expect(begin).not.toHaveBeenCalled(); }); - afterEach(() => { - vi.restoreAllMocks(); - for (const directory of policyTemporaryDirectories.splice(0)) { - fs.rmSync(directory, { recursive: true, force: true }); - } + + it.each(["none", "pending", "committed"] as const)( + "reports transaction status %s without renaming it", + async (phase) => { + vi.spyOn(process, "geteuid").mockReturnValue(0); + vi.spyOn( + sharedStateTransaction, + "getManagedStartupSharedStateTransactionStatus", + ).mockReturnValue(phase); + const output = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + await mainManagedStartupImageRuntime([ + "--shared-state-transaction-status", + "--agent", + "openclaw", + "--profile-fingerprint", + "a".repeat(64), + "--bootstrap-identity", + "b".repeat(64), + ]); + expect(output).toHaveBeenCalledExactlyOnceWith(`${phase}\n`); + }, + ); + + it("exports recovery from a profile without transporting the CA bundle again", async () => { + const encodedProfile = encodeManagedStartupProfile( + managedStartupE2eProfile("openclaw", false, true, true), + ); + vi.spyOn(process, "geteuid").mockReturnValue(0); + stdin(encodedProfile); + const handler = vi + .spyOn(sharedStateTransaction, "exportManagedStartupSharedStateRestoration") + .mockReturnValue("private original"); + vi.spyOn(process.stdout, "write").mockImplementation(() => true); + await mainManagedStartupImageRuntime([ + "--export-shared-restoration-profile-stdin", + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]); + expect(handler).toHaveBeenCalledExactlyOnceWith(decodeManagedStartupProfile(encodedProfile), { + bootstrapIdentity: "a".repeat(64), + }); + }); + + it.each([ + ["--export-shared-restoration-stdin", 998, 998], + ["--export-shared-restoration-profile-stdin", 998, 998], + ["--prepare-delegated-transaction-stdin", 998, 998], + ["--finish-shared-restoration-stdin", 998, 998], + ["--restore-shared-state-stdin", 0, 0], + ["--restore-shared-state-stdin", 998, 999], + ] as const)("rejects %s as %i:%i before consuming transport", async (action, uid, gid) => { + vi.spyOn(process, "geteuid").mockReturnValue(uid); + vi.spyOn(process, "getegid").mockReturnValue(gid); + const read = vi.spyOn(fs, "readSync"); + await expect( + mainManagedStartupImageRuntime([ + action, + "--agent", + "openclaw", + "--bootstrap-identity", + "a".repeat(64), + ]), + ).rejects.toThrow(/requires/u); + expect(read).not.toHaveBeenCalled(); }); it.each([ @@ -354,11 +606,11 @@ describe("managed startup image runtime", () => { const lstat = vi .fn() .mockReturnValueOnce({ - gid: 999, + gid: 0, isDirectory: () => true, isSymbolicLink: () => false, - mode: 0o040000 | 0o755, - uid: 999, + mode: 0o040000 | 0o700, + uid: 0, }) .mockReturnValueOnce({ gid, @@ -377,6 +629,80 @@ describe("managed startup image runtime", () => { expect(lstat).toHaveBeenCalledTimes(2); }); + it.each([ + ["openclaw", 998, 998, 0o755], + ["hermes", 998, 999, 0o755], + ["langchain-deepagents-code", 0, 999, 0o1775], + ["pi", 999, 999, 0o755], + ] as const)( + "preserves the declared %s workspace without metadata writes", + (agent, uid, gid, mode) => { + const lstat = vi.fn().mockReturnValue({ + gid, + isDirectory: () => true, + isSymbolicLink: () => false, + mode: 0o040000 | mode, + uid, + }); + const denied = () => { + throw Object.assign(new Error("operation not permitted"), { code: "EPERM" }); + }; + const chown = vi.fn(denied); + const chmod = vi.fn(denied); + + normalizeManagedStartupWorkspaceRoot(agent, { lstat, chown, chmod }); + + expect(chown).not.toHaveBeenCalled(); + expect(chmod).not.toHaveBeenCalled(); + expect(lstat).toHaveBeenCalledTimes(2); + }, + ); + + it("rejects workspace metadata drift after an initially matching observation", () => { + const initial = { + gid: 998, + isDirectory: () => true, + isSymbolicLink: () => false, + mode: 0o040755, + uid: 998, + }; + const lstat = vi + .fn() + .mockReturnValueOnce(initial) + .mockReturnValueOnce({ ...initial, mode: 0o040777 }); + const chown = vi.fn(); + const chmod = vi.fn(); + + expect(() => normalizeManagedStartupWorkspaceRoot("openclaw", { lstat, chown, chmod })).toThrow( + /workspace root does not match its declared posture/u, + ); + expect(chown).not.toHaveBeenCalled(); + expect(chmod).not.toHaveBeenCalled(); + }); + + it("fails closed when a required workspace ownership repair is denied", () => { + const lstat = vi.fn().mockReturnValue({ + gid: 0, + isDirectory: () => true, + isSymbolicLink: () => false, + mode: 0o040755, + uid: 0, + }); + const chmod = vi.fn(); + + expect(() => + normalizeManagedStartupWorkspaceRoot("openclaw", { + lstat, + chown: () => { + throw Object.assign(new Error("operation not permitted"), { code: "EPERM" }); + }, + chmod, + }), + ).toThrow(/workspace-root normalization failed/u); + expect(chmod).not.toHaveBeenCalled(); + expect(lstat).toHaveBeenCalledTimes(1); + }); + it("rejects a symlinked workspace root before mutation", () => { const chown = vi.fn(); const chmod = vi.fn(); @@ -778,6 +1104,248 @@ describe("managed startup image runtime", () => { ); }); + it("installs Switchyard before committing initial direct Hermes startup", async () => { + const routing = createHermesSwitchyardRoutingIntent({ + efficient: { provider: "nvidia-prod", model: "nvidia/efficient-model" }, + capable: { provider: "openai-api", model: "openai/capable-model" }, + }); + const original = managedStartupE2eProfile("hermes", false); + assert(original.agentConfig.agent === "hermes"); + const profile = { + ...original, + agentConfig: { + ...original.agentConfig, + agent: "hermes" as const, + switchyardRouting: routing, + }, + }; + const serialized = serializeHermesSwitchyardRouting(routing); + const sourceFiles = [ + ["/sandbox/.hermes/hermes-relay-plugins.toml", serialized.relayPluginsToml], + ["/sandbox/.hermes/hermes-relay-plugin-state.json", serialized.relayPluginStateJson], + ["/sandbox/.hermes/hermes-switchyard-v1.toml", serialized.switchyardDeploymentToml], + ["/sandbox/.hermes/managed-policy.json", "{}"], + ["/sandbox/.hermes/config.yaml", "model: test\n"], + ["/sandbox/.hermes/.env", "HERMES_TEST=1\n"], + ]; + const filesystem = mockRootReplayFilesystem( + [], + new Map([ + ...sourceFiles.map( + ([target, contents]) => + [ + target!, + { + contents: contents!, + mode: target!.endsWith("config.yaml") || target!.endsWith("/.env") ? 0o640 : 0o600, + uid: 999, + gid: 999, + }, + ] as const, + ), + ["/usr/bin/setpriv", { contents: "test executable", mode: 0o555, uid: 0, gid: 0 }], + ]), + ); + fs.mkdirSync("/usr/local/share/nemoclaw", { mode: 0o755 }); + fs.mkdirSync("/etc/nemoclaw", { mode: 0o755 }); + childProcessMock.spawnSync.mockImplementation((command) => ({ + status: 0, + stdout: command === "/usr/bin/id" ? "999" : "", + stderr: "", + })); + const prepared = preparedApplication( + createManagedStartupRootApplyRequest({ + agent: "hermes", + encodedProfile: encodeManagedStartupProfile(profile), + }), + ); + let committed = false; + coordinatorMock.coordinateManagedStartupApplication.mockImplementation( + async (_input, adapters) => { + await adapters + .find((adapter: { agent: string }) => adapter.agent === "hermes") + .apply({ + agent: "hermes", + profile, + fingerprint: prepared.fingerprint, + generationDirectory: prepared.generationDirectory, + profilePath: prepared.profilePath, + corporateCaPath: null, + }); + expect(() => + verifyHermesSwitchyardRoutingFiles({ + [HERMES_SWITCHYARD_ROUTING_ENV]: encodeHermesSwitchyardRoutingIntent(routing), + }), + ).not.toThrow(); + expect(filesystem.hasFile("/sandbox/.hermes/hermes-relay-plugins.toml")).toBe(false); + expect(filesystem.hasFile("/sandbox/.hermes/hermes-relay-plugin-state.json")).toBe(false); + expect(filesystem.hasFile("/sandbox/.hermes/hermes-switchyard-v1.toml")).toBe(false); + expect(filesystem.hasFile(MANAGED_STARTUP_COMPLETION_FILE)).toBe(false); + committed = true; + return { adapterApplied: true, application: { ...prepared, status: "committed" } }; + }, + ); + await expect( + applyManagedStartupImageProfile("hermes", { + NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION: "1", + [MANAGED_STARTUP_PROFILE_ENV]: encodeManagedStartupProfile(profile), + }), + ).resolves.toMatchObject({ adapterApplied: true, fingerprint: prepared.fingerprint }); + expect(committed).toBe(true); + expect(filesystem.hasFile(MANAGED_STARTUP_COMPLETION_FILE)).toBe(true); + }); + + it.each([ + { + state: "disabled-stale", + succeeds: false, + error: /disabled Hermes routing left a stale artifact/u, + }, + { + state: "enabled-drift", + succeeds: false, + error: /committed Hermes routing artifact drifted/u, + }, + { state: "enabled-valid", succeeds: true, error: null }, + ] as const)( + "validates Switchyard artifacts on delegated committed replay: $state", + async ({ state, succeeds, error }) => { + const routing = createHermesSwitchyardRoutingIntent({ + efficient: { provider: "nvidia-prod", model: "nvidia/efficient-model" }, + capable: { provider: "openai-api", model: "openai/capable-model" }, + }); + const original = managedStartupE2eProfile("hermes", false); + assert(original.agentConfig.agent === "hermes"); + const profile = { + ...original, + agentConfig: { + ...original.agentConfig, + agent: "hermes" as const, + switchyardRouting: state === "disabled-stale" ? null : routing, + }, + }; + const fingerprint = fingerprintManagedStartupProfile(profile); + const serialized = serializeHermesSwitchyardRouting(routing); + const filesystem = mockRootReplayFilesystem( + [], + new Map([ + [ + HERMES_RELAY_PLUGINS_FILE, + { + contents: + state === "enabled-valid" ? serialized.relayPluginsToml : "stale activation", + mode: 0o444, + }, + ], + [ + HERMES_RELAY_PLUGIN_STATE_FILE, + { contents: serialized.relayPluginStateJson, mode: 0o444 }, + ], + [ + HERMES_SWITCHYARD_DEPLOYMENT_FILE, + { contents: serialized.switchyardDeploymentToml, mode: 0o444 }, + ], + ]), + ); + coordinatorMock.coordinateManagedStartupApplication.mockResolvedValue({ + adapterApplied: false, + application: { + ...preparedApplication( + createManagedStartupRootApplyRequest({ + agent: "hermes", + encodedProfile: encodeManagedStartupProfile(profile), + }), + ), + status: "committed", + }, + }); + const replay = applyManagedStartupImageProfile( + "hermes", + { + NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION: "1", + [MANAGED_STARTUP_PROFILE_ENV]: encodeManagedStartupProfile(profile), + }, + { + publishCompletion: false, + ownerResult: { + agent: "hermes", + profileFingerprint: fingerprint, + bootstrapIdentity: "a".repeat(64), + applied: false, + hermesPolicyB64: null, + hermesConfigHash: null, + }, + }, + ); + const outcome = await replay.then( + (result) => ({ + success: true, + adapterApplied: result.adapterApplied, + fingerprint: result.fingerprint, + error: null, + }), + (failure: Error) => ({ + success: false, + adapterApplied: null, + fingerprint: null, + error: failure.message, + }), + ); + expect(outcome).toEqual({ + success: succeeds, + adapterApplied: succeeds ? false : null, + fingerprint: succeeds ? fingerprint : null, + error: error === null ? null : expect.stringMatching(error), + }); + expect(filesystem.hasFile(MANAGED_STARTUP_RUNTIME_ENV_FILE)).toBe(succeeds); + const verification = Promise.resolve().then(() => + verifyHermesSwitchyardRoutingFiles({ + [HERMES_SWITCHYARD_ROUTING_ENV]: + state === "disabled-stale" ? undefined : encodeHermesSwitchyardRoutingIntent(routing), + }), + ); + await expect( + verification.then( + () => null, + (failure: Error) => failure.message, + ), + ).resolves.toEqual(error === null ? null : expect.stringMatching(error)); + const routingTargets: readonly string[] = [ + HERMES_RELAY_PLUGINS_FILE, + HERMES_RELAY_PLUGIN_STATE_FILE, + HERMES_SWITCHYARD_DEPLOYMENT_FILE, + ]; + expect( + vi + .mocked(fs.renameSync) + .mock.calls.filter( + ([source, target]) => + routingTargets.includes(String(source)) || routingTargets.includes(String(target)), + ), + ).toEqual([]); + expect( + vi + .mocked(fs.unlinkSync) + .mock.calls.filter(([target]) => routingTargets.includes(String(target))), + ).toEqual([]); + const routingDescriptors = vi + .mocked(fs.openSync) + .mock.calls.flatMap(([target], index) => + routingTargets.includes(String(target)) + ? [vi.mocked(fs.openSync).mock.results[index]!.value] + : [], + ); + expect( + vi + .mocked(fs.writeFileSync) + .mock.calls.filter( + ([target]) => + routingTargets.includes(String(target)) || routingDescriptors.includes(target), + ), + ).toEqual([]); + }, + ); + it("verifies a host-copied transaction only through the explicit read-only receipt mode", async () => { const profile = managedStartupE2eProfile("openclaw"); const profileFingerprint = fingerprintManagedStartupProfile(profile); diff --git a/src/lib/onboard/managed-startup-onboard-profile.test.ts b/src/lib/onboard/managed-startup-onboard-profile.test.ts index 3082aa5b8a1..735b994f4d1 100644 --- a/src/lib/onboard/managed-startup-onboard-profile.test.ts +++ b/src/lib/onboard/managed-startup-onboard-profile.test.ts @@ -177,6 +177,8 @@ describe("buildManagedStartupOnboardProfile", () => { NEMOCLAW_CONTEXT_WINDOW: "262144", NEMOCLAW_MAX_TOKENS: "32000", NEMOCLAW_REASONING: "true", + NEMOCLAW_PROXY_HOST: "retired-proxy.invalid", + NEMOCLAW_PROXY_PORT: "9999", }, }), ); @@ -365,6 +367,7 @@ describe("buildManagedStartupOnboardProfile", () => { dcodeInput({ dcodeAutoApprovalMode: "thread-opt-in", observabilityEnabled: true, + environment: { NEMOCLAW_PROXY_HOST: "retired-proxy.invalid", NEMOCLAW_PROXY_PORT: "9999" }, }), ); @@ -506,7 +509,7 @@ describe("buildManagedStartupOnboardProfile", () => { expect(serialized).not.toContain("lower.internal"); }); - it("filters stale semantic env while preserving environment-owned tuning and proxy knobs", () => { + it("filters stale semantic env while preserving tuning and host proxy intent", () => { const built = buildManagedStartupOnboardProfile( openClawInput({ toolDisclosure: "direct", @@ -548,8 +551,8 @@ describe("buildManagedStartupOnboardProfile", () => { reasoningEffort: "default", }); expect(built.profile.proxy).toMatchObject({ - managedHost: "host.containers.internal", - managedPort: 3129, + managedHost: "10.200.0.1", + managedPort: 3128, hostHttpUrl: "http://proxy.example.test:8080", }); }); diff --git a/src/lib/onboard/managed-startup-profile-builder.test.ts b/src/lib/onboard/managed-startup-profile-builder.test.ts index 0a34f9e6d23..30c895c23d3 100644 --- a/src/lib/onboard/managed-startup-profile-builder.test.ts +++ b/src/lib/onboard/managed-startup-profile-builder.test.ts @@ -221,8 +221,6 @@ describe("buildManagedStartupProfile", () => { NEMOCLAW_CONTEXT_WINDOW: "262144", NEMOCLAW_MAX_TOKENS: "32000", NEMOCLAW_REASONING: "true", - NEMOCLAW_PROXY_HOST: "10.200.0.1", - NEMOCLAW_PROXY_PORT: "3128", }, }), ); @@ -472,8 +470,6 @@ describe("buildManagedStartupProfile", () => { NEMOCLAW_DASHBOARD_BIND: "0.0.0.0", NEMOCLAW_WSL_DASHBOARD_EXPOSURE: "1", NEMOCLAW_DASHBOARD_PORT: "19443", - NEMOCLAW_PROXY_HOST: "host.containers.internal", - NEMOCLAW_PROXY_PORT: "3129", NEMOCLAW_MESSAGING_PLAN_B64: encodeJson(plan), NEMOCLAW_MINIMAL_BOOTSTRAP: "1", HTTP_PROXY: "http://proxy.example.test:8080", @@ -513,8 +509,8 @@ describe("buildManagedStartupProfile", () => { }); expect(built.profile.agentConfig).not.toHaveProperty("deviceAuth"); expect(built.profile.proxy).toMatchObject({ - managedHost: "host.containers.internal", - managedPort: 3129, + managedHost: "10.200.0.1", + managedPort: 3128, hostHttpUrl: "http://proxy.example.test:8080", hostHttpsUrl: "https://connect.example.test:8443", }); @@ -654,8 +650,6 @@ describe("buildManagedStartupProfile", () => { NEMOCLAW_INFERENCE_API: "openai-completions", NEMOCLAW_TOOL_DISCLOSURE: "direct", NEMOCLAW_DCODE_AUTO_APPROVAL: "thread-opt-in", - NEMOCLAW_PROXY_HOST: "10.200.0.1", - NEMOCLAW_PROXY_PORT: "3128", NEMOCLAW_OBSERVABILITY: "1", NEMOCLAW_REASONING_EFFORT: "high", }, diff --git a/src/lib/onboard/managed-startup-profile.test.ts b/src/lib/onboard/managed-startup-profile.test.ts index a6a69c8682a..c81406c5c08 100644 --- a/src/lib/onboard/managed-startup-profile.test.ts +++ b/src/lib/onboard/managed-startup-profile.test.ts @@ -315,8 +315,6 @@ const STOCK_RUNTIME_INPUT_AGENTS = { NEMOCLAW_MINIMAL_BOOTSTRAP: ["openclaw"], NEMOCLAW_MCP_TOOLS_LIST_TIMEOUT_MS: ["openclaw"], NEMOCLAW_OBSERVABILITY: ["langchain-deepagents-code"], - NEMOCLAW_PROXY_HOST: MANAGED_STARTUP_AGENTS, - NEMOCLAW_PROXY_PORT: MANAGED_STARTUP_AGENTS, NEMOCLAW_SANDBOX_NAME: ["langchain-deepagents-code"], NO_PROXY: MANAGED_STARTUP_AGENTS, OPENCLAW_HOME: ["openclaw"], diff --git a/src/lib/onboard/managed-startup-shared-state-transaction.test.ts b/src/lib/onboard/managed-startup-shared-state-transaction.test.ts index c147061df56..92d74baac38 100644 --- a/src/lib/onboard/managed-startup-shared-state-transaction.test.ts +++ b/src/lib/onboard/managed-startup-shared-state-transaction.test.ts @@ -13,12 +13,16 @@ import type { ManagedStartupAgent, ManagedStartupProfile } from "./managed-start import { fingerprintManagedStartupProfile } from "./managed-startup/profile"; import { beginManagedStartupSharedStateTransaction, + captureManagedStartupSharedState, clearManagedStartupSharedStateCommitReceipt, commitManagedStartupSharedStateTransaction, + exportManagedStartupSharedStateRestoration, + finishManagedStartupSharedStateRestoration, getManagedStartupSharedStateTransactionStatus, MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_DIRECTORY, type ManagedStartupSharedTransactionOptions, rollbackManagedStartupSharedStateTransaction, + restoreManagedStartupCapturedSharedState, } from "./managed-startup/shared-state-transaction"; function mode(target: string): number { @@ -77,6 +81,200 @@ describe("managed startup shared-state transaction", () => { ); } + function capturedOpenClaw() { + const profile = managedStartupE2eProfile("openclaw", false); + const root = agentRoot("openclaw"); + fs.mkdirSync(root, { mode: 0o700 }); + const file = path.join(root, "openclaw.json"); + fs.writeFileSync(file, '{"synthetic":"original"}', { mode: 0o600 }); + const bootstrapIdentity = "a".repeat(64); + const capturedState = captureManagedStartupSharedState(profile, { + sandboxRoot, + bootstrapIdentity, + }); + return { profile, file, capturedState, bootstrapIdentity }; + } + + it("installs an owner capture into the trusted transaction and restores its exact original bytes", () => { + const captured = capturedOpenClaw(); + fs.writeFileSync(captured.file, "changed after capture"); + const importedOptions = { + ...options, + bootstrapIdentity: captured.bootstrapIdentity, + capturedState: captured.capturedState, + }; + expect(beginManagedStartupSharedStateTransaction(captured.profile, importedOptions)).toBe(true); + expect(mode(transactionDirectory)).toBe(0o700); + expect(mode(path.join(transactionDirectory, "backups", "000.bin"))).toBe(0o400); + expect(rollbackManagedStartupSharedStateTransaction("openclaw", importedOptions)).toBe(true); + expect(fs.readFileSync(captured.file, "utf8")).toBe('{"synthetic":"original"}'); + expect(mode(captured.file)).toBe(0o600); + }); + + it("never replaces a pending original snapshot with a replayed capture", () => { + const captured = capturedOpenClaw(); + const importedOptions = { + ...options, + bootstrapIdentity: captured.bootstrapIdentity, + capturedState: captured.capturedState, + }; + beginManagedStartupSharedStateTransaction(captured.profile, importedOptions); + fs.writeFileSync(captured.file, "partial generator output"); + expect( + beginManagedStartupSharedStateTransaction(captured.profile, { + ...importedOptions, + capturedState: "not a new receipt", + }), + ).toBe(false); + rollbackManagedStartupSharedStateTransaction("openclaw", importedOptions); + expect(fs.readFileSync(captured.file, "utf8")).toBe('{"synthetic":"original"}'); + }); + + it.each([ + ["bootstrap", { bootstrapIdentity: "b".repeat(64) }, {}, undefined, {}], + ["profile", { profileFingerprint: "b".repeat(64) }, {}, undefined, {}], + ["path", {}, { path: ".openclaw/not-selected.json" }, undefined, {}], + ["bytes", {}, {}, [Buffer.from("tampered").toString("base64")], {}], + ["metadata", {}, { mode: 0o10000 }, undefined, {}], + ["extra-field", {}, {}, undefined, { unexpected: true }], + ] as const)( + "rejects a mismatched captured %s before creating a receipt", + (_change, manifestPatch, filePatch, contents, envelopePatch) => { + const captured = capturedOpenClaw(); + const envelope = JSON.parse(captured.capturedState); + const manifest = { ...JSON.parse(envelope.manifest), ...manifestPatch }; + manifest.files[0] = { ...manifest.files[0], ...filePatch }; + envelope.contents = contents ?? envelope.contents; + Object.assign(envelope, envelopePatch); + envelope.manifest = `${JSON.stringify(manifest, null, 2)}\n`; + expect(() => + beginManagedStartupSharedStateTransaction(captured.profile, { + ...options, + bootstrapIdentity: captured.bootstrapIdentity, + capturedState: JSON.stringify(envelope), + }), + ).toThrow(/shared-state transaction failed/u); + expect(fs.existsSync(transactionDirectory)).toBe(false); + expect(fs.readFileSync(captured.file, "utf8")).toBe('{"synthetic":"original"}'); + }, + ); + + it("cannot create a trusted receipt while capturing private state", () => { + capturedOpenClaw(); + expect(fs.existsSync(transactionDirectory)).toBe(false); + }); + + it.each([ + ["openclaw", "openclaw.json"], + ["hermes", "config.yaml"], + ["langchain-deepagents-code", "config.toml"], + ["pi", "agent/models.json"], + ] as const)( + "restores %s bytes and metadata before retiring its protected receipt", + (agent, relative) => { + const profile = managedStartupE2eProfile(agent, false); + const file = path.join(agentRoot(agent), relative); + fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 }); + fs.writeFileSync(file, "original private state", { mode: 0o600 }); + const sentinel = path.join(agentRoot(agent), "unselected.txt"); + fs.writeFileSync(sentinel, "retain"); + const bound = { ...options, bootstrapIdentity: "a".repeat(64) }; + beginManagedStartupSharedStateTransaction(profile, bound); + const captured = exportManagedStartupSharedStateRestoration(profile, bound)!; + fs.writeFileSync(file, "partial generation"); + fs.chmodSync(file, 0o644); + fs.chmodSync(path.dirname(file), 0o755); + const observed = restoreManagedStartupCapturedSharedState(profile, captured, bound); + expect(fs.existsSync(transactionDirectory)).toBe(true); + expect(fs.readFileSync(file, "utf8")).toBe("original private state"); + expect(mode(file)).toBe(0o600); + expect(mode(path.dirname(file))).toBe(0o700); + expect(fs.readFileSync(sentinel, "utf8")).toBe("retain"); + expect(finishManagedStartupSharedStateRestoration(profile, observed, bound)).toBe(true); + expect(fs.existsSync(transactionDirectory)).toBe(false); + }, + ); + + it("removes newly generated selected outputs without changing unrelated files", () => { + const profile = managedStartupE2eProfile("hermes", false); + const root = agentRoot("hermes"); + fs.mkdirSync(root, { mode: 0o700 }); + const bound = { ...options, bootstrapIdentity: "a".repeat(64) }; + beginManagedStartupSharedStateTransaction(profile, bound); + const captured = exportManagedStartupSharedStateRestoration(profile, bound)!; + fs.writeFileSync(path.join(root, "config.yaml"), "new"); + fs.writeFileSync(path.join(root, ".env"), "synthetic"); + fs.writeFileSync(path.join(root, "keep"), "retained"); + const observed = restoreManagedStartupCapturedSharedState(profile, captured, bound); + expect(fs.existsSync(path.join(root, "config.yaml"))).toBe(false); + expect(fs.existsSync(path.join(root, ".env"))).toBe(false); + expect(fs.readFileSync(path.join(root, "keep"), "utf8")).toBe("retained"); + expect(finishManagedStartupSharedStateRestoration(profile, observed, bound)).toBe(true); + }); + + it("retains the original receipt after an incomplete restoration observation", () => { + const original = capturedOpenClaw(); + const bound = { ...options, bootstrapIdentity: original.bootstrapIdentity }; + beginManagedStartupSharedStateTransaction(original.profile, bound); + fs.chmodSync(original.file, 0o644); + const observed = captureManagedStartupSharedState(original.profile, bound); + expect(() => + finishManagedStartupSharedStateRestoration(original.profile, observed, bound), + ).toThrow(/does not match the protected original/u); + expect(fs.existsSync(transactionDirectory)).toBe(true); + expect(mode(original.file)).toBe(0o644); + }); + + it("rejects another restoration identity before changing any output", () => { + const original = capturedOpenClaw(); + const captured = JSON.parse(original.capturedState); + const manifest = JSON.parse(captured.manifest); + manifest.files[0].uid += 1; + captured.manifest = `${JSON.stringify(manifest, null, 2)}\n`; + fs.writeFileSync(original.file, "retain partial state"); + expect(() => + restoreManagedStartupCapturedSharedState(original.profile, JSON.stringify(captured), { + sandboxRoot, + bootstrapIdentity: original.bootstrapIdentity, + }), + ).toThrow(/different restoration identity/u); + expect(fs.readFileSync(original.file, "utf8")).toBe("retain partial state"); + }); + + it("retains recovery authority when restoring a selected directory cannot complete", () => { + const profile = managedStartupE2eProfile("openclaw", false); + const bound = { ...options, bootstrapIdentity: "a".repeat(64) }; + beginManagedStartupSharedStateTransaction(profile, bound); + const captured = exportManagedStartupSharedStateRestoration(profile, bound)!; + fs.mkdirSync(agentRoot("openclaw")); + fs.writeFileSync(path.join(agentRoot("openclaw"), "unselected"), "retain"); + expect(() => restoreManagedStartupCapturedSharedState(profile, captured, bound)).toThrow( + /could not remove newly created managed directory/u, + ); + expect(fs.existsSync(transactionDirectory)).toBe(true); + expect(fs.readFileSync(path.join(agentRoot("openclaw"), "unselected"), "utf8")).toBe("retain"); + }); + + it("refuses export after commit and rejects a changed backup before export", () => { + const original = capturedOpenClaw(); + const bound = { ...options, bootstrapIdentity: original.bootstrapIdentity }; + beginManagedStartupSharedStateTransaction(original.profile, bound); + const backup = path.join(transactionDirectory, "backups", "000.bin"); + fs.chmodSync(backup, 0o600); + fs.writeFileSync(backup, "corrupt"); + fs.chmodSync(backup, 0o400); + expect(() => exportManagedStartupSharedStateRestoration(original.profile, bound)).toThrow( + /does not match its receipt/u, + ); + fs.chmodSync(backup, 0o600); + fs.writeFileSync(backup, '{"synthetic":"original"}'); + fs.chmodSync(backup, 0o400); + commitManagedStartupSharedStateTransaction("openclaw", bound); + expect(() => exportManagedStartupSharedStateRestoration(original.profile, bound)).toThrow( + /durably committed/u, + ); + }); + function commitReceiptDirectory(): string { return path.join( path.dirname(transactionDirectory), diff --git a/src/lib/onboard/managed-startup/agent-environment.ts b/src/lib/onboard/managed-startup/agent-environment.ts index 6c213772dd4..842eb70e1e1 100644 --- a/src/lib/onboard/managed-startup/agent-environment.ts +++ b/src/lib/onboard/managed-startup/agent-environment.ts @@ -38,20 +38,14 @@ export interface ManagedStartupRootOwnedFileMaterial { | "NEMOCLAW_DCODE_AUTO_APPROVAL" | "NEMOCLAW_INFERENCE_BASE_URL" | "NEMOCLAW_INFERENCE_PROVIDER_ID" - | "NEMOCLAW_PROXY_HOST" - | "NEMOCLAW_PROXY_PORT" | "NEMOCLAW_REASONING_EFFORT" | "NEMOCLAW_UPSTREAM_PROVIDER"; readonly path: | "/usr/local/share/nemoclaw/dcode-auto-approval" | "/usr/local/share/nemoclaw/dcode-inference-base-url" - | "/usr/local/share/nemoclaw/dcode-proxy-host" - | "/usr/local/share/nemoclaw/dcode-proxy-port" | "/usr/local/share/nemoclaw/dcode-reasoning-effort" | "/usr/local/share/nemoclaw/dcode-upstream-provider" - | "/usr/local/share/nemoclaw/dcode-provider-route" - | "/usr/local/share/nemoclaw/pi-proxy-host" - | "/usr/local/share/nemoclaw/pi-proxy-port"; + | "/usr/local/share/nemoclaw/dcode-provider-route"; readonly contents: string; readonly owner: "root"; readonly group: "root"; @@ -394,8 +388,6 @@ function mapOpenClawProfile( NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE: String(profile.agentConfig.otel.sampleRate), NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME: profile.agentConfig.otel.serviceName, NEMOCLAW_PRIMARY_MODEL_REF: profile.inference?.primaryModelRef ?? "", - NEMOCLAW_PROXY_HOST: profile.proxy.managedHost, - NEMOCLAW_PROXY_PORT: String(profile.proxy.managedPort), NEMOCLAW_REASONING: String(profile.tuning.reasoning), NEMOCLAW_REASONING_EFFORT: profile.tuning.reasoningEffort, NEMOCLAW_SERVING_PRESET: profile.inference?.servingPreset ?? "", @@ -474,8 +466,6 @@ function mapHermesProfile( runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_PORT = profile.dashboard.publicPort === null ? "" : String(profile.dashboard.publicPort); runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_TUI = booleanFlag(profile.dashboard.tuiEnabled); - runtimeEnvironment.NEMOCLAW_PROXY_HOST = profile.proxy.managedHost; - runtimeEnvironment.NEMOCLAW_PROXY_PORT = String(profile.proxy.managedPort); appendHostProxyEnvironment(runtimeEnvironment, profile, { preserveAmbientWhenAbsent: true }); return Object.freeze({ @@ -557,16 +547,6 @@ function mapDcodeProfile( "/usr/local/share/nemoclaw/dcode-provider-route", profile.inference.routeProvider, ), - rootOwnedFile( - "NEMOCLAW_PROXY_HOST", - "/usr/local/share/nemoclaw/dcode-proxy-host", - profile.proxy.managedHost, - ), - rootOwnedFile( - "NEMOCLAW_PROXY_PORT", - "/usr/local/share/nemoclaw/dcode-proxy-port", - String(profile.proxy.managedPort), - ), rootOwnedFile( "NEMOCLAW_REASONING_EFFORT", "/usr/local/share/nemoclaw/dcode-reasoning-effort", @@ -623,16 +603,6 @@ function mapPiProfile( } const materials: readonly ManagedStartupAgentMaterial[] = Object.freeze([ corporateCaMaterial(profile), - rootOwnedFile( - "NEMOCLAW_PROXY_HOST", - "/usr/local/share/nemoclaw/pi-proxy-host", - profile.proxy.managedHost, - ), - rootOwnedFile( - "NEMOCLAW_PROXY_PORT", - "/usr/local/share/nemoclaw/pi-proxy-port", - String(profile.proxy.managedPort), - ), ]); return Object.freeze({ diff --git a/src/lib/onboard/managed-startup/image-runtime.ts b/src/lib/onboard/managed-startup/image-runtime.ts index ff565adc349..ef57ea459a2 100644 --- a/src/lib/onboard/managed-startup/image-runtime.ts +++ b/src/lib/onboard/managed-startup/image-runtime.ts @@ -8,6 +8,7 @@ import path from "node:path"; import { PEM_CERTIFICATE_RE_GLOBAL } from "../corporate-ca-policy"; import { managedImageRuntimeIdentity } from "../managed-image/agents"; +import { prepareManagedStartupApplication } from "./application"; import { HERMES_RELAY_PLUGIN_STATE_FILE, HERMES_RELAY_PLUGIN_STATE_STAGING_FILE, @@ -47,11 +48,16 @@ import { } from "./root-apply"; import { beginManagedStartupSharedStateTransaction, + captureManagedStartupSharedState, clearManagedStartupSharedStateCommitReceipt, commitManagedStartupSharedStateTransaction, + exportManagedStartupSharedStateRestoration, + finishManagedStartupSharedStateRestoration, getManagedStartupSharedStateTransactionStatus, MANAGED_STARTUP_SHARED_ROLLBACK_RECEIPT_DIRECTORY, + MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES, rollbackManagedStartupSharedStateTransaction, + restoreManagedStartupCapturedSharedState, } from "./shared-state-transaction"; import { managedStartupWorkspaceRoot } from "./state-roots"; import { MANAGED_STARTUP_CA_ENV, MANAGED_STARTUP_PROFILE_ENV } from "./transport"; @@ -198,8 +204,16 @@ export function normalizeManagedStartupWorkspaceRoot( agentIdentity: managedImageRuntimeIdentity(agent), }); try { - deps.chown(target, expected.uid, expected.gid); - deps.chmod(target, expected.mode); + // Capability-free root cannot rewrite metadata owned by the sandbox user, + // even when the requested ownership and mode already match. + if ( + before.uid !== expected.uid || + before.gid !== expected.gid || + (before.mode & 0o7777) !== expected.mode + ) { + deps.chown(target, expected.uid, expected.gid); + deps.chmod(target, expected.mode); + } } catch { fail("managed startup workspace-root normalization failed"); } @@ -532,6 +546,11 @@ function readSandboxIdentity(): { readonly uid: string; readonly gid: string } { export function managedStartupSandboxPrefix(): readonly string[] { if (trustedExecutable("/usr/bin/setpriv")) { const identity = readSandboxIdentity(); + if ( + process.geteuid?.() === Number(identity.uid) && + process.getegid?.() === Number(identity.gid) + ) + return []; return [ "/usr/bin/setpriv", `--reuid=${identity.uid}`, @@ -571,7 +590,7 @@ function execute( runAs: ManagedStartupImageIdentity, configurationEnvironment: Readonly>, applicationRuntime: ManagedStartupApplicationRuntimePlan, - capture = false, + capture: boolean | "private" = false, ): CommandResult { if (argv.length === 0) fail("refusing an empty managed startup command"); const command = runAs === "sandbox" ? [...managedStartupSandboxPrefix(), ...argv] : [...argv]; @@ -584,7 +603,7 @@ function execute( fail(`could not execute ${argv[0]}: ${result.error.message}`); } if (result.status !== 0) { - const detail = capture ? `: ${(result.stderr || result.stdout).trim()}` : ""; + const detail = capture === true ? `: ${(result.stderr || result.stdout).trim()}` : ""; fail(`${argv[0]} exited with status ${String(result.status ?? "unknown")}${detail}`); } return { @@ -795,6 +814,7 @@ function runInternalSandboxAction( function validateOpenClawConfiguration( configurationEnvironment: Readonly>, applicationRuntime: ManagedStartupApplicationRuntimePlan, + privateOutput = false, ): void { const validation = execute( ["/usr/local/bin/openclaw", "config", "validate", "--json"], @@ -804,7 +824,7 @@ function validateOpenClawConfiguration( OPENCLAW_CONFIG_PATH: "/sandbox/.openclaw/openclaw.json", }, applicationRuntime, - true, + privateOutput ? "private" : true, ); let parsed: unknown; try { @@ -1152,15 +1172,7 @@ function sealHermesConfiguration( configurationEnvironment: Readonly>, applicationRuntime: ManagedStartupApplicationRuntimePlan, ): void { - const configPath = "/sandbox/.hermes/config.yaml"; - const envPath = "/sandbox/.hermes/.env"; - const config = readStableRegularFile(configPath, 4 * 1024 * 1024); - const env = readStableRegularFile(envPath, 512 * 1024); - const hashText = [ - `${createHash("sha256").update(config).digest("hex")} ${configPath}`, - `${createHash("sha256").update(env).digest("hex")} ${envPath}`, - "", - ].join("\n"); + const hashText = hermesConfigurationHash(); atomicWriteRootFile("/etc/nemoclaw/hermes.config-hash", hashText, 0o444); runInternalSandboxAction( "write-hermes-compat-hash", @@ -1172,6 +1184,18 @@ function sealHermesConfiguration( ); } +function hermesConfigurationHash(): string { + const configPath = "/sandbox/.hermes/config.yaml"; + const envPath = "/sandbox/.hermes/.env"; + const config = readStableRegularFile(configPath, 4 * 1024 * 1024); + const env = readStableRegularFile(envPath, 512 * 1024); + return [ + `${createHash("sha256").update(config).digest("hex")} ${configPath}`, + `${createHash("sha256").update(env).digest("hex")} ${envPath}`, + "", + ].join("\n"); +} + function installRootOwnedMaterials(materials: readonly ManagedStartupAgentMaterial[]): void { for (const material of materials) { if (material.kind !== "root-owned-file") continue; @@ -1655,13 +1679,10 @@ export function publishManagedStartupCompletionAfterCommit( ); } -function applyAdapter( - context: ManagedStartupAdapterContext, +function applyAdapterCommands( mapped: ManagedStartupAgentEnvironment, + identity?: ManagedStartupImageIdentity, ): void { - if (mapped.agent !== context.agent) { - fail(`mapped ${mapped.agent} environment for ${context.agent}`); - } const commandPlan = buildManagedStartupImageActionPlan({ agent: mapped.agent, actions: mapped.actions, @@ -1672,6 +1693,7 @@ function applyAdapter( const command = commandPlan[commandIndex]; if (!command) fail(`missing image command for ${action.kind}`); commandIndex += 1; + if (identity !== undefined && command.runAs !== identity) continue; if (action.kind === "apply-messaging-plan") { if (action.phase === "runtime-setup") { prepareMessagingRuntimeTarget(action.mode); @@ -1681,6 +1703,7 @@ function applyAdapter( command.runAs, mapped.configurationEnvironment, mapped.applicationRuntime, + identity === undefined ? false : "private", ); if (action.phase === "runtime-setup") { verifyMessagingRuntimeTarget(action.mode); @@ -1692,27 +1715,54 @@ function applyAdapter( command.runAs, mapped.configurationEnvironment, mapped.applicationRuntime, + identity === undefined ? false : "private", ); } if (commandIndex !== commandPlan.length) { fail("image action plan contains an unmatched command"); } +} - switch (context.agent) { - case "openclaw": - validateOpenClawConfiguration(mapped.configurationEnvironment, mapped.applicationRuntime); - break; - case "hermes": - installHermesManagedPolicy(); +function applyAdapter( + context: ManagedStartupAdapterContext, + mapped: ManagedStartupAgentEnvironment, + ownerResult?: ManagedStartupOwnerResult, +): void { + if (mapped.agent !== context.agent) + fail(`mapped ${mapped.agent} environment for ${context.agent}`); + if (ownerResult) { + for (const action of mapped.actions) { + if (action.kind === "apply-messaging-plan" && action.phase === "runtime-setup") { + verifyMessagingRuntimeTarget(action.mode); + } + } + if (context.agent === "hermes") { + atomicWriteRootFile( + HERMES_INSTALLED_MANAGED_POLICY_FILE, + Buffer.from(ownerResult.hermesPolicyB64!, "base64"), + 0o444, + ); + atomicWriteRootFile("/etc/nemoclaw/hermes.config-hash", ownerResult.hermesConfigHash!, 0o444); installHermesSwitchyardRoutingFiles(mapped.configurationEnvironment); - sealHermesConfiguration(mapped.configurationEnvironment, mapped.applicationRuntime); - // Normalize before the coordinator commits a newly applied profile so - // the durable transaction never records generator-created 0600 files as - // a completed mutable image contract. - normalizeHermesManagedConfiguration(); - break; - case "langchain-deepagents-code": - break; + } + } else { + applyAdapterCommands(mapped); + switch (context.agent) { + case "openclaw": + validateOpenClawConfiguration(mapped.configurationEnvironment, mapped.applicationRuntime); + break; + case "hermes": + installHermesManagedPolicy(); + installHermesSwitchyardRoutingFiles(mapped.configurationEnvironment); + sealHermesConfiguration(mapped.configurationEnvironment, mapped.applicationRuntime); + // Normalize before the coordinator commits a newly applied profile so + // the durable transaction never records generator-created 0600 files as + // a completed mutable image contract. + normalizeHermesManagedConfiguration(); + break; + case "langchain-deepagents-code": + break; + } } installRootOwnedMaterials(mapped.materials); installCorporateCa(context.corporateCaPath); @@ -1720,17 +1770,23 @@ function applyAdapter( mergeCorporateCa(context.corporateCaPath); } -function adapters(mapped: ManagedStartupAgentEnvironment): readonly ManagedStartupAgentAdapter[] { +function adapters( + mapped: ManagedStartupAgentEnvironment, + ownerResult?: ManagedStartupOwnerResult, +): readonly ManagedStartupAgentAdapter[] { return MANAGED_STARTUP_AGENTS.map((agent) => ({ agent, - apply: (context: ManagedStartupAdapterContext) => applyAdapter(context, mapped), + apply: (context: ManagedStartupAdapterContext) => applyAdapter(context, mapped, ownerResult), })); } export async function applyManagedStartupImageProfile( expectedAgentInput: string, env: Environment = process.env, - options: { readonly publishCompletion?: boolean } = {}, + options: { + readonly publishCompletion?: boolean; + readonly ownerResult?: ManagedStartupOwnerResult; + } = {}, ): Promise { requireRoot(); const expectedAgent = exactAgent(expectedAgentInput); @@ -1762,14 +1818,16 @@ export async function applyManagedStartupImageProfile( ? {} : { corporateCaB64: env[MANAGED_STARTUP_CA_ENV] }), }, - adapters(mapped), + adapters(mapped, options.ownerResult), ); if (mapped.agent !== result.application.profile.agent) { fail(`mapped ${mapped.agent} environment for ${result.application.profile.agent}`); } if (expectedAgent === "hermes" && !result.adapterApplied) { - // Committed startup replays still repair generator-created 0600 files. - normalizeHermesManagedConfiguration(); + if (!options.ownerResult) { + // The delegated owner already normalized its configuration files. + normalizeHermesManagedConfiguration(); + } verifyHermesSwitchyardRoutingFiles(mapped.configurationEnvironment); } let corporateCaMerged: boolean; @@ -1818,6 +1876,203 @@ export async function applyManagedStartupImageProfile( }; } +interface ManagedStartupOwnerResult { + readonly agent: ManagedStartupAgent; + readonly profileFingerprint: string; + readonly bootstrapIdentity: string; + readonly applied: boolean; + readonly hermesPolicyB64: string | null; + readonly hermesConfigHash: string | null; +} + +function requestImageEnvironment( + request: ManagedStartupRootApplyRequest, + env: Environment, +): Environment { + return { + NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION: "1", + ...selectManagedStartupApplicationRuntimeEnvironment(env), + [MANAGED_STARTUP_PROFILE_ENV]: request.encodedProfile, + ...(request.corporateCaB64 === null + ? {} + : { [MANAGED_STARTUP_CA_ENV]: request.corporateCaB64 }), + }; +} + +function prepareDelegatedApplication( + request: ManagedStartupRootApplyRequest, + bootstrapIdentity: string, +) { + requireRoot(); + if ( + !SHA256_RE.test(bootstrapIdentity) || + getManagedStartupSharedStateTransactionStatus({ + agent: request.agent, + profileFingerprint: request.profileFingerprint, + bootstrapIdentity, + }) === "none" + ) + fail("delegated startup requires its protected shared-state transaction"); + requireManagedStartupExchangeDirectory(); + ensureRootOwnedDirectory(ROOT_STATE_PARENT); + ensureRootOwnedDirectory(ROOT_RUNTIME_DIRECTORY); + return prepareManagedStartupApplication({ + encodedProfile: request.encodedProfile, + expectedAgent: request.agent, + ...(request.corporateCaB64 === null ? {} : { corporateCaB64: request.corporateCaB64 }), + }); +} + +/** Run root setup before the provider starts generation as the native account. */ +export function prepareManagedStartupDelegatedRequest( + request: ManagedStartupRootApplyRequest, + bootstrapIdentity: string, + env: Environment = process.env, +): "prepared" | "already-committed" { + requireRoot(); + const profile = decodeManagedStartupProfile(request.encodedProfile); + const mapped = mapManagedStartupProfileToAgentEnvironment( + profile, + requestImageEnvironment(request, env), + ); + validateManagedStartupApplicationRuntimePlan(mapped.applicationRuntime); + const prepared = prepareDelegatedApplication(request, bootstrapIdentity); + if (prepared.status === "prepared") applyAdapterCommands(mapped, "root"); + return prepared.status; +} + +/** Provider-selected native execution; stdout is reserved for the bounded private result. */ +export function applyManagedStartupOwnerRequest( + request: ManagedStartupRootApplyRequest, + bootstrapIdentity: string, + replay = false, + env: Environment = process.env, +): ManagedStartupOwnerResult { + requireNativeStateOwner(request.agent); + if (!SHA256_RE.test(bootstrapIdentity)) fail("invalid owner bootstrap identity"); + const profile = decodeManagedStartupProfile(request.encodedProfile); + const mapped = mapManagedStartupProfileToAgentEnvironment( + profile, + requestImageEnvironment(request, env), + ); + validateManagedStartupApplicationRuntimePlan(mapped.applicationRuntime); + if (!replay) applyAdapterCommands(mapped, "sandbox"); + let hermesPolicyB64: string | null = null; + let hermesConfigHash: string | null = null; + if (request.agent === "openclaw" && !replay) + validateOpenClawConfiguration(mapped.configurationEnvironment, mapped.applicationRuntime, true); + if (request.agent === "hermes") { + if (!replay) { + const policy = readStableRegularFileSnapshot( + HERMES_GENERATED_MANAGED_POLICY_FILE, + MAX_HERMES_MANAGED_POLICY_BYTES, + ); + hermesPolicyB64 = policy.bytes.toString("base64"); + hermesConfigHash = hermesConfigurationHash(); + writeSandboxFileAtomically("/sandbox/.hermes/.config-hash", hermesConfigHash, 0o640); + const routing = readHermesSwitchyardRoutingEnvironment(mapped.configurationEnvironment); + if (routing) { + const serialized = serializeHermesSwitchyardRouting(routing); + writeSandboxFileAtomically( + `/sandbox/.hermes/${HERMES_RELAY_PLUGINS_STAGING_FILE}`, + serialized.relayPluginsToml, + 0o600, + ); + writeSandboxFileAtomically( + `/sandbox/.hermes/${HERMES_RELAY_PLUGIN_STATE_STAGING_FILE}`, + serialized.relayPluginStateJson, + 0o600, + ); + writeSandboxFileAtomically( + `/sandbox/.hermes/${HERMES_SWITCHYARD_DEPLOYMENT_STAGING_FILE}`, + serialized.switchyardDeploymentToml, + 0o600, + ); + } + + if ( + !sameStableFileMetadata( + policy.stat, + fs.lstatSync(HERMES_GENERATED_MANAGED_POLICY_FILE, { bigint: true }), + ) + ) + fail("Hermes policy changed before owner cleanup"); + fs.unlinkSync(HERMES_GENERATED_MANAGED_POLICY_FILE); + } + normalizeHermesManagedConfiguration(); + } + return { + agent: request.agent, + profileFingerprint: request.profileFingerprint, + bootstrapIdentity, + applied: !replay, + hermesPolicyB64, + hermesConfigHash, + }; +} + +function parseOwnerResult( + text: string, + request: ManagedStartupRootApplyRequest, + bootstrapIdentity: string, + applied: boolean, +): ManagedStartupOwnerResult { + if (Buffer.byteLength(text) > MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES) + fail("owner result exceeds its bounded transport"); + let value: unknown; + try { + value = JSON.parse(text); + } catch { + fail("owner result is not valid JSON"); + } + if (!value || typeof value !== "object" || Array.isArray(value)) fail("invalid owner result"); + const result = value as Record; + if ( + Object.keys(result).sort().join(",") !== + "agent,applied,bootstrapIdentity,hermesConfigHash,hermesPolicyB64,profileFingerprint" || + result.agent !== request.agent || + result.profileFingerprint !== request.profileFingerprint || + result.bootstrapIdentity !== bootstrapIdentity || + result.applied !== applied + ) + fail("owner result does not match prepared startup"); + if (request.agent === "hermes" && applied) { + if ( + typeof result.hermesPolicyB64 !== "string" || + result.hermesPolicyB64.length === 0 || + result.hermesPolicyB64.length > 4 * Math.ceil(MAX_HERMES_MANAGED_POLICY_BYTES / 3) || + Buffer.from(result.hermesPolicyB64, "base64").toString("base64") !== result.hermesPolicyB64 || + typeof result.hermesConfigHash !== "string" || + !/^[a-f0-9]{64} \/sandbox\/\.hermes\/config\.yaml\n[a-f0-9]{64} \/sandbox\/\.hermes\/\.env\n$/u.test( + result.hermesConfigHash, + ) + ) + fail("invalid Hermes owner materials"); + } else if (result.hermesPolicyB64 !== null || result.hermesConfigHash !== null) + fail("unexpected owner materials"); + return result as unknown as ManagedStartupOwnerResult; +} + +/** Accept only the result transported from the provider's pinned owner process. */ +export async function finishManagedStartupDelegatedRequest( + request: ManagedStartupRootApplyRequest, + bootstrapIdentity: string, + ownerResultText: string, + env: Environment = process.env, +): Promise { + const prepared = prepareDelegatedApplication(request, bootstrapIdentity); + const ownerResult = parseOwnerResult( + ownerResultText, + request, + bootstrapIdentity, + prepared.status === "prepared", + ); + return applyManagedStartupImageProfile(request.agent, requestImageEnvironment(request, env), { + publishCompletion: false, + ownerResult, + }); +} + function completionAlreadyPublished(request: ManagedStartupRootApplyRequest): boolean { try { verifyManagedStartupImageCompletion(request.agent, request.profileFingerprint); @@ -1882,7 +2137,7 @@ export async function applyManagedStartupRootRequest( }; } -function readBoundedRootApplyStdin(): string { +function readBoundedRootApplyStdin(maxBytes = MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES): string { const chunks: Buffer[] = []; let total = 0; while (true) { @@ -1890,7 +2145,7 @@ function readBoundedRootApplyStdin(): string { const read = fs.readSync(0, chunk, 0, chunk.length, null); if (read === 0) break; total += read; - if (total > MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES) { + if (total > maxBytes) { fail("root application stdin exceeds its bounded transport"); } chunks.push(chunk.subarray(0, read)); @@ -1985,7 +2240,179 @@ function readCliBootstrapIdentity(argv: readonly string[]): string { return argv[index + 1] as string; } +function requireNativeStateOwner(agent: ManagedStartupAgent): void { + const identity = managedImageRuntimeIdentity(agent); + if (process.geteuid?.() !== identity.uid || process.getegid?.() !== identity.gid) { + fail("shared-state capture requires the exact native state owner"); + } +} + +function readSharedStateEnvelope(agent: ManagedStartupAgent) { + let value: unknown; + try { + value = JSON.parse( + readBoundedRootApplyStdin( + MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES + MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES, + ), + ); + } catch { + fail("shared-state capture envelope is unreadable"); + } + if (!value || typeof value !== "object" || Array.isArray(value)) + fail("invalid shared-state capture envelope"); + const envelope = value as Record; + if ( + Object.keys(envelope).sort().join(",") !== "capturedState,encodedProfile" || + typeof envelope.capturedState !== "string" || + typeof envelope.encodedProfile !== "string" + ) { + fail("invalid shared-state capture envelope fields"); + } + const profile = decodeManagedStartupProfile(envelope.encodedProfile); + if (profile.agent !== agent) fail("shared-state capture envelope targets another agent"); + return { profile, capturedState: envelope.capturedState }; +} + export async function main(argv: readonly string[] = process.argv.slice(2)): Promise { + if (argv.length === 1 && argv[0] === "--startup-protocol") { + requireRoot(); + process.stdout.write("delegated-owner-v1\n"); + return; + } + if (argv.length === 5 && argv[0] === "--prepare-delegated-transaction-stdin") { + requireRoot(); + const agent = exactAgent(readCliAgent(argv, 5)); + const bootstrapIdentity = readCliBootstrapIdentity(argv); + const request = parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin()); + if (request.agent !== agent) fail("transaction preparation targets another agent"); + ensureRootOwnedDirectory(ROOT_STATE_PARENT); + process.stdout.write( + `${getManagedStartupSharedStateTransactionStatus({ + agent, + profileFingerprint: request.profileFingerprint, + bootstrapIdentity, + })}\n`, + ); + return; + } + if (argv.length === 5 && argv[0] === "--export-shared-restoration-profile-stdin") { + requireRoot(); + const agent = exactAgent(readCliAgent(argv, 5)); + const profile = decodeManagedStartupProfile(readBoundedRootApplyStdin().trim()); + if (profile.agent !== agent) fail("shared-state restoration profile targets another agent"); + const bootstrapIdentity = readCliBootstrapIdentity(argv); + process.stdout.write( + exportManagedStartupSharedStateRestoration(profile, { bootstrapIdentity }) ?? "absent\n", + ); + return; + } + if ( + (argv.length === 5 || (argv.length === 6 && argv[5] === "--replay")) && + ["--prepare-delegated-startup-stdin", "--apply-owner-startup-stdin"].includes(argv[0] ?? "") + ) { + const agent = exactAgent(readCliAgent(argv, argv.length)); + const owner = argv[0] === "--apply-owner-startup-stdin"; + if (owner) requireNativeStateOwner(agent); + else { + requireRoot(); + if (argv.length !== 5) fail("root preparation cannot request owner replay"); + } + const bootstrapIdentity = readCliBootstrapIdentity(argv); + const request = parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin()); + if (request.agent !== agent) fail("delegated startup request targets another agent"); + if (owner) + process.stdout.write( + `${JSON.stringify(applyManagedStartupOwnerRequest(request, bootstrapIdentity, argv.length === 6))}\n`, + ); + else + process.stdout.write( + `${prepareManagedStartupDelegatedRequest(request, bootstrapIdentity)}\n`, + ); + return; + } + if (argv.length === 5 && argv[0] === "--finish-delegated-startup-stdin") { + requireRoot(); + const agent = exactAgent(readCliAgent(argv, 5)); + const bootstrapIdentity = readCliBootstrapIdentity(argv); + let value: unknown; + try { + value = JSON.parse( + readBoundedRootApplyStdin( + MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES + MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES, + ), + ); + } catch { + fail("delegated completion envelope is unreadable"); + } + if (!value || typeof value !== "object" || Array.isArray(value)) + fail("invalid delegated completion envelope"); + const envelope = value as Record; + if ( + Object.keys(envelope).sort().join(",") !== "ownerResult,request" || + typeof envelope.request !== "string" || + typeof envelope.ownerResult !== "string" + ) + fail("invalid delegated completion fields"); + const request = parseManagedStartupRootApplyRequest(envelope.request); + if (request.agent !== agent) fail("delegated completion targets another agent"); + await finishManagedStartupDelegatedRequest(request, bootstrapIdentity, envelope.ownerResult); + process.stdout.write("applied\n"); + return; + } + if ( + argv.length === 5 && + (argv[0] === "--capture-shared-state-stdin" || argv[0] === "--export-shared-restoration-stdin") + ) { + const agent = exactAgent(readCliAgent(argv, 5)); + const exporting = argv[0] === "--export-shared-restoration-stdin"; + if (exporting) requireRoot(); + else requireNativeStateOwner(agent); + const request = parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin()); + if (request.agent !== agent) fail("shared-state capture request targets another agent"); + const profile = decodeManagedStartupProfile(request.encodedProfile); + const options = { bootstrapIdentity: readCliBootstrapIdentity(argv) }; + process.stdout.write( + exporting + ? (exportManagedStartupSharedStateRestoration(profile, options) ?? "absent\n") + : captureManagedStartupSharedState(profile, options), + ); + return; + } + if ( + argv.length === 5 && + [ + "--begin-shared-state-from-capture-stdin", + "--restore-shared-state-stdin", + "--finish-shared-restoration-stdin", + ].includes(argv[0] ?? "") + ) { + const agent = exactAgent(readCliAgent(argv, 5)); + const restoring = argv[0] === "--restore-shared-state-stdin"; + if (restoring) requireNativeStateOwner(agent); + else requireRoot(); + const bootstrapIdentity = readCliBootstrapIdentity(argv); + const { profile, capturedState } = readSharedStateEnvelope(agent); + if (restoring) { + process.stdout.write( + restoreManagedStartupCapturedSharedState(profile, capturedState, { bootstrapIdentity }), + ); + return; + } + if (argv[0] === "--finish-shared-restoration-stdin") { + const restored = finishManagedStartupSharedStateRestoration(profile, capturedState, { + bootstrapIdentity, + }); + process.stdout.write(restored ? "restored\n" : "absent\n"); + return; + } + ensureRootOwnedDirectory(ROOT_STATE_PARENT); + const created = beginManagedStartupSharedStateTransaction(profile, { + bootstrapIdentity, + capturedState, + }); + process.stdout.write(created ? "created\n" : "pending\n"); + return; + } if (argv.length === 1 && argv[0] === "--internal-write-hermes-compat-hash") { internalWriteHermesCompatHash(); return; diff --git a/src/lib/onboard/managed-startup/onboard-profile.ts b/src/lib/onboard/managed-startup/onboard-profile.ts index 2d8a594cbb2..0b3b033e372 100644 --- a/src/lib/onboard/managed-startup/onboard-profile.ts +++ b/src/lib/onboard/managed-startup/onboard-profile.ts @@ -41,30 +41,13 @@ const PROFILE_ENVIRONMENT_INPUTS = { "NEMOCLAW_OPENCLAW_OTEL_ENDPOINT", "NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE", "NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME", - "NEMOCLAW_PROXY_HOST", - "NEMOCLAW_PROXY_PORT", "NEMOCLAW_REASONING", "NEMOCLAW_REASONING_EFFORT", "NEMOCLAW_SERVING_PRESET", ], - hermes: [ - "NEMOCLAW_CONTEXT_WINDOW", - "NEMOCLAW_HERMES_SWITCHYARD_ROUTING_B64", - "NEMOCLAW_PROXY_HOST", - "NEMOCLAW_PROXY_PORT", - ], - "langchain-deepagents-code": [ - "NEMOCLAW_PROXY_HOST", - "NEMOCLAW_PROXY_PORT", - "NEMOCLAW_REASONING_EFFORT", - ], - pi: [ - "NEMOCLAW_CONTEXT_WINDOW", - "NEMOCLAW_MAX_TOKENS", - "NEMOCLAW_PROXY_HOST", - "NEMOCLAW_PROXY_PORT", - "NEMOCLAW_REASONING", - ], + hermes: ["NEMOCLAW_CONTEXT_WINDOW", "NEMOCLAW_HERMES_SWITCHYARD_ROUTING_B64"], + "langchain-deepagents-code": ["NEMOCLAW_REASONING_EFFORT"], + pi: ["NEMOCLAW_CONTEXT_WINDOW", "NEMOCLAW_MAX_TOKENS", "NEMOCLAW_REASONING"], } as const satisfies Record; const HOST_NO_PROXY_INPUTS = ["NO_PROXY", "no_proxy"] as const; @@ -105,9 +88,17 @@ export function resolveManagedStartupNativeLocalAttachment(input: { provider: string | null; endpointUrl: string | null; localNimSelected?: boolean; + preferredInferenceApi?: string | null; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; }): NativeLocalProviderAttachment | undefined { const attachment = input.nativeLocalProviderAttachment; + if ( + (attachment?.agent === "openclaw" || attachment?.agent === "hermes") && + attachment.provider === "compatible-endpoint" && + (attachment.api ?? "openai-completions") !== + (input.preferredInferenceApi ?? "openai-completions") + ) + throw new Error("Recorded compatible provider API does not match its native attachment."); return usesNativeLocalInferenceForAgent( input.agentName, input.provider, @@ -128,11 +119,17 @@ export function resolveManagedStartupInferenceEndpoint(input: { localNimSelected?: boolean; nativeLocalProviderAttachment?: { endpointUrl: string }; }): string | null { + if ( + input.provider === "vllm-local" && + input.localNimSelected && + input.agentName !== "pi" && + !input.nativeLocalProviderAttachment + ) + throw new Error("Local NIM inference requires its native provider attachment."); if (input.agentName === "hermes") { return input.nativeLocalProviderAttachment?.endpointUrl ?? input.endpointUrl; } if (input.agentName === "langchain-deepagents-code") { - if (input.provider === "vllm-local" && input.localNimSelected) return input.endpointUrl; if ( usesNativeLocalInference(input.provider, input.endpointUrl) && !input.nativeLocalProviderAttachment diff --git a/src/lib/onboard/managed-startup/profile-builder.ts b/src/lib/onboard/managed-startup/profile-builder.ts index d37abf24abf..6ad1e8618f9 100644 --- a/src/lib/onboard/managed-startup/profile-builder.ts +++ b/src/lib/onboard/managed-startup/profile-builder.ts @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { Buffer } from "node:buffer"; -import { createHash, X509Certificate } from "node:crypto"; +import { createHash } from "node:crypto"; import { assertNoPerAgentMaxSpawnDepth } from "../../extra-agents-validation"; import { MAX_AUTODETECTED_OLLAMA_CONTEXT_WINDOW } from "../../inference/ollama-runtime-context"; @@ -13,13 +13,8 @@ import { HERMES_SWITCHYARD_ROUTING_ENV, readHermesSwitchyardRoutingEnvironment, } from "../../hermes-switchyard-routing"; -import { - MAX_CORPORATE_CA_BYTES, - MAX_CORPORATE_CA_CERTS, - PEM_CERTIFICATE_RE_GLOBAL, -} from "../corporate-ca-policy"; import type { ResolvedCorporateCa } from "../corporate-ca-types"; -import { normalizeCertificateBlocks } from "../corporate-ca-validation"; +import { validateCorporateCaPem } from "../corporate-ca-validation"; import { encodeManagedStartupProfile, MANAGED_STARTUP_PROFILE_AFFORDANCE_INVENTORY, @@ -69,10 +64,10 @@ export const MANAGED_STARTUP_HOST_PROXY_URL_INPUTS = [ * change; otherwise construction fails before a sandbox is launched. */ const EXPECTED_AFFORDANCE_INVENTORY_SHA256 = { - openclaw: "b4a3fa7060c7b7f5654c271513398666ccf52e36dcc2ff4ac8064b1d801841cf", - hermes: "b33584cd58428ff7e3810c4a95a7b2fb43f393f13e2e8c4010816742db7ca200", - "langchain-deepagents-code": "08c75cf22495ec93a090bc5b70544eac65970e658b10fba057dea5ffef502e4a", - pi: "6302d387182c596fd67ad18577ecf82107bad6271aeeb5e69714115f91557abb", + openclaw: "d119329b26a6e73dcdb7e644f6bc618651c7a82b0698785bd9d09ca6d81d7b48", + hermes: "d324a228f042cd208102ff559e41ceb9d171a8ede9b198787a55661d05e89fb5", + "langchain-deepagents-code": "1aa0b36b1ba387a8ab67149af5cb549ca9ab729057a9fed51eb32ceb1427fc32", + pi: "822d29b6c5d8309a676a47a2e6f5dc74854dd0e59d71e6b2c8c6d148a7e6f37c", } as const satisfies Record; const INVENTORY_INPUTS = new Set( @@ -499,30 +494,11 @@ function resolveCorporateCaMaterial(corporateCa: ResolvedCorporateCa | null | un readonly corporateCaB64?: string; } { if (!corporateCa) return { bundleSha256: null }; - const bytes = Buffer.byteLength(corporateCa.pem, "utf8"); - if (bytes === 0 || bytes > MAX_CORPORATE_CA_BYTES) { - fail(`corporate CA material must be between 1 and ${String(MAX_CORPORATE_CA_BYTES)} bytes`); - } - const blocks = corporateCa.pem.match(PEM_CERTIFICATE_RE_GLOBAL); - if (!blocks || blocks.length === 0 || blocks.length > MAX_CORPORATE_CA_CERTS) { - fail( - `corporate CA material must contain between 1 and ${String(MAX_CORPORATE_CA_CERTS)} certificates`, - ); - } - for (const block of blocks) { - let certificate: X509Certificate; - try { - certificate = new X509Certificate(block); - } catch { - fail("corporate CA material contains an invalid X.509 certificate"); - } - if (!certificate.ca) { - fail("corporate CA material contains a certificate without CA basic constraints"); - } - } - const normalizedPem = normalizeCertificateBlocks(blocks); - if (corporateCa.pem.trim() !== normalizedPem.trim()) { - fail("corporate CA material must contain only normalized PEM certificate blocks"); + let normalizedPem: string; + try { + normalizedPem = validateCorporateCaPem(corporateCa.pem); + } catch (error) { + fail(error instanceof Error ? error.message : String(error)); } return { bundleSha256: createHash("sha256").update(normalizedPem, "utf8").digest("hex"), diff --git a/src/lib/onboard/managed-startup/profile.ts b/src/lib/onboard/managed-startup/profile.ts index d2933c7fbc0..8d29cc2616f 100644 --- a/src/lib/onboard/managed-startup/profile.ts +++ b/src/lib/onboard/managed-startup/profile.ts @@ -537,8 +537,6 @@ export const MANAGED_STARTUP_PROFILE_AFFORDANCE_INVENTORY = { affordance("NEMOCLAW_DASHBOARD_BIND", "dashboard.bindAddress"), affordance("NEMOCLAW_WSL_DASHBOARD_EXPOSURE", "dashboard.wslExposure"), affordance("NEMOCLAW_DASHBOARD_PORT", "dashboard.port", "runtime-env"), - affordance("NEMOCLAW_PROXY_HOST", "proxy.managedHost"), - affordance("NEMOCLAW_PROXY_PORT", "proxy.managedPort"), affordance("NEMOCLAW_MESSAGING_PLAN_B64", "messaging.plan"), affordance("NEMOCLAW_MINIMAL_BOOTSTRAP", "agentConfig.minimalBootstrap", "runtime-env"), affordance( @@ -578,8 +576,6 @@ export const MANAGED_STARTUP_PROFILE_AFFORDANCE_INVENTORY = { affordance("NEMOCLAW_HERMES_DASHBOARD_PORT", "dashboard.publicPort", "runtime-env"), affordance("NEMOCLAW_HERMES_DASHBOARD_INTERNAL_PORT", "dashboard.internalPort", "runtime-env"), affordance("NEMOCLAW_HERMES_DASHBOARD_TUI", "dashboard.tuiEnabled", "runtime-env"), - affordance("NEMOCLAW_PROXY_HOST", "proxy.managedHost", "runtime-env"), - affordance("NEMOCLAW_PROXY_PORT", "proxy.managedPort", "runtime-env"), affordance( "NEMOCLAW_CORPORATE_CA_B64", "corporateCa.bundleSha256", @@ -598,8 +594,6 @@ export const MANAGED_STARTUP_PROFILE_AFFORDANCE_INVENTORY = { affordance("NEMOCLAW_REASONING_EFFORT", "tuning.reasoningEffort"), affordance("NEMOCLAW_TOOL_DISCLOSURE", "tools.disclosure"), affordance("NEMOCLAW_DCODE_AUTO_APPROVAL", "agentConfig.autoApprovalMode"), - affordance("NEMOCLAW_PROXY_HOST", "proxy.managedHost"), - affordance("NEMOCLAW_PROXY_PORT", "proxy.managedPort"), affordance("NEMOCLAW_OBSERVABILITY", "agentConfig.observabilityEnabled", "runtime-env"), affordance( "NEMOCLAW_CORPORATE_CA_B64", @@ -619,8 +613,6 @@ export const MANAGED_STARTUP_PROFILE_AFFORDANCE_INVENTORY = { affordance("NEMOCLAW_MAX_TOKENS", "tuning.maxTokens"), affordance("NEMOCLAW_REASONING", "tuning.reasoning"), affordance("NEMOCLAW_TOOL_DISCLOSURE", "tools.disclosure"), - affordance("NEMOCLAW_PROXY_HOST", "proxy.managedHost"), - affordance("NEMOCLAW_PROXY_PORT", "proxy.managedPort"), affordance( "NEMOCLAW_CORPORATE_CA_B64", "corporateCa.bundleSha256", diff --git a/src/lib/onboard/managed-startup/provider-root-apply.test.ts b/src/lib/onboard/managed-startup/provider-root-apply.test.ts index f6817e2636a..67000db12e5 100644 --- a/src/lib/onboard/managed-startup/provider-root-apply.test.ts +++ b/src/lib/onboard/managed-startup/provider-root-apply.test.ts @@ -33,6 +33,9 @@ import { releaseProviderManagedStartupHold, } from "./provider-root-apply"; import { createManagedStartupRootApplyRequest } from "./root-apply"; +import { buildConfig } from "../../../../scripts/generate-openclaw-config.mts"; +import { mapManagedStartupProfileToAgentEnvironment } from "./agent-environment"; +import { applyManagedStartupCommandEnvironmentPlan } from "./image-runtime"; const CONTAINER_ID = "a".repeat(64); const IMAGE_ID = `sha256:${"b".repeat(64)}`; @@ -304,6 +307,62 @@ describe("provider-owned managed startup root application", () => { }, ); + it("generates the local compatible reference from the credential-free root apply environment", () => { + const { runtimeProvider } = createDockerRootApplyFixture(() => ({ + resourceHandle: CONTAINER_ID, + })); + const profile = managedStartupE2eProfile("openclaw", false, true, true); + const localProfile = { + ...profile, + inference: { + ...profile.inference!, + upstreamProvider: "compatible-endpoint", + routedBaseUrl: "http://host.openshell.internal:45173/v1", + api: "openai-completions" as const, + }, + }; + const request = createManagedStartupRootApplyRequest({ + agent: "openclaw", + encodedProfile: encodeManagedStartupProfile(localProfile), + corporateCaB64: Buffer.from(MANAGED_STARTUP_E2E_CORPORATE_CA_PEM).toString("base64"), + }); + applyProviderManagedStartupRootRequest({ + runtimeProvider, + sandboxName: SANDBOX_NAME, + sandboxId: SANDBOX_ID, + bootstrapIdentity: "c".repeat(64), + request, + environment: { + NEMOCLAW_LOCAL_INFERENCE_TOKEN: "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + COMPATIBLE_API_KEY: "synthetic-host-secret", + }, + }); + const command = vi + .mocked(runtimeProvider.lifecycle.privilegedSandboxControl.execute) + .mock.calls.map(([input]) => input.command) + .find((argv) => argv.includes("--apply-root-stdin"))!; + expect(command.slice(0, 2)).toEqual(["/usr/bin/env", "-i"]); + const rootEnvironment = Object.fromEntries( + command.slice(2, command.indexOf("/usr/local/bin/node")).map((entry) => { + const separator = entry.indexOf("="); + return [entry.slice(0, separator), entry.slice(separator + 1)]; + }), + ); + const mapped = mapManagedStartupProfileToAgentEnvironment(localProfile, rootEnvironment); + const generationEnvironment = applyManagedStartupCommandEnvironmentPlan( + { ...rootEnvironment, ...mapped.configurationEnvironment }, + mapped.applicationRuntime, + ); + expect(generationEnvironment.NEMOCLAW_LOCAL_INFERENCE_TOKEN).toBeUndefined(); + expect(generationEnvironment.COMPATIBLE_API_KEY).toBeUndefined(); + const config = buildConfig(generationEnvironment); + expect(config.models.providers[localProfile.inference.routeProvider]).toMatchObject({ + baseUrl: localProfile.inference.routedBaseUrl, + api: "openai-completions", + apiKey: "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }); + }); + it("rejects a provider without workspace preparation before root application", () => { const { request, runtimeProvider } = createDockerRootApplyFixture(() => ({ resourceHandle: CONTAINER_ID, diff --git a/src/lib/onboard/managed-startup/shared-state-transaction.ts b/src/lib/onboard/managed-startup/shared-state-transaction.ts index 38b43dcd6e3..30743aea966 100644 --- a/src/lib/onboard/managed-startup/shared-state-transaction.ts +++ b/src/lib/onboard/managed-startup/shared-state-transaction.ts @@ -23,7 +23,7 @@ const MAX_TRANSACTION_FILE_BYTES = 8 * 1024 * 1024; const MAX_TRANSACTION_TOTAL_BYTES = 32 * 1024 * 1024; const MAX_MANIFEST_BYTES = 256 * 1024; const MAX_COMMIT_RECEIPT_BYTES = 4096; -// Base64 of the existing backup budget, canonical manifest and JSON framing. +// Base64 of the existing 32 MiB backup budget, canonical manifest and JSON framing. export const MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES = 4 * Math.ceil(MAX_TRANSACTION_TOTAL_BYTES / 3) + MAX_MANIFEST_BYTES * 2; const TRANSACTION_PARENT_DIRECTORY_MODE = 0o755; @@ -89,6 +89,8 @@ interface CommitReceipt { } export interface ManagedStartupSharedTransactionOptions { + /** Trusted provider stdin only, never a workload-writable file or owner-pasted receipt. */ + readonly capturedState?: string; readonly sandboxRoot?: string; readonly transactionDirectory?: string; /** Test/helper seam. Production derives the fixed image-owned commit receipt path. */ @@ -1182,28 +1184,10 @@ export function beginManagedStartupSharedStateTransaction( verifyAllBackups(pending.files, options); return false; } - const targets = managedOutputTargets(profile, options); - if (targets.files.length > MAX_TRANSACTION_FILES) { - fail("managed startup transaction has too many file targets"); - } - const snapshots = targets.files.map((target, index) => - snapshotFile(target, index, profile.agent, options), - ); - const totalBytes = snapshots.reduce((sum, snapshot) => sum + (snapshot.bytes?.length ?? 0), 0); - if (totalBytes > MAX_TRANSACTION_TOTAL_BYTES) { - fail("managed startup transaction backup exceeds the total size limit"); - } - const directories = targets.directories.map((target) => - snapshotDirectory(target, profile.agent, options), - ); - const manifest: TransactionManifest = { - schemaVersion: TRANSACTION_SCHEMA_VERSION, - agent: profile.agent, - profileFingerprint, - bootstrapIdentity: options.bootstrapIdentity, - files: snapshots.map(({ receipt }) => receipt), - directories, - }; + const { manifest, snapshots } = + inputOptions.capturedState === undefined + ? captureSharedState(profile, options) + : parseCapturedSharedState(inputOptions.capturedState, profile, options); let createdTransactionIdentity: | { readonly dev: bigint; readonly ino: bigint; readonly uid: bigint; readonly gid: bigint } @@ -1277,6 +1261,128 @@ export function beginManagedStartupSharedStateTransaction( return true; } +function captureSharedState(profile: ManagedStartupProfile, options: ResolvedOptions) { + const targets = managedOutputTargets(profile, options); + if (targets.files.length > MAX_TRANSACTION_FILES) { + fail("managed startup transaction has too many file targets"); + } + const snapshots = targets.files.map((target, index) => + snapshotFile(target, index, profile.agent, options), + ); + const totalBytes = snapshots.reduce((sum, snapshot) => sum + (snapshot.bytes?.length ?? 0), 0); + if (totalBytes > MAX_TRANSACTION_TOTAL_BYTES) { + fail("managed startup transaction backup exceeds the total size limit"); + } + const directories = targets.directories.map((target) => + snapshotDirectory(target, profile.agent, options), + ); + const manifest: TransactionManifest = { + schemaVersion: TRANSACTION_SCHEMA_VERSION, + agent: profile.agent, + profileFingerprint: fingerprintManagedStartupProfile(profile), + bootstrapIdentity: options.bootstrapIdentity, + files: snapshots.map(({ receipt }) => receipt), + directories, + }; + + return { manifest, snapshots }; +} + +/** Read as the native state owner; this operation cannot create a trusted receipt. */ +export function captureManagedStartupSharedState( + profile: ManagedStartupProfile, + inputOptions: Pick, +): string { + const captured = captureSharedState(profile, resolveOptions(inputOptions)); + return serializeCapturedSharedState( + captured.manifest, + captured.snapshots.map(({ bytes }) => bytes), + ); +} + +function serializeCapturedSharedState( + manifest: TransactionManifest, + contents: readonly (Buffer | null)[], +): string { + const text = `${JSON.stringify({ + manifest: canonicalManifest(manifest), + contents: contents.map((bytes) => bytes?.toString("base64") ?? null), + })}\n`; + if (Buffer.byteLength(text) > MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES) { + fail("shared-state capture exceeds its bounded transport"); + } + return text; +} + +function parseCapturedSharedState( + text: string, + profile: ManagedStartupProfile, + options: ResolvedOptions, +) { + if (Buffer.byteLength(text) > MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES) { + fail("shared-state capture exceeds its bounded transport"); + } + let value: unknown; + try { + value = JSON.parse(text); + } catch { + fail("shared-state capture is not valid JSON"); + } + if (!value || typeof value !== "object" || Array.isArray(value)) + fail("invalid shared-state capture"); + const record = value as Record; + requireExactKeys(record, ["manifest", "contents"]); + if ( + typeof record.manifest !== "string" || + Buffer.byteLength(record.manifest) > MAX_MANIFEST_BYTES || + !Array.isArray(record.contents) + ) { + fail("invalid shared-state capture fields"); + } + const manifest = parseManifest(record.manifest); + const targets = managedOutputTargets(profile, options); + if ( + manifest.agent !== profile.agent || + manifest.profileFingerprint !== fingerprintManagedStartupProfile(profile) || + manifest.bootstrapIdentity !== options.bootstrapIdentity || + options.bootstrapIdentity === null || + JSON.stringify(manifest.files.map(({ path: target }) => target)) !== + JSON.stringify(targets.files.map((target) => relativeTarget(target, options))) || + JSON.stringify(manifest.directories.map(({ path: target }) => target)) !== + JSON.stringify(targets.directories.map((target) => relativeTarget(target, options))) || + record.contents.length !== manifest.files.length + ) { + fail("shared-state capture does not match the exact profile and bootstrap targets"); + } + const contents = record.contents; + let total = 0; + const snapshots = manifest.files.map((receipt, index) => { + const encoded = contents[index]; + if (receipt.state === "absent") { + if (encoded !== null) fail("absent captured output has unexpected contents"); + return { receipt, bytes: null }; + } + if ( + typeof encoded !== "string" || + encoded.length > 4 * Math.ceil(MAX_TRANSACTION_FILE_BYTES / 3) + ) { + fail("captured output exceeds its bounded encoding"); + } + const bytes = Buffer.from(encoded, "base64"); + total += bytes.length; + if ( + total > MAX_TRANSACTION_TOTAL_BYTES || + bytes.toString("base64") !== encoded || + bytes.length !== receipt.size || + createHash("sha256").update(bytes).digest("hex") !== receipt.sha256 + ) { + fail("captured output does not match its bounded receipt"); + } + return { receipt, bytes }; + }); + return { manifest, snapshots }; +} + function ensureOriginalDirectories( receipts: readonly DirectoryReceipt[], expectedAgent: ManagedStartupAgent, @@ -1416,6 +1522,20 @@ export function rollbackManagedStartupSharedStateTransaction( inputOptions: ManagedStartupSharedTransactionOptions = {}, ): boolean { const options = resolveOptions(inputOptions); + const manifest = loadRollbackManifest(expectedAgent, options); + if (!manifest) return false; + const backups = verifyAllBackups(manifest.files, options); + restoreSharedState(manifest, backups, options); + if (!options.readOnlyReceipt) { + removeTransactionDirectory(options); + } + return true; +} + +function loadRollbackManifest( + expectedAgent: ManagedStartupAgent, + options: ResolvedOptions, +): TransactionManifest | null { requireTransactionIdentity(options); const committed = loadCommitReceipt(options); if (committed) { @@ -1429,21 +1549,84 @@ export function rollbackManagedStartupSharedStateTransaction( fail("shared state is already durably committed and cannot be rolled back"); } const manifest = loadManifest(options); - if (!manifest) return false; + if (!manifest) return null; if (manifest.agent !== expectedAgent) { fail(`pending transaction targets ${manifest.agent}, expected ${expectedAgent}`); } if (manifest.bootstrapIdentity !== options.bootstrapIdentity) { fail("pending transaction belongs to a different bootstrap attempt"); } - const backups = verifyAllBackups(manifest.files, options); - ensureOriginalDirectories(manifest.directories, expectedAgent, options); - restoreFiles(manifest.files, backups, expectedAgent, options); + return manifest; +} + +function restoreSharedState( + manifest: TransactionManifest, + backups: ReadonlyMap, + options: ResolvedOptions, +): void { + ensureOriginalDirectories(manifest.directories, manifest.agent, options); + restoreFiles(manifest.files, backups, manifest.agent, options); restoreDirectoryMetadata(manifest.directories, options); verifyRestoration(manifest, options); - if (!options.readOnlyReceipt) { - removeTransactionDirectory(options); +} + +/** Export only to the provider's private process transport; keep the protected receipt. */ +export function exportManagedStartupSharedStateRestoration( + profile: ManagedStartupProfile, + inputOptions: ManagedStartupSharedTransactionOptions, +): string | null { + const options = resolveOptions(inputOptions); + const manifest = loadRollbackManifest(profile.agent, options); + if (!manifest) return null; + const backups = verifyAllBackups(manifest.files, options); + const captured = serializeCapturedSharedState( + manifest, + manifest.files.map((receipt) => (receipt.state === "file" ? backups.get(receipt.path)! : null)), + ); + // Bind legacy or externally copied receipts to the selected profile before exporting bytes. + parseCapturedSharedState(captured, profile, options); + return captured; +} + +/** Restore as the native owner. Only authenticated provider stdin may supply the root snapshot. */ +export function restoreManagedStartupCapturedSharedState( + profile: ManagedStartupProfile, + capturedState: string, + inputOptions: Pick, +): string { + const options = resolveOptions(inputOptions); + const { manifest, snapshots } = parseCapturedSharedState(capturedState, profile, options); + // A capability-free owner cannot restore another identity. Reject before any mutation. + for (const receipt of [...manifest.files, ...manifest.directories]) { + if (receipt.state === "absent") continue; + if (receipt.uid !== process.geteuid?.() || receipt.gid !== process.getegid?.()) { + fail("captured output requires a different restoration identity"); + } } + const backups = new Map(); + for (const snapshot of snapshots) { + if (snapshot.bytes !== null) backups.set(snapshot.receipt.path, snapshot.bytes); + } + restoreSharedState(manifest, backups, options); + return captureManagedStartupSharedState(profile, inputOptions); +} + +/** Retire authority only after the provider returns a fresh verified owner observation. */ +export function finishManagedStartupSharedStateRestoration( + profile: ManagedStartupProfile, + observedState: string, + inputOptions: ManagedStartupSharedTransactionOptions, +): boolean { + const options = resolveOptions(inputOptions); + if (options.readOnlyReceipt) fail("cannot retire a read-only rollback receipt"); + const original = loadRollbackManifest(profile.agent, options); + if (!original) return false; + verifyAllBackups(original.files, options); + const observed = parseCapturedSharedState(observedState, profile, options).manifest; + if (canonicalManifest(original) !== canonicalManifest(observed)) { + fail("restored state does not match the protected original snapshot"); + } + removeTransactionDirectory(options); return true; } diff --git a/src/lib/onboard/managed-workload/agent-state-volumes.ts b/src/lib/onboard/managed-workload/agent-state-volumes.ts index c808919c3d8..81e56547db2 100644 --- a/src/lib/onboard/managed-workload/agent-state-volumes.ts +++ b/src/lib/onboard/managed-workload/agent-state-volumes.ts @@ -2,7 +2,10 @@ // SPDX-License-Identifier: Apache-2.0 import { isManagedImageAgent, managedImageRuntimeIdentity } from "../managed-image/agents"; -import { managedStartupStateRoots } from "../managed-startup/state-roots"; +import { + managedStartupStateRoots, + type ManagedStartupStateRoot, +} from "../managed-startup/state-roots"; import type { RuntimeProviderBundle, RuntimeProviderBundleRegistry, @@ -62,6 +65,7 @@ export function managedStateVolumeDeps( const runtimeProvider = deps.runtimeProvider ?? selectedRuntimeProvider(runtimeProviderId, deps.runtimeProviders); return { + workspace: deps.workspace, ...(deps.runDocker && !runtimeProvider ? { runContainerEngine: deps.runDocker } : {}), ...(runtimeProvider ? { runtimeProvider } : {}), ...(deps.registerExitCleanup ? { registerExitCleanup: deps.registerExitCleanup } : {}), @@ -86,10 +90,12 @@ function managedAgentStateRoots(context: ManagedAgentStateVolumeContext) { /** Remove every state root declared for a managed-image agent, gated by exact ownership labels. */ export function removeManagedAgentStateVolumes( - context: ManagedAgentStateVolumeContext, + context: ManagedAgentStateVolumeContext & { + readonly retainedRoots?: readonly ManagedStartupStateRoot[]; + }, deps: Pick< ManagedAgentStateVolumeDeps, - "runDocker" | "runtimeProvider" | "runtimeProviders" + "runDocker" | "workspace" | "runtimeProvider" | "runtimeProviders" > = {}, ): readonly ManagedAgentStateVolumeCleanupResult[] { const runtimeProvider = selectedRuntimeProvider( @@ -103,7 +109,14 @@ export function removeManagedAgentStateVolumes( : supportsContainerEngineOperation(runtimeProvider, "workload-cleanup"); if (!hasCleanupAuthority) return []; return removeManagedStateVolumes( - { roots: managedAgentStateRoots(context) }, + { + roots: managedAgentStateRoots(context).filter( + (source) => + !context.retainedRoots?.some( + (target) => target.resourceIdentity === source.resourceIdentity, + ), + ), + }, managedStateVolumeDeps(deps, context.runtimeProviderId), ); } diff --git a/src/lib/onboard/managed-workload/hermes-state-volume.test.ts b/src/lib/onboard/managed-workload/hermes-state-volume.test.ts index ce30e5280e4..9980f107e77 100644 --- a/src/lib/onboard/managed-workload/hermes-state-volume.test.ts +++ b/src/lib/onboard/managed-workload/hermes-state-volume.test.ts @@ -1,28 +1,22 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it, vi } from "vitest"; +import assert from "node:assert/strict"; +import { afterEach, describe, expect, it, vi } from "vitest"; import type { PodmanBoundContainerEngine, PodmanContainerEngine } from "../../adapters/podman"; import { createHermesStateVolumeDockerHarness as dockerHarness } from "../__test-helpers__/hermes-state-volume"; -import { - createDockerRuntimeProviderBundle, - createKubernetesRuntimeProviderBundle, -} from "../runtime-provider/docker"; +import { createDockerRuntimeProviderBundle } from "../runtime-provider/docker"; import { createPodmanRuntimeProviderBundle } from "../runtime-provider/podman"; import { MANAGED_HERMES_STATE_ROOT, + managedHermesStateVolumeName, MANAGED_OPENCLAW_STATE_ROOT, managedStartupStateRoots, } from "../managed-startup/state-roots"; import { managedImageRuntimeIdentity } from "../managed-image/agents"; -import { prepareManagedStateVolumes } from "./managed-state-volumes"; +import { preflightManagedStateVolumes, prepareManagedStateVolumes } from "./managed-state-volumes"; import { removeManagedAgentStateVolumes } from "./agent-state-volumes"; -import { - managedHermesStateVolumeName, - prepareManagedHermesStateVolume, - removeManagedHermesStateVolume, -} from "./hermes-state-volume"; const context = { agentName: "hermes", @@ -31,29 +25,149 @@ const context = { workloadKind: "managed-image", } as const; +const roots = managedStartupStateRoots({ + agent: "hermes", + sandboxName: context.sandboxName, + agentIdentity: managedImageRuntimeIdentity("hermes"), +}); + describe("managed Hermes state volume", () => { + afterEach(() => vi.unstubAllEnvs()); + + it.each([ + [undefined, undefined], + ["team-ml", undefined], + ["other", "selected"], + ])("approves a new volume for ambient %s and selected workspace %s", (workspace, selected) => { + vi.stubEnv("OPENSHELL_WORKSPACE", workspace); + const docker = dockerHarness(); + const scope = prepareManagedStateVolumes( + { roots }, + { + workspace: selected, + runContainerEngine: docker.runDocker as never, + registerExitCleanup: () => () => undefined, + }, + ); + expect(docker.volume?.labels).toMatchObject({ + "openshell.ai/sandbox-attachable": "true", + "openshell.ai/sandbox-attachable-workspace": selected ?? workspace ?? "default", + }); + expect(scope?.reused).toEqual([false]); + }); + + it.each([ + ["missing", {}], + [ + "other workspace", + { + "openshell.ai/sandbox-attachable": "true", + "openshell.ai/sandbox-attachable-workspace": "another", + }, + ], + [ + "not approved", + { + "openshell.ai/sandbox-attachable": "false", + "openshell.ai/sandbox-attachable-workspace": "default", + }, + ], + ] as const)("preserves an owned retained volume with %s approval", (_problem, approval) => { + vi.stubEnv("OPENSHELL_WORKSPACE", "default"); + const created = dockerHarness(); + prepareManagedStateVolumes( + { roots }, + { + runContainerEngine: created.runDocker as never, + registerExitCleanup: () => () => undefined, + }, + )!.commit(); + const retained = created.volume!; + delete retained.labels["openshell.ai/sandbox-attachable"]; + delete retained.labels["openshell.ai/sandbox-attachable-workspace"]; + Object.assign(retained.labels, approval); + const docker = dockerHarness(retained); + const before = JSON.stringify(docker.volume); + expect(() => + prepareManagedStateVolumes({ roots }, { runContainerEngine: docker.runDocker as never }), + ).toThrow("Retained data was not changed"); + expect(JSON.stringify(docker.volume)).toBe(before); + expect(docker.calls.every((args) => args[0] === "inspect")).toBe(true); + expect( + removeManagedAgentStateVolumes(context, { runDocker: docker.runDocker as never }), + ).toEqual([{ status: "removed" }]); + }); + + it.each(["openclaw", "hermes"] as const)( + "rejects legacy %s state through the provider without changing the volume", + (agent) => { + const roots = managedStartupStateRoots({ + agent, + sandboxName: "alpha", + agentIdentity: managedImageRuntimeIdentity(agent), + }); + const root = roots[0]!; + const docker = dockerHarness({ + name: root.resourceIdentity, + labels: { ...root.ownershipLabels }, + }); + const provider = createDockerRuntimeProviderBundle(); + assert(provider.containerEngine.supported, "Docker engine is unavailable"); + vi.spyOn(provider.containerEngine, "capture").mockImplementation((_operation, args) => { + expect(args[0]).toBe("volume"); + const result = docker.runDocker(args.slice(1)); + return { + status: result.status, + stdout: String(result.stdout ?? ""), + stderr: String(result.stderr ?? ""), + }; + }); + const before = JSON.stringify(docker.volume); + expect(() => preflightManagedStateVolumes({ roots }, { runtimeProvider: provider })).toThrow( + "lacks OpenShell attachment approval", + ); + expect(() => prepareManagedStateVolumes({ roots }, { runtimeProvider: provider })).toThrow( + "lacks OpenShell attachment approval", + ); + expect(JSON.stringify(docker.volume)).toBe(before); + expect(docker.calls.every((args) => args[0] === "inspect")).toBe(true); + }, + ); + + it.each([" default", "default\n"])("rejects invalid workspace %j before effects", (workspace) => { + vi.stubEnv("OPENSHELL_WORKSPACE", workspace); + const docker = dockerHarness(); + expect(() => + prepareManagedStateVolumes({ roots }, { runContainerEngine: docker.runDocker as never }), + ).toThrow("exact OpenShell workspace"); + expect(docker.calls).toEqual([]); + }); it("creates and mounts one labeled writable volume for managed Docker Hermes", () => { const docker = dockerHarness(); let exitCleanup: (() => void) | null = null; const unregister = vi.fn(); - const scope = prepareManagedHermesStateVolume(context, { - runDocker: docker.runDocker as never, - registerExitCleanup: (cleanup) => { - exitCleanup = cleanup; - return unregister; + const scope = prepareManagedStateVolumes( + { roots }, + { + runContainerEngine: docker.runDocker as never, + registerExitCleanup: (cleanup) => { + exitCleanup = cleanup; + return unregister; + }, }, - }); + ); expect(scope).toMatchObject({ - reused: false, - volumeName: "nemoclaw-hermes-state-v1-alpha", - mount: { - type: "volume", - source: "nemoclaw-hermes-state-v1-alpha", - target: MANAGED_HERMES_STATE_ROOT, - read_only: false, - }, + reused: [false], + mounts: [ + { + type: "volume", + source: "nemoclaw-hermes-state-v1-alpha", + target: MANAGED_HERMES_STATE_ROOT, + read_only: false, + }, + ], }); expect(docker.volume?.labels).toMatchObject({ "io.nvidia.nemoclaw.hermes-state.managed": "true", @@ -68,22 +182,6 @@ describe("managed Hermes state volume", () => { expect(unregister).not.toHaveBeenCalled(); }); - it("uses the registered native provider for the same managed Hermes volume contract", () => { - const runtime = dockerHarness(); - const scope = prepareManagedHermesStateVolume( - { ...context, runtimeProviderId: "podman" }, - { - runDocker: runtime.runDocker as never, - registerExitCleanup: () => () => undefined, - }, - ); - - expect(scope?.mount).toMatchObject({ - source: "nemoclaw-hermes-state-v1-alpha", - target: MANAGED_HERMES_STATE_ROOT, - }); - }); - it("dispatches native volume lifecycle through the selected provider operation", () => { const runtime = dockerHarness(); const workloadCleanupCapture = vi.fn((args: readonly string[]) => { @@ -126,15 +224,15 @@ describe("managed Hermes state volume", () => { }, }); - const scope = prepareManagedHermesStateVolume( - { ...context, runtimeProviderId: "podman" }, + const scope = prepareManagedStateVolumes( + { roots }, { - runtimeProviders: { podman: provider }, + runtimeProvider: provider, registerExitCleanup: () => () => undefined, }, ); - expect(scope?.mount.source).toBe("nemoclaw-hermes-state-v1-alpha"); + expect(scope?.mounts[0]?.source).toBe("nemoclaw-hermes-state-v1-alpha"); expect(workloadCleanupCapture).toHaveBeenCalled(); expect(workloadCleanupCapture.mock.calls.every(([args]) => args[0] === "volume")).toBe(true); }); @@ -143,13 +241,16 @@ describe("managed Hermes state volume", () => { const docker = dockerHarness(); let exitCleanup: (() => void) | null = null; const unregister = vi.fn(); - const scope = prepareManagedHermesStateVolume(context, { - runDocker: docker.runDocker as never, - registerExitCleanup: (cleanup) => { - exitCleanup = cleanup; - return unregister; + const scope = prepareManagedStateVolumes( + { roots }, + { + runContainerEngine: docker.runDocker as never, + registerExitCleanup: (cleanup) => { + exitCleanup = cleanup; + return unregister; + }, }, - }); + ); scope!.commit(); exitCleanup!(); @@ -161,23 +262,29 @@ describe("managed Hermes state volume", () => { it("reuses the exact owned volume across rebuild without arming failure cleanup", () => { const created = dockerHarness(); - const first = prepareManagedHermesStateVolume(context, { - runDocker: created.runDocker as never, - registerExitCleanup: () => () => undefined, - }); + const first = prepareManagedStateVolumes( + { roots }, + { + runContainerEngine: created.runDocker as never, + registerExitCleanup: () => () => undefined, + }, + ); const reused = dockerHarness(created.volume); const registerExitCleanup = vi.fn(); - const second = prepareManagedHermesStateVolume(context, { - runDocker: reused.runDocker as never, - registerExitCleanup, - }); + const second = prepareManagedStateVolumes( + { roots }, + { + runContainerEngine: reused.runDocker as never, + registerExitCleanup, + }, + ); - expect(first?.volumeName).toBe(second?.volumeName); - expect(second?.reused).toBe(true); + expect(first?.mounts[0]?.source).toBe(second?.mounts[0]?.source); + expect(second?.reused).toEqual([true]); expect(registerExitCleanup).not.toHaveBeenCalled(); expect(reused.calls.some((args) => args[0] === "create")).toBe(false); - expect(second?.cleanupIncompleteCreate()).toEqual({ status: "not-applicable" }); + expect(second?.cleanupIncompleteCreate()).toEqual([]); expect(reused.volume).not.toBeNull(); }); @@ -186,7 +293,7 @@ describe("managed Hermes state volume", () => { const docker = dockerHarness({ name, labels: { "com.example.owner": "foreign" } }); expect(() => - prepareManagedHermesStateVolume(context, { runDocker: docker.runDocker as never }), + prepareManagedStateVolumes({ roots }, { runContainerEngine: docker.runDocker as never }), ).toThrow(/exact NemoClaw ownership labels do not match/u); expect(docker.volume).not.toBeNull(); expect(docker.calls.some((args) => args[0] === "rm")).toBe(false); @@ -194,77 +301,83 @@ describe("managed Hermes state volume", () => { it("removes only an exactly owned volume during sandbox destroy", () => { const owned = dockerHarness(); - const scope = prepareManagedHermesStateVolume(context, { - runDocker: owned.runDocker as never, - registerExitCleanup: () => () => undefined, - }); - scope!.commit(); - - expect( - removeManagedHermesStateVolume(context, { runDocker: owned.runDocker as never }), - ).toEqual({ status: "removed" }); - expect(owned.volume).toBeNull(); - - const name = managedHermesStateVolumeName(context.sandboxName); - const foreign = dockerHarness({ name, labels: { "com.example.owner": "foreign" } }); - expect( - removeManagedHermesStateVolume(context, { runDocker: foreign.runDocker as never }), - ).toMatchObject({ status: "not-owned", volumeName: name }); - expect(foreign.volume).not.toBeNull(); - }); - - it("projects and retires the declared OpenClaw state root through the generic volume path", () => { - const docker = dockerHarness(); - const roots = managedStartupStateRoots({ - agent: "openclaw", - sandboxName: "alpha", - agentIdentity: managedImageRuntimeIdentity("openclaw"), - }); const scope = prepareManagedStateVolumes( { roots }, { - runContainerEngine: docker.runDocker as never, + runContainerEngine: owned.runDocker as never, registerExitCleanup: () => () => undefined, }, ); - - expect(scope?.mounts).toEqual([ - { - type: "volume", - source: "nemoclaw-openclaw-state-v1-alpha", - target: MANAGED_OPENCLAW_STATE_ROOT, - read_only: false, - }, - ]); scope!.commit(); + expect( - removeManagedAgentStateVolumes( - { ...context, agentName: "openclaw" }, - { runDocker: docker.runDocker as never }, - ), + removeManagedAgentStateVolumes(context, { runDocker: owned.runDocker as never }), ).toEqual([{ status: "removed" }]); - expect(docker.volume).toBeNull(); + expect(owned.volume).toBeNull(); + + const name = managedHermesStateVolumeName(context.sandboxName); + const foreign = dockerHarness({ name, labels: { "com.example.owner": "foreign" } }); + expect( + removeManagedAgentStateVolumes(context, { runDocker: foreign.runDocker as never }), + ).toMatchObject([{ status: "not-owned", volumeName: name }]); + expect(foreign.volume).not.toBeNull(); }); it.each([ - ["agent", { ...context, agentName: "openclaw" }], - ["provider", { ...context, runtimeProviderId: "kubernetes" }], - ["workload", { ...context, workloadKind: "legacy-dockerfile" }], - ])( - "does not provision outside the managed container-engine Hermes %s boundary", - (_boundary, input) => { + ["openclaw", null], + ["openclaw", "hermes"], + ["openclaw", "openclaw"], + ["hermes", "openclaw"], + ["hermes", "hermes"], + ] as const)( + "retires only %s source roots not owned by the recreated target %s", + (sourceAgent, targetAgent) => { const docker = dockerHarness(); + const roots = managedStartupStateRoots({ + agent: sourceAgent, + sandboxName: "alpha", + agentIdentity: managedImageRuntimeIdentity(sourceAgent), + }); + const scope = prepareManagedStateVolumes( + { roots }, + { + runContainerEngine: docker.runDocker as never, + registerExitCleanup: () => () => undefined, + }, + ); + expect(scope?.mounts).toEqual([ + { + type: "volume", + source: `nemoclaw-${sourceAgent}-state-v1-alpha`, + target: + sourceAgent === "openclaw" ? MANAGED_OPENCLAW_STATE_ROOT : MANAGED_HERMES_STATE_ROOT, + read_only: false, + }, + ]); + scope!.commit(); expect( - prepareManagedHermesStateVolume(input, { - runDocker: docker.runDocker as never, - runtimeProviders: { - docker: createDockerRuntimeProviderBundle(), - kubernetes: createKubernetesRuntimeProviderBundle(), + removeManagedAgentStateVolumes( + { + ...context, + agentName: sourceAgent, + retainedRoots: targetAgent + ? managedStartupStateRoots({ + agent: targetAgent, + sandboxName: "alpha", + agentIdentity: managedImageRuntimeIdentity(targetAgent), + }) + : [], }, - }), - ).toBeNull(); - expect(docker.calls).toEqual([]); + { runDocker: docker.runDocker as never }, + ), + ).toEqual(targetAgent === sourceAgent ? [] : [{ status: "removed" }]); + expect(docker.volume?.name ?? null).toBe( + targetAgent === sourceAgent ? `nemoclaw-${sourceAgent}-state-v1-alpha` : null, + ); + expect(docker.calls.filter((args) => args[0] === "rm")).toEqual( + targetAgent === sourceAgent ? [] : [["rm", `nemoclaw-${sourceAgent}-state-v1-alpha`]], + ); }, ); }); diff --git a/src/lib/onboard/managed-workload/hermes-state-volume.ts b/src/lib/onboard/managed-workload/hermes-state-volume.ts deleted file mode 100644 index 098cb2d5308..00000000000 --- a/src/lib/onboard/managed-workload/hermes-state-volume.ts +++ /dev/null @@ -1,119 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -import { managedImageRuntimeIdentity } from "../managed-image/agents"; -import { - managedHermesStateVolumeLabels, - managedHermesStateVolumeName, - managedStartupStateRoots, - MANAGED_HERMES_STATE_ROOT, -} from "../managed-startup/state-roots"; -import type { - RuntimeProviderBundle, - RuntimeProviderBundleRegistry, -} from "../runtime-provider/contract"; -import { - managedStateVolumeDeps, - selectedRuntimeProvider, - supportsContainerEngineOperation, - type ManagedAgentStateVolumeContext, - type ManagedAgentStateVolumeDeps, -} from "./agent-state-volumes"; -import { - prepareManagedStateVolumes, - removeManagedStateVolumes, - type ManagedStateVolumeCleanupResult, - type ManagedStateVolumeMount, -} from "./managed-state-volumes"; - -export { managedHermesStateVolumeLabels, managedHermesStateVolumeName, MANAGED_HERMES_STATE_ROOT }; - -export type ManagedHermesStateVolumeContext = ManagedAgentStateVolumeContext; - -export type ManagedHermesStateVolumeMount = ManagedStateVolumeMount & { - readonly target: typeof MANAGED_HERMES_STATE_ROOT; - readonly read_only: false; -}; - -export type ManagedHermesStateVolumeCleanupResult = ManagedStateVolumeCleanupResult; - -export type ManagedHermesStateVolumeDeps = ManagedAgentStateVolumeDeps; - -export type ManagedHermesStateVolumeScope = { - readonly mount: ManagedHermesStateVolumeMount; - readonly reused: boolean; - readonly volumeName: string; - cleanupIncompleteCreate(): ManagedHermesStateVolumeCleanupResult; - commit(): void; -}; - -function hermesStateRoots(sandboxName: string) { - return managedStartupStateRoots({ - agent: "hermes", - sandboxName, - agentIdentity: managedImageRuntimeIdentity("hermes"), - }); -} - -export function requiresManagedHermesStateVolume( - context: ManagedHermesStateVolumeContext, - providers?: RuntimeProviderBundleRegistry, - runtimeProvider?: RuntimeProviderBundle, -): boolean { - const hasLifecycleAuthority = - !providers && !runtimeProvider - ? true - : supportsContainerEngineOperation( - selectedRuntimeProvider(context.runtimeProviderId, providers, runtimeProvider), - "sandbox-lifecycle", - ); - return ( - context.agentName === "hermes" && - hasLifecycleAuthority && - context.workloadKind === "managed-image" - ); -} - -export function prepareManagedHermesStateVolume( - context: ManagedHermesStateVolumeContext, - deps: ManagedHermesStateVolumeDeps = {}, -): ManagedHermesStateVolumeScope | null { - if (!requiresManagedHermesStateVolume(context, deps.runtimeProviders, deps.runtimeProvider)) { - return null; - } - const root = hermesStateRoots(context.sandboxName)[0]; - if (!root) throw new Error("Hermes managed state-root declaration is unavailable."); - const scope = prepareManagedStateVolumes( - { roots: [root] }, - managedStateVolumeDeps(deps, context.runtimeProviderId), - ); - if (!scope || !scope.mounts[0]) { - throw new Error("Hermes managed state-volume scope is unavailable."); - } - return { - mount: scope.mounts[0] as ManagedHermesStateVolumeMount, - reused: scope.reused[0] === true, - volumeName: root.resourceIdentity, - cleanupIncompleteCreate: () => - scope.cleanupIncompleteCreate()[0] ?? { status: "not-applicable" }, - commit: () => scope.commit(), - }; -} - -export function removeManagedHermesStateVolume( - context: ManagedHermesStateVolumeContext, - deps: Pick< - ManagedHermesStateVolumeDeps, - "runDocker" | "runtimeProvider" | "runtimeProviders" - > = {}, -): ManagedHermesStateVolumeCleanupResult { - if (!requiresManagedHermesStateVolume(context, deps.runtimeProviders, deps.runtimeProvider)) { - return { status: "not-applicable" }; - } - return ( - removeManagedStateVolumes( - { roots: hermesStateRoots(context.sandboxName) }, - managedStateVolumeDeps(deps, context.runtimeProviderId), - )[0] ?? { status: "not-applicable" } - ); -} diff --git a/src/lib/onboard/managed-workload/managed-state-volumes-projected.test.ts b/src/lib/onboard/managed-workload/managed-state-volumes-projected.test.ts index 026d4ac3234..97bfdec762a 100644 --- a/src/lib/onboard/managed-workload/managed-state-volumes-projected.test.ts +++ b/src/lib/onboard/managed-workload/managed-state-volumes-projected.test.ts @@ -24,7 +24,7 @@ const roots = managedStartupStateRoots({ afterEach(() => vi.unstubAllEnvs()); -describe.each(["0.1.3", "0.1.5"])("OpenShell %s projection", (version) => { +describe.each(["0.1.3", "0.1.5", "0.1.7"])("OpenShell %s projection", (version) => { beforeEach(() => { runtime.version = version; }); @@ -75,7 +75,7 @@ describe.each(["0.1.3", "0.1.5"])("OpenShell %s projection", (version) => { it("binds lifecycle volume approval to the selected workspace", () => { vi.stubEnv("OPENSHELL_WORKSPACE", "ambient"); const docker = createHermesStateVolumeDockerHarness(); - createManagedStateVolumeOnboardLifecycle( + const lifecycle = createManagedStateVolumeOnboardLifecycle( { roots, workspace: "selected", @@ -92,6 +92,7 @@ describe.each(["0.1.3", "0.1.5"])("OpenShell %s projection", (version) => { }, { runContainerEngine: docker.runDocker }, ); + lifecycle.materializeSandboxCreatePlan({} as never, () => ({}) as never); expect(docker.volume?.labels["openshell.ai/sandbox-attachable-workspace"]).toBe("selected"); }); diff --git a/src/lib/onboard/managed-workload/managed-state-volumes.ts b/src/lib/onboard/managed-workload/managed-state-volumes.ts index 5b78992b51a..b4afe9a7884 100644 --- a/src/lib/onboard/managed-workload/managed-state-volumes.ts +++ b/src/lib/onboard/managed-workload/managed-state-volumes.ts @@ -221,6 +221,32 @@ function managedStateVolumeApproval(deps: ManagedStateVolumeDeps): Record { - const prepared = prepareProfile(agentName, { - endpointUrl: nativeBinding.endpointUrl, - }); +it.each([ + { agentName: "openclaw" as const, endpointUrl: nativeBinding.endpointUrl }, + { agentName: "hermes" as const, endpointUrl: nativeBinding.endpointUrl }, + { agentName: "openclaw" as const, endpointUrl: "http://host.openshell.internal:42103/v1" }, + { agentName: "hermes" as const, endpointUrl: "http://host.openshell.internal:42103/v1" }, +])( + "$agentName keeps unattached $endpointUrl as its legacy route (#12558)", + ({ agentName, endpointUrl }) => { + const prepared = prepareProfile(agentName, { endpointUrl }); expect(prepared?.profile.inference?.routedBaseUrl).toBe("https://inference.local/v1"); }, ); -it("rejects unattached Deep Agents Code native inference (#13197)", () => { - expect(() => - prepareProfile("langchain-deepagents-code", { endpointUrl: nativeBinding.endpointUrl }), - ).toThrow("requires its native provider attachment"); +it.each([nativeBinding.endpointUrl, "http://host.openshell.internal:42103/v1"])( + "rejects unattached Deep Agents Code native inference at %s (#13197)", + (endpointUrl) => { + expect(() => prepareProfile("langchain-deepagents-code", { endpointUrl })).toThrow( + "requires its native provider attachment", + ); + }, +); + +it.each(["openclaw", "hermes", "langchain-deepagents-code"] as const)( + "%s refuses to start NIM without its native attachment", + (agentName) => { + expect(() => + prepareProfile(agentName, { + provider: "vllm-local", + endpointUrl: nativeBinding.endpointUrl, + localNimSelected: true, + }), + ).toThrow("Local NIM inference requires its native provider attachment"); + }, +); + +it("preserves the published Pi NIM shared startup path on OpenShell 0.0.x", () => { + const prepared = prepareProfile("pi", { + provider: "vllm-local", + endpointUrl: nativeBinding.endpointUrl, + localNimSelected: true, + }); + expect(prepared?.profile.inference?.routedBaseUrl).toBe("https://inference.local/v1"); }); -it.each(["openclaw", "langchain-deepagents-code"] as const)( - "keeps owned Local NIM on the %s shared route without an attachment (#13116)", +it.each(["openclaw", "hermes", "pi", "langchain-deepagents-code"] as const)( + "uses the %s Local NIM native attachment", (agentName) => { + const nimBinding = { + ...nativeBinding, + provider: "vllm-local" as const, + authMode: "sentinel" as const, + ...(agentName === "langchain-deepagents-code" ? {} : { agent: agentName }), + }; + const nimAttachment = { + ...nimBinding, + ...nativeLocalIdentity(nimBinding), + schemaVersion: 1 as const, + providerId: "nim-provider", + }; const prepared = prepareProfile(agentName, { provider: "vllm-local", - endpointUrl: "http://127.0.0.1:8000/v1", + endpointUrl: nativeBinding.endpointUrl, localNimSelected: true, - nativeProvider: false, + attachment: nimAttachment, }); expect(prepared?.profile.inference).toMatchObject({ upstreamProvider: "vllm-local", - routedBaseUrl: "https://inference.local/v1", + routedBaseUrl: nativeBinding.endpointUrl, }); }, ); @@ -227,3 +263,47 @@ it.each([ routedBaseUrl: expected, }); }); + +it.each([ + { agent: "openclaw" as const, api: "openai-completions" as const }, + { agent: "openclaw" as const, api: "openai-responses" as const }, + { agent: "hermes" as const, api: "openai-completions" as const }, + { agent: "hermes" as const, api: "openai-responses" as const }, +])("rebuilds compatible $agent with its recorded native $api API", ({ agent, api }) => { + const binding = { + ...nativeBinding, + agent, + ...(api === "openai-responses" ? { api } : {}), + }; + const attachment = { ...nativeAttachment, ...binding, ...nativeLocalIdentity(binding) }; + const prepared = prepareProfile(agent, { + endpointUrl: binding.endpointUrl, + attachment, + api, + }); + expect(prepared?.profile.inference).toMatchObject({ + api, + upstreamProvider: "compatible-endpoint", + routedBaseUrl: binding.endpointUrl, + model: "fixture-model", + }); +}); + +it.each(["openclaw", "hermes"] as const)( + "rejects a rebuilt compatible %s API that differs from its native provider", + (agent) => { + const binding = { + ...nativeBinding, + agent, + api: "openai-responses" as const, + }; + const attachment = { ...nativeAttachment, ...binding, ...nativeLocalIdentity(binding) }; + expect(() => + prepareProfile(agent, { + endpointUrl: binding.endpointUrl, + attachment, + api: "openai-completions", + }), + ).toThrow(/API does not match/); + }, +); diff --git a/src/lib/onboard/managed-workload/onboard-orchestration.test.ts b/src/lib/onboard/managed-workload/onboard-orchestration.test.ts index 65efd06a883..1e84169af88 100644 --- a/src/lib/onboard/managed-workload/onboard-orchestration.test.ts +++ b/src/lib/onboard/managed-workload/onboard-orchestration.test.ts @@ -515,6 +515,38 @@ describe("managed workload onboard orchestration", () => { expect(ensurePreparedProfile).not.toHaveBeenCalled(); }); + it.each(["openclaw", "hermes"] as const)( + "refuses legacy %s state before returning the recreation lifecycle", + (agent) => { + const roots = managedStartupStateRoots({ + agent, + sandboxName: "alpha", + agentIdentity: { uid: 1000, gid: 1000 }, + }); + const root = roots[0]!; + const docker = createHermesStateVolumeDockerHarness({ + name: root.resourceIdentity, + labels: { ...root.ownershipLabels }, + }); + const before = JSON.stringify(docker.volume); + expect(() => + createManagedStateVolumeOnboardLifecycle( + { + roots, + workspace: "default", + runtimeProvider: { + workload: { managedStateMountDriverId: "docker" }, + containerEngine: { supported: true }, + } as never, + }, + { runContainerEngine: docker.runDocker as never }, + ), + ).toThrow("lacks OpenShell attachment approval"); + expect(JSON.stringify(docker.volume)).toBe(before); + expect(docker.calls.every((args) => args[0] === "inspect")).toBe(true); + }, + ); + it("preserves prepared named state when exit cleanup runs before registration", () => { const docker = createHermesStateVolumeDockerHarness(); let exitCleanup: (() => void) | null = null; @@ -857,6 +889,47 @@ describe("managed workload onboard orchestration", () => { }); }); + it("managed receipts retain shared ownership and catalog provenance", () => { + const reference = `ghcr.io/nvidia/nemoclaw-openclaw@sha256:${"a".repeat(64)}`; + const source = { release: "v0.0.100", revision: "b".repeat(40), cohort: "ghrun-123-1" }; + const { workloadReceipt } = resolveOnboardSandboxWorkloadReceipt({ + runtime: { + ensurePreparedProfile: () => ({ + encodedProfile: "fixture-profile", + startupProfileSha256: "c".repeat(64), + credentialProxyReplayRequired: false, + }), + }, + workload: { + source: { + kind: "managed-image", + reference, + contract: { + platform: "linux/amd64", + source, + capabilityContractVersion: 1, + startupProfileContractVersion: 1, + }, + }, + }, + registryImageRef: null, + prebuildImageRef: null, + firstCreateOutput: "", + createOutput: "", + buildId: "unused", + extractBuiltImageRef: vi.fn(), + resolveSandboxImageTagFromCreateOutput: vi.fn(), + } as never); + expect(workloadReceipt).toMatchObject({ + kind: "managed-image", + reference, + shared: true, + release: source.release, + sourceRevision: source.revision, + sourceCohort: source.cohort, + }); + }); + it("requires the registered external-image receipt to match the requested digest", () => { const first = `ghcr.io/example/openclaw@sha256:${"a".repeat(64)}`; const second = `ghcr.io/example/openclaw@sha256:${"b".repeat(64)}`; diff --git a/src/lib/onboard/managed-workload/onboard-orchestration.ts b/src/lib/onboard/managed-workload/onboard-orchestration.ts index a85af82983c..f4ae2e55958 100644 --- a/src/lib/onboard/managed-workload/onboard-orchestration.ts +++ b/src/lib/onboard/managed-workload/onboard-orchestration.ts @@ -84,6 +84,7 @@ import { resolveSandboxWorkloadRuntimeCapabilities } from "../workload/runtime"; import type { ExternalImageWorkloadSource } from "../workload/source"; import { prepareManagedStateVolumes, + preflightManagedStateVolumes, removeManagedStateVolumes, type ManagedStateVolumeDeps, } from "./managed-state-volumes"; @@ -152,23 +153,27 @@ export function createManagedStateVolumeOnboardLifecycle( }, deps: ManagedStateVolumeDeps = {}, ): ManagedStateVolumeOnboardLifecycle { - const scope = prepareManagedStateVolumes( - { roots: input.roots }, - { - ...deps, - workspace: input.workspace, - ...(input.runtimeProvider ? { runtimeProvider: input.runtimeProvider } : {}), - }, - ); - const managedStateMountDriverId = scope - ? input.runtimeProvider?.workload.managedStateMountDriverId - : undefined; - if (scope && !managedStateMountDriverId) { + const volumeDeps = { + ...deps, + workspace: input.workspace, + ...(input.runtimeProvider ? { runtimeProvider: input.runtimeProvider } : {}), + }; + const roots = input.roots; + const mountDriverId = volumeDeps.runtimeProvider?.workload.managedStateMountDriverId; + if ( + roots.length > 0 && + volumeDeps.runtimeProvider?.containerEngine.supported !== false && + !mountDriverId + ) throw new Error("Managed state volumes require provider-owned mount projection."); - } + preflightManagedStateVolumes({ roots: input.roots }, volumeDeps); + let scope: ReturnType | undefined; return { roots: input.roots, materializeSandboxCreatePlan(input, materialize) { + // Recheck attachment approval after source deletion and before selecting mounts. + scope ??= prepareManagedStateVolumes({ roots }, volumeDeps); + const managedStateMountDriverId = scope ? mountDriverId : undefined; return materialize({ ...input, managedStateMounts: scope?.mounts, @@ -204,6 +209,7 @@ export interface CreateManagedWorkloadOnboardRuntimeInput { readonly localNimSelected?: boolean; readonly nativeLocalProviderAttachment?: SandboxEntry["nativeLocalProviderAttachment"]; readonly nativeProvider?: boolean; + readonly registeredInferenceProviderAttachment?: SandboxEntry["registeredInferenceProviderAttachment"]; readonly startupProfile: ManagedProfileInput; readonly note: (message: string) => void; readonly fallbackBuildEstimate: () => string | null; @@ -432,6 +438,7 @@ export function createManagedWorkloadOnboardRuntime( : (input.nativeProvider ?? true), endpointUrl, input.agentName, + input.registeredInferenceProviderAttachment, ); preparedProfile = buildManagedStartupOnboardProfile({ agentName: input.agentName, @@ -608,6 +615,7 @@ export async function prepareOnboardSandboxWorkloadLaunch( log, reverifyBridgeReachability: () => verifySandboxBridgeGatewayReachableOrExit(true, { + gatewayRuntime, skip: false, port: input.gpu.gatewayPort, reachabilityImpl: (options) => diff --git a/src/lib/onboard/messaging-prep.test.ts b/src/lib/onboard/messaging-prep.test.ts index cd6e6901aee..8c6148e80ad 100644 --- a/src/lib/onboard/messaging-prep.test.ts +++ b/src/lib/onboard/messaging-prep.test.ts @@ -62,7 +62,7 @@ describe("prepareCreateSandboxMessaging", () => { const providerMatchesGatewayCredential = vi.fn( (name: string, type: string, credentialKey: string) => name === "demo-slack-bridge" && - type === "nemoclaw-mcp-v1" && + type === "nemoclaw-messaging-v1" && credentialKey === "SLACK_BOT_TOKEN", ); @@ -88,7 +88,7 @@ describe("prepareCreateSandboxMessaging", () => { expect(result.reusableMessagingChannels).toEqual(["slack"]); expect(providerMatchesGatewayCredential).toHaveBeenCalledWith( "demo-slack-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "SLACK_BOT_TOKEN", ); }); @@ -97,7 +97,7 @@ describe("prepareCreateSandboxMessaging", () => { const providerMatchesGatewayCredential = vi.fn( (name: string, type: string, credentialKey: string) => name === "demo-discord-bridge" && - type === "nemoclaw-mcp-v1" && + type === "nemoclaw-messaging-v1" && credentialKey === "DISCORD_BOT_TOKEN", ); @@ -115,13 +115,13 @@ describe("prepareCreateSandboxMessaging", () => { expect.arrayContaining([ expect.objectContaining({ name: "demo-discord-bridge", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }), ]), ); expect(providerMatchesGatewayCredential).toHaveBeenCalledWith( "demo-discord-bridge", - "nemoclaw-mcp-v1", + "nemoclaw-messaging-v1", "DISCORD_BOT_TOKEN", ); }); @@ -452,17 +452,17 @@ describe("prepareCreateSandboxMessaging", () => { { name: "demo-discord-bridge", envKey: "DISCORD_BOT_TOKEN", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, { name: "demo-slack-bridge", envKey: "SLACK_BOT_TOKEN", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, { name: "demo-slack-app", envKey: "SLACK_APP_TOKEN", - providerType: "nemoclaw-mcp-v1", + providerType: "nemoclaw-messaging-v1", }, ]); }); diff --git a/src/lib/onboard/model-router-reconcile.test.ts b/src/lib/onboard/model-router-reconcile.test.ts index 6d2419b29fc..f7216246367 100644 --- a/src/lib/onboard/model-router-reconcile.test.ts +++ b/src/lib/onboard/model-router-reconcile.test.ts @@ -4,6 +4,10 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import { reconcileModelRouter } from "./model-router"; +import { + probeHostServiceSandboxReachability, + formatHostServiceUnreachableMessage, +} from "./host-service-reachability"; import { saveCredential } from "../credentials/store"; const RECORDED_ROUTER_PID = 4321; @@ -73,7 +77,9 @@ vi.mock("./host-service-reachability", () => ({ holder.reachabilityProbes += 1; return { ok: true }; }), - formatHostServiceUnreachableMessage: () => "", + formatHostServiceUnreachableMessage: vi.fn( + () => "Inspect the selected bridge and its firewall rules.", + ), })); describe("model router reconciliation", () => { @@ -100,6 +106,48 @@ describe("model router reconciliation", () => { expect(holder.updatedRouterPorts).toEqual([]); }); + it("reuses a healthy native router when the historical Docker bridge is blocked", async () => { + holder.snapshotBody = JSON.stringify({ healthy_endpoints: [{}] }); + vi.mocked(probeHostServiceSandboxReachability).mockImplementationOnce(async (options) => { + // Native supervisor capability is supplied by the runtime provider. A + // bridge-only probe would report the stale UFW failure for this host. + return (options as { nativeSupervisorNetwork?: boolean }).nativeSupervisorNetwork + ? { ok: true, reason: "ok", networkName: "host", gatewayIp: "127.0.0.1" } + : { + ok: false, + reason: "tcp_failed", + networkName: "openshell-cluster-nemoclaw", + detail: "UFW blocks bridge", + }; + }); + await expect(reconcileModelRouter()).resolves.toBeUndefined(); + expect(probeHostServiceSandboxReachability).toHaveBeenCalledWith({ + port: 4000, + nativeSupervisorNetwork: true, + }); + expect(holder.stopped).toEqual([]); + }); + + it.each([ + ["host", " Check that Model Router listens on 127.0.0.1:4000 for the native supervisor.", 0], + ["openshell-cluster-nemoclaw", "Inspect the selected bridge and its firewall rules.", 1], + ] as const)( + "reports the correct repair for unreachable %s routing", + async (networkName, message, bridgeCalls) => { + holder.snapshotBody = JSON.stringify({ healthy_endpoints: [{}] }); + const error = vi.spyOn(console, "error").mockImplementation(() => {}); + vi.mocked(probeHostServiceSandboxReachability).mockResolvedValueOnce({ + ok: false, + reason: "tcp_failed", + networkName, + }); + await expect(reconcileModelRouter()).rejects.toThrow("cannot reach the Model Router"); + expect(error).toHaveBeenCalledWith(message); + expect(formatHostServiceUnreachableMessage).toHaveBeenCalledTimes(bridgeCalls); + expect(holder.stopped).toEqual([]); + }, + ); + it("backfills the cleanup port when reusing a legacy router session", async () => { holder.routerPort = null; holder.snapshotBody = JSON.stringify({ diff --git a/src/lib/onboard/model-router.ts b/src/lib/onboard/model-router.ts index b8261a159d7..0bb8f599c76 100644 --- a/src/lib/onboard/model-router.ts +++ b/src/lib/onboard/model-router.ts @@ -439,6 +439,7 @@ export async function startModelRouter( const startupDeadline = deps.now() + ROUTER_STARTUP_TIMEOUT_MS; let healthAttempts = 0; let lastSnapshot: RouterHealthSnapshot | null = null; + let lastCompletedSnapshot: RouterHealthSnapshot | null = null; while (healthAttempts < ROUTER_HEALTH_RETRIES && deps.now() < startupDeadline) { const delayMs = Math.min(ROUTER_HEALTH_INTERVAL_MS, startupDeadline - deps.now()); await deps.sleep(delayMs); @@ -462,6 +463,7 @@ export async function startModelRouter( } healthAttempts += 1; lastSnapshot = observation.snapshot; + if (lastSnapshot.outcome === "complete") lastCompletedSnapshot = lastSnapshot; const healthy = isRouterSnapshotReady(lastSnapshot); const processAlive = deps.isProcessAlive(pid); if (healthy && processAlive) return pid; @@ -476,7 +478,9 @@ export async function startModelRouter( } catch { // already dead } - const lastHealthError = firstUnhealthyEndpointError(lastSnapshot?.body ?? null); + const lastHealthError = firstUnhealthyEndpointError( + lastCompletedSnapshot?.body ?? lastSnapshot?.body ?? null, + ); const lastHealthObservation = formatHealthObservation(lastSnapshot); const logTail = routerLog === null ? "" : deps.readRouterLogTail(logPath, routerLog.startOffset); throw new Error( @@ -582,26 +586,18 @@ export function isRoutedInferenceProvider(provider: string | null | undefined): const MODEL_ROUTER_SERVICE_LABEL = "Model Router"; -/** - * Verify the host Model Router is reachable from the OpenShell Docker network. - * - * A healthy /health answer only proves the router responds on the host loopback. On - * Linux Docker-driver hosts with UFW default-deny, a sandbox container can - * still fail to reach `host.openshell.internal:` even though the - * host curl succeeds (#4564). This mirrors the Ollama auth-proxy probe: on a - * `tcp_failed` result print the concrete `ufw allow` remediation and fail so - * onboarding does not declare an unreachable router healthy. A - * `probe_unavailable` result (Docker Desktop, missing network during fresh - * setup before the sandbox network exists, DNS) is non-fatal. - */ +/** Probe the selected runtime path; native Linux Docker uses the supervisor network. */ async function verifyModelRouterSandboxReachability(routerPort: number): Promise { - const reachability = await probeHostServiceSandboxReachability({ port: routerPort }); + const options = { port: routerPort, nativeSupervisorNetwork: true }; + const reachability = await probeHostServiceSandboxReachability(options); if (!reachability.ok && reachability.reason === "tcp_failed") { console.error( - formatHostServiceUnreachableMessage(reachability, { - serviceLabel: MODEL_ROUTER_SERVICE_LABEL, - port: routerPort, - }), + reachability.networkName === "host" + ? ` Check that Model Router listens on 127.0.0.1:${routerPort} for the native supervisor.` + : formatHostServiceUnreachableMessage(reachability, { + serviceLabel: MODEL_ROUTER_SERVICE_LABEL, + port: routerPort, + }), ); throw new Error( `Sandbox containers cannot reach the Model Router at host.openshell.internal:${routerPort}.`, @@ -692,3 +688,18 @@ export async function reconcileModelRouter(): Promise { }); await verifyModelRouterSandboxReachability(routerPort); } + +/** Verify the retained router before reusing its sandbox provider. */ +export function assertNativeModelRouterOwner(endpoint: string, credentialValue: string): void { + const port = Number(new URL(endpoint).port); + const session = onboardSession.loadSession(); + if ( + !credentialValue || + !session || + session.routerPort !== port || + session.routerCredentialHash !== hashCredential(credentialValue) || + !doesModelRouterProcessOwnPort(session.routerPid ?? null, port) + ) { + throw new Error("Model Router process and credential ownership could not be verified."); + } +} diff --git a/src/lib/onboard/native-model-router.test.ts b/src/lib/onboard/native-model-router.test.ts new file mode 100644 index 00000000000..9a2c2dd20f1 --- /dev/null +++ b/src/lib/onboard/native-model-router.test.ts @@ -0,0 +1,242 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { nativeModelRouterEndpoint } from "../inference/native-local/model-router"; +import { normalizeNativeLocalProviderAttachment } from "../inference/native-local/contract"; +import { isProtectedLocalInferencePort } from "../inference/gateway/local-endpoint-ports"; +import { prepareNativeModelRouterSelection } from "./routed-inference"; +import { loadSession } from "../state/onboard-session"; +import { doesModelRouterProcessOwnPort } from "./model-router-process"; +import { ensureNativeLocalProvider } from "../inference/native-local/profile"; +import { hashCredential } from "../security/credential-hash"; +import type { BlueprintInferenceProfile } from "../core/model-router-port"; +import type { OpenShellProviderAdapter } from "../adapters/openshell/provider-adapter"; + +vi.mock("../state/onboard-session", () => ({ loadSession: vi.fn() })); +vi.mock("./model-router-process", () => ({ doesModelRouterProcessOwnPort: vi.fn() })); +vi.mock("../inference/native-local/profile", async (importOriginal) => ({ + ...(await importOriginal()), + ensureNativeLocalProvider: vi.fn(), +})); + +const profile: BlueprintInferenceProfile = { + provider_name: "custom-router", + endpoint: "http://localhost:4000/v1", + model: "routed", + router: { enabled: true, port: 4000 }, +}; +const input = { + adapter: {} as OpenShellProviderAdapter, + binding: { + provider: "nvidia-router", + agent: "openclaw" as const, + endpointUrl: "http://host.openshell.internal:4000/v1", + authMode: "authenticated" as const, + gatewayName: "nemoclaw", + sandboxName: "alice", + }, + credentialValue: "test-router-key", + readAuthority: vi.fn(), + writeAuthority: vi.fn(), +}; + +beforeEach(() => { + vi.mocked(loadSession).mockReturnValue({ + routerPort: 4000, + routerPid: 123, + routerCredentialHash: hashCredential(input.credentialValue), + } as ReturnType); + vi.mocked(doesModelRouterProcessOwnPort).mockReturnValue(true); + vi.mocked(ensureNativeLocalProvider).mockImplementation( + async (request) => ({ ...request.binding, providerId: "owned" }) as never, + ); +}); + +describe("native Model Router ownership", () => { + it("checks Router ownership while preserving the credential before sandbox backup", async () => { + await prepareNativeModelRouterSelection({ ...input, preserveProviderCredential: true }); + expect(doesModelRouterProcessOwnPort).toHaveBeenCalledTimes(2); + expect(ensureNativeLocalProvider).toHaveBeenCalledWith( + expect.objectContaining({ credentialValue: null }), + ); + }); + + it("rejects a changed Router credential even when provider mutation is deferred", async () => { + await expect( + prepareNativeModelRouterSelection({ + ...input, + credentialValue: "changed-key", + preserveProviderCredential: true, + }), + ).rejects.toThrow(); + expect(ensureNativeLocalProvider).not.toHaveBeenCalled(); + }); + + it("preserves the configured provider name and endpoint without admitting another host or service", () => { + expect(nativeModelRouterEndpoint("custom-router", "http://127.0.0.1:4000/v1", profile)).toBe( + "http://host.openshell.internal:4000/v1", + ); + }); + + it.each([ + ["other-router", "http://localhost:4000/v1"], + ["custom-router", "http://localhost:8080/v1"], + ["custom-router", "http://10.0.0.1:4000/v1"], + ["custom-router", "http://host.openshell.internal:4000/admin"], + ["custom-router", "http://user:pass@localhost:4000/v1"], + ["custom-router", "http://localhost:4000/v1?token=x"], + ])("rejects router %s at unowned endpoint %s", (provider, endpoint) => { + expect(nativeModelRouterEndpoint(provider, endpoint, profile)).toBeNull(); + }); + + it("requires the recorded live process and credential before registering the provider", async () => { + await prepareNativeModelRouterSelection(input); + expect(ensureNativeLocalProvider).toHaveBeenCalledWith( + expect.objectContaining({ + credentialValue: "test-router-key", + binding: expect.objectContaining({ + provider: "nvidia-router", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }), + }), + ); + expect(doesModelRouterProcessOwnPort).toHaveBeenCalledWith(123, 4000); + }); + + it("rejects a forged or stale router identity before touching provider state", async () => { + vi.mocked(doesModelRouterProcessOwnPort).mockReturnValue(false); + await expect(prepareNativeModelRouterSelection(input)).rejects.toThrow( + "ownership could not be verified", + ); + expect(ensureNativeLocalProvider).not.toHaveBeenCalled(); + vi.mocked(doesModelRouterProcessOwnPort).mockReturnValue(true); + vi.mocked(loadSession).mockReturnValue({ + routerPort: 4000, + routerPid: 123, + routerCredentialHash: "other", + } as never); + await expect(prepareNativeModelRouterSelection(input)).rejects.toThrow( + "ownership could not be verified", + ); + expect(ensureNativeLocalProvider).not.toHaveBeenCalled(); + }); + + it("retains provider recovery authority when the router changes during registration", async () => { + vi.mocked(doesModelRouterProcessOwnPort).mockReturnValueOnce(true).mockReturnValue(false); + await expect(prepareNativeModelRouterSelection(input)).rejects.toThrow( + "ownership could not be verified", + ); + expect(ensureNativeLocalProvider).toHaveBeenCalledOnce(); + }); + + it("keeps router ports reserved for ordinary endpoints and never exempts control ports", () => { + expect(isProtectedLocalInferencePort(4000)).toBe(true); + expect( + normalizeNativeLocalProviderAttachment({ + ...input.binding, + routeKind: "model-router", + endpointUrl: "http://host.openshell.internal:8080/v1", + }), + ).toBeUndefined(); + }); + it.each([8080, 8990, 18789, 8642, 11435, 11436, 11437, 11438])( + "rejects protected port %i for Model Router", + (port) => { + expect(isProtectedLocalInferencePort(port, { ownedModelRouterPort: port })).toBe(true); + }, + ); +}); + +describe("Model Router native setup", () => { + it.each(["openclaw", "hermes", "langchain-deepagents-code", "pi"])( + "publishes %s provider ownership inside both lifecycle locks", + async (agentName) => { + const { createSetupInference } = await import("./setup-inference"); + const { upsertRoutedProvider } = await import("./routed-inference"); + let gatewayLocked = false; + let routerLocked = false; + const updateSandbox = vi.fn(() => { + expect(gatewayLocked && routerLocked).toBe(true); + return true; + }); + const retiredCommand = vi.fn(() => { + throw new Error("retired shared route command"); + }); + const setup = createSetupInference({ + getGatewayName: () => "nemoclaw", + withSandboxMutationLock: async (_name: string, action: () => Promise) => + await action(), + withGatewayRouteMutationLock: async (_name: string, action: () => Promise) => { + gatewayLocked = true; + try { + return await action(); + } finally { + gatewayLocked = false; + } + }, + withModelRouterPortLifecycleLock: async (_port: number, action: () => Promise) => { + expect(gatewayLocked).toBe(true); + routerLocked = true; + try { + return await action(); + } finally { + routerLocked = false; + } + }, + getModelRouterPort: () => 4000, + isRoutedInferenceProvider: () => true, + requireNativeProviderPolicy: async () => {}, + checkGatewayRouteCompatibility: () => ({ ok: true }), + getSandbox: () => null, + updateSandbox, + providerAdapter: input.adapter, + getNativeLocalProviderAuthority: () => undefined, + setNativeLocalProviderAuthority: vi.fn(), + reconcileModelRouter: async () => { + expect(gatewayLocked && routerLocked).toBe(true); + }, + routedInference: { upsertRoutedProvider }, + hydrateCredentialEnv: () => input.credentialValue, + runOpenshell: retiredCommand, + inferenceRouteObserver: { observeInferenceRoute: retiredCommand }, + inferenceRouteMutator: { setInferenceRoute: retiredCommand }, + verifyInferenceRoute: retiredCommand, + verifyOnboardInferenceSmoke: async () => {}, + isNonInteractive: () => true, + hermesProviderAuth: { HERMES_PROVIDER_NAME: "hermes-provider" }, + localInference: { loadPendingOllamaModelCleanup: () => [] }, + withOllamaModelOwnershipLock: (action: () => unknown) => action(), + step: vi.fn(), + log: vi.fn(), + error: vi.fn(), + redact: (s: string) => s, + compactText: (s: string) => s, + exitProcess: (code: number) => { + throw new Error(`exit ${code}`); + }, + } as unknown as import("./setup-inference").SetupInferenceDeps); + await expect( + setup( + "alice", + "nvidia-routed", + "nvidia-router", + "http://localhost:4000/v1", + "NVIDIA_INFERENCE_API_KEY", + null, + [], + { agentName, revalidateSandboxIdentity: () => {} }, + ), + ).resolves.toMatchObject({ ok: true }); + expect(updateSandbox).toHaveBeenCalledWith( + "alice", + expect.objectContaining({ + provider: "nvidia-router", + nativeLocalProviderAttachment: expect.objectContaining({ providerId: "owned" }), + }), + ); + expect(JSON.stringify(updateSandbox.mock.calls)).not.toContain(input.credentialValue); + expect(retiredCommand).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/src/lib/onboard/openclaw-setup.test.ts b/src/lib/onboard/openclaw-setup.test.ts index bae90d08f1d..573d10a5b38 100644 --- a/src/lib/onboard/openclaw-setup.test.ts +++ b/src/lib/onboard/openclaw-setup.test.ts @@ -2,10 +2,18 @@ // SPDX-License-Identifier: Apache-2.0 import { beforeEach, describe, expect, it, vi } from "vitest"; +import { registeredInferenceFixture } from "../../../test/helpers/registered-inference"; import { isSandboxGatewayHttpReachableForStatus, waitForStartedNativeGatewayProcess, } from "../actions/sandbox/process-recovery"; +import { + verifyRegisteredInferenceProviderAttachment, + registeredInferenceProviderRevision, +} from "../inference/native-provider/registered"; + +import * as sandboxRecreateProbe from "./sandbox-recreate-probe"; +import type { SandboxEntry } from "../state/registry/types"; const configMocks = vi.hoisted(() => ({ readSandboxConfig: vi.fn(), @@ -27,6 +35,8 @@ import { isOpenclawGatewayReady, } from "./openclaw-setup"; import { + type InitializeOpenclawInferenceRoute, + refreshRestoredRegisteredOpenclawInferenceRoute, createInitialOpenclawInferenceRoute, createOpenclawInferenceRouteWriter, } from "./openclaw/initial-inference-route"; @@ -491,6 +501,8 @@ describe("initial OpenClaw inference route", () => { false, undefined, true, + undefined, + undefined, ); }); @@ -673,3 +685,182 @@ describe("initial Hermes provider selection", () => { }, ); }); + +describe("restored registered OpenClaw selection", () => { + it.each(["attached", "replaced", "detached", "wrong-gateway", "changed-revision"])( + "refreshes only a verified selection: %s", + async (state) => { + const fixture = registeredInferenceFixture(true); + const overrides: Record> = { + attached: {}, + replaced: { id: "replacement" }, + detached: { attached: false }, + "wrong-gateway": {}, + "changed-revision": {}, + }; + Object.assign(fixture.state, overrides[state]); + const binding = fixture.binding; + const selectedModel = { + id: "selected-model", + contextWindow: 131072, + maxTokens: 8192, + compat: { supportsStore: false }, + params: { custom: "retained" }, + }; + const config = { + agents: { defaults: { model: { primary: "old/old-model" } } }, + models: { + providers: { + [binding.providerName]: { + baseUrl: binding.endpointUrl, + api: binding.inferenceApi, + apiKey: "openshell:resolve:env:v1_VENDOR_API_KEY", + models: [selectedModel], + }, + }, + }, + channels: { retained: true }, + }; + const writeConfig = vi.fn(); + const readHandle = vi.fn(async () => { + fixture.state.revision += Number(state === "changed-revision"); + return fixture.handle; + }); + const readConfig = vi.fn(() => config); + const writer = createOpenclawInferenceRouteWriter({ + readOpenclawConfig: readConfig, + patchOpenclawInferenceConfig: patchOpenClawInferenceConfig, + writeOpenclawInferenceConfigNatively: writeConfig, + verifyRegisteredAttachment: async (expected) => { + await verifyRegisteredInferenceProviderAttachment({ adapter: fixture.adapter, expected }); + return registeredInferenceProviderRevision({ adapter: fixture.adapter, expected }); + }, + readRegisteredHandle: readHandle, + }); + const operation = writer( + binding.sandboxName, + "selected-model", + binding.providerName, + binding.inferenceApi, + state === "wrong-gateway" ? "other" : binding.gatewayName, + undefined, + { + nativeProvider: true, + endpointUrl: binding.endpointUrl, + registeredInferenceProviderAttachment: binding, + }, + ); + const outcome = await operation.then( + () => "written", + () => "refused", + ); + expect(outcome).toBe(state === "attached" ? "written" : "refused"); + expect(readHandle.mock.calls).toEqual( + ["attached", "changed-revision"].includes(state) ? [[binding]] : [], + ); + expect(readConfig.mock.calls).toEqual( + state === "attached" ? [[binding.sandboxName, binding.gatewayName]] : [], + ); + expect(writeConfig.mock.calls).toEqual( + state === "attached" + ? [ + [ + binding.sandboxName, + config, + expect.objectContaining({ + providerKey: binding.providerName, + inferenceBaseUrl: binding.endpointUrl, + inferenceApi: binding.inferenceApi, + }), + binding.gatewayName, + ], + ] + : [], + ); + expect(config.models.providers[binding.providerName]).toMatchObject({ + models: [selectedModel], + apiKey: state === "attached" ? fixture.handle : "openshell:resolve:env:v1_VENDOR_API_KEY", + }); + expect(config.channels).toEqual({ retained: true }); + }, + ); +}); + +it.each([ + "verified", + "changed-registration", + "changed-agent", + "changed-live-id", + "wrong-runtime", + "unrelated-update", +])("checks outer restored OpenClaw identity before projection: %s", async (state) => { + const f = registeredInferenceFixture(true); + const entry = { + name: f.binding.sandboxName, + agent: "openclaw", + provider: f.binding.providerName, + model: "selected-model", + endpointUrl: f.binding.endpointUrl, + preferredInferenceApi: f.binding.inferenceApi, + credentialEnv: f.binding.credentialEnv, + gatewayName: f.binding.gatewayName, + registeredInferenceProviderAttachment: f.binding, + lifecycleGeneration: "generation", + lifecycleLiveIdentityFingerprint: "fingerprint", + } as SandboxEntry; + const observe = vi.spyOn(sandboxRecreateProbe, "observeSandboxOnGateway").mockReturnValue({ + state: "not_ready", + liveIdentityFingerprint: state === "changed-live-id" ? "other" : "fingerprint", + }); + const read = vi.fn(() => + state === "changed-registration" + ? { ...entry, model: "other-model" } + : state === "changed-agent" + ? { ...entry, agent: "hermes" } + : state === "unrelated-update" + ? { ...entry, dashboardPort: 9001 } + : entry, + ); + const write = vi.fn( + async (_name, _model, _provider, _api, _gateway, assertCurrent) => { + assertCurrent?.("write restored config"); + }, + ); + const runtime = { + gatewayName: state === "wrong-runtime" ? "other" : f.binding.gatewayName, + } as never; + const result = await refreshRestoredRegisteredOpenclawInferenceRoute( + entry, + runtime, + read, + write, + ).then( + () => "written", + () => "refused", + ); + expect(result).toBe(["verified", "unrelated-update"].includes(state) ? "written" : "refused"); + expect(write.mock.calls).toEqual( + ["verified", "unrelated-update"].includes(state) + ? [ + [ + entry.name, + entry.model, + entry.provider, + entry.preferredInferenceApi, + entry.gatewayName, + expect.any(Function), + { + nativeProvider: true, + endpointUrl: f.binding.endpointUrl, + registeredInferenceProviderAttachment: f.binding, + }, + ], + ] + : [], + ); + expect( + observe.mock.calls.every( + ([target, , selected]) => target.gatewayName === entry.gatewayName && selected === runtime, + ), + ).toBe(true); +}); diff --git a/src/lib/onboard/openclaw/initial-inference-route.ts b/src/lib/onboard/openclaw/initial-inference-route.ts index 393bd7084a5..f07b8e34acc 100644 --- a/src/lib/onboard/openclaw/initial-inference-route.ts +++ b/src/lib/onboard/openclaw/initial-inference-route.ts @@ -1,10 +1,19 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import type { SandboxInferenceConfig } from "../../inference/config"; +import type { + SandboxInferenceConfig, + RegisteredInferenceProviderAttachment, +} from "../../inference/config"; import type { ReasoningEffortRequest } from "../../inference/selection"; import type { ConfigObject } from "../../security/credential-filter"; +import { isDeepStrictEqual } from "node:util"; +import type { SandboxEntry } from "../../state/registry/types"; +import type { OpenShellRuntimeSelection } from "../../adapters/openshell/runtime-selection"; +import { createRegisteredSandboxIdentityRevalidation } from "../sandbox-recreate-transaction"; +import { observeSandboxOnGateway } from "../sandbox-recreate-probe"; + const initialOpenclawInferenceRouteRuntime = { loadInferenceSet: () => require("../../actions/inference-set") as typeof import("../../actions/inference-set"), @@ -16,9 +25,12 @@ const initialOpenclawInferenceRouteRuntime = { export type InitialInferenceSelection = { endpointUrl?: string | null; nativeProvider: boolean; + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; }; export interface InitialOpenclawInferenceRouteDeps { + verifyRegisteredAttachment?: (expected: RegisteredInferenceProviderAttachment) => Promise; + readRegisteredHandle?: (expected: RegisteredInferenceProviderAttachment) => Promise; readInferenceSelection?( sandboxName: string, gatewayName: string, @@ -36,6 +48,8 @@ export interface InitialOpenclawInferenceRouteDeps { inheritPrimaryReplyBudget: false, nativeEndpointUrl?: string, nativeProvider?: boolean, + registered?: RegisteredInferenceProviderAttachment, + registeredHandle?: string, ): { route: SandboxInferenceConfig }; writeOpenclawInferenceConfigNatively( sandboxName: string, @@ -80,6 +94,27 @@ export function createOpenclawInferenceRouteWriter( ): Promise { revalidateSandboxIdentity?.(`read native OpenClaw config in sandbox '${sandboxName}'`); const selected = selection ?? deps.readInferenceSelection?.(sandboxName, gatewayName, provider); + const registered = selected?.registeredInferenceProviderAttachment; + let handle: string | undefined; + if (registered) { + if ( + registered.sandboxName !== sandboxName || + registered.gatewayName !== gatewayName || + registered.providerName !== provider || + registered.inferenceApi !== preferredInferenceApi + ) + throw new Error("Restored registered provider does not match its selected sandbox route."); + if (!deps.verifyRegisteredAttachment || !deps.readRegisteredHandle) + throw new Error( + "Registered provider restore requires live attachment and credential verification.", + ); + const revision = await deps.verifyRegisteredAttachment(registered); + handle = await deps.readRegisteredHandle(registered); + if ((await deps.verifyRegisteredAttachment(registered)) !== revision) + throw new Error( + "Registered provider changed while acquiring its runtime credential reference.", + ); + } const config = deps.readOpenclawConfig(sandboxName, gatewayName); const patched = deps.patchOpenclawInferenceConfig( config, @@ -92,6 +127,8 @@ export function createOpenclawInferenceRouteWriter( false, selected?.endpointUrl ?? undefined, selected?.nativeProvider ?? true, + registered, + handle, ); revalidateSandboxIdentity?.( @@ -134,6 +171,23 @@ export function createInitialOpenclawInferenceRoute( } const nativeInferenceRouteDeps: InitialOpenclawInferenceRouteDeps = { + verifyRegisteredAttachment: async (expected) => { + const { verifyRegisteredInferenceProviderAttachment, registeredInferenceProviderRevision } = + require("../../inference/native-provider/registered") as typeof import("../../inference/native-provider/registered"); + const { createManagedProviderAdapter } = + require("../../adapters/openshell/managed-provider-adapter") as typeof import("../../adapters/openshell/managed-provider-adapter"); + const adapter = createManagedProviderAdapter(); + await verifyRegisteredInferenceProviderAttachment({ adapter, expected }); + return registeredInferenceProviderRevision({ adapter, expected }); + }, + readRegisteredHandle: (expected) => { + const { readRegisteredInferenceCredentialHandle } = + require("../../actions/inference-set/registered") as typeof import("../../actions/inference-set/registered"); + return readRegisteredInferenceCredentialHandle( + expected, + (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)), + ); + }, readInferenceSelection: (sandboxName, gatewayName, provider) => { const { load } = require("../../state/registry/persistence") as typeof import("../../state/registry/persistence"); @@ -143,7 +197,9 @@ const nativeInferenceRouteDeps: InitialOpenclawInferenceRouteDeps = { } return { endpointUrl: entry.endpointUrl, + registeredInferenceProviderAttachment: entry.registeredInferenceProviderAttachment, nativeProvider: Boolean( + entry.registeredInferenceProviderAttachment || entry.nativeHostedProviderAttachment || entry.nativeNvidiaProviderAttachment || entry.nativeLocalProviderAttachment, @@ -182,3 +238,59 @@ export const initializeOpenclawInferenceRoute = /** The caller owns the offline restore window and its subsequent gateway restart. */ export const writeRestoredOpenclawInferenceRoute = createOpenclawInferenceRouteWriter(nativeInferenceRouteDeps); + +/** Reapply the recorded operator route after the outer rebuild restores its backup. */ +export async function refreshRestoredRegisteredOpenclawInferenceRoute( + entry: SandboxEntry, + runtimeSelection: OpenShellRuntimeSelection | undefined, + readRegistration: (name: string) => SandboxEntry | null, + write: InitializeOpenclawInferenceRoute = writeRestoredOpenclawInferenceRoute, +): Promise { + const binding = entry.registeredInferenceProviderAttachment; + if (!binding) return; + if ( + !entry.gatewayName || + !entry.provider || + !entry.model || + (entry.agent && entry.agent !== "openclaw") || + (runtimeSelection && runtimeSelection.gatewayName !== entry.gatewayName) + ) + throw new Error("Restored OpenClaw registered route does not match its selected runtime."); + const expected = structuredClone(entry); + const revalidate = createRegisteredSandboxIdentityRevalidation(expected, { + readRegistration, + observe: (sandboxName, gatewayName) => + observeSandboxOnGateway({ sandboxName, gatewayName }, undefined, runtimeSelection, 15_000), + }); + if (!revalidate) + throw new Error("Restored OpenClaw inference requires verified sandbox identity."); + const assertCurrent = (operation: string) => { + revalidate(operation); + const current = readRegistration(entry.name); + const fields = [ + "agent", + "provider", + "model", + "endpointUrl", + "preferredInferenceApi", + "credentialEnv", + "registeredInferenceProviderAttachment", + ] as const; + if (!current || fields.some((field) => !isDeepStrictEqual(current[field], expected[field]))) + throw new Error("Restored OpenClaw provider selection changed before configuration write."); + }; + assertCurrent("restore registered OpenClaw inference"); + await write( + entry.name, + entry.model, + entry.provider, + entry.preferredInferenceApi ?? null, + entry.gatewayName, + assertCurrent, + { + nativeProvider: true, + endpointUrl: binding.endpointUrl, + registeredInferenceProviderAttachment: binding, + }, + ); +} diff --git a/src/lib/onboard/openshell-docker-sandbox-containers.test.ts b/src/lib/onboard/openshell-docker-sandbox-containers.test.ts index dd0365dc562..ad8ba008a84 100644 --- a/src/lib/onboard/openshell-docker-sandbox-containers.test.ts +++ b/src/lib/onboard/openshell-docker-sandbox-containers.test.ts @@ -148,10 +148,14 @@ function querySnapshot(fields: unknown, nvidiaVisibleDevices?: string) { }; } -// OpenShell 0.0.116 sends supervisor startup failures only to the sandbox -// container's stderr (#13178). Remove once the OpenShell pin reaches 0.1.0. +// Preserve startup output when create fails before sandbox status is readable +// and rollback removes the container (#13178). describe("captureOpenShellDockerSandboxFailureOutput (#13178)", () => { const CONTAINER_ID = "f".repeat(64); + const identity = (id: string, role = "") => + [id, "openshell", role === "supervisor" ? "" : "default", "sandbox-id", "namespace", role].join( + "\t", + ); const STARTUP_FAILURE = "Error: × Failed to prepare sandbox: Landlock path unavailable in hard_requirement mode"; const dockerResult = (stdout: string, status = 0, stderr = "") => @@ -161,7 +165,7 @@ describe("captureOpenShellDockerSandboxFailureOutput (#13178)", () => { it("reads redacted output from the one OpenShell container labeled with the sandbox name", () => { const opaqueSecret = "opaque-value-9f8e7d6c5b4a"; const dockerRun = vi.fn((_args: readonly string[], _options?: { timeout?: number }) => - dockerResult(`${CONTAINER_ID}\n`), + dockerResult(`${identity(CONTAINER_ID)}\n`), ); const dockerCapture = vi.fn((args: readonly string[]) => args.includes("--format") ? "" : inspectWithEnv([`PROVIDER_TOKEN=${opaqueSecret}`]), @@ -172,13 +176,11 @@ describe("captureOpenShellDockerSandboxFailureOutput (#13178)", () => { dockerRun, dockerCapture, dockerLogs, + expectedSandboxId: "sandbox-id", }); expect(dockerRun.mock.calls[0][0]).toEqual( - expect.arrayContaining([ - "label=openshell.ai/managed-by=openshell", - "label=openshell.ai/sandbox-name=my-assistant", - ]), + expect.arrayContaining(["label=openshell.ai/sandbox-name=my-assistant"]), ); expect(dockerLogs).toHaveBeenCalledWith(CONTAINER_ID, expect.anything()); // A stalled lookup must not hold the failed sandbox before rollback. @@ -187,11 +189,30 @@ describe("captureOpenShellDockerSandboxFailureOutput (#13178)", () => { expect(output).not.toContain(opaqueSecret); }); + it.each([null, CONTAINER_ID])( + "refuses logs from a changed sandbox identity with runtime %s", + (expectedContainerId) => { + const dockerRun = vi.fn(() => dockerResult(`${identity(CONTAINER_ID)}\n`)); + const dockerCapture = vi.fn(); + const dockerLogs = vi.fn(); + expect( + captureOpenShellDockerSandboxFailureOutput("my-assistant", expectedContainerId, { + dockerRun, + dockerCapture, + dockerLogs, + expectedSandboxId: "different-sandbox-id", + }), + ).toBeNull(); + expect(dockerCapture).not.toHaveBeenCalled(); + expect(dockerLogs).not.toHaveBeenCalled(); + }, + ); + it("starts no Docker call after the shared capture deadline before rollback", () => { let clock = 1_000; const dockerRun = vi.fn((_args: readonly string[], options?: { timeout?: number }) => { clock += options?.timeout ?? 0; - return dockerResult(`${CONTAINER_ID}\n`); + return dockerResult(`${identity(CONTAINER_ID)}\n`); }); const dockerCapture = vi.fn((_args: readonly string[], options?: { timeout?: number }) => { clock += options?.timeout ?? 0; @@ -214,6 +235,145 @@ describe("captureOpenShellDockerSandboxFailureOutput (#13178)", () => { expect(clock - 1_000).toBeLessThanOrEqual(4_000); }); + it.each(["opaque-private-token", "opaque-" + "z".repeat(1600) + "-secret"])( + "learns both role inventories before truncating any log", + (secret) => { + const supervisor = "e".repeat(64); + const dockerLogs = vi.fn((id: string) => `${id}: ${secret}\n${STARTUP_FAILURE}`); + const output = captureOpenShellDockerSandboxFailureOutput("my-assistant", null, { + dockerRun: vi.fn(() => + dockerResult( + [identity(CONTAINER_ID, "sandbox"), identity(supervisor, "supervisor")].join("\n"), + ), + ), + dockerCapture: () => + JSON.stringify([ + { Id: CONTAINER_ID, Config: { Env: [] }, State: {} }, + { + Id: supervisor, + Config: { + Env: [ + `OPENSHELL_MAIN_PROCESS_SPEC=${JSON.stringify({ version: 1, command: ["env", `TOKEN=${secret}`, "nemoclaw-start"], tty: false, await_main_process_attachment: false })}`, + ], + }, + State: {}, + }, + ]), + dockerLogs, + }); + expect(dockerLogs.mock.calls.map(([id]) => id)).toEqual([CONTAINER_ID, supervisor]); + expect(output).toContain("== supervisor"); + expect(output).toContain("== sandbox"); + expect(output).toContain(STARTUP_FAILURE); + expect(output).not.toContain(secret); + expect(output).not.toContain("z".repeat(20)); + }, + ); + + it.each([ + ["incomplete inventory", []], + ["foreign ID", [{ Id: "b".repeat(64), Config: { Env: [] } }]], + ["missing environment", [{ Id: "e".repeat(64), Config: {} }]], + ["duplicate ID", [{ Id: CONTAINER_ID, Config: { Env: [] } }]], + ])("reads no role logs with %s", (_reason, remainder) => { + const dockerLogs = vi.fn(); + expect( + captureOpenShellDockerSandboxFailureOutput("my-assistant", null, { + dockerRun: () => + dockerResult( + [identity(CONTAINER_ID, "sandbox"), identity("e".repeat(64), "supervisor")].join("\n"), + ), + dockerCapture: () => + JSON.stringify([{ Id: CONTAINER_ID, Config: { Env: [] } }, ...remainder]), + dockerLogs, + }), + ).toBeNull(); + expect(dockerLogs).not.toHaveBeenCalled(); + }); + + it("retains bounded role state with empty logs without copying inspect secrets", () => { + const supervisor = "e".repeat(64); + const output = captureOpenShellDockerSandboxFailureOutput("my-assistant", null, { + dockerRun: () => + dockerResult( + [identity(CONTAINER_ID, "sandbox"), identity(supervisor, "supervisor")].join("\n"), + ), + dockerCapture: () => + JSON.stringify([ + { + Id: CONTAINER_ID, + Config: { Env: ["TOKEN=private-inspect-secret"] }, + State: { + Running: false, + ExitCode: 7, + OOMKilled: true, + Error: "private-inspect-secret", + }, + }, + { + Id: supervisor, + Config: { Env: [] }, + State: { Running: true, ExitCode: "private-inspect-secret", OOMKilled: false }, + }, + ]), + dockerLogs: () => "", + }); + expect(output).toBe( + `== sandbox ${CONTAINER_ID} ==\nrunning=false exit_code=7 oom_killed=true\n` + + `== supervisor ${supervisor} ==\nrunning=true oom_killed=false`, + ); + }); + + it("reads no role logs when the complete inspect inventory exceeds the shared deadline", () => { + let clock = 0; + const dockerLogs = vi.fn(); + const output = captureOpenShellDockerSandboxFailureOutput("my-assistant", null, { + now: () => clock, + dockerRun: vi.fn((_args, options) => { + clock += options?.timeout ?? 0; + return dockerResult( + [identity(CONTAINER_ID, "sandbox"), identity("e".repeat(64), "supervisor")].join("\n"), + ); + }), + dockerCapture: vi.fn((_args, options) => { + clock += options?.timeout ?? 0; + return JSON.stringify([ + { Id: CONTAINER_ID, Config: { Env: [] } }, + { Id: "e".repeat(64), Config: { Env: [] } }, + ]); + }), + dockerLogs, + }); + expect(output).toBeNull(); + expect(dockerLogs).not.toHaveBeenCalled(); + expect(clock).toBe(4000); + }); + + it.each([ + ["different sandbox", identity("e".repeat(64), "supervisor").replace("sandbox-id", "other")], + ["different namespace", identity("e".repeat(64), "supervisor").replace("namespace", "other")], + [ + "different workspace", + identity("e".repeat(64), "supervisor").replace("\t\tsandbox-id", "\tother\tsandbox-id"), + ], + ["foreign owner", identity("e".repeat(64), "supervisor").replace("openshell", "foreign")], + ["duplicate role", identity("e".repeat(64), "sandbox")], + ["duplicate identity", identity(CONTAINER_ID, "supervisor")], + ["missing namespace", identity("e".repeat(64), "supervisor").replace("namespace", "")], + ["malformed identity", "malformed"], + ])("refuses native diagnostics with %s", (_name, second) => { + const dockerLogs = vi.fn(); + expect( + captureOpenShellDockerSandboxFailureOutput("my-assistant", null, { + dockerRun: vi.fn(() => + dockerResult([identity(CONTAINER_ID, "sandbox"), second].join("\n")), + ), + dockerLogs, + }), + ).toBeNull(); + expect(dockerLogs).not.toHaveBeenCalled(); + }); + it("reads the exact failed replacement without a label lookup", () => { const dockerRun = vi.fn(); const dockerLogs = vi.fn(() => STARTUP_FAILURE); @@ -282,6 +442,51 @@ describe("captureOpenShellDockerSandboxFailureOutput (#13178)", () => { }); describe("queryOpenShellDockerSandboxRuntimeSnapshot", () => { + it.each([ + { + label: "one workload and its supervisor", + roles: ["sandbox", "supervisor"], + expected: { ok: true, containerId: "1".repeat(64) }, + calls: 2, + }, + { + label: "only a supervisor", + roles: ["supervisor"], + expected: { ok: false, error: "expected one labeled sandbox container, found 0" }, + calls: 1, + }, + { + label: "two workloads and a supervisor", + roles: ["sandbox", "sandbox", "supervisor"], + expected: { ok: false, error: "expected one labeled sandbox container, found 2" }, + calls: 1, + }, + ])("requires one workload when Docker contains $label", ({ roles, expected, calls }) => { + const containers = roles.map((role, index) => ({ + id: String(index + 1).repeat(64), + labels: [ + "openshell.ai/managed-by=openshell", + "openshell.ai/sandbox-name=alpha", + `openshell.ai/isolation-role=${role}`, + ], + })); + const dockerRun = vi + .fn() + .mockImplementationOnce((args: readonly string[]) => { + const labels = args.filter((arg) => arg.startsWith("label=")).map((arg) => arg.slice(6)); + const ids = containers + .filter((container) => labels.every((label) => container.labels.includes(label))) + .map((container) => container.id); + return { status: 0, stdout: ids.join("\n"), stderr: "" }; + }) + .mockReturnValueOnce({ status: 0, stdout: JSON.stringify(EMPTY_RUNTIME_FIELDS), stderr: "" }); + + expect(queryOpenShellDockerSandboxRuntimeSnapshot("alpha", { dockerRun })).toMatchObject( + expected, + ); + expect(dockerRun).toHaveBeenCalledTimes(calls); + }); + it("does not start container discovery after the shared deadline expires", () => { const dockerRun = vi.fn(); const now = vi.fn().mockReturnValueOnce(1_000).mockReturnValue(1_001); diff --git a/src/lib/onboard/openshell-docker-sandbox-containers.ts b/src/lib/onboard/openshell-docker-sandbox-containers.ts index 2e8822cbde5..ace2ab91839 100644 --- a/src/lib/onboard/openshell-docker-sandbox-containers.ts +++ b/src/lib/onboard/openshell-docker-sandbox-containers.ts @@ -7,7 +7,8 @@ import { type DockerSandboxIdentityObservation, } from "../adapters/docker/inspect"; import { captureDockerContainerFailureEvidence } from "./compute/docker-container-failure-evidence"; -import type { DockerGpuPatchDeps } from "./docker-gpu-patch-types"; +import { createDockerGpuDiagnosticRedactor } from "./docker-gpu-diagnostic-redaction"; +import type { DockerContainerInspect, DockerGpuPatchDeps } from "./docker-gpu-patch-types"; export const OPENSHELL_MANAGED_BY_LABEL = "openshell.ai/managed-by"; export const OPENSHELL_MANAGED_BY_VALUE = "openshell"; @@ -15,6 +16,7 @@ export const OPENSHELL_SANDBOX_NAME_LABEL = "openshell.ai/sandbox-name"; export const OPENSHELL_SANDBOX_ID_LABEL = "openshell.ai/sandbox-id"; export const OPENSHELL_SANDBOX_NAMESPACE_LABEL = "openshell.ai/sandbox-namespace"; export const OPENSHELL_SANDBOX_WORKSPACE_LABEL = "openshell.ai/sandbox-workspace"; +export const OPENSHELL_ISOLATION_ROLE_LABEL = "openshell.ai/isolation-role"; export type OpenShellSandboxOwnershipLabel = { readonly label: string; @@ -88,24 +90,24 @@ const SANDBOX_FAILURE_OUTPUT_BUDGET_MS = 4_000; type SandboxFailureOutputDeps = Pick< DockerGpuPatchDeps, "dockerCapture" | "dockerLogs" | "dockerRun" -> & { readonly now?: () => number }; +> & { readonly now?: () => number; readonly expectedSandboxId?: string }; /** - * Capture redacted recent output from one sandbox container before rollback + * Capture redacted recent output from the sandbox runtime before rollback * can remove or replace it. Use the exact replacement container when the - * caller owns one; otherwise require exactly one OpenShell-labeled container. + * caller owns one; otherwise require one legacy container or one exact native role pair. * * SOURCE_OF_TRUTH_REVIEW - * invalidState: OpenShell 0.0.116 writes supervisor startup failures, such as - * a Landlock hard_requirement error, only to the sandbox container's stderr; - * neither `openshell logs` nor sandbox status carries the reason (#13178). + * invalidState: sandbox create fails before its status provides the startup + * reason, and rollback removes the container's output (#13178). * sourceBoundary: captureDockerContainerFailureEvidence owns bounded Docker * inspect, log, and redaction behavior. - * whyNotSourceFix: the supervisor runs in the pinned external OpenShell - * release; OpenShell 0.1.x reports the supervisor log tail in sandbox status. + * whyNotSourceFix: OpenShell status can report a supervisor log tail, but a + * failed create may have no readable sandbox status before rollback. * regressionTest: openshell-docker-sandbox-containers.test.ts and * onboard-sandbox-create-failure.test.ts. - * removalCondition: remove once the OpenShell pin reaches 0.1.0. + * removalCondition: remove when sandbox status reliably includes startup + * failures before rollback on every supported Docker create path. */ export function captureOpenShellDockerSandboxFailureOutput( sandboxName: string, @@ -115,28 +117,104 @@ export function captureOpenShellDockerSandboxFailureOutput( try { const now = deps.now ?? Date.now; const deadlineMs = now() + SANDBOX_FAILURE_OUTPUT_BUDGET_MS; - let containerId = expectedContainerId; - if (!containerId) { - const containers = queryOpenShellDockerSandboxContainers( + let containers = [{ id: expectedContainerId ?? "", role: "" }]; + if (!expectedContainerId || deps.expectedSandboxId) { + const observation = inspectDockerSandboxNameLabeledContainers( sandboxName, - { dockerRun: deps.dockerRun ?? dockerRun }, - Math.min(2_000, deadlineMs - now()), + (args, options) => { + const remaining = Math.min(2_000, deadlineMs - now()); + if (remaining <= 0) return { status: 1 }; + const result = (deps.dockerRun ?? dockerRun)(args, { ...options, timeout: remaining }); + return { + status: result.status, + stdout: result.stdout ?? undefined, + stderr: result.stderr ?? undefined, + error: result.error ?? undefined, + }; + }, ); - if (!containers.ok || containers.ids.length !== 1) return null; - containerId = containers.ids[0]; + if (observation.status !== "observed" || observation.malformedRows !== 0) return null; + const rows = observation.rows; + const first = rows[0]; + if ( + !first || + rows.some( + (row) => + !/^[0-9a-f]{64}$/u.test(row.id) || + row.managedBy !== OPENSHELL_MANAGED_BY_VALUE || + !row.sandboxId || + row.sandboxId !== first.sandboxId || + (deps.expectedSandboxId !== undefined && row.sandboxId !== deps.expectedSandboxId) || + row.namespace !== first.namespace, + ) + ) + return null; + const legacy = + rows.length === 1 && first.isolationRole === undefined && Boolean(first.workspace); + const native = + rows.length === 2 && + Boolean(first.namespace) && + new Set(rows.map((row) => row.id)).size === 2 && + rows.some((row) => row.isolationRole === "sandbox" && Boolean(row.workspace)) && + rows.some((row) => row.isolationRole === "supervisor" && row.workspace === ""); + if (!legacy && !native) return null; + if (expectedContainerId && !rows.some((row) => row.id === expectedContainerId)) return null; + containers = rows.map((row) => ({ id: row.id, role: row.isolationRole ?? "" })); } - const evidence = captureDockerContainerFailureEvidence( - containerId, - { - dockerCapture: deps.dockerCapture ?? dockerCapture, - dockerLogs: deps.dockerLogs ?? dockerLogs, - }, - { deadlineMs, now }, - ); - // Opaque credentials are redacted only after inspect identifies them; - // without that, pattern redaction alone could print an opaque value. - if (!evidence.inspectRedactionApplied) return null; - return evidence.redactedLogTail || null; + const outputs: string[] = []; + const redactor = createDockerGpuDiagnosticRedactor(); + const inspections = new Map(); + if (containers.length > 1) { + const remaining = Math.min(2_000, deadlineMs - now()); + if (remaining <= 0) return null; + const inventory = JSON.parse( + (deps.dockerCapture ?? dockerCapture)(["inspect", ...containers.map(({ id }) => id)], { + ignoreError: true, + timeout: remaining, + }), + ) as DockerContainerInspect[]; + if (!Array.isArray(inventory) || inventory.length !== containers.length) return null; + for (const inspect of inventory) { + if ( + !inspect.Id || + !containers.some(({ id }) => id === inspect.Id) || + inspections.has(inspect.Id) || + !Array.isArray(inspect.Config?.Env) + ) + return null; + inspections.set(inspect.Id, inspect); + redactor.rememberInspect(inspect); + } + } + for (const container of containers) { + const evidence = captureDockerContainerFailureEvidence( + container.id, + { + dockerCapture: deps.dockerCapture ?? dockerCapture, + dockerLogs: deps.dockerLogs ?? dockerLogs, + }, + { deadlineMs, now, redactor, inspect: inspections.get(container.id) }, + ); + // Inspect must identify opaque environment values before any output is retained. + if (!evidence.inspectRedactionApplied) return null; + const state = evidence.state; + const stateFields = Object.entries({ + running: typeof state?.Running === "boolean" ? state.Running : undefined, + exit_code: Number.isSafeInteger(state?.ExitCode) ? state?.ExitCode : undefined, + oom_killed: typeof state?.OOMKilled === "boolean" ? state.OOMKilled : undefined, + }).filter(([, value]) => value !== undefined); + const stateText = stateFields.map(([key, value]) => `${key}=${value}`).join(" "); + const output = [stateText, evidence.redactedLogTail] + .filter(Boolean) + .join("\n") + .slice(0, 1200); + if (output) { + outputs.push( + container.role ? `== ${container.role} ${container.id} ==\n${output}` : output, + ); + } + } + return outputs.join("\n") || null; } catch { return null; } @@ -295,6 +373,8 @@ export function inspectDockerSandboxNameLabeledContainers( managedBy: OPENSHELL_MANAGED_BY_LABEL, workspace: OPENSHELL_SANDBOX_WORKSPACE_LABEL, sandboxId: OPENSHELL_SANDBOX_ID_LABEL, + namespace: OPENSHELL_SANDBOX_NAMESPACE_LABEL, + isolationRole: OPENSHELL_ISOLATION_ROLE_LABEL, }, inspect, ); @@ -641,7 +721,17 @@ export function queryOpenShellDockerSandboxRuntimeSnapshot( ); const deadlineMs = now() + totalTimeoutMs; const remainingTimeoutMs = () => Math.floor(deadlineMs - now()); - const containers = queryOpenShellDockerSandboxContainers(sandboxName, deps, remainingTimeoutMs()); + // OpenShell labels both the workload and its supervisor with the sandbox name. + // Snapshot identity and GPU state belong to the workload container. + const containers = queryDockerSandboxContainerIds( + [ + ...sandboxContainerFilterArgs(sandboxName), + "--filter", + `label=${OPENSHELL_ISOLATION_ROLE_LABEL}=sandbox`, + ], + deps, + remainingTimeoutMs(), + ); if (!containers.ok) return { ok: false, error: containers.error }; const inspectTimeoutMs = remainingTimeoutMs(); if (inspectTimeoutMs <= 0) { diff --git a/src/lib/onboard/openshell-feature-gate.test.ts b/src/lib/onboard/openshell-feature-gate.test.ts index 1d7be53a29c..7c9a8dab662 100644 --- a/src/lib/onboard/openshell-feature-gate.test.ts +++ b/src/lib/onboard/openshell-feature-gate.test.ts @@ -1,10 +1,11 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import crypto from "node:crypto"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { hasRequiredOpenshellMessagingFeatures, @@ -27,6 +28,144 @@ exit 0 } describe("OpenShell MCP feature gate", () => { + it.each([ + { + digest: "b4ae1d60b238c84e290bcd08a962997e5c36ed04237fa59b09b5799b11363feb", + version: "0.1.7", + cliVersion: "0.1.7", + markers: true, + accepted: true, + }, + { + digest: "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21", + version: "0.1.7", + cliVersion: "0.1.7", + markers: true, + accepted: true, + }, + { + digest: "a048b1fd2acd78f44f50a4cf0e02cdefec691656f3941e92c5ec0e09a69990e2", + version: "0.1.7", + cliVersion: "0.1.7", + markers: true, + accepted: false, + }, + { + digest: "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21", + version: "0.1.5", + cliVersion: "0.1.5", + markers: true, + accepted: false, + }, + { + digest: "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21", + version: "0.1.7", + cliVersion: "0.1.7", + markers: false, + accepted: false, + }, + { + digest: "a048b1fd2acd78f44f50a4cf0e02cdefec691656f3941e92c5ec0e09a69990e2", + version: "0.1.5", + cliVersion: "0.1.5", + markers: true, + accepted: true, + }, + { + digest: "65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc", + version: "0.1.5", + cliVersion: "0.1.5", + markers: true, + accepted: true, + }, + { + digest: "5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1", + version: "0.1.5", + cliVersion: "0.1.5", + markers: true, + accepted: false, + }, + { + digest: "65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc", + version: "0.1.2", + markers: true, + accepted: false, + }, + { + digest: "65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc", + version: "0.1.5", + cliVersion: "0.1.5", + markers: false, + accepted: false, + }, + { + digest: "0".repeat(64), + version: "0.1.5", + cliVersion: "0.1.5", + markers: true, + accepted: false, + }, + { + digest: "5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1", + version: "0.1.2", + markers: true, + accepted: true, + }, + { + digest: "9b527c257e7917d11cee34075369cdfb69a57764198da6e72cc0847cb9b427aa", + version: "0.1.2", + markers: true, + accepted: true, + }, + { digest: "0".repeat(64), version: "0.1.2", markers: true, accepted: false }, + { + digest: "326ee26df8f8575ba761470757a12fe5c1cdc904ba064b81946692dd0328dd40", + version: "0.1.2", + markers: true, + accepted: false, + }, + { + digest: "5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1", + version: "0.0.116", + markers: true, + accepted: false, + }, + { + digest: "5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1", + version: "0.1.2", + markers: false, + accepted: false, + }, + ])( + "checks pinned split sandbox identity, component coherence and CLI capabilities [%j]", + ({ digest, version, cliVersion = "0.1.2", markers, accepted }) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-split-")); + const openshell = path.join(root, "openshell"); + const sandbox = path.join(root, "openshell-sandbox"); + const digestSpy = vi.spyOn(crypto, "createHash").mockReturnValue({ + update: () => ({ digest: () => digest }), + } as unknown as ReturnType); + try { + writeExecutable( + openshell, + markers ? REQUIRED_OPENSHELL_MCP_FEATURES.join(" ") : "", + cliVersion, + ); + writeExecutable(sandbox, "split sandbox without policy markers", version); + expect( + hasRequiredOpenshellMessagingFeatures({ + openshellBin: openshell, + gatewayBin: null, + sandboxBin: sandbox, + }), + ).toBe(accepted); + } finally { + digestSpy.mockRestore(); + fs.rmSync(root, { recursive: true, force: true }); + } + }, + ); + it("identifies the pinned v0.0.72 sandbox artifacts without executing them", () => { const sandbox = path.join( fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-features-")), diff --git a/src/lib/onboard/openshell-feature-gate.ts b/src/lib/onboard/openshell-feature-gate.ts index c07becedd91..46e45f077ad 100644 --- a/src/lib/onboard/openshell-feature-gate.ts +++ b/src/lib/onboard/openshell-feature-gate.ts @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { spawnSync } from "node:child_process"; -import { createHash } from "node:crypto"; +import crypto from "node:crypto"; import fs from "node:fs"; import path from "node:path"; @@ -68,6 +68,18 @@ const PINNED_SANDBOX_BUILD_VERSIONS = new Map([ // OpenShell v0.0.116 standalone sandbox binaries. ["326ee26df8f8575ba761470757a12fe5c1cdc904ba064b81946692dd0328dd40", "0.0.116"], ["7052a87d2b46ef52ecc0f7c64b9bac008dd3010c467881b0648045334eb0ed1d", "0.0.116"], + // OpenShell v0.1.2 standalone sandbox binaries. + ["5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1", "0.1.2"], + ["9b527c257e7917d11cee34075369cdfb69a57764198da6e72cc0847cb9b427aa", "0.1.2"], + // OpenShell v0.1.3 standalone sandbox binaries. + ["38fe063f19c72680a1bc4b0c343a3ec5b7a5cb087a627faab5b2aefd9d50d094", "0.1.3"], + ["23147b42d31bd0b13e1a9ed7b3834b8e6075836cba0872fffa9e7d481f6dd281", "0.1.3"], + // OpenShell v0.1.5 standalone sandbox binaries. + ["a048b1fd2acd78f44f50a4cf0e02cdefec691656f3941e92c5ec0e09a69990e2", "0.1.5"], + ["65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc", "0.1.5"], + // OpenShell v0.1.7 standalone sandbox binaries. + ["b4ae1d60b238c84e290bcd08a962997e5c36ed04237fa59b09b5799b11363feb", "0.1.7"], + ["d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21", "0.1.7"], ]); export function pinnedOpenShellSandboxBuildVersion(sha256: string): string | null { @@ -76,7 +88,7 @@ export function pinnedOpenShellSandboxBuildVersion(sha256: string): string | nul function executableSha256(candidate: string): string | null { try { - return createHash("sha256").update(fs.readFileSync(candidate)).digest("hex"); + return crypto.createHash("sha256").update(fs.readFileSync(candidate)).digest("hex"); } catch { return null; } @@ -207,13 +219,25 @@ export function hasRequiredOpenshellMessagingFeatures(options: { } if (!REQUIRED_OPENSHELL_MCP_FEATURES.every((marker) => foundMarkers.has(marker))) return false; - // MCP policy enforcement and credential replacement execute in the sandbox - // supervisor. When that exact host artifact is available, require its native - // MCP marker rather than accepting a union of unrelated binaries. + // OpenShell 0.1.2, 0.1.3, 0.1.5, and 0.1.7 enforce policy in the supervisor image. Only + // their exact pinned sandbox artifacts use that split here; version text alone + // cannot exempt another sandbox from the legacy marker check. Runtime policy + // verification remains mandatory before credential or provider changes. const sandboxMarker = Buffer.from(REQUIRED_OPENSHELL_SANDBOX_MCP_FEATURE); if (sandboxBin) { try { - return fs.readFileSync(sandboxBin).includes(sandboxMarker); + const content = fs.readFileSync(sandboxBin); + const pinnedVersion = pinnedOpenShellSandboxBuildVersion( + crypto.createHash("sha256").update(content).digest("hex"), + ); + return ( + content.includes(sandboxMarker) || + ((pinnedVersion === "0.1.2" || + pinnedVersion === "0.1.3" || + pinnedVersion === "0.1.5" || + pinnedVersion === "0.1.7") && + pinnedVersion === openshellVersion) + ); } catch { return false; } diff --git a/src/lib/onboard/openshell-install.test.ts b/src/lib/onboard/openshell-install.test.ts index 037bf2e84cb..21f5d3b6468 100644 --- a/src/lib/onboard/openshell-install.test.ts +++ b/src/lib/onboard/openshell-install.test.ts @@ -137,7 +137,7 @@ describe("ensureOpenshellForOnboard", () => { ); }); - it("applies the 0.0.116 floor during final validation when the blueprint omits a minimum", () => { + it("applies the 0.1.7 floor during final validation when the blueprint omits a minimum", () => { const deps = makeDeps({ isOpenshellInstalled: () => false, getInstalledOpenshellVersion: () => "0.0.81", @@ -151,22 +151,22 @@ describe("ensureOpenshellForOnboard", () => { expect(deps.error).toHaveBeenCalledWith( " \u2717 openshell 0.0.81 is below the minimum required by this NemoClaw release.", ); - expect(deps.error).toHaveBeenCalledWith(" blueprint.yaml min_openshell_version: 0.0.116"); + expect(deps.error).toHaveBeenCalledWith(" blueprint.yaml min_openshell_version: 0.1.7"); }); - it("applies the exact 0.0.116 ceiling when the blueprint omits a maximum", () => { + it("applies the exact 0.1.7 ceiling when the blueprint omits a maximum", () => { const deps = makeDeps({ - getInstalledOpenshellVersion: () => "0.0.117", + getInstalledOpenshellVersion: () => "0.1.8", getBlueprintMinOpenshellVersion: () => null, getBlueprintMaxOpenshellVersion: () => null, - runCaptureOpenshell: () => "openshell 0.0.117", + runCaptureOpenshell: () => "openshell 0.1.8", }); expect(() => ensureOpenshellForOnboard(deps)).toThrow("exit 1"); expect(deps.error).toHaveBeenCalledWith( - " \u2717 openshell 0.0.117 is above the maximum supported by this NemoClaw release.", + " \u2717 openshell 0.1.8 is above the maximum supported by this NemoClaw release.", ); - expect(deps.error).toHaveBeenCalledWith(" blueprint.yaml max_openshell_version: 0.0.116"); + expect(deps.error).toHaveBeenCalledWith(" blueprint.yaml max_openshell_version: 0.1.7"); }); it("fails closed when the installed version remains unknown after installation", () => { @@ -181,8 +181,6 @@ describe("ensureOpenshellForOnboard", () => { expect(deps.error).toHaveBeenCalledWith( " \u2717 OpenShell version could not be determined after installation.", ); - expect(deps.error).toHaveBeenCalledWith( - " Install exact stable OpenShell 0.0.116 and retry.", - ); + expect(deps.error).toHaveBeenCalledWith(" Install exact stable OpenShell 0.1.7 and retry."); }); }); diff --git a/src/lib/onboard/openshell-install.ts b/src/lib/onboard/openshell-install.ts index 67079bd4701..b84104aebaf 100644 --- a/src/lib/onboard/openshell-install.ts +++ b/src/lib/onboard/openshell-install.ts @@ -194,12 +194,12 @@ export function ensureOpenshellForOnboard( localBin: null, futureShellPathHint: null, }; - const minOpenshellVersion = deps.getBlueprintMinOpenshellVersion() ?? "0.0.116"; + const minOpenshellVersion = deps.getBlueprintMinOpenshellVersion() ?? "0.1.7"; if (deps.shouldUseOpenshellDevChannel()) { deps.error(""); deps.error( - " ✗ NemoClaw requires exact stable OpenShell 0.0.116; the dev channel is not supported.", + " ✗ NemoClaw requires exact stable OpenShell 0.1.7; the dev channel is not supported.", ); deps.error(""); deps.exit(1); @@ -240,7 +240,7 @@ export function ensureOpenshellForOnboard( if (needsUpgrade) { if (needsStableRelease) { deps.log( - " OpenShell development builds are unsupported. Reinstalling exact stable 0.0.116...", + " OpenShell development builds are unsupported. Reinstalling exact stable 0.1.7...", ); } else if (needsDockerDriverBinaries) { const required = platform === "linux" ? "gateway and sandbox" : "gateway"; @@ -272,14 +272,14 @@ export function ensureOpenshellForOnboard( if (!installedOpenshellVersion) { deps.error(""); deps.error(" \u2717 OpenShell version could not be determined after installation."); - deps.error(" Install exact stable OpenShell 0.0.116 and retry."); + deps.error(" Install exact stable OpenShell 0.1.7 and retry."); deps.error(""); deps.exit(1); } if (deps.isOpenshellDevVersion(openshellVersionOutput)) { deps.error(""); deps.error(" ✗ OpenShell development builds are not supported by this NemoClaw release."); - deps.error(" Install exact stable OpenShell 0.0.116 and retry."); + deps.error(" Install exact stable OpenShell 0.1.7 and retry."); deps.error(""); deps.exit(1); } @@ -314,7 +314,7 @@ export function ensureOpenshellForOnboard( deps.exit(1); } - const maxOpenshellVersion = deps.getBlueprintMaxOpenshellVersion() ?? "0.0.116"; + const maxOpenshellVersion = deps.getBlueprintMaxOpenshellVersion() ?? "0.1.7"; if ( installedOpenshellVersion && maxOpenshellVersion && diff --git a/src/lib/onboard/openshell-pin.ts b/src/lib/onboard/openshell-pin.ts index 4d3112254ec..6cbac37aaae 100644 --- a/src/lib/onboard/openshell-pin.ts +++ b/src/lib/onboard/openshell-pin.ts @@ -174,9 +174,7 @@ export function computeOpenshellInstallEnv( if (channel === "dev") { const error = deps.error ?? ((m: string) => console.error(m)); error(""); - error( - " ✗ NemoClaw requires exact stable OpenShell 0.0.116; the dev channel is not supported.", - ); + error(" ✗ NemoClaw requires exact stable OpenShell 0.1.7; the dev channel is not supported."); error(""); return { env: null }; } @@ -231,7 +229,7 @@ export function prependInstalledUserLocalOpenshellPath( } /** - * Execute `scripts/install-openshell.sh`, wiring in the blueprint-driven pin + * Execute the guarded product installer, wiring in the blueprint-driven pin * resolution and the host-side state updates onboard.ts cares about (binary * path, PATH augmentation, `OPENSHELL_BIN` cache). Lives in this submodule so * the top-level onboard entrypoint stays net-neutral (#3404 follow-up). diff --git a/src/lib/onboard/openshell-version.ts b/src/lib/onboard/openshell-version.ts index 49319ef8a44..595108aec6a 100644 --- a/src/lib/onboard/openshell-version.ts +++ b/src/lib/onboard/openshell-version.ts @@ -8,10 +8,10 @@ import { cliOpenShellInstalledVersionObserver } from "../adapters/openshell/inst import { parseOpenShellVersionFromText } from "../adapters/openshell/version-text"; import { ROOT } from "../runner"; -export const SUPPORTED_OPENSHELL_FALLBACK_VERSION = "0.0.116"; +export const SUPPORTED_OPENSHELL_FALLBACK_VERSION = "0.1.7"; export function usesOpenShellNativeProjection(version: string): boolean { - return version === "0.1.3" || version === "0.1.5"; + return version === "0.1.3" || version === "0.1.5" || version === "0.1.7"; } export function getInstalledOpenshellVersion(versionOutput: string | null = null): string | null { diff --git a/src/lib/onboard/policy-selection-application.test.ts b/src/lib/onboard/policy-selection-application.test.ts index 520e0e6c449..d07bd6bb505 100644 --- a/src/lib/onboard/policy-selection-application.test.ts +++ b/src/lib/onboard/policy-selection-application.test.ts @@ -28,7 +28,7 @@ vi.mock("../policy", () => ({ customPresetOwnsNetworkPolicyKey: vi.fn(() => false), filterSetupPolicyPresets: vi.fn(), getAppliedPresets: vi.fn(() => []), - hasNativeCustomProvider: vi.fn(() => false), + getNativeCustomProviderAttachment: vi.fn(() => undefined), getPersonalSubsumedCustomPresetNames: vi.fn(async () => []), listCustomPresets: vi.fn(() => []), listSetupPolicyPresets: vi.fn(() => [ diff --git a/src/lib/onboard/policy-selection-host-local-route.test.ts b/src/lib/onboard/policy-selection-host-local-route.test.ts index b83de311b9a..af64773155a 100644 --- a/src/lib/onboard/policy-selection-host-local-route.test.ts +++ b/src/lib/onboard/policy-selection-host-local-route.test.ts @@ -4,6 +4,65 @@ import { describe, expect, it, vi } from "vitest"; import { type SetupPolicySelectionDeps, setupPoliciesWithSelection } from "./policy-selection"; +import { hasNativeCustomProviderWebConflict } from "./policy-selection"; +import { customNativeProvider } from "../inference/native-provider/custom"; +import * as registry from "../state/registry"; + +function customAttachment(endpointUrl: string) { + const definition = customNativeProvider({ + provider: "compatible-endpoint", + endpointUrl, + api: "openai-completions", + credentialEnv: "CUSTOM_API_KEY", + allowedIps: ["10.1.0.219"], + }); + return { + schemaVersion: 1 as const, + providerId: "provider-id", + profileId: definition.profileId, + providerName: definition.providerName, + endpointUrl: definition.endpointUrl, + allowedIps: definition.allowedIps, + custom: definition.custom, + }; +} + +describe("Personal custom provider ports", () => { + it.each([ + ["http://10.1.0.219/v1", true], + ["https://10.1.0.219/v1", true], + ["http://10.1.0.219:443/v1", true], + ["https://10.1.0.219:80/v1", true], + ["http://10.1.0.219:33573/v1", false], + ["https://10.1.0.219:8443/v1", false], + ] as const)("checks the attached endpoint %s for web overlap", (endpointUrl, conflict) => { + const attachment = customAttachment(endpointUrl); + vi.spyOn(registry, "getSandbox").mockReturnValue({ + name: "alpha", + nativeHostedProviderAttachment: attachment, + } as never); + expect(hasNativeCustomProviderWebConflict("alpha")).toBe(conflict); + }); + + it("refuses unverified endpoints and leaves the attachment unchanged", () => { + const attachment = customAttachment("http://10.1.0.219:33573/v1"); + const original = structuredClone(attachment); + attachment.endpointUrl = "https://10.1.0.219/v1"; + const getSandbox = vi.spyOn(registry, "getSandbox").mockReturnValue({ + name: "alpha", + nativeHostedProviderAttachment: attachment, + } as never); + expect(hasNativeCustomProviderWebConflict("alpha")).toBe(true); + attachment.custom = { ...attachment.custom, endpointUrl: "invalid-url" }; + expect(hasNativeCustomProviderWebConflict("alpha")).toBe(true); + getSandbox.mockReturnValue({ + name: "alpha", + nativeHostedProviderAttachment: original, + } as never); + expect(hasNativeCustomProviderWebConflict("alpha")).toBe(false); + expect(original).toEqual(customAttachment("http://10.1.0.219:33573/v1")); + }); +}); function createHarness() { const syncPresetSelection = vi.fn(); @@ -60,7 +119,7 @@ describe("Personal selection with a native custom provider", () => { const { deps, syncPresetSelection } = createHarness(); deps.isNonInteractive.mockReturnValue(false); deps.selectPolicyTier.mockResolvedValue("personal"); - const hasNativeCustomProvider = vi.fn(() => true); + const hasNativeCustomProviderWebConflict = vi.fn(() => true); const errors: string[] = []; const errorSpy = vi.spyOn(console, "error").mockImplementation((message: unknown) => { errors.push(String(message)); @@ -70,7 +129,7 @@ describe("Personal selection with a native custom provider", () => { }); try { await expect( - setupPoliciesWithSelection({ ...deps, hasNativeCustomProvider }, "alpha", { + setupPoliciesWithSelection({ ...deps, hasNativeCustomProviderWebConflict }, "alpha", { selectedPresets: null, }), ).rejects.toThrow("process.exit"); @@ -80,22 +139,22 @@ describe("Personal selection with a native custom provider", () => { exitSpy.mockRestore(); } - expect(hasNativeCustomProvider).toHaveBeenCalledWith("alpha"); - expect(errors.join("\n")).toContain("NEMOCLAW_POLICY_TIER=personal"); + expect(hasNativeCustomProviderWebConflict).toHaveBeenCalledWith("alpha"); + expect(errors.join("\n")).toContain("Choose another tier."); expect(deps.selectTierPresetsAndAccess).not.toHaveBeenCalled(); expect(syncPresetSelection).not.toHaveBeenCalled(); }); it("refuses a resumed Personal selection before readiness or policy mutation", async () => { const { deps, syncPresetSelection } = createHarness(); - const hasNativeCustomProvider = vi.fn(() => true); + const hasNativeCustomProviderWebConflict = vi.fn(() => true); const errorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined); const exitSpy = vi.spyOn(process, "exit").mockImplementation(() => { throw new Error("process.exit"); }); try { await expect( - setupPoliciesWithSelection({ ...deps, hasNativeCustomProvider }, "alpha", { + setupPoliciesWithSelection({ ...deps, hasNativeCustomProviderWebConflict }, "alpha", { selectedPresets: ["personal-open-internet"], }), ).rejects.toThrow("process.exit"); @@ -105,7 +164,7 @@ describe("Personal selection with a native custom provider", () => { exitSpy.mockRestore(); } - expect(hasNativeCustomProvider).toHaveBeenCalledWith("alpha"); + expect(hasNativeCustomProviderWebConflict).toHaveBeenCalledWith("alpha"); expect(deps.waitForSandboxReady).not.toHaveBeenCalled(); expect(deps.recordPendingSelection).not.toHaveBeenCalled(); expect(syncPresetSelection).not.toHaveBeenCalled(); @@ -118,14 +177,14 @@ describe("Personal selection with a native custom provider", () => { NEMOCLAW_POLICY_PRESETS: "personal-open-internet", }; deps.parsePolicyPresetEnv.mockReturnValue(["personal-open-internet"]); - const hasNativeCustomProvider = vi.fn(() => true); + const hasNativeCustomProviderWebConflict = vi.fn(() => true); const errorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined); const exitSpy = vi.spyOn(process, "exit").mockImplementation(() => { throw new Error("process.exit"); }); try { await expect( - setupPoliciesWithSelection({ ...deps, hasNativeCustomProvider }, "alpha", { + setupPoliciesWithSelection({ ...deps, hasNativeCustomProviderWebConflict }, "alpha", { selectedPresets: null, }), ).rejects.toThrow("process.exit"); @@ -135,7 +194,7 @@ describe("Personal selection with a native custom provider", () => { exitSpy.mockRestore(); } - expect(hasNativeCustomProvider).toHaveBeenCalledWith("alpha"); + expect(hasNativeCustomProviderWebConflict).toHaveBeenCalledWith("alpha"); expect(deps.recordPendingSelection).not.toHaveBeenCalled(); expect(syncPresetSelection).not.toHaveBeenCalled(); }); diff --git a/src/lib/onboard/policy-selection.ts b/src/lib/onboard/policy-selection.ts index d0c47dab9f9..3d4f14b62d3 100644 --- a/src/lib/onboard/policy-selection.ts +++ b/src/lib/onboard/policy-selection.ts @@ -3,6 +3,8 @@ import { type WebSearchConfig, webSearchProviderForConfig } from "../inference/web-search"; import * as policies from "../policy"; +import { customNativeProvider } from "../inference/native-provider/custom"; +import { normalizeNativeProviderAttachment } from "../inference/native-provider/contract"; import * as tiers from "../policy/tiers"; import { PERSONAL_OPEN_INTERNET_PRESET_NAME, @@ -168,7 +170,7 @@ export type SetupPolicySelectionDeps = { initialSelected: string[], ) => Promise>; parsePolicyPresetEnv: (raw: string) => string[]; - hasNativeCustomProvider?: (sandboxName: string) => boolean; + hasNativeCustomProviderWebConflict?: (sandboxName: string) => boolean; env?: NodeJS.ProcessEnv; }; @@ -238,7 +240,7 @@ export function createOnboardPolicyApplication(deps: OnboardPolicyApplicationDep selectPolicyTier, selectTierPresetsAndAccess, parsePolicyPresetEnv: deps.parsePolicyPresetEnv, - hasNativeCustomProvider: (name) => policies.hasNativeCustomProvider(name), + hasNativeCustomProviderWebConflict, env: deps.env, }; @@ -452,11 +454,26 @@ async function requireSandboxReady( } } +/** Reject overlapping web ports or an unverified custom attachment. */ +export function hasNativeCustomProviderWebConflict(sandboxName: string): boolean { + const attachment = policies.getNativeCustomProviderAttachment(sandboxName); + if (!attachment) return false; + try { + const definition = customNativeProvider(attachment.custom); + if (!normalizeNativeProviderAttachment(attachment, definition)) return true; + const endpoint = new URL(definition.endpoint); + const port = Number(endpoint.port || (endpoint.protocol === "https:" ? 443 : 80)); + return port === 80 || port === 443; + } catch { + return true; + } +} + // OpenShell rejects Personal's hostless web rule beside an attached provider's -// port-443 endpoint (NVIDIA/OpenShell#4662). Refuse before submitting the policy. +// web endpoint (NVIDIA/OpenShell#4662). Refuse before submitting the policy. function refusePersonalBesideNativeCustomProvider(): never { console.error( - " The Personal tier cannot be applied because this sandbox's custom endpoint uses an attached provider. Choose another tier, or create a new sandbox with NEMOCLAW_POLICY_TIER=personal set before onboarding.", + " The Personal tier cannot be applied because this sandbox's attached custom provider overlaps ports 80 or 443, or its endpoint cannot be verified. OpenShell 0.1.7 requires a compatible runtime policy for overlapping endpoints. Choose another tier.", ); process.exit(1); } @@ -511,7 +528,7 @@ async function setupPoliciesWithSelectionInner( const refusePersonalBesideNativeCustomProviderIfSelected = (selected: readonly string[]) => { if ( selected.includes(PERSONAL_OPEN_INTERNET_PRESET_NAME) && - deps.hasNativeCustomProvider?.(sandboxName) + deps.hasNativeCustomProviderWebConflict?.(sandboxName) ) { refusePersonalBesideNativeCustomProvider(); } @@ -612,7 +629,7 @@ async function setupPoliciesWithSelectionInner( if ( (requestedTierName === PERSONAL_POLICY_TIER_NAME || chosen?.includes(PERSONAL_OPEN_INTERNET_PRESET_NAME)) && - deps.hasNativeCustomProvider?.(sandboxName) + deps.hasNativeCustomProviderWebConflict?.(sandboxName) ) { refusePersonalBesideNativeCustomProvider(); } @@ -637,7 +654,10 @@ async function setupPoliciesWithSelectionInner( } const tierName = requestedTierName ?? (await deps.selectPolicyTier()); - if (tierName === PERSONAL_POLICY_TIER_NAME && deps.hasNativeCustomProvider?.(sandboxName)) { + if ( + tierName === PERSONAL_POLICY_TIER_NAME && + deps.hasNativeCustomProviderWebConflict?.(sandboxName) + ) { refusePersonalBesideNativeCustomProvider(); } if (personalAlreadyActive && tierName !== PERSONAL_POLICY_TIER_NAME) { diff --git a/src/lib/onboard/providers.test.ts b/src/lib/onboard/providers.test.ts index 70641a0de63..9414b1fec89 100644 --- a/src/lib/onboard/providers.test.ts +++ b/src/lib/onboard/providers.test.ts @@ -733,3 +733,11 @@ describe("onboard provider helpers", () => { ]); }); }); + +it("supplies the persisted native local authority owner to production onboarding", () => { + const owner = require("../state/registry/native-local-provider-authority"); + const providers = require("./providers"); + const deps = providers.setupInferenceProviderDeps(() => ({ status: 1 })); + expect(deps.getNativeLocalProviderAuthority).toBe(owner.getNativeLocalProviderAuthority); + expect(deps.setNativeLocalProviderAuthority).toBe(owner.setNativeLocalProviderAuthority); +}); diff --git a/src/lib/onboard/providers.ts b/src/lib/onboard/providers.ts index 0a925f7dc54..4d103b2f25b 100644 --- a/src/lib/onboard/providers.ts +++ b/src/lib/onboard/providers.ts @@ -24,6 +24,10 @@ const { setNativeHostedProviderAuthority, setNativeNvidiaProviderAuthority, } = require("../state/registry/native-provider-authority"); +const { + getNativeLocalProviderAuthority, + setNativeLocalProviderAuthority, +} = require("../state/registry/native-local-provider-authority"); const { LLAMA_CPP_CREDENTIAL_ENV, LLAMA_CPP_HOST_OPENAI_BASE_URL, @@ -241,7 +245,7 @@ function getEffectiveProviderName(providerKey) { } switch (providerKey) { case "nim-local": - return "nvidia-nim"; + return "vllm-local"; case "ollama": return "ollama-local"; case "vllm": @@ -400,6 +404,8 @@ function setupInferenceProviderDeps(runOpenshell) { return { providerExistsInGateway, providerAdapter: createCliOpenShellProviderAdapter({ run: runOpenshell }), + getNativeLocalProviderAuthority, + setNativeLocalProviderAuthority, getNativeNvidiaProviderAuthority, getNativeHostedProviderAuthority, setNativeHostedProviderAuthority, @@ -436,7 +442,7 @@ function identityCheckedRunner(runOpenshell, revalidateSandboxIdentity, operatio * the provider is detached from any live sandbox before opting in: OpenShell * rejects `provider delete` on attached providers. * @param {string} name - Provider name (e.g. "discord-bridge", "inference"). - * @param {string} type - Provider type (for example, "openai", "brave", or "nemoclaw-mcp-v1"). + * @param {string} type - Provider type (for example, "openai", "brave", or "nemoclaw-messaging-v1"). * @param {string} credentialEnv - Environment variable name for the credential. * @param {string|null} baseUrl - Optional base URL for the provider endpoint. * @param {Record} env - Environment variables for the openshell command. diff --git a/src/lib/onboard/resume-config.test.ts b/src/lib/onboard/resume-config.test.ts index 477fd445c01..326fcc5968b 100644 --- a/src/lib/onboard/resume-config.test.ts +++ b/src/lib/onboard/resume-config.test.ts @@ -12,6 +12,17 @@ afterEach(() => { }); describe("authoritative rebuild resume config", () => { + it("matches Local NIM selection to its persisted vLLM provider without changing the model", () => { + vi.stubEnv("NEMOCLAW_PROVIDER", "nim-local"); + vi.stubEnv("NEMOCLAW_MODEL", "local/nim-model"); + expect( + getResumeConfigConflicts( + { provider: "vllm-local", model: "local/nim-model" }, + { nonInteractive: true, authoritativeResumeConfig: true }, + ), + ).toEqual([]); + }); + it("ignores a hosted credential alias rehydrated after ambient env isolation", () => { vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "legacy-hosted-source-key"); vi.stubEnv("NEMOCLAW_PROVIDER", ""); diff --git a/src/lib/onboard/resume-provider-recovery.test.ts b/src/lib/onboard/resume-provider-recovery.test.ts index 41702f4f578..fc2276625dc 100644 --- a/src/lib/onboard/resume-provider-recovery.test.ts +++ b/src/lib/onboard/resume-provider-recovery.test.ts @@ -81,6 +81,7 @@ describe("ensureResumeProviderReady", () => { ["anthropic-prod", "ANTHROPIC_API_KEY"], ["gemini-api", "GEMINI_API_KEY"], ["openrouter-api", "OPENROUTER_API_KEY"], + ["hermes-provider", "OPENAI_API_KEY"], ])( "revalidates native ownership for %s during resume (#12589)", async (provider, credentialEnv) => { @@ -97,6 +98,18 @@ describe("ensureResumeProviderReady", () => { }, ); + it("recovers native router ownership without looking up the logical provider", async () => { + const { deps } = makeDeps({ nonInteractive: true }); + deps.isRoutedInferenceProvider = (provider) => provider === "nvidia-router"; + deps.providerExistsInGateway = () => { + throw new Error("retired logical provider lookup"); + }; + expect(await ensureResumeProviderReady("nvidia-router", "ROUTER_KEY", deps)).toEqual({ + forceInferenceSetup: true, + credentialEnv: "ROUTER_KEY", + }); + }); + it("returns false-forced when no provider is set (nothing to recover)", async () => { const { deps } = makeDeps({ providerExists: false }); const result = await ensureResumeProviderReady(null, null, deps); diff --git a/src/lib/onboard/resume-provider-recovery.ts b/src/lib/onboard/resume-provider-recovery.ts index f20e066f86a..edf421ca7a9 100644 --- a/src/lib/onboard/resume-provider-recovery.ts +++ b/src/lib/onboard/resume-provider-recovery.ts @@ -50,6 +50,8 @@ export type ResumeProviderRecoveryDeps = { export type ResumeProviderRecoveryResult = { /** Route recovery through native setup; this is not evidence of credential ownership. */ nativeCustomSetup?: boolean; + nativeLocalSetup?: boolean; + registeredNativeSetup?: boolean; forceInferenceSetup: boolean; credentialEnv: string | null; }; @@ -76,7 +78,7 @@ export function getResumeProviderCredentialEnv( * inference setup step (provider was missing and credential was hydrated or * just re-entered). `credentialEnv` is the env var resolved for that recovery. * Fixed native hosted providers always re-enter setup to verify their persisted - * ownership; NVIDIA and Hermes retain their separate credential recovery paths. + * ownership; NVIDIA retains its separate credential recovery path. * * In non-interactive mode with a missing credential, calls `deps.exit(1)`. */ @@ -85,12 +87,15 @@ export async function ensureResumeProviderReady( credentialEnv: string | null | undefined, deps: ResumeProviderRecoveryDeps, ): Promise { + if (provider && deps.isRoutedInferenceProvider(provider)) { + // The logical routed name is not the sandbox-owned native provider name. + return { + forceInferenceSetup: true, + credentialEnv: credentialEnv || deps.defaultRouteCredentialEnv, + }; + } const native = fixedNativeProvider(provider); - if ( - native && - native.logicalProvider !== "nvidia-prod" && - native.logicalProvider !== "hermes-provider" - ) { + if (native && native.logicalProvider !== "nvidia-prod") { // Native setup verifies the persisted provider identity. The logical name // is not a gateway provider, and its presence cannot authorize skipping setup. return { forceInferenceSetup: true, credentialEnv: native.credentialEnv }; diff --git a/src/lib/onboard/routed-inference.test.ts b/src/lib/onboard/routed-inference.test.ts index 7473b8a6f05..19c31145e0f 100644 --- a/src/lib/onboard/routed-inference.test.ts +++ b/src/lib/onboard/routed-inference.test.ts @@ -25,6 +25,21 @@ import { setupRoutedInference } from "./inference-providers/routed"; import type { RoutedDeps } from "./inference-providers/types"; describe("normalizeRoutedEndpointUrl (#4564)", () => { + it("rewrites the IPv6 loopback host", () => { + expect(normalizeRoutedEndpointUrl("http://[::1]:4000/v1")).toBe( + "http://host.openshell.internal:4000/v1", + ); + }); + + it.each([ + "http://localhost.example:4000/v1", + "http://example.com/localhost", + "https://localhost:4000/v1", + "http://user:pass@localhost:4000/v1", + ])("preserves the non-router URL %s", (url) => { + expect(normalizeRoutedEndpointUrl(url)).toBe(url); + }); + it("rewrites localhost to the sandbox-facing host alias", async () => { expect(normalizeRoutedEndpointUrl("http://localhost:4000/v1")).toBe( "http://host.openshell.internal:4000/v1", @@ -37,15 +52,6 @@ describe("normalizeRoutedEndpointUrl (#4564)", () => { ); }); - it.each([ - "http://localhost.attacker.test:4000/v1", - "http://attacker.test:4000/localhost", - "http://user:secret@localhost:4000/v1", - "https://localhost:4000/v1", - ])("does not grant the host alias to %s (#13560)", (endpoint) => { - expect(normalizeRoutedEndpointUrl(endpoint)).toBe(endpoint); - }); - it("omits the colon when the endpoint has no explicit port and preserves query/hash", async () => { expect(normalizeRoutedEndpointUrl("http://localhost/v1?x=1#frag")).toBe( "http://host.openshell.internal/v1?x=1#frag", @@ -219,21 +225,13 @@ describe("setupRoutedInference route mutation", () => { credentialEnv: "NVIDIA_INFERENCE_API_KEY", }; - it("awaits native registration without changing the shared route (#13560)", async () => { - const deps = makeRoutedSetupDeps(); - const events: string[] = []; - deps.reconcileModelRouter.mockImplementation(async () => { - events.push("router"); - }); - deps.routedInference.upsertRoutedProvider.mockImplementation(async () => { - events.push("native-provider"); - return { ok: true, result: { ok: true } }; + it("registers an attached native provider without mutating the retired shared route", async () => { + const deps = { ...makeRoutedSetupDeps(), nativeLocalInference: true }; + await expect(setupRoutedInference(args, deps as unknown as RoutedDeps)).resolves.toEqual({ + done: false, }); - await setupRoutedInference(args, { - ...deps, - nativeLocalInference: true, - } as unknown as RoutedDeps); - expect(events).toEqual(["router", "native-provider"]); + expect(deps.reconcileModelRouter).toHaveBeenCalledOnce(); + expect(deps.routedInference.upsertRoutedProvider).toHaveBeenCalledOnce(); expect(deps.inferenceRouteMutator.setInferenceRoute).not.toHaveBeenCalled(); }); diff --git a/src/lib/onboard/routed-inference.ts b/src/lib/onboard/routed-inference.ts index 68c7622726c..db7acf0b43f 100644 --- a/src/lib/onboard/routed-inference.ts +++ b/src/lib/onboard/routed-inference.ts @@ -14,9 +14,25 @@ * left behind by an earlier run (#4564). */ +import { nativeModelRouterEndpoint } from "../inference/native-local/model-router"; +import { isProtectedLocalInferencePort } from "../inference/gateway/local-endpoint-ports"; +import { NATIVE_LOCAL_CREDENTIAL_ENV } from "../inference/native-local/agent-config"; +import { + ensureNativeLocalProvider, + prepareNativeLocalProfile, +} from "../inference/native-local/profile"; +import { + getNativeLocalProviderAuthority, + setNativeLocalProviderAuthority, +} from "../state/registry/native-local-provider-authority"; +import type { prepareNativeLocalSelection } from "../inference/native-local/selection"; +export { assertNativeModelRouterOwner } from "./model-router"; +export { isModelRouterProvider } from "../inference/native-local/model-router"; + import { HOST_GATEWAY_URL } from "../inference/local"; import { withModelRouterPortLifecycleLock } from "../inference/gateway-route-mutation-lock"; import { + assertNativeModelRouterOwner, DEFAULT_MODEL_ROUTER_CREDENTIAL_ENV, loadBlueprintProfile, resolveModelRouterPort, @@ -144,3 +160,36 @@ export function withRoutedInferencePortLock( operation, ); } + +/** Caller holds the gateway and Model Router port locks through route publication. */ +export async function prepareNativeModelRouterSelection( + input: Parameters[0] & { + preserveProviderCredential?: boolean; + }, +) { + const endpoint = nativeModelRouterEndpoint(input.binding.provider, input.binding.endpointUrl); + if (!endpoint || endpoint !== input.binding.endpointUrl || !input.credentialValue) + throw new Error( + "Model Router selection does not match its configured endpoint and credential.", + ); + const credentialValue = input.credentialValue; + const port = Number(new URL(endpoint).port); + if (isProtectedLocalInferencePort(port, { ownedModelRouterPort: port })) + throw new Error("Model Router cannot use a gateway or another protected service port."); + assertNativeModelRouterOwner(endpoint, credentialValue); + const binding = { ...input.binding, credentialEnv: NATIVE_LOCAL_CREDENTIAL_ENV }; + const identity = prepareNativeLocalProfile(binding); + const readAuthority = input.readAuthority ?? getNativeLocalProviderAuthority; + const writeAuthority = input.writeAuthority ?? setNativeLocalProviderAuthority; + const attachment = await ensureNativeLocalProvider({ + adapter: input.adapter, + binding, + credentialValue: input.preserveProviderCredential ? null : input.credentialValue, + expected: readAuthority(identity.providerName), + readAuthority, + writeAuthority, + }); + // Keep durable provider authority when router ownership changes during registration. + assertNativeModelRouterOwner(endpoint, credentialValue); + return attachment; +} diff --git a/src/lib/onboard/runtime-provider/contract.ts b/src/lib/onboard/runtime-provider/contract.ts index f6d05f0fd2e..976bf62b9f2 100644 --- a/src/lib/onboard/runtime-provider/contract.ts +++ b/src/lib/onboard/runtime-provider/contract.ts @@ -159,10 +159,13 @@ export interface RuntimeProviderGatewayHostRuntime { readonly sandboxNamespace: "scoped" | "omitted"; readonly hostGatewayIp: string | null; readonly includeSupervisorBin: boolean; + /** Driver layout projected by the provider, independent of gateway schema version. */ readonly driverConfigLayout: "split-supervisor" | "inline-supervisor"; readonly processOwnership: "scoped-namespace" | "runtime-marker"; }; readonly network: { + /** Supervisor topology available to host-service probes without a sandbox bridge. */ + readonly supervisorNetwork?: { readonly name: string; readonly hostAddress: string }; /** Provider-owned sandbox source ranges authorized to reach host services. */ sandboxSourceCidrs(): readonly string[]; inspect(networkName: string): RuntimeProviderGatewayNetworkInfo | undefined; @@ -461,7 +464,6 @@ export interface RuntimeProviderPrivilegedSandboxControl { readonly workspace: { readonly uid: number; readonly gid: number; readonly mode: number }; }, ): void; - resolveTarget( input: Pick< RuntimeProviderPrivilegedSandboxCommandInput, diff --git a/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle-engine.test-support.ts b/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle-engine.test-support.ts index b7f52c1ac64..544dfce087d 100644 --- a/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle-engine.test-support.ts +++ b/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle-engine.test-support.ts @@ -33,7 +33,6 @@ export interface DockerFixture { readonly status: number; readonly stderr?: string; }) => void; - readonly setOpenShellBridgeSubnet: (value: string) => void; readonly driftHardening: () => void; readonly driftEntrypoint: () => void; readonly dropTmpfs: () => void; @@ -86,7 +85,6 @@ export function createDockerFixture( readonly status: number; readonly stderr?: string; } | null = null; - let openShellBridgeSubnet = "172.29.0.0/16"; let hardeningDrift = false; let tmpfs: Record | null = { "/tmp": "rw,noexec,nosuid,nodev,size=1024,uid=1001,gid=1001,mode=1777", @@ -197,24 +195,6 @@ export function createDockerFixture( case "network": switch (args[1]) { case "inspect": - switch (args[2]) { - case "openshell-docker": - return { - status: 0, - stdout: JSON.stringify([ - { - Name: "openshell-docker", - Internal: false, - Driver: "bridge", - Scope: "local", - IPAM: { - Config: [{ Subnet: openShellBridgeSubnet, Gateway: "172.29.0.1" }], - }, - }, - ]), - stderr: "", - }; - } switch (networkPresent) { case false: absentNetworkInspectHook?.(); @@ -348,10 +328,7 @@ export function createDockerFixture( case "run": invariant(args[1] === "--rm", unexpected); probeHook?.(); - if ( - sandboxBridgeProbeFailure !== null && - args[args.indexOf("--network") + 1] === "openshell-docker" - ) { + if (sandboxBridgeProbeFailure !== null && args[args.indexOf("--network") + 1] === "host") { return { ...sandboxBridgeProbeFailure, stdout: "", @@ -386,7 +363,6 @@ export function createDockerFixture( failCreateUncertain: () => (createUncertain = true), failProbe: () => (probeFails = true), failSandboxBridgeProbe: (result) => (sandboxBridgeProbeFailure = result), - setOpenShellBridgeSubnet: (value) => (openShellBridgeSubnet = value), driftHardening: () => (hardeningDrift = true), driftEntrypoint: () => { invariant(container !== undefined, "cannot drift an absent fixture container"); @@ -450,9 +426,3 @@ export function createDockerFixture( export function dockerCommandPrefixes(fixture: DockerFixture): unknown[] { return fixture.capture.mock.calls.map((call) => call[0]?.slice(0, 2)); } - -export function hostNetworkRuns(fixture: DockerFixture): readonly (readonly string[])[] { - return fixture.capture.mock.calls - .map(([argv]) => argv as readonly string[]) - .filter((argv) => argv[0] === "run" && argv[argv.indexOf("--network") + 1] === "host"); -} diff --git a/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.test.ts b/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.test.ts index 0ba7013f2b2..0ece696ac5e 100644 --- a/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.test.ts +++ b/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.test.ts @@ -40,7 +40,6 @@ import { import { createDockerFixture, dockerCommandPrefixes, - hostNetworkRuns, type DockerFixture, } from "./docker-llama-cpp-managed-lifecycle-engine.test-support"; import type { HostLocalCreateJournalRecord } from "./host-local-create-journal"; @@ -299,21 +298,50 @@ describe("dormant Docker llama.cpp managed lifecycle", () => { expect(lifecycle.runtime.destroy(receipt).status).toBe("already-absent"); }); - it("accepts managed reachability through the authoritative OpenShell bridge (#8712)", () => { + it("accepts managed reachability through the native supervisor network (#8712)", () => { const fixture = dockerFixture(); - createLifecycle(options(fixture)).start(receiptWriter()); + const bridge = privateBridgeFixture(); + createLifecycle(options(fixture), {}, bridge).start(receiptWriter()); + expect(bridge.start).toHaveBeenCalledWith( + expect.objectContaining({ bindAddresses: ["127.0.0.1"] }), + ); expect(fixture.capture.mock.calls.map((call) => call[0])).toContainEqual( expect.arrayContaining([ "--network", - "openshell-docker", + "host", "--add-host", - "host.openshell.internal:172.29.0.1", + "host.openshell.internal:127.0.0.1", + "--cap-drop=ALL", + "--security-opt=no-new-privileges", + "--read-only", "http://host.openshell.internal:8081/health", ]), ); }); + it("keeps the Desktop host-gateway probe and loopback proxy binding", () => { + const fixture = dockerFixture(); + const bridge = privateBridgeFixture(); + const lifecycle = createLifecycle( + options(fixture, journalStore(), { ...bindings(), bridgeLoopbackOnly: true }), + {}, + bridge, + ); + lifecycle.start(receiptWriter()); + expect(bridge.start).toHaveBeenCalledWith( + expect.objectContaining({ bindAddresses: ["127.0.0.1"] }), + ); + expect(fixture.capture.mock.calls.map(([argv]) => argv)).toContainEqual( + expect.arrayContaining([ + "--network", + "bridge", + "--add-host", + "host.openshell.internal:host-gateway", + ]), + ); + }); + it.each([ { failure: "connection refusal", status: 7, stderr: "curl: (7) failed to connect" }, { failure: "timeout", status: 28, stderr: "curl: (28) timed out" }, @@ -329,12 +357,8 @@ describe("dormant Docker llama.cpp managed lifecycle", () => { failure = error instanceof Error ? error : new Error(String(error)); } - expect(failure?.message).toContain( - "host-loopback health check passed, but the OpenShell Docker bridge health check failed", - ); - expect(failure?.message).toContain("sudo ufw"); - expect(failure?.message).toContain("172.29.0.0/16"); - expect(failure?.message).toContain("172.29.0.1"); + expect(failure?.message).toContain(`private supervisor endpoint probe failed (exit ${status})`); + expect(failure?.message).not.toContain("sudo ufw"); expect(failure?.message).not.toContain("test-only-secret"); expect(store.list()).toEqual([]); expect(dockerCommandPrefixes(fixture)).toContainEqual(["rm", "--force"]); @@ -353,28 +377,9 @@ describe("dormant Docker llama.cpp managed lifecycle", () => { } expect(failure?.message).toBe( - "Docker llama.cpp private sandbox bridge probe failed (exit 125).", - ); - expect(failure?.message).not.toContain("sudo ufw"); - }); - - it("does not print unvalidated bridge topology in UFW remediation (#8712)", () => { - const fixture = dockerFixture(); - fixture.setOpenShellBridgeSubnet("0.0.0.0/0"); - fixture.failSandboxBridgeProbe({ status: 28, stderr: "curl: (28) timed out" }); - - let failure: Error | undefined; - try { - createLifecycle(options(fixture)).start(receiptWriter()); - } catch (error) { - failure = error instanceof Error ? error : new Error(String(error)); - } - - expect(failure?.message).toBe( - "Docker llama.cpp private sandbox bridge probe failed (exit 28).", + "Docker llama.cpp private supervisor endpoint probe failed (exit 125).", ); expect(failure?.message).not.toContain("sudo ufw"); - expect(failure?.message).not.toContain("0.0.0.0/0"); }); it("probes the private bridge endpoint through the default host curl boundary", () => { @@ -393,13 +398,9 @@ describe("dormant Docker llama.cpp managed lifecycle", () => { { timeoutMs: 86_415_000, pinnedAddresses: [] }, ); expect(fixture.capture.mock.calls.map(([argv]) => argv[0])).not.toContain("curl"); - expect(hostNetworkRuns(fixture)).toEqual([]); + expect(fixture.capture.mock.calls.map(([argv]) => argv)).toContainEqual( - expect.arrayContaining([ - "--network", - "openshell-docker", - "http://host.openshell.internal:8081/health", - ]), + expect.arrayContaining(["--network", "host", "http://host.openshell.internal:8081/health"]), ); probe.mockReturnValue({ ok: false, @@ -467,9 +468,9 @@ describe("dormant Docker llama.cpp managed lifecycle", () => { "http://127.0.0.1:8081/health", 1_800, ); - expect(hostNetworkRuns(fixture)).toEqual([]); + expect(fixture.capture.mock.calls.map(([argv]) => argv)).toContainEqual( - expect.arrayContaining(["--network", "openshell-docker"]), + expect.arrayContaining(["--network", "host"]), ); }, ); @@ -516,7 +517,6 @@ describe("dormant Docker llama.cpp managed lifecycle", () => { "http://127.0.0.1:8081/health", 1_800, ); - expect(hostNetworkRuns(fixture)).toEqual([]); }); it.each([true, false] as const)( @@ -561,9 +561,7 @@ describe("dormant Docker llama.cpp managed lifecycle", () => { failure = error instanceof Error ? error : new Error(String(error)); } - expect(failure?.message).toContain( - "host-loopback health check passed, but the OpenShell Docker bridge health check failed", - ); + expect(failure?.message).toContain("private supervisor endpoint probe failed (exit 7)"); expect(store.load(TRANSACTION_ID)).toMatchObject({ phase: "finalized", runtimeId: RUNTIME_ID, diff --git a/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.ts b/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.ts index 8c306cc89a4..80f60a07c87 100644 --- a/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.ts +++ b/src/lib/onboard/runtime-provider/docker-llama-cpp-managed-lifecycle.ts @@ -10,8 +10,6 @@ import type { ContainerEngineCommandResult, } from "../../adapters/container-engine"; import { runCurlProbe } from "../../adapters/http/probe"; -import { formatHostServiceUnreachableMessage } from "../reachability/host-service-message"; -import { validateUfwRuleOperands } from "../ufw-auto-apply"; import { assertLlamaCppGgufCachePlanDigest } from "../../inference/llama-cpp/gguf-cache-plan"; import { assertLlamaCppVerifiedLocalModelArtifact, @@ -56,7 +54,6 @@ import { const PROVIDER_ID = "docker"; const SERVICE = "llama-cpp"; const ENDPOINT_HOST = "host.openshell.internal"; -const OPENSHELL_DOCKER_NETWORK = "openshell-docker"; const FULL_ID = /^[a-f0-9]{64}$/u; const SHA256 = /^[a-f0-9]{64}$/u; const SAFE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/u; @@ -94,12 +91,6 @@ interface DockerNetworkAuthority { readonly name: string; } -interface DockerGatewayBridgeAuthority { - readonly gatewayIp: string; - readonly name: typeof OPENSHELL_DOCKER_NETWORK; - readonly subnet?: string; -} - interface DockerContainerInspection { readonly id: string; readonly name: string; @@ -283,48 +274,6 @@ function inspectNetworkIfPresent( }); } -function inspectGatewayBridge(engine: ContainerEngine): DockerGatewayBridgeAuthority { - const output = requireSuccess( - "OpenShell bridge inspection", - engine.capture(["network", "inspect", OPENSHELL_DOCKER_NETWORK], INSPECT_TIMEOUT_MS), - ); - let parsed: unknown; - try { - parsed = JSON.parse(output); - } catch { - throw new Error("Docker llama.cpp OpenShell bridge inspection returned unreadable JSON."); - } - if (!Array.isArray(parsed) || parsed.length !== 1) { - throw new Error("Docker llama.cpp requires exactly one OpenShell Docker bridge."); - } - const source = record(parsed[0], "Docker llama.cpp OpenShell bridge inspection"); - const ipam = record(source.IPAM, "Docker llama.cpp OpenShell bridge IPAM"); - const configs = ipam.Config; - if ( - source.Name !== OPENSHELL_DOCKER_NETWORK || - source.Driver !== "bridge" || - source.Scope !== "local" || - source.Internal !== false || - !Array.isArray(configs) - ) { - throw new Error("Docker llama.cpp requires the native OpenShell Docker bridge."); - } - const gateways = configs.flatMap((candidate) => { - const config = record(candidate, "Docker llama.cpp OpenShell bridge IPAM entry"); - const gatewayIp = config.Gateway; - const subnet = config.Subnet; - return typeof gatewayIp === "string" && - (/^172\.(?:1[6-9]|2[0-9]|3[01])\.(?:[0-9]{1,3})\.(?:[0-9]{1,3})$/u.test(gatewayIp) || - /^(?:10\.|192\.168\.)/u.test(gatewayIp)) - ? [{ gatewayIp, ...(typeof subnet === "string" ? { subnet } : {}) }] - : []; - }); - if (gateways.length !== 1) { - throw new Error("Docker llama.cpp requires one private IPv4 OpenShell bridge gateway."); - } - return Object.freeze({ ...gateways[0]!, name: OPENSHELL_DOCKER_NETWORK }); -} - function inspectNetwork( engine: ContainerEngine, name: string, @@ -988,7 +937,6 @@ function privateBridgeAuthority( options: DockerLlamaCppManagedLifecycleOptions, journal: HostLocalCreateJournalRecord, container: DockerContainerInspection, - gateway: DockerGatewayBridgeAuthority, ): DockerLlamaCppPrivateBridgeAuthority { if (container.containerIp === null || options.bindings.hostPort !== options.contract.serve.port) throw new Error("Docker llama.cpp private bridge authority is invalid."); @@ -1000,9 +948,8 @@ function privateBridgeAuthority( containerId: container.id, dockerAuthorityId: options.engine.authorityId, listenPort: options.bindings.hostPort, - bindAddresses: options.bindings.bridgeLoopbackOnly - ? Object.freeze(["127.0.0.1"] as const) - : Object.freeze(["127.0.0.1", gateway.gatewayIp] as const), + // The native supervisor reaches the authenticated host proxy through loopback. + bindAddresses: Object.freeze(["127.0.0.1"] as const), }); } @@ -1015,8 +962,7 @@ function probePrivateBridge( execution: MutationExecutionState, hostLoopbackProbe: (url: string, timeoutSeconds: number) => ContainerEngineCommandResult, ): void { - const gateway = inspectGatewayBridge(options.engine); - const authority = privateBridgeAuthority(options, journal, container, gateway); + const authority = privateBridgeAuthority(options, journal, container); options.journalStore.assertExecution(lease); bridge.start(authority); bridge.assertRunning(authority); @@ -1036,9 +982,12 @@ function probePrivateBridge( "--rm", "--pull=never", "--network", - gateway.name, + options.bindings.bridgeLoopbackOnly ? "bridge" : "host", + "--cap-drop=ALL", + "--security-opt=no-new-privileges", + "--read-only", "--add-host", - `${ENDPOINT_HOST}:${options.bindings.bridgeLoopbackOnly ? "host-gateway" : gateway.gatewayIp}`, + `${ENDPOINT_HOST}:${options.bindings.bridgeLoopbackOnly ? "host-gateway" : "127.0.0.1"}`, "--entrypoint", "curl", options.probeImageReference, @@ -1051,40 +1000,7 @@ function probePrivateBridge( ], timeoutSeconds * 1_000 + INSPECT_TIMEOUT_MS, ); - if (sandboxProbe.error || sandboxProbe.status !== 0) { - const port = options.bindings.hostPort; - const subnet = gateway.subnet; - if ( - !sandboxProbe.error && - [7, 28].includes(sandboxProbe.status) && - subnet && - !options.bindings.bridgeLoopbackOnly && - !validateUfwRuleOperands(subnet, gateway.gatewayIp, port) - ) { - const remediation = formatHostServiceUnreachableMessage( - { - ok: false, - reason: "tcp_failed", - port, - networkName: gateway.name, - subnet, - gatewayIp: gateway.gatewayIp, - }, - { serviceLabel: "managed llama.cpp server", port }, - ); - throw new Error( - [ - "Managed llama.cpp host-loopback health check passed, but the OpenShell Docker bridge health check failed.", - ` OpenShell Docker network: ${gateway.name}`, - ` Source subnet: ${subnet}`, - ` Gateway IP address: ${gateway.gatewayIp}`, - ` TCP port: ${String(port)}`, - remediation, - ].join("\n"), - ); - } - requireSuccess("private sandbox bridge probe", sandboxProbe); - } + requireSuccess("private supervisor endpoint probe", sandboxProbe); bridge.assertRunning(authority); options.journalStore.assertExecution(lease); } @@ -1503,12 +1419,7 @@ export function createDockerLlamaCppManagedLifecycle( assertModelFilesystemAuthority(options); if (inspected.container.running) { privateBridge.assertRunning( - privateBridgeAuthority( - options, - inspected.journal, - inspected.container, - inspectGatewayBridge(options.engine), - ), + privateBridgeAuthority(options, inspected.journal, inspected.container), ); } else privateBridge.assertStopped(inspected.journal.transactionId); return Object.freeze({ diff --git a/src/lib/onboard/runtime-provider/docker.test.ts b/src/lib/onboard/runtime-provider/docker.test.ts index d0d627934e5..04b4b4f5ad1 100644 --- a/src/lib/onboard/runtime-provider/docker.test.ts +++ b/src/lib/onboard/runtime-provider/docker.test.ts @@ -184,6 +184,20 @@ describe("Docker runtime provider NVIDIA container capture", () => { }); describe("Docker network command bounds", () => { + it.each(["linux", "darwin"] as const)("projects supervisor probe topology on %s", (platform) => { + const gateway = createDockerRuntimeProviderBundle().gateway as Extract< + ReturnType["gateway"], + { supported: true } + >; + const runtime = gateway.observeHostRuntime({ + environment: { NEMOCLAW_GATEWAY_BIND_ADDRESS: "0.0.0.0" }, + platform, + }); + expect(runtime.network.supervisorNetwork).toEqual( + platform === "linux" ? { name: "host", hostAddress: "127.0.0.1" } : undefined, + ); + }); + it("uses the selected socket, output limit, and forced timeout for provisioning (#11606)", () => { const dockerRun = vi.spyOn(dockerCommands, "dockerRun").mockReturnValue({ status: null, diff --git a/src/lib/onboard/runtime-provider/docker.ts b/src/lib/onboard/runtime-provider/docker.ts index 7cbacd84280..95c4ed4d2c9 100644 --- a/src/lib/onboard/runtime-provider/docker.ts +++ b/src/lib/onboard/runtime-provider/docker.ts @@ -382,6 +382,9 @@ export function createDockerRuntimeProviderBundle( processOwnership: "scoped-namespace", }, network: { + ...(input.platform === "linux" + ? { supervisorNetwork: { name: "host", hostAddress: connectHost } } + : {}), sandboxSourceCidrs: () => { const network = inspectDockerGatewayNetwork( resolveDockerDriverNetworkName(input.environment), diff --git a/src/lib/onboard/runtime-provider/podman-lifecycle.ts b/src/lib/onboard/runtime-provider/podman-lifecycle.ts index b134ab5c82e..274ad77a822 100644 --- a/src/lib/onboard/runtime-provider/podman-lifecycle.ts +++ b/src/lib/onboard/runtime-provider/podman-lifecycle.ts @@ -8,6 +8,7 @@ import type { import { isValidName } from "../../name-validation"; export const PODMAN_MANAGED_LABEL = "openshell.managed"; +export const PODMAN_ISOLATION_ROLE_LABEL = "openshell.ai/isolation-role"; export const PODMAN_SANDBOX_ID_LABEL = "openshell.ai/sandbox-id"; export const PODMAN_SANDBOX_NAME_LABEL = "openshell.ai/sandbox-name"; export const PODMAN_SANDBOX_NAMESPACE_LABEL = "openshell.ai/sandbox-namespace"; @@ -119,6 +120,10 @@ function parsePodmanManagedContainer( const name = safeText(entry.Name, "Podman inspect Name"); const config = record(entry.Config, "Podman inspect Config"); const containerLabels = labels(config.Labels); + const isolationRole = containerLabels[PODMAN_ISOLATION_ROLE_LABEL]; + if (isolationRole !== undefined && isolationRole !== "sandbox") { + throw new Error("Podman sandbox has an unexpected isolation role."); + } if (containerLabels[PODMAN_MANAGED_LABEL] !== "true") { throw new Error(`Podman sandbox is missing exact label ${PODMAN_MANAGED_LABEL}=true.`); } @@ -239,6 +244,9 @@ export function observePodmanManagedContainer( `label=${PODMAN_SANDBOX_NAME_LABEL}=${sandboxName}`, "--filter", `label=${PODMAN_SANDBOX_WORKSPACE_LABEL}=${PODMAN_SANDBOX_WORKSPACE}`, + // Keep legacy workloads without a role label; exclude the 0.1.2 companion. + "--filter", + `label!=${PODMAN_ISOLATION_ROLE_LABEL}=supervisor`, "--format", "{{.ID}}", ], diff --git a/src/lib/onboard/runtime-provider/podman-runtime-surfaces.test.ts b/src/lib/onboard/runtime-provider/podman-runtime-surfaces.test.ts index 1bd689c6275..dcdaecccd1a 100644 --- a/src/lib/onboard/runtime-provider/podman-runtime-surfaces.test.ts +++ b/src/lib/onboard/runtime-provider/podman-runtime-surfaces.test.ts @@ -4,8 +4,11 @@ import { describe, expect, it, vi } from "vitest"; import type { PodmanBoundContainerEngine } from "../../adapters/podman"; +import { createPodmanPrivilegedSandboxControl } from "./podman-privileged-sandbox-control"; import { createCurrentPodmanRuntimeProviderBundle } from "./podman"; import { + observePodmanManagedContainer, + PODMAN_ISOLATION_ROLE_LABEL, PODMAN_MANAGED_LABEL, PODMAN_SANDBOX_CONTAINER_PREFIX, PODMAN_SANDBOX_ID_LABEL, @@ -284,8 +287,8 @@ describe("current Podman runtime provider", () => { sandboxNamespace: "omitted", hostGatewayIp: NATIVE_PODMAN_SANDBOX_HOST_ADDRESS, includeSupervisorBin: false, - processOwnership: "runtime-marker", driverConfigLayout: "inline-supervisor", + processOwnership: "runtime-marker", }, network: { sandboxSourceCidrs: expect.any(Function), @@ -370,3 +373,134 @@ describe("current Podman runtime provider", () => { expect(capture).toHaveBeenCalledWith(["network", "inspect", "openshell-docker"], 30_000); }); }); + +describe("Podman split workload discovery", () => { + it("excludes the supervisor companion before requiring one workload", () => { + const engine = runtimeEngine(() => ({ + ...exactLabels(), + [PODMAN_ISOLATION_ROLE_LABEL]: "sandbox", + })); + const capture = engine.capture; + vi.mocked(engine.capture).mockImplementationOnce((args) => ({ + status: 0, + stdout: args.includes("label!=openshell.ai/isolation-role=supervisor") + ? CONTAINER_ID + "\n" + : CONTAINER_ID + "\n" + "b".repeat(64) + "\n", + stderr: "", + })); + expect(observePodmanManagedContainer(engine, SANDBOX_NAME)?.containerId).toBe(CONTAINER_ID); + expect(capture).toHaveBeenCalledWith( + ["container", "inspect", CONTAINER_ID], + expect.any(Number), + ); + }); + + it("retains discovery of legacy workloads without an isolation role", () => { + const engine = runtimeEngine(exactLabels); + expect(observePodmanManagedContainer(engine, SANDBOX_NAME)?.containerId).toBe(CONTAINER_ID); + }); + + it.each(["supervisor", "unknown", ""])("rejects an inspected role of %j", (role) => { + const engine = runtimeEngine(() => ({ + ...exactLabels(), + [PODMAN_ISOLATION_ROLE_LABEL]: role, + })); + expect(() => observePodmanManagedContainer(engine, SANDBOX_NAME)).toThrow( + "unexpected isolation role", + ); + }); + + it("rejects multiple workloads after excluding supervisors", () => { + const engine = runtimeEngine(exactLabels); + vi.mocked(engine.capture).mockReturnValueOnce({ + status: 0, + stdout: CONTAINER_ID + "\n" + "b".repeat(64) + "\n", + stderr: "", + }); + expect(() => observePodmanManagedContainer(engine, SANDBOX_NAME)).toThrow( + "has 2 managed containers", + ); + expect(engine.capture).toHaveBeenCalledTimes(1); + }); +}); + +function workspaceFixture() { + const wanted = { uid: 998, gid: 998, mode: 0o755 }; + const state = { + labels: exactLabels(), + metadata: { uid: 0, gid: 0, mode: 0o700, dev: "1", ino: "2", mtime: 100 }, + copyStatus: 0, + }; + const observe = vi.fn(); + const engine = { ...runtimeEngine(() => state.labels), operation: "sandbox-lifecycle" as const }; + const capture = engine.capture; + engine.capture = vi.fn((args, timeoutMs, input) => { + switch (args.slice(0, 2).join(" ")) { + case "container exec": { + const stdout = JSON.stringify(state.metadata); + observe(); + return { status: 0, stdout, stderr: "" }; + } + case "container cp": + state.metadata = { ...state.metadata, ...wanted }; + return { status: state.copyStatus, stdout: "", stderr: "" }; + default: + return capture(args, timeoutMs, input); + } + }); + const control = createPodmanPrivilegedSandboxControl(engine); + const prepare = () => + control.prepareManagedStartupWorkspace!({ + sandbox: sandbox(), + sandboxName: SANDBOX_NAME, + registeredSandboxNames: [SANDBOX_NAME], + expectedResourceHandle: CONTAINER_ID, + timeoutMs: 2000, + workspace: wanted, + }); + return { engine, state, wanted, prepare, observe }; +} + +describe("Podman managed workspace preparation", () => { + it("restores only directory metadata through the pinned provider engine", () => { + const { engine, state, wanted, prepare } = workspaceFixture(); + prepare(); + const copies = vi.mocked(engine.capture).mock.calls.filter(([args]) => args[1] === "cp"); + expect(copies).toHaveLength(1); + expect(copies[0]?.[0]).toEqual([ + "container", + "cp", + "--archive=false", + "-", + `${CONTAINER_ID}:/`, + ]); + expect(copies[0]?.[1]).toBe(2000); + const archive = copies[0]![2]!; + expect(archive.length).toBe(1536); + expect(archive.subarray(0, 8).toString()).toBe("sandbox/"); + expect(archive.subarray(512).equals(Buffer.alloc(1024))).toBe(true); + expect(state.metadata).toEqual({ ...wanted, dev: "1", ino: "2", mtime: 100 }); + }); + + it("leaves a workspace with matching metadata unchanged", () => { + const { engine, state, wanted, prepare } = workspaceFixture(); + state.metadata = { ...state.metadata, ...wanted }; + prepare(); + expect(vi.mocked(engine.capture).mock.calls.some(([args]) => args[1] === "cp")).toBe(false); + }); + + it("rejects a failed metadata repair", () => { + const { state, prepare } = workspaceFixture(); + state.copyStatus = 125; + expect(prepare).toThrow(/could not be restored/u); + }); + + it("rechecks runtime ownership before the directory write", () => { + const { engine, state, prepare, observe } = workspaceFixture(); + observe.mockImplementationOnce(() => { + state.labels = { ...state.labels, [PODMAN_SANDBOX_WORKSPACE_LABEL]: "other" }; + }); + expect(prepare).toThrow(); + expect(vi.mocked(engine.capture).mock.calls.some(([args]) => args[1] === "cp")).toBe(false); + }); +}); diff --git a/src/lib/onboard/runtime-provider/podman.test.ts b/src/lib/onboard/runtime-provider/podman.test.ts index 66cfb9ee278..ad036be2e17 100644 --- a/src/lib/onboard/runtime-provider/podman.test.ts +++ b/src/lib/onboard/runtime-provider/podman.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { spawnSync } from "node:child_process"; import { beforeAll, describe, expect, it, vi } from "vitest"; import { DEEP_AGENTS_CODE_SANDBOX_USER } from "../../agent/deep-agents-code-runtime-identity"; import { createPodmanHostLocalInferenceTestHarness } from "../../../../test/helpers/podman-host-local-inference-test-harness"; @@ -450,6 +451,37 @@ describe("managed Podman runtime provider", () => { ).not.toContain("docker"); }); + it("runs sanitized cleanup commands without an image PATH (#12603)", () => { + const runtime = providerHarness("hermes"); + runtime.lifecycle.capture(["start", CONTAINER_ID]); + const lifecycle = runtime.providers.podman!.lifecycle; + const supportedLifecycle = lifecycle as Extract; + supportedLifecycle.privilegedSandboxControl.execute({ + registeredSandboxNames: [runtime.sandboxName], + sandbox: runtime.entry, + sandboxName: runtime.sandboxName, + expectedResourceHandle: CONTAINER_ID, + command: ["sh", "-c", 'test -n "$(command -v rm)" && printf cleanup-ready'], + sanitizeEnvironment: true, + timeoutMs: 9000, + }); + const args = vi.mocked(runtime.lifecycle.capture).mock.lastCall![0]; + const commandIndex = args.indexOf(CONTAINER_ID) + 1; + // crun cannot resolve a bare executable after OpenShell clears the image environment. + const helperPath = + args + .slice(0, commandIndex - 1) + .join("\n") + .match(/(?:^|\n)--env\nPATH=([^\n]*)/u)?.[1] ?? ""; + const result = spawnSync(args[commandIndex]!, args.slice(commandIndex + 1), { + env: { PATH: helperPath }, + encoding: "utf8", + timeout: 5000, + }); + expect(result.status).toBe(0); + expect(result.stdout).toBe("cleanup-ready"); + }); + it("pins Deep Agents sandbox-user cleanup to the owned Podman resource", () => { const runtime = providerHarness("langchain-deepagents-code"); const lifecycle = runtime.providers.podman?.lifecycle; @@ -487,6 +519,19 @@ describe("managed Podman runtime provider", () => { 9000, undefined, ); + const privateInput = Buffer.from("synthetic private snapshot"); + control.executeAsSandboxUser?.({ ...input, input: privateInput, maxOutputBytes: 2048 }); + expect(runtime.lifecycle.capture).toHaveBeenLastCalledWith( + expect.arrayContaining([ + "--interactive", + "--user", + DEEP_AGENTS_CODE_SANDBOX_USER, + CONTAINER_ID, + ]), + 9000, + privateInput, + { maxInputBytes: privateInput.length, maxOutputBytes: 2048 }, + ); expect(() => control.executeAsSandboxUser?.({ ...input, diff --git a/src/lib/onboard/runtime-provider/privileged-sandbox-environment.ts b/src/lib/onboard/runtime-provider/privileged-sandbox-environment.ts index 50aabc90149..ebc4b28c396 100644 --- a/src/lib/onboard/runtime-provider/privileged-sandbox-environment.ts +++ b/src/lib/onboard/runtime-provider/privileged-sandbox-environment.ts @@ -1,8 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -/** Remove interpreter and loader hooks before entering a privileged container helper. */ +/** Supply helper executables and remove interpreter and loader hooks in privileged commands. */ export const SANITIZED_PRIVILEGED_ENV = [ + // OpenShell clears the image environment; privileged engine exec needs its own search path. + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "BASH_ENV=", "ENV=", "GCONV_PATH=", diff --git a/src/lib/onboard/runtime-provider/stopped-native-home-cleanup.test.ts b/src/lib/onboard/runtime-provider/stopped-native-home-cleanup.test.ts index 03256d28c35..f3ca5d0bfcf 100644 --- a/src/lib/onboard/runtime-provider/stopped-native-home-cleanup.test.ts +++ b/src/lib/onboard/runtime-provider/stopped-native-home-cleanup.test.ts @@ -95,7 +95,7 @@ describe("stopped native-home cleanup", () => { ).toEqual({ cleared: false, failure: "runtime-revalidation-failed" }); }); - it("pulls the pinned cleanup image when a mounted stopped home needs it", () => { + it("pulls the pinned cleanup image and mounts the stopped home without copying image data", () => { const stateResource = { type: "volume" as const, source: "openclaw-state", @@ -135,11 +135,10 @@ describe("stopped native-home cleanup", () => { ["pull", "--quiet", expect.stringContaining("node:24.22.1-trixie-slim")], 120_000, ); - // Podman 6.1 rejects Docker's --mount volume-nocopy option; both engines - // accept --volume with nocopy for this exact, validated named resource. - expect(capture).toHaveBeenCalledWith( - expect.arrayContaining(["--volume", "openclaw-state:/sandbox:nocopy"]), - ); + const createArgs = capture.mock.calls.find(([args]) => args[0] === "create")![0]; + expect( + createArgs.slice(createArgs.indexOf("--volume"), createArgs.indexOf("--volume") + 2), + ).toEqual(["--volume", "openclaw-state:/sandbox:nocopy"]); }); it("removes complete native state while preserving exact user-managed paths", () => { diff --git a/src/lib/onboard/sandbox-create-intent-resolution.ts b/src/lib/onboard/sandbox-create-intent-resolution.ts index f2f48a8e784..e3d46962d14 100644 --- a/src/lib/onboard/sandbox-create-intent-resolution.ts +++ b/src/lib/onboard/sandbox-create-intent-resolution.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import type { RegisteredInferenceProviderAttachment } from "../inference/native-provider/registered"; + import type { NativeLocalProviderAttachment } from "../inference/native-local/contract"; import type { NativeProviderAttachment } from "../inference/native-provider/contract"; @@ -30,6 +32,7 @@ import { export type CompleteSandboxCreateIntentInput = { sandboxName: string; inferenceProvider?: string | null; + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; nativeNvidiaProviderAttachment?: NativeNvidiaProviderAttachment; nativeHostedProviderAttachment?: NativeProviderAttachment; @@ -151,6 +154,7 @@ export function createSandboxCreateIntentResolver< basePolicyPath: deps.getAgentPolicyPath(input.agent) || deps.defaultPolicyPath, sandboxName: input.sandboxName, inferenceProvider: input.inferenceProvider, + registeredInferenceProviderAttachment: input.registeredInferenceProviderAttachment, nativeLocalProviderAttachment: input.nativeLocalProviderAttachment, nativeNvidiaProviderAttachment: input.nativeNvidiaProviderAttachment, nativeHostedProviderAttachment: input.nativeHostedProviderAttachment, diff --git a/src/lib/onboard/sandbox-create-intent-types.ts b/src/lib/onboard/sandbox-create-intent-types.ts index 54aec18f9af..63b5f2aa8c3 100644 --- a/src/lib/onboard/sandbox-create-intent-types.ts +++ b/src/lib/onboard/sandbox-create-intent-types.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import type { RegisteredInferenceProviderAttachment } from "../inference/native-provider/registered"; + import type { NativeLocalProviderAttachment } from "../inference/native-local/contract"; import type { NativeProviderAttachment } from "../inference/native-provider/contract"; @@ -51,6 +53,7 @@ export type SandboxCreatePolicyRequest = { export type SandboxCreateIntent = { readonly sandboxName: string; readonly inferenceProvider: string | null; + readonly registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; readonly nativeLocalProviderAttachment?: NativeLocalProviderAttachment; readonly nativeNvidiaProviderAttachment?: NativeNvidiaProviderAttachment; readonly nativeHostedProviderAttachment?: NativeProviderAttachment; @@ -75,6 +78,7 @@ export type ResolveSandboxCreateIntentInput = { basePolicyPath: string; sandboxName: string; inferenceProvider?: string | null; + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; nativeNvidiaProviderAttachment?: NativeNvidiaProviderAttachment; nativeHostedProviderAttachment?: NativeProviderAttachment; diff --git a/src/lib/onboard/sandbox-create-intent.ts b/src/lib/onboard/sandbox-create-intent.ts index 7868a5530a1..e7bb21b1aaa 100644 --- a/src/lib/onboard/sandbox-create-intent.ts +++ b/src/lib/onboard/sandbox-create-intent.ts @@ -140,6 +140,7 @@ export function resolveSandboxCreateIntent({ basePolicyPath, sandboxName, inferenceProvider, + registeredInferenceProviderAttachment, nativeLocalProviderAttachment, nativeNvidiaProviderAttachment, nativeHostedProviderAttachment, @@ -190,6 +191,7 @@ export function resolveSandboxCreateIntent({ return { sandboxName, inferenceProvider: normalizedInferenceProvider, + ...(registeredInferenceProviderAttachment ? { registeredInferenceProviderAttachment } : {}), ...(nativeLocalProviderAttachment ? { nativeLocalProviderAttachment } : {}), ...(nativeNvidiaProviderAttachment ? { nativeNvidiaProviderAttachment } : {}), ...(nativeHostedProviderAttachment ? { nativeHostedProviderAttachment } : {}), diff --git a/src/lib/onboard/sandbox-create/orchestration-native-nvidia.test.ts b/src/lib/onboard/sandbox-create/orchestration-native-nvidia.test.ts index a03ed3a7371..a06de40b819 100644 --- a/src/lib/onboard/sandbox-create/orchestration-native-nvidia.test.ts +++ b/src/lib/onboard/sandbox-create/orchestration-native-nvidia.test.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../../test/helpers/registered-inference"; + import { describe, expect, it, vi } from "vitest"; import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; @@ -169,3 +171,45 @@ describe("native NVIDIA post-create provider verification", () => { expect(adapter.listProviderAttachments).not.toHaveBeenCalled(); }); }); + +it.each([false, true])( + "restores external provider membership without adopting its profile (attached=%s)", + async (attached) => { + const f = registeredInferenceFixture(attached); + const boundary = createProviderEffectBoundary({ + deferred: false, + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + expectedRegisteredInferenceProviderAttachment: f.binding, + preparationInput: { + openshellDriver: "docker", + inferenceProvider: f.binding.providerName, + messagingProviders: [], + messagingProviderRequests: [], + extraProviders: [], + gatewayName: f.binding.gatewayName, + }, + preparationDeps: { + providerAdapter: f.adapter, + runOpenshell: vi.fn() as never, + cleanupCreateSources: vi.fn(), + }, + runVerifiedSandboxCreateEffects: null, + activateDeferredProviderEffects: null, + revalidateSandboxIdentityBeforeCreate: vi.fn(), + }); + await boundary.validateBeforeCreate(); + await boundary.runAfterVerifiedCreate?.({ + ...verifiedCreateContext(), + gatewayName: f.binding.gatewayName, + }); + expect(f.adapter.attachProvider).toHaveBeenCalledTimes(attached ? 0 : 1); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.updateProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + f.state.id = "replacement"; + await expect(boundary.validateBeforeCreate()).rejects.toThrow(); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + }, +); diff --git a/src/lib/onboard/sandbox-create/orchestration.test.ts b/src/lib/onboard/sandbox-create/orchestration.test.ts index 26027d48175..5cc49542d94 100644 --- a/src/lib/onboard/sandbox-create/orchestration.test.ts +++ b/src/lib/onboard/sandbox-create/orchestration.test.ts @@ -10,11 +10,13 @@ import { describe, expect, it, vi } from "vitest"; import * as processRecovery from "../../actions/sandbox/process-recovery"; import { createHermesCredentialEnvReconciliationRuntime } from "../../actions/sandbox/runtime/hermes-lifecycle"; import type { SandboxEntry } from "../../state/registry"; +import { managedStartupStateRoots } from "../managed-startup/state-roots"; import { releaseManagedStartupHoldWithRetry } from "../managed-startup/provider-root-apply"; import { assertApfCreateIntent, activateManagedStartupCorporateCaTrustAfterSandboxCreate, completeHermesPortableSandboxRegistration, + finalizeRecreatedSourceStateVolumes, createProviderEffectBoundary, finalizeCreatedSandboxBeforeHermesCredentialReconciliation, hasManagedMcpRebuildHandoff, @@ -35,6 +37,76 @@ const UNVERIFIED_RECOVERY_CONTEXT = { lifecycleGeneration: "generation-1", createAttemptNonce: "a".repeat(62), } as const; +describe("recreated managed state retention reporting", () => { + it.each([ + [true, () => [{ status: "removed" as const }], ["cleanup", "registry"], null], + [ + true, + () => [{ status: "failed" as const, volumeName: "source", detail: "engine unavailable" }], + ["cleanup"], + expect.stringContaining("engine unavailable"), + ], + [ + true, + () => { + throw new Error("engine unavailable"); + }, + ["cleanup"], + "engine unavailable", + ], + [false, () => [{ status: "removed" as const }], [], null], + ] as const)( + "preserves registry authority for source absence %s and cleanup %j", + (sourceConfirmedAbsent, cleanup, expectedEvents, expectedError) => { + const events: string[] = []; + const targetStateRoots = managedStartupStateRoots({ + agent: "hermes", + sandboxName: "alpha", + agentIdentity: { uid: 1000, gid: 1000 }, + }); + let error: string | null = null; + try { + finalizeRecreatedSourceStateVolumes( + { + sandboxName: "alpha", + sourceConfirmedAbsent, + targetStateRoots, + sourceEntry: { + name: "alpha", + agent: "openclaw", + openshellDriver: "docker", + workload: { kind: "managed-image" }, + } as SandboxEntry, + }, + { + normalizeRuntimeProviderIdentity: (driver) => driver ?? "docker", + removeManagedAgentStateVolumes: (context) => { + expect(context).toEqual({ + agentName: "openclaw", + runtimeProviderId: "docker", + sandboxName: "alpha", + workloadKind: "managed-image", + retainedRoots: targetStateRoots, + }); + events.push("cleanup"); + return cleanup(); + }, + removeSourceRegistryEntry: () => { + events.push("registry"); + }, + note: vi.fn(), + warn: vi.fn(), + redact: (message) => message, + }, + ); + } catch (caught) { + error = (caught as Error).message; + } + expect(error).toEqual(expectedError); + expect(events).toEqual(expectedEvents); + }, + ); +}); describe("managed startup corporate CA onboarding orchestration", () => { it("wires the verified create boundary into the corporate CA refresh", async () => { const events: string[] = []; @@ -1254,125 +1326,52 @@ describe("sandbox create identity checks", () => { ]); }); - it("withholds checkpoint and effects when post-create boundary capture fails (#9833)", async () => { - const persistCreateIdentity = vi.fn(); - const runVerifiedCreateEffects = vi.fn(); - - const error = await runSandboxCreateWithIdentityVerification({ - sandboxName: "alpha", - revalidate: vi.fn(), - create: async (verifyCreatedSandbox) => { - await verifyCreatedSandbox("created"); - return "created"; - }, - captureCreatedSandboxIdentity: () => exactIdentity, - persistCreatedSandboxIdentity: vi.fn(), - revalidateCreatedSandboxIdentity: vi.fn(), - captureVerifiedCreateBoundary: () => { - throw new Error("identity boundary capture failed"); - }, - persistCreateIdentity, - revalidateVerifiedCreateIdentity: vi.fn(), - runVerifiedCreateEffects, - cleanupTemporarySources: vi.fn(), - }).catch((caught: unknown) => caught); - - expect(error).toBeInstanceOf(AggregateError); - expect((error as AggregateError).errors).toEqual( - expect.arrayContaining([ - expect.objectContaining({ - message: "identity boundary capture failed", - }), - ]), - ); - expect(persistCreateIdentity).not.toHaveBeenCalled(); - expect(runVerifiedCreateEffects).not.toHaveBeenCalled(); - }); - - it("withholds effects when durable checkpoint persistence fails (#9833)", async () => { - const revalidateVerifiedCreateIdentity = vi.fn(); - const runVerifiedCreateEffects = vi.fn(); - - await expect( - runSandboxCreateWithIdentityVerification({ - sandboxName: "alpha", - revalidate: vi.fn(), - create: async (verifyCreatedSandbox) => { - await verifyCreatedSandbox("created"); - return "created"; - }, - captureCreatedSandboxIdentity: () => exactIdentity, - persistCreatedSandboxIdentity: vi.fn(), - revalidateCreatedSandboxIdentity: vi.fn(), - captureVerifiedCreateBoundary: () => "verified", - persistCreateIdentity: () => { - throw new Error("checkpoint persistence failed"); - }, - revalidateVerifiedCreateIdentity, - runVerifiedCreateEffects, - cleanupTemporarySources: vi.fn(), - }), - ).rejects.toThrow("automatic sandbox cleanup was not safe"); - - expect(revalidateVerifiedCreateIdentity).not.toHaveBeenCalled(); - expect(runVerifiedCreateEffects).not.toHaveBeenCalled(); - }); - - it("retains the checkpoint and withholds effects when its reread fails (#9833)", async () => { - const persistCreateIdentity = vi.fn(); - const runVerifiedCreateEffects = vi.fn(); - - await expect( - runSandboxCreateWithIdentityVerification({ - sandboxName: "alpha", - revalidate: vi.fn(), - create: async (verifyCreatedSandbox) => { - await verifyCreatedSandbox("created"); - return "created"; - }, - captureCreatedSandboxIdentity: () => exactIdentity, - persistCreatedSandboxIdentity: vi.fn(), - revalidateCreatedSandboxIdentity: vi.fn(), - captureVerifiedCreateBoundary: () => "verified", - persistCreateIdentity, - revalidateVerifiedCreateIdentity: () => { - throw new Error("durable checkpoint missing"); - }, - runVerifiedCreateEffects, - cleanupTemporarySources: vi.fn(), - }), - ).rejects.toThrow("automatic sandbox cleanup was not safe"); - - expect(persistCreateIdentity).toHaveBeenCalledOnce(); - expect(runVerifiedCreateEffects).not.toHaveBeenCalled(); - }); - - it("retains the durable checkpoint when a deferred effect fails (#9833)", async () => { - const persistCreateIdentity = vi.fn(); - - await expect( - runSandboxCreateWithIdentityVerification({ + it.each([ + ["capture", "identity boundary capture failed", [1, 0, 0, 0]], + ["persist", "checkpoint persistence failed", [1, 1, 0, 0]], + ["reread", "durable checkpoint missing", [1, 1, 1, 0]], + ["effects", "provider effect failed", [1, 1, 1, 1]], + ] as const)( + "retains authority and withholds later effects when %s fails (#9833)", + async (failedPhase, message, expectedCalls) => { + const steps = { + capture: vi.fn(() => "verified"), + persist: vi.fn(), + reread: vi.fn(), + effects: vi.fn(), + }; + steps[failedPhase].mockImplementationOnce(() => { + throw new Error(message); + }); + const error = await runSandboxCreateWithIdentityVerification({ sandboxName: "alpha", revalidate: vi.fn(), create: async (verifyCreatedSandbox) => { await verifyCreatedSandbox("created"); return "created"; }, - captureCreatedSandboxIdentity: () => exactIdentity, - persistCreatedSandboxIdentity: vi.fn(), - revalidateCreatedSandboxIdentity: vi.fn(), - captureVerifiedCreateBoundary: () => "verified", - persistCreateIdentity, - revalidateVerifiedCreateIdentity: vi.fn(), + ...exactIdentityBoundary(), + captureVerifiedCreateBoundary: steps.capture, + persistCreateIdentity: steps.persist, + revalidateVerifiedCreateIdentity: steps.reread, runVerifiedCreateEffects: async () => { - throw new Error("provider effect failed"); + steps.effects(); }, cleanupTemporarySources: vi.fn(), - }), - ).rejects.toThrow("automatic sandbox cleanup was not safe"); - - expect(persistCreateIdentity).toHaveBeenCalledOnce(); - }); + }).catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(AggregateError); + expect((error as AggregateError).message).toContain("automatic sandbox cleanup was not safe"); + expect((error as AggregateError).errors).toEqual( + expect.arrayContaining([expect.objectContaining({ message })]), + ); + expect([ + steps.capture.mock.calls.length, + steps.persist.mock.calls.length, + steps.reread.mock.calls.length, + steps.effects.mock.calls.length, + ]).toEqual(expectedCalls); + }, + ); it("refuses continuation when identity changes during boundary capture (#9833)", async () => { const continuationEffect = vi.fn(); diff --git a/src/lib/onboard/sandbox-create/orchestration.ts b/src/lib/onboard/sandbox-create/orchestration.ts index 67a224f7904..b2511c77ae0 100644 --- a/src/lib/onboard/sandbox-create/orchestration.ts +++ b/src/lib/onboard/sandbox-create/orchestration.ts @@ -1,6 +1,12 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + attachRegisteredInferenceProvider, + detachRegisteredInferenceProvider, + registeredInferenceProviderRevision, +} from "./provider-publication"; + import { isDeepStrictEqual } from "node:util"; import fs from "node:fs"; @@ -50,11 +56,7 @@ import { assertProviderlessInterceptorEnvironment, enforceRemovedImmutabilityMigrationBoundary, } from "../entry-options"; -import type { - ManagedHermesStateVolumeCleanupResult, - ManagedHermesStateVolumeContext, -} from "../managed-workload/hermes-state-volume"; -import { removeManagedHermesStateVolume } from "../managed-workload/hermes-state-volume"; +import { removeManagedAgentStateVolumes } from "../managed-workload/agent-state-volumes"; import { createOnboardRecreateGatewayAuthorityRevalidator, shouldReconcileRestoredOpenClawSelection, @@ -64,7 +66,7 @@ import { managedImageRuntimeIdentity } from "../managed-image/agents"; import type { ProviderManagedStartupTransaction } from "../managed-startup/provider-root-apply"; import { managedStartupStateRoots, - MANAGED_HERMES_STATE_ROOT, + type ManagedStartupStateRoot, } from "../managed-startup/state-roots"; import type { SandboxGpuConfig } from "../sandbox-gpu-mode"; import { cliName } from "../branding"; @@ -945,63 +947,64 @@ type SandboxRecreateReasonInput = { existingSandboxState: string; }; -type RecreatedSourceHermesStateVolumeCleanupInput = { +type RecreatedSourceStateVolumeCleanupInput = { readonly sandboxName: string; readonly sourceEntry: SandboxEntry | null; - readonly targetKeepsManagedHermesStateVolume: boolean; + readonly targetStateRoots: readonly ManagedStartupStateRoot[]; }; -type RecreatedSourceHermesStateVolumeCleanupDeps = { +type RecreatedSourceStateVolumeCleanupDeps = { readonly normalizeRuntimeProviderIdentity: (driverName: string | null | undefined) => string; - readonly removeManagedHermesStateVolume: ( - context: ManagedHermesStateVolumeContext, - ) => ManagedHermesStateVolumeCleanupResult; + readonly removeManagedAgentStateVolumes: typeof removeManagedAgentStateVolumes; readonly note: (message: string) => void; readonly warn: (message: string) => void; readonly redact: (message: string) => string; }; -type RecreatedSourceHermesStateVolumeFinalizationInput = - RecreatedSourceHermesStateVolumeCleanupInput & { - readonly sourceConfirmedAbsent: boolean; - }; +type RecreatedSourceStateVolumeFinalizationInput = RecreatedSourceStateVolumeCleanupInput & { + readonly sourceConfirmedAbsent: boolean; +}; -type RecreatedSourceHermesStateVolumeFinalizationDeps = - RecreatedSourceHermesStateVolumeCleanupDeps & { - readonly removeSourceRegistryEntry: (entry: SandboxEntry, sandboxName: string) => void; - }; +type RecreatedSourceStateVolumeFinalizationDeps = RecreatedSourceStateVolumeCleanupDeps & { + readonly removeSourceRegistryEntry: (entry: SandboxEntry, sandboxName: string) => void; +}; -export function cleanupRecreatedSourceHermesStateVolume( - input: RecreatedSourceHermesStateVolumeCleanupInput, - deps: RecreatedSourceHermesStateVolumeCleanupDeps, +export function cleanupRecreatedSourceStateVolumes( + input: RecreatedSourceStateVolumeCleanupInput, + deps: RecreatedSourceStateVolumeCleanupDeps, ): void { - if (!input.sourceEntry || input.targetKeepsManagedHermesStateVolume) return; + if (!input.sourceEntry) return; - const cleanup = deps.removeManagedHermesStateVolume({ + const results = deps.removeManagedAgentStateVolumes({ agentName: input.sourceEntry.agent, runtimeProviderId: deps.normalizeRuntimeProviderIdentity(input.sourceEntry.openshellDriver), sandboxName: input.sandboxName, workloadKind: input.sourceEntry.workload?.kind ?? "", + retainedRoots: input.targetStateRoots, }); - if (cleanup.status === "failed") { - throw new Error( - `OpenShell confirmed that sandbox '${input.sandboxName}' is absent, but Docker could not remove its managed Hermes state volume '${cleanup.volumeName}': ${deps.redact(cleanup.detail)}. NemoClaw preserved the sandbox registry entry so a subsequent recreation can retry the volume removal.`, - ); - } - if (cleanup.status === "not-owned") { - deps.warn(` Left Docker volume '${cleanup.volumeName}' untouched because ${cleanup.detail}.`); - } else if (cleanup.status === "removed") { - deps.note(` Removed managed Hermes state volume for '${input.sandboxName}'.`); + for (const cleanup of results) { + if (cleanup.status === "failed") { + throw new Error( + `OpenShell confirmed that sandbox '${input.sandboxName}' is absent, but the runtime could not remove its managed state volume '${cleanup.volumeName}': ${deps.redact(cleanup.detail)}. NemoClaw preserved the sandbox registry entry so a subsequent recreation can retry the volume removal.`, + ); + } + if (cleanup.status === "not-owned") { + deps.warn( + ` Left managed state volume '${cleanup.volumeName}' untouched because ${cleanup.detail}.`, + ); + } else if (cleanup.status === "removed") { + deps.note(` Removed managed state volume for '${input.sandboxName}'.`); + } } } -export function finalizeRecreatedSourceHermesStateVolume( - input: RecreatedSourceHermesStateVolumeFinalizationInput, - deps: RecreatedSourceHermesStateVolumeFinalizationDeps, +export function finalizeRecreatedSourceStateVolumes( + input: RecreatedSourceStateVolumeFinalizationInput, + deps: RecreatedSourceStateVolumeFinalizationDeps, ): void { if (!input.sourceConfirmedAbsent || !input.sourceEntry) return; - cleanupRecreatedSourceHermesStateVolume(input, deps); + cleanupRecreatedSourceStateVolumes(input, deps); deps.removeSourceRegistryEntry(input.sourceEntry, input.sandboxName); } @@ -1440,6 +1443,9 @@ type ProviderPreparationDeps = Parameters< >[1]; type ProviderEffectBoundary = { + readonly registeredInferenceProviderAttachment: SandboxEntry["registeredInferenceProviderAttachment"]; + readonly markRegistrationPublished: () => void; + readonly withRegisteredAttachmentRollback: (operation: () => Promise) => Promise; readonly validateBeforeCreate: () => Promise; readonly publishBeforeCreate: () => Promise; readonly runAfterVerifiedCreate: @@ -1452,6 +1458,7 @@ export function createProviderEffectBoundary(input: { readonly resumingVerifiedCreate?: boolean; readonly sandboxName: string; readonly gatewayName: string; + readonly expectedRegisteredInferenceProviderAttachment?: SandboxEntry["registeredInferenceProviderAttachment"]; readonly expectedNativeLocalProviderAttachment?: SandboxEntry["nativeLocalProviderAttachment"]; readonly expectedNativeNvidiaProviderAttachment?: SandboxEntry["nativeNvidiaProviderAttachment"]; readonly expectedNativeHostedProviderAttachment?: SandboxEntry["nativeHostedProviderAttachment"]; @@ -1465,9 +1472,43 @@ export function createProviderEffectBoundary(input: { }): ProviderEffectBoundary { const adapter = resolveProviderAdapter(input.preparationDeps); const deps = { ...input.preparationDeps, providerAdapter: adapter }; + let registeredAttachment = input.expectedRegisteredInferenceProviderAttachment; + let rollbackRegisteredAttachment: (() => Promise) | undefined; + const attachmentLifecycle = { + markRegistrationPublished: () => { + rollbackRegisteredAttachment = undefined; + }, + withRegisteredAttachmentRollback: async (operation: () => Promise): Promise => { + try { + return await operation(); + } catch (error) { + try { + await rollbackRegisteredAttachment?.(); + } catch (rollbackError) { + throw new AggregateError( + [error, rollbackError], + "Sandbox creation failed and registered provider attachment cleanup was not confirmed.", + ); + } + throw error; + } + }, + }; let verifiedOwnedProfileIds: readonly string[] = []; let verifiedHermesToolGatewayProfileId: string | null = null; + const validateRegistered = async () => { + const expected = input.expectedRegisteredInferenceProviderAttachment; + if (!expected) return; + if ( + expected.sandboxName !== input.sandboxName || + expected.gatewayName !== input.gatewayName || + expected.providerName !== input.preparationInput.inferenceProvider + ) + throw new Error("Registered provider binding does not match sandbox creation."); + await registeredInferenceProviderRevision({ adapter, expected }); + }; const validate = async () => { + await validateRegistered(); await validateAttachedMessagingProvidersBeforeSandboxCreation(input.preparationInput, deps); if (!input.deferred && !input.resumingVerifiedCreate) { verifiedOwnedProfileIds = await validateAttachedOwnedProfilesBeforeSandboxCreation( @@ -1488,7 +1529,14 @@ export function createProviderEffectBoundary(input: { } }; const publish = async () => - publishAttachedProvidersBeforeDockerSandboxCreation(input.preparationInput, deps); + publishAttachedProvidersBeforeDockerSandboxCreation( + { + ...input.preparationInput, + registeredInferenceProviderName: + input.expectedRegisteredInferenceProviderAttachment?.providerName, + }, + deps, + ); const verifiedDirectCreateProfileIds = () => { const attachedNames = new Set([ ...(input.preparationInput.directCreateProviderNames ?? [ @@ -1513,6 +1561,31 @@ export function createProviderEffectBoundary(input: { const attachAndVerifyNativeInferenceProvider = async ( context: VerifiedSandboxCreateEffectsContext, ) => { + if (input.expectedRegisteredInferenceProviderAttachment) { + context.revalidateSandboxIdentity("restoring registered inference attachment"); + await validateRegistered(); + const receipt = await attachRegisteredInferenceProvider({ + adapter, + expected: input.expectedRegisteredInferenceProviderAttachment, + revalidateSandboxIdentity: context.revalidateSandboxIdentity, + }); + registeredAttachment = receipt.attachment; + if (receipt.changed) { + rollbackRegisteredAttachment = async () => { + context.revalidateSandboxIdentity( + "removing newly attached registered inference provider", + ); + await detachRegisteredInferenceProvider({ + adapter, + expected: receipt.attachment, + revalidateSandboxIdentity: context.revalidateSandboxIdentity, + }); + rollbackRegisteredAttachment = undefined; + }; + } + context.revalidateSandboxIdentity("verifying restored registered inference attachment"); + return; + } if (input.expectedNativeLocalProviderAttachment) { context.revalidateSandboxIdentity( `attaching native local provider for sandbox '${input.sandboxName}'`, @@ -1550,6 +1623,10 @@ export function createProviderEffectBoundary(input: { }; if (!input.deferred) { return { + ...attachmentLifecycle, + get registeredInferenceProviderAttachment() { + return registeredAttachment; + }, validateBeforeCreate: validate, publishBeforeCreate: async () => { input.revalidateSandboxIdentityBeforeCreate(); @@ -1568,7 +1645,11 @@ export function createProviderEffectBoundary(input: { }; } return { - validateBeforeCreate: async () => undefined, + ...attachmentLifecycle, + get registeredInferenceProviderAttachment() { + return registeredAttachment; + }, + validateBeforeCreate: validateRegistered, publishBeforeCreate: async () => undefined, runAfterVerifiedCreate: async (context) => { context.revalidateSandboxIdentity( @@ -2140,15 +2221,15 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche preferredInferenceApi, endpointUrl: createIntent?.endpointUrl ?? null, localNimSelected: createIntent?.localNimSelected === true, + registeredInferenceProviderAttachment: + resolvedCreateIntent.registeredInferenceProviderAttachment, nativeLocalProviderAttachment: resolvedCreateIntent.nativeLocalProviderAttachment, - nativeProvider: - createIntent?.localNimSelected === true - ? false - : Boolean( - resolvedCreateIntent.nativeLocalProviderAttachment || - resolvedCreateIntent.nativeHostedProviderAttachment || - resolvedCreateIntent.nativeNvidiaProviderAttachment, - ), + nativeProvider: Boolean( + resolvedCreateIntent.registeredInferenceProviderAttachment || + resolvedCreateIntent.nativeLocalProviderAttachment || + resolvedCreateIntent.nativeHostedProviderAttachment || + resolvedCreateIntent.nativeNvidiaProviderAttachment, + ), startupProfile: { chatUiUrl, effectiveDashboardPort: effectivePort, @@ -2207,22 +2288,22 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche workspace: process.env.OPENSHELL_WORKSPACE || "default", }); }; - const finalizeRecreatedSourceHermesVolume = ( + const finalizeRecreatedSourceVolumes = ( sourceConfirmedAbsent: boolean, sourceEntry: SandboxEntry | null, - targetKeepsManagedHermesStateVolume: boolean, + targetStateRoots: readonly ManagedStartupStateRoot[], ) => - finalizeRecreatedSourceHermesStateVolume( + finalizeRecreatedSourceStateVolumes( { sandboxName, sourceConfirmedAbsent, sourceEntry, - targetKeepsManagedHermesStateVolume, + targetStateRoots, }, { normalizeRuntimeProviderIdentity: managedWorkloadOnboard.normalizeRuntimeProviderIdentity, - removeManagedHermesStateVolume: (context) => - removeManagedHermesStateVolume(context, { + removeManagedAgentStateVolumes: (context) => + removeManagedAgentStateVolumes(context, { runtimeProvider: requireManagedWorkloadRuntime().runtimeProvider ?? undefined, }), removeSourceRegistryEntry: sandboxLifecycle.removeSandboxUnlessSessionReservation, @@ -2518,6 +2599,7 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche createIntent, nativeLocalProviderAttachment: resolvedCreateIntent.nativeLocalProviderAttachment, nativeProvider: Boolean( + existingSandboxEntry?.registeredInferenceProviderAttachment || existingSandboxEntry?.nativeLocalProviderAttachment || existingSandboxEntry?.nativeHostedProviderAttachment || existingSandboxEntry?.nativeNvidiaProviderAttachment, @@ -2789,13 +2871,7 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche // Keep the source route published through every pre-delete check. A failure // before confirmed deletion must leave the live source route untouched. await createIntent?.reservePublishedRouteAfterBackup?.(); - finalizeRecreatedSourceHermesVolume( - true, - previousEntry, - managedStateVolumeLifecycle.roots.some( - (root) => root.mountTarget === MANAGED_HERMES_STATE_ROOT, - ), - ); + finalizeRecreatedSourceVolumes(true, previousEntry, managedStateVolumeLifecycle.roots); await Promise.all([ dashboardPortReservationScope.rebindAfterOwnedForwardDelete(), hermesApiPortReservationScope.rebindAfterOwnedForwardDelete(hermesApiPortReservationInput), @@ -2809,13 +2885,7 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche await hermesApiPortReservationScope.selectAndReserve(hermesApiPortReservationInput); preparedSandboxWorkload = await ensurePreparedSandboxWorkload(); managedStateVolumeLifecycle = prepareManagedStateVolumeLifecycle(preparedSandboxWorkload); - finalizeRecreatedSourceHermesVolume( - !liveExists, - existingEntry, - managedStateVolumeLifecycle.roots.some( - (root) => root.mountTarget === MANAGED_HERMES_STATE_ROOT, - ), - ); + finalizeRecreatedSourceVolumes(!liveExists, existingEntry, managedStateVolumeLifecycle.roots); } if (!liveExists && existingEntry) { await createIntent?.reservePublishedRouteAfterBackup?.(); @@ -2885,6 +2955,7 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche createIntent?.endpointUrl ?? null, nativeProvider: Boolean( + resolvedCreateIntent.registeredInferenceProviderAttachment || resolvedCreateIntent.nativeLocalProviderAttachment || resolvedCreateIntent.nativeHostedProviderAttachment || resolvedCreateIntent.nativeNvidiaProviderAttachment, @@ -3684,6 +3755,9 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche { createIntent, resolvedCreateIntent, + getRegisteredInferenceProviderAttachment: () => + providerEffectBoundary.registeredInferenceProviderAttachment, + onRegistered: () => providerEffectBoundary.markRegistrationPublished(), reconcileOpenClawInference: onboardSession.loadSession()?.checkpoint?.sandboxRecreate?.reconcileOpenClawInference === true, @@ -3795,6 +3869,8 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche resumingVerifiedCreate: Boolean(resumeVerifiedCreateInput), sandboxName, gatewayName: GATEWAY_NAME, + expectedRegisteredInferenceProviderAttachment: + resolvedCreateIntent.registeredInferenceProviderAttachment, expectedNativeLocalProviderAttachment: resolvedCreateIntent.nativeLocalProviderAttachment, expectedNativeNvidiaProviderAttachment: resolvedCreateIntent.nativeNvidiaProviderAttachment, expectedNativeHostedProviderAttachment: resolvedCreateIntent.nativeHostedProviderAttachment, @@ -3810,181 +3886,183 @@ export function createSandboxWithBaseImageResolution(runtime: SandboxCreateOrche }); await providerEffectBoundary.validateBeforeCreate(); - if (hermesPortableAuthority) { - if (!portableRuntimeContext?.environmentScope) { - throw new Error("Hermes portable onboarding is missing runtime environment authority."); - } - if (managedStartupRootApplyRequest || !["none", "native-only"].includes(gpuRoutePlan)) { - throw new Error( - "Hermes portable onboarding cannot use managed startup root application or Docker GPU compatibility.", - ); - } - if (!inferenceRouteReservationAuthority?.sessionId) { - throw new Error( - "Hermes portable onboarding is missing current inference route reservation authority.", - ); - } - const inferenceRouteReservation = { - sessionId: inferenceRouteReservationAuthority.sessionId, - selection: inferenceRouteReservationAuthority.selection, - }; - await sandboxGpuCreateFlow.runHermesPortableOnboardingFromOnboard< - import("../sandbox-gpu-create-flow").SandboxGpuCreateFlowResult - >({ - sandboxName, - gatewayName: GATEWAY_NAME, - lifecycleGeneration: createdSandboxLifecycle.generation, - portableRuntime: portableRuntimeContext, - createRequest: finalizeOrdinaryCreateRequest({ + await providerEffectBoundary.withRegisteredAttachmentRollback(async () => { + if (hermesPortableAuthority) { + if (!portableRuntimeContext?.environmentScope) { + throw new Error("Hermes portable onboarding is missing runtime environment authority."); + } + if (managedStartupRootApplyRequest || !["none", "native-only"].includes(gpuRoutePlan)) { + throw new Error( + "Hermes portable onboarding cannot use managed startup root application or Docker GPU compatibility.", + ); + } + if (!inferenceRouteReservationAuthority?.sessionId) { + throw new Error( + "Hermes portable onboarding is missing current inference route reservation authority.", + ); + } + const inferenceRouteReservation = { + sessionId: inferenceRouteReservationAuthority.sessionId, + selection: inferenceRouteReservationAuthority.selection, + }; + await sandboxGpuCreateFlow.runHermesPortableOnboardingFromOnboard< + import("../sandbox-gpu-create-flow").SandboxGpuCreateFlowResult + >({ + sandboxName, + gatewayName: GATEWAY_NAME, + lifecycleGeneration: createdSandboxLifecycle.generation, + portableRuntime: portableRuntimeContext, + createRequest: finalizeOrdinaryCreateRequest({ + plan: createRequestPlan, + gatewayName: GATEWAY_NAME, + startupCommand: sandboxStartupCommand, + environment: createFlowEnvironment, + compatibilityPolicyPath, + compatibility: false, + rebuildPolicySourcePath: createIntent?.rebuildPolicySourcePath, + }), + createPolicyPath: initialSandboxPolicy.policyPath, + startup: { + agent: hermesPortableAuthority.agent, + sandboxName, + startupArgv: intendedSandboxStartupCommand, + }, + inferenceRouteReservation, + withLifecycleLock: sandboxGpuCreateFlow.bindHermesPortableOnboardingLifecycleLock( + sandboxMutationLock.withMcpLifecycleLock, + ), + childEnv: sandboxEnv, + openshellArgv, + createSandbox: ( + attemptRequest, + readyCapture, + readyRunner, + lifecycleCreateSandbox, + effectivePolicySourcePath, + ) => + runSandboxCreateWithProviderEffects({ + resumingVerifiedCreate: Boolean(resumeVerifiedCreateInput), + providerEffectBoundary, + create: (runAfterVerifiedCreate) => + runCreateFlow( + attemptRequest, + readyCapture, + readyRunner, + lifecycleCreateSandbox, + effectivePolicySourcePath, + runAfterVerifiedCreate, + ), + }), + readRegistry: () => registry.getSandbox(sandboxName), + revalidatePendingCreateRegistry: () => + revalidateVerifiedCreateIdentity( + requireVerifiedCreateBoundary(), + `requalify verified create checkpoint for sandbox '${sandboxName}'`, + ), + compareAndSetRegistryGatewayPort: registry.compareAndSetSandboxGatewayPort, + registerSandbox: async ( + created, + receipt, + liveIdentityFingerprint, + revalidate, + routeReservation, + ) => + completeHermesPortableSandboxRegistration({ + sandboxName, + completeRegistration: () => + completeCreatedSandboxRegistration( + created, + receipt, + liveIdentityFingerprint, + revalidate, + routeReservation, + ), + readRegistry: registry.getSandbox, + }), + sourceRoot: ROOT, + buildContextSettings: { + nativeEndpointUrl: resolvedCreateIntent.nativeLocalProviderAttachment?.endpointUrl, + model, + provider, + preferredInferenceApi, + toolDisclosure: effectiveToolDisclosure, + }, + cleanupTemporaryPolicy: cleanupInitialCreateSource, + createPolicySourceBytes: initialSandboxPolicy.sourceBytes, + }); + cleanupBuildContext(); + } else { + const routedCreateRequest = finalizeOrdinaryCreateRequest({ plan: createRequestPlan, gatewayName: GATEWAY_NAME, startupCommand: sandboxStartupCommand, environment: createFlowEnvironment, compatibilityPolicyPath, - compatibility: false, + compatibility: initialGpuRoute === "compatibility", rebuildPolicySourcePath: createIntent?.rebuildPolicySourcePath, - }), - createPolicyPath: initialSandboxPolicy.policyPath, - startup: { - agent: hermesPortableAuthority.agent, - sandboxName, - startupArgv: intendedSandboxStartupCommand, - }, - inferenceRouteReservation, - withLifecycleLock: sandboxGpuCreateFlow.bindHermesPortableOnboardingLifecycleLock( - sandboxMutationLock.withMcpLifecycleLock, - ), - childEnv: sandboxEnv, - openshellArgv, - createSandbox: ( - attemptRequest, - readyCapture, - readyRunner, - lifecycleCreateSandbox, - effectivePolicySourcePath, - ) => - runSandboxCreateWithProviderEffects({ - resumingVerifiedCreate: Boolean(resumeVerifiedCreateInput), - providerEffectBoundary, - create: (runAfterVerifiedCreate) => - runCreateFlow( - attemptRequest, - readyCapture, - readyRunner, - lifecycleCreateSandbox, - effectivePolicySourcePath, - runAfterVerifiedCreate, - ), - }), - readRegistry: () => registry.getSandbox(sandboxName), - revalidatePendingCreateRegistry: () => - revalidateVerifiedCreateIdentity( - requireVerifiedCreateBoundary(), - `requalify verified create checkpoint for sandbox '${sandboxName}'`, - ), - compareAndSetRegistryGatewayPort: registry.compareAndSetSandboxGatewayPort, - registerSandbox: async ( - created, - receipt, - liveIdentityFingerprint, - revalidate, - routeReservation, - ) => - completeHermesPortableSandboxRegistration({ - sandboxName, - completeRegistration: () => - completeCreatedSandboxRegistration( - created, - receipt, - liveIdentityFingerprint, - revalidate, - routeReservation, - ), - readRegistry: registry.getSandbox, - }), - sourceRoot: ROOT, - buildContextSettings: { - nativeEndpointUrl: resolvedCreateIntent.nativeLocalProviderAttachment?.endpointUrl, - model, - provider, - preferredInferenceApi, - toolDisclosure: effectiveToolDisclosure, - }, - cleanupTemporaryPolicy: cleanupInitialCreateSource, - createPolicySourceBytes: initialSandboxPolicy.sourceBytes, - }); - cleanupBuildContext(); - } else { - const routedCreateRequest = finalizeOrdinaryCreateRequest({ - plan: createRequestPlan, - gatewayName: GATEWAY_NAME, - startupCommand: sandboxStartupCommand, - environment: createFlowEnvironment, - compatibilityPolicyPath, - compatibility: initialGpuRoute === "compatibility", - rebuildPolicySourcePath: createIntent?.rebuildPolicySourcePath, - }); - try { - const created = await activateManagedStartupCorporateCaTrustAfterSandboxCreate({ - create: runSandboxCreateWithProviderEffects({ - resumingVerifiedCreate: Boolean(resumeVerifiedCreateInput), - providerEffectBoundary, - create: (runAfterVerifiedCreate) => - runCreateFlow( - routedCreateRequest, - undefined, - undefined, - undefined, - undefined, - runAfterVerifiedCreate, - ), - }), - corporateCaB64: managedStartupRootApplyRequest?.corporateCaB64 ?? null, - sandboxName, - requireVerifiedCreateBoundary, - refreshCorporateCaTrust: (request) => - managedWorkloadOnboard.refreshManagedStartupCorporateCaTrust(request), - revalidateSandboxIdentity: (boundary, operation) => { - revalidateVerifiedCreateIdentity(boundary, operation); - }, - recordRecovery: () => recordPostCreateRecovery("onboarding finalization"), - noteActivation: () => - console.log(" Activating corporate CA trust in the OpenShell supervisor..."), }); - await finalizeCreatedSandboxBeforeHermesCredentialReconciliation( - async () => { - const registration = await runAsyncWithPostCreateRecovery( - () => completeCreatedSandboxRegistration(created, null), - () => recordPostCreateRecovery("registry publication"), - ); - if ( - managedStartupProtocol && - !registry.updateSandbox(sandboxName, { managedStartupProtocol }) - ) { - throw new Error( - `Sandbox '${sandboxName}' registered without its managed startup protocol.`, + try { + const created = await activateManagedStartupCorporateCaTrustAfterSandboxCreate({ + create: runSandboxCreateWithProviderEffects({ + resumingVerifiedCreate: Boolean(resumeVerifiedCreateInput), + providerEffectBoundary, + create: (runAfterVerifiedCreate) => + runCreateFlow( + routedCreateRequest, + undefined, + undefined, + undefined, + undefined, + runAfterVerifiedCreate, + ), + }), + corporateCaB64: managedStartupRootApplyRequest?.corporateCaB64 ?? null, + sandboxName, + requireVerifiedCreateBoundary, + refreshCorporateCaTrust: (request) => + managedWorkloadOnboard.refreshManagedStartupCorporateCaTrust(request), + revalidateSandboxIdentity: (boundary, operation) => { + revalidateVerifiedCreateIdentity(boundary, operation); + }, + recordRecovery: () => recordPostCreateRecovery("onboarding finalization"), + noteActivation: () => + console.log(" Activating corporate CA trust in the OpenShell supervisor..."), + }); + await finalizeCreatedSandboxBeforeHermesCredentialReconciliation( + async () => { + const registration = await runAsyncWithPostCreateRecovery( + () => completeCreatedSandboxRegistration(created, null), + () => recordPostCreateRecovery("registry publication"), ); - } - createEffectsFinalized = true; - return registration; - }, - () => - reconcileCreatedHermesCredentialEnvironment( - { - sandboxName, - plan: plannedMessagingState?.plan ?? null, - }, - createHermesCredentialEnvReconciliationRuntime( - (args, options) => runOpenshell([...args], options), - (operation) => revalidateSandboxIdentity(true, operation), + if ( + managedStartupProtocol && + !registry.updateSandbox(sandboxName, { managedStartupProtocol }) + ) { + throw new Error( + `Sandbox '${sandboxName}' registered without its managed startup protocol.`, + ); + } + createEffectsFinalized = true; + return registration; + }, + () => + reconcileCreatedHermesCredentialEnvironment( + { + sandboxName, + plan: plannedMessagingState?.plan ?? null, + }, + createHermesCredentialEnvReconciliationRuntime( + (args, options) => runOpenshell([...args], options), + (operation) => revalidateSandboxIdentity(true, operation), + ), + () => recordPostCreateRecovery("onboarding finalization"), ), - () => recordPostCreateRecovery("onboarding finalization"), - ), - ); - } finally { - cleanupInitialCreateSource(); + ); + } finally { + cleanupInitialCreateSource(); + } } - } + }); return runAsyncWithPostCreateRecovery( async () => { managedStateVolumeLifecycle.commit(); diff --git a/src/lib/onboard/sandbox-create/provider-publication.test.ts b/src/lib/onboard/sandbox-create/provider-publication.test.ts index 3775199e02f..94fed3c1298 100644 --- a/src/lib/onboard/sandbox-create/provider-publication.test.ts +++ b/src/lib/onboard/sandbox-create/provider-publication.test.ts @@ -5,6 +5,7 @@ import { describe, expect, it, vi } from "vitest"; import type { OpenShellProviderAdapter } from "../../adapters/openshell/provider-adapter"; import { endpointlessProviderProfilePath } from "../../adapters/openshell/provider-profile"; +import { credentialFamilyProfileType } from "../../messaging/applier/credential-family-profile"; import { REPOSITORY_ROOT } from "../../core/repository-root"; import { nativeNvidiaProviderProfilePath } from "../../inference/native-nvidia"; import { @@ -221,7 +222,18 @@ describe("sandbox provider preparation", () => { ); it.each( - ["langfuse-hermes-v1", "nemoclaw-nvidia-inference-v1"].flatMap((type) => + [ + { type: "langfuse-hermes-v1", credentialKeys: ["TOKEN"] }, + { type: "nemoclaw-nvidia-inference-v1", credentialKeys: ["TOKEN"] }, + { + type: credentialFamilyProfileType( + MESSAGING_CREDENTIAL_PROVIDER_TYPE, + "TELEGRAM_BOT_TOKEN", + ["TELEGRAM_BOT_TOKEN", "TELEGRAM_BOT_TOKEN_WORKSPACE"], + ), + credentialKeys: ["TELEGRAM_BOT_TOKEN", "TELEGRAM_BOT_TOKEN_WORKSPACE"], + }, + ].flatMap(({ type, credentialKeys }) => ( [ ["profile_incompatible", "Resolve the conflicting same-ID profile"], @@ -230,18 +242,19 @@ describe("sandbox provider preparation", () => { ] as const ).map(([reason, recovery]) => ({ type, + credentialKeys, reason, recovery, })), ), )( "blocks $type on $reason and verifies the profile on retry (#13345)", - async ({ type, reason, recovery }) => { + async ({ type, credentialKeys, reason, recovery }) => { const secret = "unrecognized-private-provider-value"; const adapter = typedProviderAdapter({ getProvider: vi.fn(async () => ({ ok: true as const, - value: { name: "extra-provider", type, credentialKeys: ["TOKEN"], configKeys: [] }, + value: { name: "extra-provider", type, credentialKeys, configKeys: [] }, })), importProviderProfile: vi .fn() @@ -604,6 +617,11 @@ describe("sandbox provider preparation", () => { value: { ...exactMetadata, name: request.providerName, + type: credentialFamilyProfileType( + MESSAGING_CREDENTIAL_PROVIDER_TYPE, + "TELEGRAM_BOT_TOKEN", + ["TELEGRAM_BOT_TOKEN", "TELEGRAM_BOT_TOKEN_WORKSPACE"], + ), credentialKeys: ["TELEGRAM_BOT_TOKEN", "TELEGRAM_BOT_TOKEN_WORKSPACE"], }, })); diff --git a/src/lib/onboard/sandbox-create/provider-publication.ts b/src/lib/onboard/sandbox-create/provider-publication.ts index 1894fb72c23..03ca7125a4a 100644 --- a/src/lib/onboard/sandbox-create/provider-publication.ts +++ b/src/lib/onboard/sandbox-create/provider-publication.ts @@ -27,6 +27,12 @@ import { endpointlessProviderProfilePath } from "../../adapters/openshell/provid import { namedOpenShellGateway } from "../../adapters/openshell/sandbox-observer"; import { REPOSITORY_ROOT } from "../../core/repository-root"; import { messagingProviderProfileForType } from "../../messaging/applier/provider-application"; +import { + credentialFamilyBaseType, + credentialFamilyPrimary, + credentialFamilyProfileContract, + withCredentialFamilyProfile, +} from "../../messaging/applier/credential-family-profile"; import { ensureNativeNvidiaProviderAttached, nativeNvidiaProviderProfilePath, @@ -48,6 +54,7 @@ type ProviderPreparationInput = { readonly openshellDriver: SandboxEntry["openshellDriver"]; readonly inferenceProvider: string | null; readonly transactionBoundInferenceProvider?: string | null; + readonly registeredInferenceProviderName?: string; readonly messagingProviders: readonly string[]; readonly messagingProviderRequests: SandboxCreateIntent["messagingProviderRequests"]; readonly extraProviders: readonly string[]; @@ -129,10 +136,6 @@ export async function validateAttachedOwnedProfilesBeforeSandboxCreation( null; const profileIds = new Set(); for (const providerName of new Set(providerNames)) { - if (expectedMessaging.has(providerName)) { - profileIds.add(MESSAGING_CREDENTIAL_PROVIDER_TYPE); - continue; - } const observed = await adapter.getProvider({ target, providerName }); if (!observed.ok) { throwAfterCleanup( @@ -141,11 +144,9 @@ export async function validateAttachedOwnedProfilesBeforeSandboxCreation( ); } if (observed.value.type === "generic") continue; - let directProfile = messagingProviderProfileForType( - REPOSITORY_ROOT, - observed.value.type, - undefined, - ); + const baseType = + expectedMessaging.get(providerName)?.type ?? credentialFamilyBaseType(observed.value.type); + let directProfile = messagingProviderProfileForType(REPOSITORY_ROOT, baseType, undefined); if (observed.value.type === "langfuse-hermes-v1") { directProfile = { profileType: observed.value.type, @@ -168,23 +169,42 @@ export async function validateAttachedOwnedProfilesBeforeSandboxCreation( } const profile = directProfile ?? - messagingProviderProfileForType( - REPOSITORY_ROOT, - observed.value.type, - bridgeProfiles?.get(observed.value.type), - ); + messagingProviderProfileForType(REPOSITORY_ROOT, baseType, bridgeProfiles?.get(baseType)); if (!profile) continue; - const imported = await adapter.importProviderProfile({ - target, - profilePath: profile.profilePath, - }); + const primary = + expectedMessaging.get(providerName)?.credentialKey ?? + credentialFamilyPrimary(observed.value.credentialKeys); + if (observed.value.type !== baseType && !primary) { + throwAfterCleanup(deps, `Provider '${providerName}' has invalid credential names.`); + } + const imported = + observed.value.type !== baseType && primary + ? await withCredentialFamilyProfile( + profile.profilePath, + primary, + observed.value.credentialKeys, + (generated) => { + if (generated.profileType !== observed.value.type) { + throw new Error( + `Provider '${providerName}' has an invalid credential profile identity.`, + ); + } + return adapter.importProviderProfile({ target, profilePath: generated.profilePath }); + }, + ).catch(() => { + throwAfterCleanup( + deps, + `Could not verify the credential profile for provider '${providerName}'.`, + ); + }) + : await adapter.importProviderProfile({ target, profilePath: profile.profilePath }); if (!imported.ok) { throwAfterCleanup( deps, - `Could not verify provider profile '${profile.profileType}' before sandbox creation. ${ownedProfileFailureDetail(imported)}`, + `Could not verify provider profile '${observed.value.type}' before sandbox creation. ${ownedProfileFailureDetail(imported)}`, ); } - profileIds.add(profile.profileType); + profileIds.add(observed.value.type); } return [...profileIds]; } @@ -352,11 +372,23 @@ async function inspectExpectedMessagingBinding( providerName, }); if (!result.ok) return { kind: "failed", error: result.error }; - return { - kind: matchesGatewayCredentialFamilyProviderBinding(result.value, expected) - ? "exact" - : "mismatch", - }; + if (!matchesGatewayCredentialFamilyProviderBinding(result.value, expected)) { + return { kind: "mismatch" }; + } + if (result.value.type !== expected.type) { + const expectedProfile = credentialFamilyProfileContract( + messagingCredentialProviderProfilePath(REPOSITORY_ROOT), + expected.credentialKey, + result.value.credentialKeys, + ); + const inspected = await adapter.inspectProviderProfile({ + target: namedOpenShellGateway(input.gatewayName), + profileType: result.value.type, + expectedProfile, + }); + if (!inspected.ok) return { kind: "failed", error: inspected.error }; + } + return { kind: "exact" }; } function throwAfterCleanup(deps: ProviderPreparationDeps, message: string): never { @@ -453,7 +485,11 @@ export async function publishAttachedProvidersBeforeDockerSandboxCreation( ...providersRequiringExistenceProbe, ...input.messagingProviders, ...input.extraProviders, - ].filter((provider) => provider !== input.transactionBoundInferenceProvider), + ].filter( + (provider) => + provider !== input.transactionBoundInferenceProvider && + provider !== input.registeredInferenceProviderName, + ), ); const adapter = resolveProviderAdapter(deps); const target = namedOpenShellGateway(input.gatewayName); @@ -488,3 +524,9 @@ export async function publishAttachedProvidersBeforeDockerSandboxCreation( ); } } + +export { + attachRegisteredInferenceProvider, + detachRegisteredInferenceProvider, + registeredInferenceProviderRevision, +} from "../../inference/native-provider/registered"; diff --git a/src/lib/onboard/sandbox-create/registered-provider-recreation.test.ts b/src/lib/onboard/sandbox-create/registered-provider-recreation.test.ts new file mode 100644 index 00000000000..dd98e640820 --- /dev/null +++ b/src/lib/onboard/sandbox-create/registered-provider-recreation.test.ts @@ -0,0 +1,300 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { + registeredInferenceFixture, + registeredIdentityFailureAt, +} from "../../../../test/helpers/registered-inference"; +import { createProviderEffectBoundary } from "./orchestration"; + +describe("registered provider recreation", () => { + it.each([ + ["ordinary", false, false], + ["deferred", true, false], + ["resumed", true, true], + ] as const)( + "preserves operator ownership and attachment receipt during %s creation", + async (_route, deferred, resumingVerifiedCreate) => { + const f = registeredInferenceFixture(); + f.adapter.updateProvider.mockResolvedValue({ ok: true }); + const revalidateSandboxIdentity = vi.fn(); + const boundary = createProviderEffectBoundary({ + deferred, + resumingVerifiedCreate, + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + expectedRegisteredInferenceProviderAttachment: f.binding, + preparationInput: { + openshellDriver: "docker", + inferenceProvider: f.binding.providerName, + messagingProviders: ["messaging-extra"], + messagingProviderRequests: [], + extraProviders: [f.binding.providerName, "owned-extra"], + directCreateProviderNames: [], + gatewayName: f.binding.gatewayName, + }, + preparationDeps: { + runOpenshell: vi.fn(), + cleanupCreateSources: vi.fn(), + providerAdapter: f.adapter, + }, + runVerifiedSandboxCreateEffects: null, + activateDeferredProviderEffects: async () => [], + revalidateSandboxIdentityBeforeCreate: revalidateSandboxIdentity, + }); + await boundary.validateBeforeCreate(); + expect(f.adapter.getProvider).toHaveBeenCalled(); + expect(f.adapter.inspectProviderProfile).toHaveBeenCalled(); + await boundary.publishBeforeCreate(); + await boundary.runAfterVerifiedCreate!({ + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + gatewayPort: 18080, + lifecycleGeneration: "generation-1", + lifecycleLiveIdentityFingerprint: "a".repeat(64), + route: "direct" as never, + revalidateSandboxIdentity, + }); + expect(f.adapter.updateProvider.mock.calls.map(([request]) => request.providerName)).toEqual([ + "messaging-extra", + "owned-extra", + ]); + expect(f.adapter.attachProvider).toHaveBeenCalledExactlyOnceWith({ + target: { kind: "named", gatewayName: f.binding.gatewayName }, + sandboxName: f.binding.sandboxName, + providerName: f.binding.providerName, + }); + expect(f.state.attached).toBe(true); + expect(boundary.registeredInferenceProviderAttachment).toEqual({ + ...f.binding, + attachedBySelection: true, + }); + expect(f.binding.attachedBySelection).toBe(false); + expect(f.adapter.createProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + expect(f.adapter.importProviderProfile).not.toHaveBeenCalled(); + }, + ); +}); + +function rollbackFixture( + attached: boolean, + deferred: boolean, + resumingVerifiedCreate: boolean, + previouslyOwned = false, +) { + const f = registeredInferenceFixture(attached); + const revalidateSandboxIdentity = vi.fn(); + const boundary = createProviderEffectBoundary({ + deferred, + resumingVerifiedCreate, + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + expectedRegisteredInferenceProviderAttachment: { + ...f.binding, + attachedBySelection: previouslyOwned, + }, + preparationInput: { + openshellDriver: "docker", + inferenceProvider: f.binding.providerName, + messagingProviders: [], + messagingProviderRequests: [], + extraProviders: [], + directCreateProviderNames: [], + gatewayName: f.binding.gatewayName, + }, + preparationDeps: { + runOpenshell: vi.fn(), + cleanupCreateSources: vi.fn(), + providerAdapter: f.adapter, + }, + runVerifiedSandboxCreateEffects: null, + activateDeferredProviderEffects: async () => [], + revalidateSandboxIdentityBeforeCreate: revalidateSandboxIdentity, + }); + const create = () => + boundary.runAfterVerifiedCreate!({ + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + gatewayPort: 18080, + lifecycleGeneration: "generation-1", + lifecycleLiveIdentityFingerprint: "a".repeat(64), + route: "native", + revalidateSandboxIdentity, + }); + return { ...f, boundary, revalidateSandboxIdentity, create }; +} + +function expectGuardedDetach(f: ReturnType, rollbackOperation: string) { + const checkOrder = (operation: string) => { + const index = f.revalidateSandboxIdentity.mock.calls.findIndex(([name]) => name === operation); + expect(index, operation).toBeGreaterThanOrEqual(0); + return f.revalidateSandboxIdentity.mock.invocationCallOrder[index]!; + }; + const rollbackOrder = checkOrder(rollbackOperation); + const beforeDetach = checkOrder("detach registered inference provider"); + const afterDetach = checkOrder("verify registered inference provider detach"); + expect(f.adapter.detachProvider).toHaveBeenCalledOnce(); + const detachOrder = f.adapter.detachProvider.mock.invocationCallOrder[0]!; + const absenceOrder = f.adapter.listProviderAttachments.mock.invocationCallOrder.at(-1)!; + expect(rollbackOrder).toBeLessThan(beforeDetach); + expect(beforeDetach).toBeLessThan(detachOrder); + expect(detachOrder).toBeLessThan(absenceOrder); + expect(absenceOrder).toBeLessThan(afterDetach); + expect(f.state.attached).toBe(false); +} + +describe.each([ + ["ordinary", false, false], + ["deferred", true, false], + ["resumed", true, true], +] as const)("failed %s creation", (_route, deferred, resuming) => { + it("rolls back newly selected access after revalidating identity", async () => { + const f = rollbackFixture(false, deferred, resuming); + const failure = new Error("registration failed"); + await expect( + f.boundary.withRegisteredAttachmentRollback(async () => { + await f.create(); + throw failure; + }), + ).rejects.toBe(failure); + expectGuardedDetach(f, "removing newly attached registered inference provider"); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + }); + + it("preserves preexisting access", async () => { + const f = rollbackFixture(true, deferred, resuming); + const failure = new Error("registration failed"); + await expect( + f.boundary.withRegisteredAttachmentRollback(async () => { + await f.create(); + throw failure; + }), + ).rejects.toBe(failure); + expect(f.state.attached).toBe(true); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + }); +}); + +it.each([ + { + identity: "sandbox", + replace: (f: ReturnType) => { + f.revalidateSandboxIdentity.mockImplementation(() => { + throw new Error("sandbox changed"); + }); + }, + }, + { + identity: "provider", + replace: (f: ReturnType) => { + f.state.id = "replacement-provider"; + }, + }, +])( + "retains access when $identity identity becomes uncertain before rollback", + async ({ replace }) => { + const f = rollbackFixture(false, false, false); + await expect( + f.boundary.withRegisteredAttachmentRollback(async () => { + await f.create(); + replace(f); + throw new Error("registration failed"); + }), + ).rejects.toThrow("cleanup was not confirmed"); + expect(f.state.attached).toBe(true); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); + }, +); + +it("keeps published attachment ownership when later finalization fails", async () => { + const f = rollbackFixture(false, false, false); + const failure = new Error("retirement failed after registration"); + await expect( + f.boundary.withRegisteredAttachmentRollback(async () => { + await f.create(); + f.boundary.markRegistrationPublished(); + throw failure; + }), + ).rejects.toBe(failure); + expect(f.state.attached).toBe(true); + expect(f.boundary.registeredInferenceProviderAttachment?.attachedBySelection).toBe(true); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); +}); + +it("does not roll back an attachment owned by an earlier selection", async () => { + const f = rollbackFixture(true, true, true, true); + await expect( + f.boundary.withRegisteredAttachmentRollback(async () => { + await f.create(); + throw new Error("registration failed"); + }), + ).rejects.toThrow("registration failed"); + expect(f.boundary.registeredInferenceProviderAttachment?.attachedBySelection).toBe(true); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); +}); + +it("rechecks identity before cleanup when post-attachment verification fails", async () => { + const f = rollbackFixture(false, false, false); + const operation = "verifying restored registered inference attachment"; + const failure = new Error("identity check unavailable"); + f.revalidateSandboxIdentity.mockImplementation(registeredIdentityFailureAt(operation, failure)); + await expect(f.boundary.withRegisteredAttachmentRollback(f.create)).rejects.toBe(failure); + expect(f.adapter.attachProvider).toHaveBeenCalledOnce(); + const failureIndex = f.revalidateSandboxIdentity.mock.calls.findIndex( + ([name]) => name === operation, + ); + expect(failureIndex).toBeGreaterThanOrEqual(0); + expect(f.revalidateSandboxIdentity.mock.results[failureIndex]).toEqual({ + type: "throw", + value: failure, + }); + expect(f.adapter.attachProvider.mock.invocationCallOrder[0]!).toBeLessThan( + f.revalidateSandboxIdentity.mock.invocationCallOrder[failureIndex]!, + ); + expect(f.revalidateSandboxIdentity.mock.invocationCallOrder[failureIndex]!).toBeLessThan( + f.adapter.detachProvider.mock.invocationCallOrder[0]!, + ); + expectGuardedDetach(f, "removing newly attached registered inference provider"); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); +}); + +it("revalidates the sandbox before rolling back an unverified attachment", async () => { + const f = rollbackFixture(false, false, false); + f.adapter.attachProvider.mockImplementation(async () => { + f.state.attached = true; + f.state.revision++; + return { ok: true }; + }); + await expect(f.boundary.withRegisteredAttachmentRollback(f.create)).rejects.toThrow( + "changed during attachment", + ); + expectGuardedDetach(f, "rolling back unverified registered inference attachment"); + expect(f.adapter.detachProvider).toHaveBeenCalledOnce(); + expect(f.state.attached).toBe(false); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); +}); + +it("retains an unverified attachment when the sandbox was replaced before rollback", async () => { + const f = rollbackFixture(false, false, false); + f.adapter.attachProvider.mockImplementation(async () => { + f.state.attached = true; + f.state.revision++; + f.revalidateSandboxIdentity.mockImplementation(() => { + throw new Error("sandbox replaced"); + }); + return { ok: true }; + }); + await expect(f.boundary.withRegisteredAttachmentRollback(f.create)).rejects.toThrow( + "sandbox replaced", + ); + expect(f.revalidateSandboxIdentity).toHaveBeenLastCalledWith( + "rolling back unverified registered inference attachment", + ); + expect(f.adapter.detachProvider).not.toHaveBeenCalled(); + expect(f.state.attached).toBe(true); + expect(f.adapter.deleteProvider).not.toHaveBeenCalled(); +}); diff --git a/src/lib/onboard/sandbox-gpu-create-attempt.ts b/src/lib/onboard/sandbox-gpu-create-attempt.ts index c94ed464738..a6ccb40dec9 100644 --- a/src/lib/onboard/sandbox-gpu-create-attempt.ts +++ b/src/lib/onboard/sandbox-gpu-create-attempt.ts @@ -22,6 +22,44 @@ import { STABLE_ABSENCE_CHECKS, } from "./sandbox-gpu-fallback-constants"; +import { redactOnboardCommandDiagnosticText } from "./diagnostics/redaction"; +import { + DOCKER_GPU_PATCH_STOP_TIMEOUT_MS, + DOCKER_GPU_PATCH_TIMEOUT_MS, +} from "./docker-gpu-patch-constants"; +import type { DockerGpuPatchDeps } from "./docker-gpu-patch-types"; + +export function requireCompatibilityLifecycleCommand( + action: "start" | "stop", + sandboxName: string, + gatewayName: string, + deps: { runOpenshell: NonNullable }, +): void { + const result = deps.runOpenshell(["sandbox", action, "-g", gatewayName, sandboxName], { + ignoreError: true, + killProcessTreeOnTimeout: true, + killSignal: "SIGKILL", + suppressOutput: true, + // Lifecycle commands wait for a state transition, unlike readiness reads. + timeout: action === "stop" ? DOCKER_GPU_PATCH_STOP_TIMEOUT_MS : DOCKER_GPU_PATCH_TIMEOUT_MS, + }); + if (Number(result.status ?? 1) !== 0 || result.error || ("signal" in result && result.signal)) { + const detail = redactOnboardCommandDiagnosticText( + [ + `status=${String(result.status)}, signal=${String("signal" in result ? result.signal : null)}`, + result.error?.message, + result.stderr, + result.stdout, + ] + .filter(Boolean) + .join("\n"), + ).slice(0, 2_000); + throw new Error( + `OpenShell could not ${action} sandbox '${sandboxName}' for the exact compatibility cutover.\n${detail}`, + ); + } +} + export type SandboxGpuCreateFailureStage = "create" | "readiness" | "gpu-proof"; export { getSandboxFailurePhase, hasSandboxListEntry, isSandboxReady }; diff --git a/src/lib/onboard/sandbox-gpu-create-failure-classification.test.ts b/src/lib/onboard/sandbox-gpu-create-failure-classification.test.ts index 9a81f8ca9e1..da73085eddc 100644 --- a/src/lib/onboard/sandbox-gpu-create-failure-classification.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-failure-classification.test.ts @@ -1,7 +1,66 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => ({ + streamSandboxCreate: vi.fn(), + waitForCreatedSandboxReadyWithTrace: vi.fn(), + printReadinessFailure: vi.fn(), + enforceDockerGpuPatchPreserveNetwork: vi.fn(), + verifyGpuSandboxAccessAfterReady: vi.fn(), + createDockerGpuSandboxCreatePatch: vi.fn(), + printSandboxCreateFailureDiagnostics: vi.fn(), + collectDockerGpuPatchDiagnostics: vi.fn(), + queryOpenShellDockerSandboxContainers: vi.fn(), + queryOpenShellDockerSandboxRuntimeSnapshot: vi.fn(), +})); + +vi.mock("../sandbox/create-stream", () => ({ + streamSandboxCreate: mocks.streamSandboxCreate, +})); + +vi.mock("./sandbox-readiness-tracing", async (importOriginal) => ({ + ...(await importOriginal()), + waitForCreatedSandboxReadyWithTrace: mocks.waitForCreatedSandboxReadyWithTrace, + printReadinessFailure: mocks.printReadinessFailure, +})); + +vi.mock("./docker-gpu-local-inference", () => ({ + enforceDockerGpuPatchPreserveNetwork: mocks.enforceDockerGpuPatchPreserveNetwork, + verifyGpuSandboxAccessAfterReady: mocks.verifyGpuSandboxAccessAfterReady, +})); + +vi.mock("./docker-gpu-sandbox-create", () => ({ + createDockerGpuSandboxCreatePatch: mocks.createDockerGpuSandboxCreatePatch, +})); + +vi.mock("./sandbox-create-failure", () => ({ + printSandboxCreateFailureDiagnostics: mocks.printSandboxCreateFailureDiagnostics, +})); + +vi.mock("./docker-gpu-patch", async (importOriginal) => ({ + ...(await importOriginal()), + collectDockerGpuPatchDiagnostics: mocks.collectDockerGpuPatchDiagnostics, +})); + +vi.mock("./openshell-docker-sandbox-containers", async (importOriginal) => ({ + ...(await importOriginal()), + queryOpenShellDockerSandboxContainers: mocks.queryOpenShellDockerSandboxContainers, + queryOpenShellDockerSandboxRuntimeSnapshot: mocks.queryOpenShellDockerSandboxRuntimeSnapshot, +})); + +import { NEMOCLAW_CREATE_ATTEMPT_LABEL } from "../adapters/openshell/sandbox-identity"; +import { + rejectChangedDiagnosticIdentity, + diagnosticResult, + createGpuFlowInput, + createGpuFlowDeps, + createGpuPatchFixture, + setupGpuFlowMocks, + resetGpuFlowMocks, +} from "./__test-helpers__/sandbox-gpu-create-flow"; +import { runSandboxGpuCreateFlow } from "./sandbox-gpu-create-flow"; import { isNativeGpuCreatePreBuildRejection, @@ -92,3 +151,181 @@ describe("native GPU create failure classification", () => { }, ); }); + +describe("compatibility lifecycle failure diagnostics", () => { + beforeEach(() => setupGpuFlowMocks(mocks)); + afterEach(resetGpuFlowMocks); + + it.each([ + { + action: "stop", + status: 1, + signal: null, + error: undefined, + timeout: 90_000, + applied: 0, + diagnostic: "pass", + }, + { + action: "stop", + status: null, + signal: "SIGKILL", + error: new Error("spawnSync openshell ETIMEDOUT"), + timeout: 90_000, + applied: 0, + diagnostic: "pass", + }, + { + action: "start", + status: 1, + signal: null, + error: undefined, + timeout: 30_000, + applied: 1, + diagnostic: "pass", + }, + { + action: "stop", + status: 1, + signal: null, + error: undefined, + timeout: 90_000, + applied: 0, + diagnostic: "capture-fails", + }, + { + action: "start", + status: 1, + signal: null, + error: undefined, + timeout: 30_000, + applied: 1, + diagnostic: "writer-fails", + }, + { + action: "stop", + status: 1, + signal: null, + error: undefined, + timeout: 90_000, + applied: 0, + diagnostic: "identity-changed", + }, + ])( + "retains bounded redacted $action failure evidence before further mutation [case %#]", + async ({ action, status, signal, error, timeout, applied, diagnostic }) => { + const input = createGpuFlowInput(); + input.gpuRoutePlan = "compatibility-only"; + input.initialGpuRoute = "compatibility"; + input.persistRetainedSandboxRecovery = vi.fn(() => true); + const events: string[] = []; + let originalFailure: unknown; + input.revalidateVerifiedSandboxBeforeEffect = vi.fn((operation) => { + events.push(operation); + rejectChangedDiagnosticIdentity(diagnostic, operation); + }); + input.verifyCreatedSandboxBeforeEffects = vi.fn(async (_identity, before, after) => { + try { + await before?.(); + await after?.(); + } catch (caught) { + originalFailure = caught; + events.push("unwind"); + throw caught; + } + }); + const patch = createGpuPatchFixture(); + patch.ensureApplied.mockImplementation(() => { + patch.replacementRuntimeId.mockReturnValue("b".repeat(64)); + }); + mocks.createDockerGpuSandboxCreatePatch.mockReturnValue(patch); + const deps = createGpuFlowDeps(); + let nonce = ""; + input.beforeCreateAttempt = (value) => { + nonce = value; + }; + vi.mocked(deps.runCaptureOpenshell).mockImplementation((args) => + args.includes("--selector") + ? JSON.stringify([ + { + id: "alpha-sandbox-id", + name: "alpha", + labels: { [NEMOCLAW_CREATE_ATTEMPT_LABEL]: nonce }, + resource_version: 1, + created_at: "2026-08-25T00:00:00Z", + phase: "Ready", + current_policy_version: 1, + }, + ]) + : "alpha Ready", + ); + deps.captureSandboxContainerOutput = vi.fn(() => { + events.push("capture"); + return diagnosticResult(diagnostic, "capture-fails", "redacted container output"); + }); + deps.printCreateFailureDiagnostics = vi.fn(() => { + events.push("write"); + diagnosticResult(diagnostic, "writer-fails", ""); + }); + const secret = "private-lifecycle-diagnostic-canary"; + vi.stubEnv("NVIDIA_API_KEY", secret); + const originalRun = vi.mocked(deps.runOpenshell).getMockImplementation()!; + vi.mocked(deps.runOpenshell).mockImplementation((args) => + args[1] === action + ? { + status, + signal, + error, + stdout: "x".repeat(3_000), + stderr: `runtime transition refused\nAuthorization: Bearer ${secret}\nhttps://user:private-password@example.com`, + } + : originalRun(args), + ); + + const failure = await runSandboxGpuCreateFlow(input, deps).catch((caught: Error) => caught); + expect(failure).toBeInstanceOf(Error); + expect(failure).toBe(originalFailure); + expect(events.at(-1)).toBe("unwind"); + expect(events).toContain(`capture failed compatibility ${action} for sandbox 'alpha'`); + expect(vi.mocked(deps.captureSandboxContainerOutput).mock.calls).toEqual( + diagnostic === "identity-changed" + ? [] + : [["alpha", action === "start" ? "b".repeat(64) : null, "alpha-sandbox-id"]], + ); + expect(vi.mocked(deps.printCreateFailureDiagnostics).mock.calls).toEqual( + ["identity-changed", "capture-fails"].includes(diagnostic) + ? [] + : [ + [ + "alpha", + { + backupPath: null, + sandboxContainerOutput: "redacted container output", + }, + ], + ], + ); + const message = (failure as Error).message; + expect(message).toContain(`could not ${action} sandbox 'alpha'`); + expect(message).toContain(`status=${String(status)}, signal=${String(signal)}`); + expect(message).toContain("runtime transition refused"); + expect(message.includes("ETIMEDOUT")).toBe(Boolean(error)); + expect(message).not.toContain(secret); + expect(message).not.toContain("private-password"); + expect(message.length).toBeLessThanOrEqual(2_100); + expect(deps.runOpenshell).toHaveBeenCalledWith( + ["sandbox", action, "-g", "nemoclaw", "alpha"], + expect.objectContaining({ + timeout, + suppressOutput: true, + killSignal: "SIGKILL", + killProcessTreeOnTimeout: true, + }), + ); + expect(patch.ensureApplied).toHaveBeenCalledTimes(applied); + expect(patch.commitAfterReady).not.toHaveBeenCalled(); + expect(mocks.waitForCreatedSandboxReadyWithTrace).not.toHaveBeenCalled(); + expect(mocks.streamSandboxCreate).toHaveBeenCalledTimes(1); + }, + ); +}); diff --git a/src/lib/onboard/sandbox-gpu-create-flow.test.ts b/src/lib/onboard/sandbox-gpu-create-flow.test.ts index 93ea4698199..06afab79568 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.test.ts @@ -822,42 +822,44 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { ); }); - it("detaches a managed startup hold before post-create setup without waiting for its banner", async () => { - const input = createInput(); - input.sandboxStartupCommand = ["/usr/local/bin/nemoclaw-managed-startup-hold"]; - input.sandboxEnv = { OPENSHELL_DRIVERS: "docker" }; - - await expect(runSandboxGpuCreateFlow(input, createDeps())).resolves.toMatchObject({ - route: "native", - }); - expect(mocks.streamSandboxCreate.mock.calls[0]?.[3]).toEqual( - expect.objectContaining({ - readyCheckOutputPatterns: [], - waitForReadyTermination: true, - }), - ); - }); - - it("waits for native non-terminal startup output before detaching the create client", async () => { - const input = createInput(); - input.sandboxEnv = { OPENSHELL_DRIVERS: "docker" }; - - await expect(runSandboxGpuCreateFlow(input, createDeps())).resolves.toMatchObject({ - route: "native", - }); + it.each([ + { held: false, startup: {} }, + { + held: true, + startup: { + startupCommand: [ + "env", + "/usr/local/bin/nemoclaw-managed-startup-hold", + "--", + "nemoclaw-start", + ], + }, + }, + ])( + "gates native startup output only outside a managed hold ($held)", + async ({ held, startup }) => { + const input = createInput(); + input.sandboxEnv = { OPENSHELL_DRIVERS: "docker" }; + input.createRequest = { ...input.createRequest, ...startup }; + + await expect(runSandboxGpuCreateFlow(input, createDeps())).resolves.toMatchObject({ + route: "native", + }); - const streamOptions = mocks.streamSandboxCreate.mock.calls[0]?.[3]; - expect(streamOptions).toEqual( - expect.objectContaining({ - readyCheckOutputPatterns: [expect.any(RegExp)], - }), - ); - expect( - streamOptions?.readyCheckOutputPatterns?.some((pattern: RegExp) => - pattern.test("Setting up NemoClaw (Hermes)..."), - ), - ).toBe(true); - }); + const streamOptions = mocks.streamSandboxCreate.mock.calls[0]?.[3]; + expect(streamOptions).toEqual( + expect.objectContaining({ + readyCheckOutputPatterns: held ? [] : [expect.any(RegExp)], + waitForReadyTermination: held, + }), + ); + expect( + streamOptions?.readyCheckOutputPatterns?.some((pattern: RegExp) => + pattern.test("Setting up NemoClaw (Hermes)..."), + ), + ).toBe(!held); + }, + ); it("applies exact required limits while preserving the native GPU route", async () => { const input = createInput(); diff --git a/src/lib/onboard/sandbox-gpu-create-flow.ts b/src/lib/onboard/sandbox-gpu-create-flow.ts index 56dcdf4ff3f..55084709a84 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.ts @@ -294,6 +294,7 @@ export interface SandboxGpuCreateFlowDeps { captureSandboxContainerOutput?: ( sandboxName: string, expectedContainerId: string | null, + expectedSandboxId?: string, ) => string | null; /** Production callers configure the hidden portable lifecycle through the default implementation. */ installPortableDemoLifecycle?: typeof installPortableDemoSandboxLifecycle; diff --git a/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts b/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts index 77d7a44ea89..3d4a7ee423e 100644 --- a/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts @@ -554,7 +554,7 @@ describe("created sandbox identity gate", () => { `${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}`, "--output", "json", - "--limit", + "--page-size", "2", ], { @@ -878,7 +878,7 @@ describe("created sandbox identity gate", () => { `${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}`, "--output", "json", - "--limit", + "--page-size", "2", ].join("\0")]: () => { events.push("selector-settled"); diff --git a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts index fd8faa53098..0d1cc5f196e 100644 --- a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts +++ b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts @@ -15,7 +15,6 @@ import { import { namedOpenShellGateway } from "../adapters/openshell/sandbox-observer"; import { printSandboxCreateRecoveryHints } from "../build-context"; import type { StreamSandboxCreateResult } from "../sandbox/create-stream"; -import { managedStartupCreateReadiness } from "./managed-startup/hold"; import type { SandboxGpuProofResult } from "../state/registry"; import { classifySandboxCreateFailure } from "../validation"; import { @@ -47,6 +46,7 @@ import { SANDBOX_RECREATE_PROBE_TIMEOUT_MS } from "./sandbox-recreate-probe"; import type { CreatedSandboxReadyIdentityCheck } from "./sandbox-readiness-tracing"; import * as sandboxReadinessTracing from "./sandbox-readiness-tracing"; import { addTraceEvent } from "./tracing"; +import { managedStartupCreateReadiness } from "./managed-startup/hold"; type NativeRuntimeSnapshot = Readonly<{ imageId: string | null; @@ -194,26 +194,6 @@ async function verifyCreatedSandboxBeforeEffects( ); } -function requireCompatibilityLifecycleCommand( - action: "start" | "stop", - sandboxName: string, - gatewayName: string, - deps: SandboxGpuCreateFlowDeps, -): void { - const result = deps.runOpenshell(["sandbox", action, "-g", gatewayName, sandboxName], { - ignoreError: true, - killProcessTreeOnTimeout: true, - killSignal: "SIGKILL", - suppressOutput: true, - timeout: SANDBOX_READY_PROBE_TIMEOUT_MS, - }); - if (Number(result.status ?? 1) !== 0 || result.error || ("signal" in result && result.signal)) { - throw new Error( - `OpenShell could not ${action} sandbox '${sandboxName}' for the exact compatibility cutover.`, - ); - } -} - function resolveCreateAttemptNonce( input: SandboxGpuCreateFlowInput, deferPostCreateEffects: boolean, @@ -681,9 +661,46 @@ export function createSandboxGpuCreateAttemptRunner( }; let createdSandboxVerified = false; let compatibilityCreatePollError: unknown = null; + let compatibilitySandboxId: string | null = null; + const runCompatibilityLifecycle = (action: "start" | "stop"): void => { + try { + sandboxGpuCreateAttempt.requireCompatibilityLifecycleCommand( + action, + input.sandboxName, + input.gatewayName, + deps, + ); + } catch (error) { + try { + // Recheck current authority before reading a failed runtime; never rediscover ownership by name. + revalidatePostCreateEffect( + `capture failed compatibility ${action} for sandbox '${input.sandboxName}'`, + ); + if (compatibilitySandboxId) { + const containerId = runtimePatch.replacementRuntimeId?.() ?? null; + const sandboxContainerOutput = deps.captureSandboxContainerOutput + ? deps.captureSandboxContainerOutput( + input.sandboxName, + containerId, + compatibilitySandboxId, + ) + : captureOpenShellDockerSandboxFailureOutput(input.sandboxName, containerId, { + expectedSandboxId: compatibilitySandboxId, + }); + printCreateFailureDiagnostics(input.sandboxName, { + backupPath: input.restoreBackupPath, + sandboxContainerOutput, + }); + } + } catch { + // Evidence collection must not replace the original lifecycle failure. + } + throw error; + } + }; const applyVerifiedCompatibilityCutover = async (): Promise => { revalidatePostCreateEffect(`apply runtime patch for sandbox '${input.sandboxName}'`); - requireCompatibilityLifecycleCommand("stop", input.sandboxName, input.gatewayName, deps); + runCompatibilityLifecycle("stop"); revalidatePostCreateEffect(`confirm stopped compatibility sandbox '${input.sandboxName}'`); await runtimePatch.ensureApplied(); await runtimePatch.exitOnPatchError(); @@ -691,7 +708,7 @@ export function createSandboxGpuCreateAttemptRunner( }; const publishVerifiedCompatibilityCutover = async (): Promise => { revalidatePostCreateEffect(`publish replacement runtime for sandbox '${input.sandboxName}'`); - requireCompatibilityLifecycleCommand("start", input.sandboxName, input.gatewayName, deps); + runCompatibilityLifecycle("start"); }; const verifyAndPatchCompatibilityDuringCreate = async (): Promise => { if (!compatibility || !deferPostCreateEffects || !createAttemptNonce) return; @@ -732,6 +749,7 @@ export function createSandboxGpuCreateAttemptRunner( deps.verifyExactFinalHandoffRuntime ?? isExactOpenShellDockerSandboxReplacement; if (!runtimeId || !verifyExactRuntime(input.sandboxName, runtimeId, true)) return; waitForCreatedSandboxPublication(sandboxId); + compatibilitySandboxId = sandboxId; await verifyCreatedSandboxBeforeEffects( sandboxId, createAttemptNonce, @@ -805,7 +823,7 @@ export function createSandboxGpuCreateAttemptRunner( }, }), ...managedStartupCreateReadiness({ - startupCommand: input.sandboxStartupCommand, + startupCommand: attemptRequest.startupCommand, isTerminalAgent: input.terminalAgent, env: input.sandboxEnv, deferPostCreateEffects: deferRestartSafeCutover || deferPostCreateEffects, @@ -968,6 +986,7 @@ export function createSandboxGpuCreateAttemptRunner( ); } waitForCreatedSandboxPublication(sandboxId); + compatibilitySandboxId = sandboxId; await verifyCreatedSandboxBeforeEffects( sandboxId, createAttemptNonce!, diff --git a/src/lib/onboard/sandbox-provider-cleanup.ts b/src/lib/onboard/sandbox-provider-cleanup.ts index 34cc74eaae5..f96e37c1099 100644 --- a/src/lib/onboard/sandbox-provider-cleanup.ts +++ b/src/lib/onboard/sandbox-provider-cleanup.ts @@ -15,15 +15,6 @@ export { planRegisteredExtraProviders, type ReconcileExtraProvidersDeps, } from "./extra-provider-reconciliation"; -export function removeManagedHermesStateVolume( - context: import("./managed-workload/hermes-state-volume").ManagedHermesStateVolumeContext, - deps: import("./managed-workload/hermes-state-volume").ManagedHermesStateVolumeDeps = {}, -): import("./managed-workload/hermes-state-volume").ManagedHermesStateVolumeCleanupResult { - const volumeModule = - require("./managed-workload/hermes-state-volume") as typeof import("./managed-workload/hermes-state-volume"); - return volumeModule.removeManagedHermesStateVolume(context, deps); -} - export function removeManagedAgentStateVolumes( context: import("./managed-workload/agent-state-volumes").ManagedAgentStateVolumeContext, deps: import("./managed-workload/agent-state-volumes").ManagedAgentStateVolumeDeps = {}, diff --git a/src/lib/onboard/sandbox-recreate-probe.ts b/src/lib/onboard/sandbox-recreate-probe.ts index 3a3c55333af..bdc8aeb7da0 100644 --- a/src/lib/onboard/sandbox-recreate-probe.ts +++ b/src/lib/onboard/sandbox-recreate-probe.ts @@ -120,7 +120,7 @@ export function observeSandboxPresenceOnGateway( } export function observeSandboxOnGateway( - target: SandboxRecreateTarget, + target: SandboxGatewayPresenceTarget & { readonly gatewayPort?: number }, capture: SandboxRecreateCapture = captureOpenshell, runtimeSelection?: OpenShellRuntimeSelection, timeoutMs?: number, diff --git a/src/lib/onboard/sandbox-registration.test.ts b/src/lib/onboard/sandbox-registration.test.ts index f48ed817a66..4d9be01d77e 100644 --- a/src/lib/onboard/sandbox-registration.test.ts +++ b/src/lib/onboard/sandbox-registration.test.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { registeredInferenceFixture } from "../../../test/helpers/registered-inference"; + import { createHash } from "node:crypto"; import { createRequire } from "node:module"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -96,7 +98,7 @@ function createdRegistryEntryInput( } describe("buildCreatedSandboxRegistryEntry", () => { - it("omits stale Local NIM native attachments while retaining standalone vLLM attachments", () => { + it("retains sentinel Local NIM and standalone vLLM attachments", () => { const { nativeLocalIdentity } = requireDist( "../inference/native-local/contract.ts", ) as typeof import("../inference/native-local/contract"); @@ -131,18 +133,59 @@ describe("buildCreatedSandboxRegistryEntry", () => { const nim = makeEntry("nemoclaw-nim-owned"); const standalone = makeEntry(null); - expect(nim.nativeLocalProviderAttachment).toBeUndefined(); + expect(nim.nativeLocalProviderAttachment).toEqual(attachment); expect(standalone.nativeLocalProviderAttachment).toEqual(attachment); expect( nativeInferenceCreateIntentFields("vllm-local", nim).nativeLocalProviderAttachment, - ).toBeUndefined(); + ).toEqual(attachment); expect( nativeInferenceCreateIntentFields("vllm-local", standalone).nativeLocalProviderAttachment, ).toEqual(attachment); expect( nativeInferenceCreateIntentFields("vllm-local", standalone, true) .nativeLocalProviderAttachment, - ).toBeUndefined(); + ).toEqual(attachment); + }); + + it("rejects authenticated vLLM authority for Local NIM registration and creation", () => { + const { nativeLocalIdentity } = requireDist( + "../inference/native-local/contract.ts", + ) as typeof import("../inference/native-local/contract"); + const binding = { + provider: "vllm-local" as const, + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated" as const, + gatewayName: "nemoclaw", + sandboxName: "demo", + }; + const attachment = { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1 as const, + providerId: "managed-vllm-provider", + }; + const base = createdRegistryEntryInput(); + const input = { + ...base, + nativeLocalProviderAttachment: attachment, + inferenceSelection: { + ...base.inferenceSelection, + provider: "vllm-local", + nimContainer: "owned-nim", + endpointUrl: binding.endpointUrl, + }, + }; + expect(() => buildCreatedSandboxRegistryEntry(input)).toThrow( + "Local NIM cannot reuse authenticated managed vLLM provider authority", + ); + const entry = buildCreatedSandboxRegistryEntry({ + ...input, + inferenceSelection: { ...input.inferenceSelection, nimContainer: null }, + }); + expect(() => nativeInferenceCreateIntentFields("vllm-local", entry, true)).toThrow( + "Local NIM cannot reuse authenticated managed vLLM provider authority", + ); }); it("records only an exact native NVIDIA provider identity", () => { @@ -1004,3 +1047,28 @@ describe("Hermes native attachment registration", () => { ); }); }); + +it("retains the external binding and namespace through sandbox recreation", () => { + const f = registeredInferenceFixture(); + const defaults = createdRegistryEntryInput(); + const input = createdRegistryEntryInput({ + sandboxName: f.binding.sandboxName, + gatewayName: f.binding.gatewayName, + inferenceSelection: { + ...defaults.inferenceSelection, + provider: f.binding.providerName, + model: "model-a", + endpointUrl: f.binding.endpointUrl, + credentialEnv: f.binding.credentialEnv, + preferredInferenceApi: f.binding.inferenceApi, + }, + registeredInferenceProviderAttachment: f.binding, + }); + const entry = buildCreatedSandboxRegistryEntry(input); + expect(nativeInferenceCreateIntentFields(f.binding.providerName, entry)).toMatchObject({ + inferenceProvider: f.binding.providerName, + registeredInferenceProviderAttachment: f.binding, + }); + expect(() => buildCreatedSandboxRegistryEntry({ ...input, gatewayName: "wrong" })).toThrow(); + expect(() => nativeInferenceCreateIntentFields("other-provider", entry)).toThrow(); +}); diff --git a/src/lib/onboard/sandbox-registration.ts b/src/lib/onboard/sandbox-registration.ts index f4802305762..8e4e5b71c23 100644 --- a/src/lib/onboard/sandbox-registration.ts +++ b/src/lib/onboard/sandbox-registration.ts @@ -12,6 +12,7 @@ import type { InferenceSelectionInput, } from "../inference/selection"; import { + recordedRegisteredInferenceProviderAttachment, inferenceSelectionRegistryFields, normalizeNativeLocalProviderAttachment, normalizeInferenceSelection, @@ -91,6 +92,7 @@ export interface CreatedSandboxRegistryEntryInput { workload?: SandboxEntry["workload"]; hostLocalInferenceReceipt?: SandboxEntry["hostLocalInferenceReceipt"]; hostLocalInferenceProvenance?: SandboxEntry["hostLocalInferenceProvenance"]; + registeredInferenceProviderAttachment?: SandboxEntry["registeredInferenceProviderAttachment"]; nativeLocalProviderAttachment?: SandboxEntry["nativeLocalProviderAttachment"]; nativeNvidiaProviderAttachment?: SandboxEntry["nativeNvidiaProviderAttachment"]; nativeHostedProviderAttachment?: SandboxEntry["nativeHostedProviderAttachment"]; @@ -249,11 +251,23 @@ export function buildCreatedSandboxRegistryEntry( const hostLocalInferenceProvenance = cloneSandboxHostLocalInferenceProvenance( input.hostLocalInferenceProvenance, ); + const registeredInferenceProviderAttachment = recordedRegisteredInferenceProviderAttachment({ + ...input, + ...input.inferenceSelection, + name: input.sandboxName, + }); const nativeLocalProviderAttachment = normalizeNativeLocalProviderAttachment( input.nativeLocalProviderAttachment, ); if (input.nativeLocalProviderAttachment !== undefined && !nativeLocalProviderAttachment) throw new Error("Invalid native local provider attachment."); + if ( + input.inferenceSelection.provider === "vllm-local" && + input.inferenceSelection.nimContainer && + nativeLocalProviderAttachment && + nativeLocalProviderAttachment.authMode !== "sentinel" + ) + throw new Error("Local NIM cannot reuse authenticated managed vLLM provider authority."); if ( nativeLocalProviderAttachment && (requestedAgent === "hermes" || nativeLocalProviderAttachment.agent === "hermes") && @@ -334,10 +348,8 @@ export function buildCreatedSandboxRegistryEntry( workload, ...(hostLocalInferenceReceipt !== undefined ? { hostLocalInferenceReceipt } : {}), ...(hostLocalInferenceProvenance ? { hostLocalInferenceProvenance } : {}), - ...(nativeLocalProviderAttachment && - !(input.inferenceSelection.provider === "vllm-local" && input.inferenceSelection.nimContainer) - ? { nativeLocalProviderAttachment } - : {}), + ...(registeredInferenceProviderAttachment ? { registeredInferenceProviderAttachment } : {}), + ...(nativeLocalProviderAttachment ? { nativeLocalProviderAttachment } : {}), ...(nativeNvidiaProviderAttachment ? { nativeNvidiaProviderAttachment } : {}), ...(nativeHostedProviderAttachment ? { nativeHostedProviderAttachment } : {}), ...(deferredN1xManagedVllmAccepted ? { deferredN1xManagedVllmAccepted: true as const } : {}), @@ -433,6 +445,8 @@ export function prepareCreatedSandboxRegistration( input.nativeHostedProviderAttachment ?? pending?.nativeHostedProviderAttachment; const entry = buildCreatedSandboxRegistryEntry({ ...input, + registeredInferenceProviderAttachment: + input.registeredInferenceProviderAttachment ?? pending?.registeredInferenceProviderAttachment, nativeHostedProviderAttachment: pendingNativeHostedProviderAttachment, inferenceSelection: pendingRoute ? { ...input.inferenceSelection, ...pendingRoute } @@ -532,6 +546,15 @@ export function registerCreatedSandbox(input: CreatedSandboxRegistrationInput): return publishCreatedSandboxRegistration(input, prepareCreatedSandboxRegistration(input)); } +/** Compare staged authority with the validated attachment in its source registry entry. */ +export function retainsNativeLocalProviderAttachment( + entry: SandboxEntry, + attachment: unknown, +): boolean { + const recorded = normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment); + return recorded !== undefined && isDeepStrictEqual(recorded, attachment); +} + /** The sandbox phase may bypass a shared route only for its exact native local reservation. */ export function nativeLocalReservationStatus(input: { entry: SandboxEntry; @@ -581,15 +604,29 @@ export function nativeInferenceCreateIntentFields( localNimSelected = false, ): { inferenceProvider: string | null; + registeredInferenceProviderAttachment?: SandboxEntry["registeredInferenceProviderAttachment"]; nativeLocalProviderAttachment?: SandboxEntry["nativeLocalProviderAttachment"]; nativeNvidiaProviderAttachment?: SandboxEntry["nativeNvidiaProviderAttachment"]; nativeHostedProviderAttachment?: SandboxEntry["nativeHostedProviderAttachment"]; } { - // Local NIM keeps its shared route even if a stale vLLM attachment exists. - const nativeLocalProviderAttachment = - provider === "vllm-local" && (localNimSelected || entry?.nimContainer) - ? undefined - : normalizeNativeLocalProviderAttachment(entry?.nativeLocalProviderAttachment); + const registeredInferenceProviderAttachment = entry + ? recordedRegisteredInferenceProviderAttachment(entry) + : undefined; + if ( + registeredInferenceProviderAttachment && + provider !== registeredInferenceProviderAttachment.providerName + ) + throw new Error("Registered provider binding does not match the selected provider."); + const nativeLocalProviderAttachment = normalizeNativeLocalProviderAttachment( + entry?.nativeLocalProviderAttachment, + ); + if ( + provider === "vllm-local" && + (localNimSelected || entry?.nimContainer) && + nativeLocalProviderAttachment && + nativeLocalProviderAttachment.authMode !== "sentinel" + ) + throw new Error("Local NIM cannot reuse authenticated managed vLLM provider authority."); const nativeNvidiaProviderAttachment = normalizeNativeNvidiaProviderAttachment( entry?.nativeNvidiaProviderAttachment, ); @@ -599,9 +636,11 @@ export function nativeInferenceCreateIntentFields( ); return { inferenceProvider: + registeredInferenceProviderAttachment?.providerName ?? nativeLocalProviderAttachment?.providerName ?? hostedAttachment?.providerName ?? nativeInferenceProviderForSandbox(provider), + ...(registeredInferenceProviderAttachment ? { registeredInferenceProviderAttachment } : {}), ...(nativeLocalProviderAttachment ? { nativeLocalProviderAttachment } : {}), nativeHostedProviderAttachment: hostedAttachment, ...(nativeNvidiaProviderAttachment ? { nativeNvidiaProviderAttachment } : {}), diff --git a/src/lib/onboard/sandbox-workload-preparation.test.ts b/src/lib/onboard/sandbox-workload-preparation.test.ts index f2de5af422a..5ecc995a218 100644 --- a/src/lib/onboard/sandbox-workload-preparation.test.ts +++ b/src/lib/onboard/sandbox-workload-preparation.test.ts @@ -23,6 +23,8 @@ import { type ManagedImageContractV1, type ManagedImageAgent, SHIPPED_MANAGED_IMAGE_AGENTS, + managedImagePlatformForNodeArchitecture, + parseManagedImageContractV1, } from "./managed-image/contract"; import { createRuntimeProviderBundleRegistry } from "./runtime-provider/registry"; import { @@ -65,6 +67,37 @@ const CATALOG: ManagedImageContractCatalog = Object.fromEntries( SHIPPED_MANAGED_IMAGE_AGENTS.map((agent, index) => [agent, contract(agent, index)]), ); +it.each(["x64", "amd64"])( + "contract parsing accepts the matching %s host architecture", + (architecture) => { + const platform = managedImagePlatformForNodeArchitecture(architecture); + const candidate = { ...contract("openclaw", 0), platform: "linux/amd64" }; + expect(parseManagedImageContractV1(candidate, "openclaw", platform!).platform).toBe( + "linux/amd64", + ); + }, +); + +it.each(["arm64", "ppc64"])( + "contract parsing rejects an amd64 image on the %s host architecture", + (architecture) => { + const platform = managedImagePlatformForNodeArchitecture(architecture); + const candidate = { ...contract("openclaw", 0), platform: "linux/amd64" }; + expect(() => parseManagedImageContractV1(candidate, "openclaw", platform!)).toThrow( + "contract.platform must be", + ); + }, +); + +it("the live catalog loader rejects extra entries outside the shipped agent set", () => { + expect(() => + readLiveE2eManagedImageCatalogContracts({ + catalog: { ...CATALOG, unused: contract("openclaw", 0) }, + revision: REVISION, + }), + ).toThrow("must contain only the shipped agent contracts"); +}); + function runtime(driverName = "docker"): SandboxWorkloadRuntimeCapabilities { return { driverName, diff --git a/src/lib/onboard/setup-inference-gateway-scope.test.ts b/src/lib/onboard/setup-inference-gateway-scope.test.ts index 76fd359ceb8..9722104dd6c 100644 --- a/src/lib/onboard/setup-inference-gateway-scope.test.ts +++ b/src/lib/onboard/setup-inference-gateway-scope.test.ts @@ -31,6 +31,31 @@ describe("gateway-scoped onboarding OpenShell commands", () => { expect(scopeGatewayOpenshellArgs(input, GATEWAY)).toEqual(expected); }); + it.each(["list", "get"])("binds global policy %s to the named gateway", (verb) => { + const command = ["policy", verb, "--global", "--output", "json"]; + const expected = ["policy", verb, "-g", GATEWAY, "--global", "--output", "json"]; + const run = vi.fn(() => ({ status: 0 })); + createGatewayScopedOpenshellRunner(run, GATEWAY)(command); + expect(run).toHaveBeenCalledExactlyOnceWith(expected); + expect(command).toEqual(["policy", verb, "--global", "--output", "json"]); + expect(scopeGatewayOpenshellArgs(expected, GATEWAY)).toEqual(expected); + expect(() => + scopeGatewayOpenshellArgs(["policy", verb, "--gateway", "other", "--global"], GATEWAY), + ).toThrow(/instead of 'nemoclaw-9090'/); + expect(() => scopeGatewayOpenshellArgs([...expected, "--gateway", GATEWAY], GATEWAY)).toThrow( + /multiple gateway targets/, + ); + expect(() => scopeGatewayOpenshellArgs(["policy", verb, "-g"], GATEWAY)).toThrow( + /instead of 'nemoclaw-9090'/, + ); + expect(() => + scopeGatewayOpenshellArgs( + [...command, "--gateway-endpoint=https://other.example.test"], + GATEWAY, + ), + ).toThrow(/--gateway-endpoint may bypass the gateway recorded/); + }); + it("targets sandbox execution at the same gateway", () => { expect( scopeGatewayOpenshellArgs(["sandbox", "exec", "-n", "alpha", "--", "true"], GATEWAY), diff --git a/src/lib/onboard/setup-inference.test.ts b/src/lib/onboard/setup-inference.test.ts index b143190dd65..354e533c001 100644 --- a/src/lib/onboard/setup-inference.test.ts +++ b/src/lib/onboard/setup-inference.test.ts @@ -2,6 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 import { describe, expect, it, vi } from "vitest"; +import { getInstalledOpenshellVersion } from "./openshell-version"; + +import { cliOpenShellInstalledVersionObserver } from "../adapters/openshell/installed-version-cli"; import type { OpenShellProviderAdapter } from "../adapters/openshell/provider-adapter"; import type { OpenShellSandboxPolicyReader } from "../adapters/openshell/sandbox-policy"; @@ -57,6 +60,46 @@ describe("recorded Pi local inference route", () => { }); }); +it.each([ + { label: "0.1.3", observation: { ok: true, version: "0.1.3" } }, + { label: "unknown", observation: { ok: false, error: { kind: "timeout", message: "timeout" } } }, +] as const)( + "rejects direct retained Pi setup before route mutation on OpenShell $label", + async ({ observation }) => { + vi.spyOn(cliOpenShellInstalledVersionObserver, "observeInstalledVersion").mockReturnValue( + observation, + ); + const recorded = { + agent: "pi", + provider: "ollama-local", + model: "qwen3:8b", + endpointUrl: "http://host.openshell.internal:11435/v1", + createdAt: "2026-09-01T00:00:00Z", + }; + const withGatewayRouteMutationLock = vi.fn(); + const requireNativeProviderPolicy = vi.fn(); + const updateSandbox = vi.fn(); + const setup = createSetupInference({ + getGatewayName: () => "nemoclaw", + getSandbox: () => recorded, + getInstalledOpenshellVersion, + withSandboxMutationLock: (_name: string, operation: () => unknown) => operation(), + withGatewayRouteMutationLock, + requireNativeProviderPolicy, + updateSandbox, + } as unknown as SetupInferenceDeps); + await expect( + setup("alpha", recorded.model, recorded.provider, recorded.endpointUrl, null, null, [], { + agentName: "pi", + preserveLegacyPiSharedLocalRoute: true, + }), + ).rejects.toThrow("Cannot resume a legacy Pi shared inference route"); + expect(withGatewayRouteMutationLock).not.toHaveBeenCalled(); + expect(requireNativeProviderPolicy).not.toHaveBeenCalled(); + expect(updateSandbox).not.toHaveBeenCalled(); + }, +); + describe("createProviderReviewDeps", () => { it("prepares the Ollama proxy after review acceptance", async () => { const updateSession = vi.fn(); @@ -228,99 +271,6 @@ describe("createProviderReviewDeps", () => { }); }); -describe("Personal custom endpoint onboarding", () => { - it("resolves Personal before provider setup and keeps the shared route", async () => { - const events: string[] = []; - const createProvider = vi.fn(); - const attachProvider = vi.fn(); - const upsertProvider = vi.fn(async () => { - events.push("shared-provider"); - return { ok: true }; - }); - const inferenceRouteMutator = { - setInferenceRoute: vi.fn(async () => ({ ok: true as const })), - }; - const setup = createSetupInference({ - checkGatewayRouteCompatibility: vi.fn(() => ({ ok: true as const })), - withSandboxMutationLock: async (_name: string, operation: () => Promise | T) => { - events.push("sandbox-lock"); - return await operation(); - }, - withGatewayRouteMutationLock: async (_name: string, operation: () => Promise | T) => { - events.push("gateway-lock"); - return await operation(); - }, - step: vi.fn(), - getGatewayName: () => "nemoclaw", - getSandbox: () => null, - runOpenshell: vi.fn(() => ({ status: 0, stdout: "", stderr: "" })), - inferenceRouteMutator, - inferenceRouteObserver: { - observeInferenceRoute: vi.fn(async () => ({ - ok: true as const, - value: { state: "unconfigured" as const }, - })), - }, - updateSandbox: vi.fn(() => true), - upsertProvider, - verifyInferenceRoute: vi.fn(), - verifyOnboardInferenceSmoke: vi.fn(), - resolveEndpointHost: vi.fn(async () => [{ address: "93.184.216.34", family: 4 as const }]), - resolvePolicyTier: async () => { - events.push("policy-tier"); - return "personal"; - }, - bedrockRuntimeOnboard: { - setupBedrockRuntimeInference: vi.fn(async () => ({ handled: false as const })), - }, - openrouterRuntimeOnboard: { - setupOpenRouterRuntimeInference: vi.fn(async () => ({ handled: false as const })), - }, - isNonInteractive: () => false, - hermesProviderAuth: { HERMES_PROVIDER_NAME: "hermes-provider" }, - providerAdapter: { createProvider, attachProvider } as unknown as OpenShellProviderAdapter, - REMOTE_PROVIDER_CONFIG: { - custom: { - label: "Compatible endpoint", - providerName: "compatible-endpoint", - providerType: "openai", - credentialEnv: "COMPATIBLE_API_KEY", - endpointUrl: "https://api.example/v1", - helpUrl: null, - modelMode: "input", - defaultModel: "model-a", - }, - }, - hydrateCredentialEnv: vi.fn(() => "secret"), - redact: (value: string) => value, - compactText: (value: string) => value, - log: vi.fn(), - error: vi.fn(), - exitProcess: vi.fn((code: number): never => { - throw new Error(`exit ${code}`); - }), - } as unknown as SetupInferenceDeps); - - await expect( - setup( - "alpha", - "model-a", - "compatible-endpoint", - "https://api.example/v1", - "COMPATIBLE_API_KEY", - null, - [], - { skipHostInferenceSmoke: true }, - ), - ).resolves.toEqual({ ok: true }); - - expect(events).toEqual(["policy-tier", "sandbox-lock", "gateway-lock", "shared-provider"]); - expect(createProvider).not.toHaveBeenCalled(); - expect(attachProvider).not.toHaveBeenCalled(); - expect(inferenceRouteMutator.setInferenceRoute).toHaveBeenCalledOnce(); - }); -}); - describe("custom native onboarding policy admission", () => { function fixture( document: string, diff --git a/src/lib/onboard/setup-inference.ts b/src/lib/onboard/setup-inference.ts index a31b94a6f77..252b59cd00b 100644 --- a/src/lib/onboard/setup-inference.ts +++ b/src/lib/onboard/setup-inference.ts @@ -23,7 +23,7 @@ import { } from "../inference/native-provider/hosted"; import { requireHostedProviderAttachment } from "../inference/native-provider/hosted-attachment"; import { - isHostedCustomEndpoint, + isCustomNativeEndpoint, prepareHostedNativeProvider, prepareCustomNativeProvider, readCustomNativeSandboxPolicy, @@ -85,6 +85,8 @@ import type { Session } from "../state/onboard-session"; import type { SandboxEntry } from "../state/registry/types"; import { createSandboxHostLocalInferenceProvenance } from "../state/registry/host-local-inference"; import { + isModelRouterProvider, + prepareNativeModelRouterSelection, withRoutedInferencePortLock, type RoutedProviderDeps, upsertRoutedProvider as upsertRoutedInferenceProvider, @@ -258,6 +260,7 @@ export type SetupInferenceDeps = ProviderBranchDeps & { withSandboxMutationLock: typeof withSandboxMutationLock; step: (current: number, total: number, label: string) => void; getGatewayName: () => string; + getInstalledOpenshellVersion?: () => string | null; runOpenshell: import("./openshell-cli").OpenshellCliHelpers["runOpenshell"]; inferenceRouteMutator: OpenShellInferenceRouteMutator; inferenceRouteObserver: OpenShellInferenceRouteObserver; @@ -769,19 +772,13 @@ async function resolveHostedSelection(input: { const recordedSandbox = sandboxName ? deps.getSandbox?.(sandboxName) : null; const freshCustomSelection = Boolean( sandboxName && - isHostedCustomEndpoint(provider, endpointUrl) && + isCustomNativeEndpoint(provider, endpointUrl) && !recordedSandbox?.nativeHostedProviderAttachment?.custom && (!recordedSandbox || recordedSandbox.pendingRouteReservation), ); const resolvePolicyTier = options.resolvePolicyTier ?? deps.resolvePolicyTier; - const policyTier = freshCustomSelection ? await resolvePolicyTier?.() : undefined; - const custom = usesCustomNativeOnboarding( - provider, - endpointUrl, - sandboxName, - recordedSandbox, - policyTier, - ); + if (freshCustomSelection) await resolvePolicyTier?.(); + const custom = usesCustomNativeOnboarding(provider, endpointUrl, sandboxName, recordedSandbox); const authority = provider === "hermes-provider" ? ((recordedSandbox?.provider === provider @@ -1103,6 +1100,17 @@ function createGatewayRouteStager(input: { }; } +function nativeRouterSetupResult( + routedProvider: boolean, + attachment: NativeLocalProviderAttachment | undefined, + credentialEnv: string | undefined, +): Pick, "nativeInference"> { + if (!routedProvider || !attachment) return {}; + return { + nativeInference: { endpointUrl: attachment.endpointUrl, credentialEnv: credentialEnv! }, + }; +} + export function createSetupInference( defaults: SetupInferenceDeps, overrides: Partial = {}, @@ -1129,7 +1137,7 @@ export function createSetupInference( ? () => (policyTierResolution ??= policyTierResolver()) : undefined; const hostedSelectionOptions = resolvePolicyTier ? { ...options, resolvePolicyTier } : options; - if (sandboxName && isHostedCustomEndpoint(provider, endpointUrl)) { + if (sandboxName && isCustomNativeEndpoint(provider, endpointUrl)) { await resolvePolicyTier?.(); } let selectingNativeLocal = false; @@ -1138,6 +1146,7 @@ export function createSetupInference( const usesBedrockRuntimeAdapter = provider === "compatible-anthropic-endpoint" && isBedrockRuntimeEndpoint(endpointUrl); let shouldLogSuccessfulRoute = false; + let selectedLocalNim = false; const withInferenceMutationLocks = (operation: () => Promise | T): Promise => deps.withGatewayRouteMutationLock(gatewayName, () => withRoutedInferencePortLock(routedProvider, operation, deps), @@ -1364,6 +1373,9 @@ export function createSetupInference( throw new Error("Native local inference requires a provider adapter."); const [name, , sourceCredentialEnv, selectedEndpoint, env] = args; const recorded = deps.getSandbox?.(sandboxName); + const recordedAttachment = normalizeNativeLocalProviderAttachment( + recorded?.nativeLocalProviderAttachment, + ); if ( recorded && recorded.pendingRouteReservation !== true && @@ -1373,7 +1385,7 @@ export function createSetupInference( recorded.provider, recorded.endpointUrl, )) && - !normalizeNativeLocalProviderAttachment(recorded.nativeLocalProviderAttachment) + !recordedAttachment ) { if ( !options.recreatePublishedSandbox || @@ -1394,7 +1406,43 @@ export function createSetupInference( gatewayReachableCompatibleEndpointUrl(name, selectedEndpoint); if (typeof hostReachableEndpoint !== "string") throw new Error("Native local inference endpoint is missing."); - nativeLocalProviderAttachment = await prepareNativeLocalSelection({ + const authMode = + (name === "ollama-local" && value === "ollama") || + (name === "vllm-local" && value === "dummy") + ? "sentinel" + : "authenticated"; + if ( + recorded && + recorded.pendingRouteReservation !== true && + recordedAttachment && + options.deferPublishedRouteReservation + ) { + if ( + recorded.name !== sandboxName || + recorded.provider !== name || + recorded.gatewayName !== gatewayName || + (recorded.credentialEnv ?? null) !== credentialEnv || + recordedAttachment.sandboxName !== sandboxName || + recordedAttachment.gatewayName !== gatewayName || + recordedAttachment.provider !== name || + recordedAttachment.authMode !== authMode || + deps.getNativeLocalProviderAuthority?.(recordedAttachment.providerName) + ?.providerId !== recordedAttachment.providerId || + (recordedAttachment.agent ?? "langchain-deepagents-code") !== nativeLocalAgent || + recordedAttachment.endpointUrl !== hostReachableEndpoint || + (recordedAttachment.api ?? "openai-completions") !== + (selectedInferenceApi ?? "openai-completions") + ) + throw new Error( + "Cannot change the published native provider binding before sandbox backup. Re-run the model change with the recorded provider configuration.", + ); + // Keep the source row and gateway credential usable if backup fails. + // The sandbox phase reserves the replacement after backup succeeds. + nativeLocalRecreationSource = structuredClone(recorded); + } + nativeLocalProviderAttachment = await ( + routedProvider ? prepareNativeModelRouterSelection : prepareNativeLocalSelection + )({ adapter: deps.providerAdapter, binding: { provider: name as NativeLocalProvider, @@ -1404,20 +1452,19 @@ export function createSetupInference( ? { agent: nativeLocalAgent } : {}), endpointUrl: hostReachableEndpoint, - authMode: - (name === "ollama-local" && value === "ollama") || - (name === "vllm-local" && value === "dummy") - ? "sentinel" - : "authenticated", + ...((nativeLocalAgent === "openclaw" || nativeLocalAgent === "hermes") && + name === "compatible-endpoint" && + selectedInferenceApi === "openai-responses" + ? { api: "openai-responses" as const } + : {}), + authMode, gatewayName, sandboxName, }, - credentialValue: value, - ...(name === "nvidia-router" - ? { - ownedModelRouterPort: - deps.getModelRouterPort?.() ?? Number(new URL(hostReachableEndpoint).port), - } + credentialValue: + recordedAttachment && nativeLocalRecreationSource && !routedProvider ? null : value, + ...(recordedAttachment && nativeLocalRecreationSource && routedProvider + ? { preserveProviderCredential: true } : {}), ownedProxy: (name === "ollama-local" && !hostLocalRoute && deps.shouldFrontOllamaWithProxy()) || @@ -1426,7 +1473,14 @@ export function createSetupInference( readAuthority: deps.getNativeLocalProviderAuthority, writeAuthority: deps.setNativeLocalProviderAuthority, }); - if (name === "nvidia-router") { + if ( + recordedAttachment && + nativeLocalRecreationSource && + (nativeLocalProviderAttachment.providerId !== recordedAttachment.providerId || + nativeLocalProviderAttachment.providerName !== recordedAttachment.providerName) + ) + throw new Error("Published native provider ownership changed before sandbox backup."); + if (routedProvider) { nativeProviderEndpoint = hostReachableEndpoint; const hostSmokeUrl = new URL(hostReachableEndpoint); hostSmokeUrl.hostname = "127.0.0.1"; @@ -1624,7 +1678,7 @@ export function createSetupInference( { ...commonDeps, getNativeProviderAuthority: () => hermesAuthority, - prepareNativeProvider: async (apiKey, returnedEndpoint) => { + prepareNativeProvider: async (apiKey, returnedEndpoint, retainedHermesEndpoint) => { if ( !deps.providerAdapter || !deps.getNativeHostedProviderAuthority || @@ -1640,6 +1694,7 @@ export function createSetupInference( adapter: deps.providerAdapter, credentialValue: apiKey, endpointUrl: returnedEndpoint, + retainedHermesEndpoint, lookup: deps.resolveEndpointHost, recordedSandbox: sandboxName ? deps.getSandbox?.(sandboxName) : null, reuseExistingCredential: apiKey === null, @@ -1765,7 +1820,7 @@ export function createSetupInference( if (outcome.done) return outcome.result; } else if (provider === "vllm-local") { const outcome = await inferenceProviders.setupVllmLocalInference( - { model, provider }, + { model, provider, localNimSelected: selectedLocalNim }, { ...commonDeps, validateLocalProvider: hostLocalRoute @@ -2013,25 +2068,29 @@ export function createSetupInference( } revalidateSandboxIdentity?.("report successful inference provider setup"); shouldLogSuccessfulRoute = true; - const result = - nativeLocalRecreationSource && nativeLocalProviderAttachment - ? { - ok: true as const, - stagedNativeLocalRecreation: { - source: nativeLocalRecreationSource, - attachment: nativeLocalProviderAttachment, - }, - } - : routeStager.successResult(); - return { - ...result, - ...(nativeLocalProviderAttachment?.provider === "nvidia-router" && { - nativeInference: { - endpointUrl: nativeLocalProviderAttachment.endpointUrl, - credentialEnv: nativeHostedCredentialEnv!, + const nativeRouterInference = nativeRouterSetupResult( + routedProvider, + nativeLocalProviderAttachment, + nativeHostedCredentialEnv, + ); + + if ( + nativeLocalRecreationSource && + nativeLocalProviderAttachment && + (!nativeLocalRecreationSource.nativeLocalProviderAttachment || + nativeLocalRecreationSource.model !== model || + options.recreatePublishedSandbox) + ) { + return { + ok: true, + ...nativeRouterInference, + stagedNativeLocalRecreation: { + source: nativeLocalRecreationSource, + attachment: nativeLocalProviderAttachment, }, - }), - }; + }; + } + return { ...routeStager.successResult(), ...nativeRouterInference }; } catch (error) { if (!hostLocalRoute || !hostLocalSelection) { return rejectNativeLocalSetupFailure({ @@ -2044,7 +2103,9 @@ export function createSetupInference( }); } await cleanupUnreservedNativeLocalAfterHostFailure({ - selectingNativeLocal, + // A runtime-owned native provider must roll back through its publication journal. + selectingNativeLocal: + selectingNativeLocal && !hostLocalGatewayMutation?.registerNativeProvider, attachment: nativeLocalProviderAttachment, reservationEntered: nativeLocalReservationEntered, sandboxName, @@ -2106,6 +2167,8 @@ export function createSetupInference( if (!selectedAgent && usesNativeLocalInference(provider, endpointUrl)) throw error; /* An unreadable registry skips GPU release; it must not fail onboarding. */ } + selectedLocalNim = + provider === "vllm-local" && (options.localNimSelected ?? Boolean(recordedNimContainer)); nativeLocalAgent = selectedAgent ?? recordedAgent; selectingNativeLocal = !options.hostLocalInference?.sharedInferenceRoute && @@ -2115,6 +2178,10 @@ export function createSetupInference( endpointUrl, options.localNimSelected ?? (provider === "vllm-local" && Boolean(recordedNimContainer)), ); + if (isModelRouterProvider(provider) && !selectingNativeLocal) + throw new Error( + "Model Router requires a native provider for the selected agent and configured endpoint.", + ); const preservingLegacyPiRoute = options.preserveLegacyPiSharedLocalRoute === true && isPersistedPiSharedLocalInferenceRoute( @@ -2123,6 +2190,13 @@ export function createSetupInference( ); if (options.preserveLegacyPiSharedLocalRoute && !preservingLegacyPiRoute) throw new Error("Recorded Pi shared local inference route changed before setup."); + if ( + preservingLegacyPiRoute && + !/^0\.0\.\d+$/.test(deps.getInstalledOpenshellVersion?.() ?? "") + ) + throw new Error( + "Cannot resume a legacy Pi shared inference route on this OpenShell runtime. Back up the sandbox and recreate it with native inference; existing data was not changed.", + ); selectingNativeLocal = selectingNativeLocal && !preservingLegacyPiRoute; if (selectingNativeLocal) await (deps.requireNativeProviderPolicy ?? requireNativeProviderPolicy)(gatewayName); diff --git a/src/lib/onboard/setup-native-hosted.test.ts b/src/lib/onboard/setup-native-hosted.test.ts index e79109a8d82..1eeb08fa911 100644 --- a/src/lib/onboard/setup-native-hosted.test.ts +++ b/src/lib/onboard/setup-native-hosted.test.ts @@ -6,6 +6,7 @@ import type { OpenShellProviderAdapter } from "../adapters/openshell/provider-ad import { customNativeProvider } from "../inference/native-provider/custom"; import { HOSTED_NATIVE_PROVIDERS, + hostedNativeProvider, type HostedProviderDefinition, } from "../inference/native-provider/hosted"; import { createSetupInference, type SetupInferenceDeps } from "./setup-inference"; @@ -17,6 +18,14 @@ vi.mock("./sandbox-lifecycle", async (importOriginal) => ({ const providers: HostedProviderDefinition[] = [ ...HOSTED_NATIVE_PROVIDERS, + ...[ + { endpoint: "https://8.8.8.8/team/v1", allowedIps: ["8.8.8.8"] }, + { endpoint: "http://8.8.8.8/team/v1", allowedIps: ["8.8.8.8"] }, + { endpoint: "http://host.openshell.internal:42103/team/v1", allowedIps: undefined }, + ].map(({ endpoint, allowedIps }) => ({ + ...hostedNativeProvider("hermes-provider", endpoint)!, + allowedIps, + })), ...[ ["compatible-endpoint", "openai-completions", "https://custom.example/v1"], ["compatible-endpoint", "openai-responses", "https://custom.example/v1"], @@ -46,7 +55,9 @@ describe.each(providers)("native $label onboarding", (provider) => { ? ["default-api"] : []), ] - : ["explicit", "omitted"]; + : provider.endpointUrl + ? ["explicit"] + : ["explicit", "omitted"]; it.each(endpointModes)("uses %s native endpoint (#12589)", async (endpointMode) => { const receipt = { schemaVersion: 1, diff --git a/src/lib/onboard/setup-nim-local-route.test.ts b/src/lib/onboard/setup-nim-local-route.test.ts index e42d54a85d6..a46322c88c3 100644 --- a/src/lib/onboard/setup-nim-local-route.test.ts +++ b/src/lib/onboard/setup-nim-local-route.test.ts @@ -14,7 +14,7 @@ afterEach(() => { vi.unstubAllEnvs(); }); -it("marks Local NIM as shared before its container starts", async () => { +it("marks Local NIM before its container starts", async () => { const routeGuard = vi.fn(() => ({ requiredModel: null, requiredEndpointUrl: null, diff --git a/src/lib/onboard/setup-nim-vllm.test.ts b/src/lib/onboard/setup-nim-vllm.test.ts index f112724caaa..f03bf7618ff 100644 --- a/src/lib/onboard/setup-nim-vllm.test.ts +++ b/src/lib/onboard/setup-nim-vllm.test.ts @@ -5,7 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { requireValue } from "../core/require-value"; import * as localModelDocker from "../adapters/docker/run"; import { resolveVllmModelAlias } from "../inference/vllm-models"; -import { getLegacyHostLocalManagedVllmBridgeHost } from "../inference/serving/vllm-host-local-lifecycle"; +import { getLegacyHostLocalManagedVllmLoopbackHost } from "../inference/serving/vllm-host-local-lifecycle"; import { bindStationExpressProviderSelection } from "./station-express-resume"; import type { SetupNimSelectionState } from "./setup-nim-flow"; import { @@ -346,9 +346,7 @@ describe("setupNim vLLM route containment", () => { deps({ runCapture, getManagedVllmProviderBinding: () => { - getLegacyHostLocalManagedVllmBridgeHost(8000, () => - JSON.stringify([{ Gateway: "172.18.0.1" }]), - ); + getLegacyHostLocalManagedVllmLoopbackHost(8000); return null; }, }), diff --git a/src/lib/onboard/workload/rebuild.ts b/src/lib/onboard/workload/rebuild.ts index b239a9494ae..06ab081d745 100644 --- a/src/lib/onboard/workload/rebuild.ts +++ b/src/lib/onboard/workload/rebuild.ts @@ -270,6 +270,7 @@ export function managedWorkloadRebuildHandoffMatchesEntry( } export interface ManagedWorkloadRebuildProfileOverrides { + readonly openClawSelectedModel?: Readonly>; readonly openClawContextWindow?: number; readonly openClawReasoning?: boolean; readonly openClawReasoningEffort?: ManagedStartupReasoningEffort; @@ -293,19 +294,30 @@ export function managedWorkloadRebuildProfileEnvironment( const previous = handoff.previousProfile; if (previous.agent === "openclaw" && previous.agentConfig.agent === "openclaw") { const config = previous.agentConfig; - const contextWindow = overrides.openClawContextWindow ?? previous.tuning.contextWindow; + const selectedModel = overrides.openClawSelectedModel; + const contextWindow = selectedModel + ? (selectedModel.contextWindow ?? null) + : (overrides.openClawContextWindow ?? previous.tuning.contextWindow); if (contextWindow !== null) result.NEMOCLAW_CONTEXT_WINDOW = String(contextWindow); - if (previous.tuning.maxTokens !== null) { - result.NEMOCLAW_MAX_TOKENS = String(previous.tuning.maxTokens); - } - const reasoning = overrides.openClawReasoning ?? previous.tuning.reasoning; + const maxTokens = selectedModel ? (selectedModel.maxTokens ?? null) : previous.tuning.maxTokens; + if (maxTokens !== null) result.NEMOCLAW_MAX_TOKENS = String(maxTokens); + const reasoning = selectedModel + ? (selectedModel.reasoning ?? false) + : (overrides.openClawReasoning ?? previous.tuning.reasoning); if (reasoning !== null) result.NEMOCLAW_REASONING = String(reasoning); - const reasoningEffort = overrides.openClawReasoningEffort ?? previous.tuning.reasoningEffort; + const reasoningEffort = selectedModel + ? "default" + : (overrides.openClawReasoningEffort ?? previous.tuning.reasoningEffort); if (reasoningEffort !== null) result.NEMOCLAW_REASONING_EFFORT = reasoningEffort; - if (previous.inference?.inputModalities != null) { - result.NEMOCLAW_INFERENCE_INPUTS = previous.inference.inputModalities.join(","); + const inputs = selectedModel + ? (selectedModel.input ?? ["text"]) + : previous.inference?.inputModalities; + if (inputs != null) { + if (!Array.isArray(inputs) || inputs.some((value) => typeof value !== "string")) + throw new Error("Selected OpenClaw model has invalid input modalities."); + result.NEMOCLAW_INFERENCE_INPUTS = inputs.join(","); } - if (previous.inference?.servingPreset) { + if (!selectedModel && previous.inference?.servingPreset) { result.NEMOCLAW_SERVING_PRESET = previous.inference.servingPreset; } result.NEMOCLAW_AGENT_TIMEOUT = String(config.agentTimeoutSeconds); diff --git a/src/lib/policy/index.ts b/src/lib/policy/index.ts index 4d1a46d2290..adcf9c87f7b 100644 --- a/src/lib/policy/index.ts +++ b/src/lib/policy/index.ts @@ -1287,13 +1287,10 @@ function mergePresetIntoPolicy(currentPolicy: string, presetEntries: string): st return normalizePersonalOpenInternetPolicy(YAML.stringify(output)); } -/** - * An attached custom provider adds an exact endpoint on port 443 that NemoClaw - * cannot remove. OpenShell rejects Personal's hostless web rule beside it - * (NVIDIA/OpenShell#4662), so Personal cannot be applied to such a sandbox. - */ -export function hasNativeCustomProvider(sandboxName: string): boolean { - return Boolean(registry.getSandbox(sandboxName)?.nativeHostedProviderAttachment?.custom); +/** Read the custom provider attachment used by onboarding policy selection. */ +export function getNativeCustomProviderAttachment(sandboxName: string) { + const attachment = registry.getSandbox(sandboxName)?.nativeHostedProviderAttachment; + return attachment?.custom ? attachment : undefined; } /** diff --git a/src/lib/policy/preset-scope-render.ts b/src/lib/policy/preset-scope-render.ts index f9d23d2b98e..2e7e042d62a 100644 --- a/src/lib/policy/preset-scope-render.ts +++ b/src/lib/policy/preset-scope-render.ts @@ -55,8 +55,8 @@ export function escapeTerminalText(value: string): string { .join(""); } -/** Redact credential-shaped content before rendering untrusted YAML scalars. */ -function renderTerminalText(value: string): string { +/** Redact credentials and escape terminal controls in untrusted text. */ +export function renderTerminalText(value: string): string { return escapeTerminalText(redactFullWithUrls(value)); } diff --git a/src/lib/policy/sandbox-policy-validation.test.ts b/src/lib/policy/sandbox-policy-validation.test.ts new file mode 100644 index 00000000000..0cf90946435 --- /dev/null +++ b/src/lib/policy/sandbox-policy-validation.test.ts @@ -0,0 +1,55 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import { parseAndValidateSandboxPolicy } from "./sandbox-policy-validation"; + +function policyWithMcpOptions(options: Record) { + return { + version: 1, + network_policies: { + mcp_bridge_fake: { + name: "MCP fixture", + binaries: [{ path: "/usr/bin/node" }], + endpoints: [ + { + host: "mcp.example.test", + port: 443, + protocol: "mcp", + mcp: { allow_all_known_mcp_methods: true, ...options }, + }, + ], + }, + }, + }; +} + +describe("MCP policy version preservation during rebuild", () => { + it.each([ + {}, + { versions: ["2025-03-26"] }, + { versions: ["2025-06-18"] }, + { versions: ["2025-11-25"] }, + { versions: ["2026-07-28"] }, + { versions: ["2025-11-25", "2025-03-26", "2025-06-18", "2026-07-28"] }, + ])("preserves supported MCP options %j", (options) => { + const policy = policyWithMcpOptions(options); + expect(parseAndValidateSandboxPolicy(JSON.stringify(policy))).toEqual(policy); + }); + + it.each([ + { versions: [] }, + { versions: null }, + { versions: "2025-11-25" }, + { versions: ["2025-11-25", "2025-11-25"] }, + { versions: ["2025-11-26"] }, + { versions: [" 2025-11-25"] }, + { versions: ["2025-11-25 "] }, + { versions: [20251125] }, + { unknown_option: true }, + ])("rejects invalid MCP options %j", (options) => { + expect(() => + parseAndValidateSandboxPolicy(JSON.stringify(policyWithMcpOptions(options))), + ).toThrow("does not satisfy the shipped sandbox policy schema"); + }); +}); diff --git a/src/lib/private-networks.ts b/src/lib/private-networks.ts index b695f84feae..224d195b260 100644 --- a/src/lib/private-networks.ts +++ b/src/lib/private-networks.ts @@ -19,7 +19,7 @@ import type { PrivateNetworkMatcher, } from "../../nemoclaw/dist/shared/private-networks-boundary.cjs"; -import { ROOT } from "./runner"; +import { ROOT } from "./state/paths"; const NETWORKS_FILE = path.join(ROOT, "nemoclaw-blueprint", "private-networks.yaml"); diff --git a/src/lib/registry-recovery-seeded-paths.test.ts b/src/lib/registry-recovery-seeded-paths.test.ts index 3d38aedd028..1477ad1c856 100644 --- a/src/lib/registry-recovery-seeded-paths.test.ts +++ b/src/lib/registry-recovery-seeded-paths.test.ts @@ -3,8 +3,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; -import { runInferenceSet } from "./actions/inference-set.js"; -import { createDeps } from "../../test/helpers/inference-set.js"; +import { prepareInferenceSetRoute } from "./actions/inference-set-route-containment"; import type { SandboxEntry } from "./state/registry.js"; interface MockRegistryState { @@ -255,7 +254,7 @@ describe("recoverRegistryEntries seeded recovery paths", () => { expect(mockRegistryState.defaultSandbox).toBeNull(); }); - it("blocks route mutation after seeded recovery persists a live row without route metadata (#6315)", async () => { + it("blocks shared route preflight after seeded recovery persists a live row without route metadata (#6315)", async () => { mockRegistryState.sandboxes.gamma = gammaEntry([]); mockRegistryState.defaultSandbox = "gamma"; vi.mocked(captureOpenshell).mockReturnValue({ @@ -270,28 +269,26 @@ describe("recoverRegistryEntries seeded recovery paths", () => { model: null, }); - const deps = createDeps({ - config: {}, - entries: Object.values(mockRegistryState.sandboxes), - defaultSandbox: "gamma", - }); - await expect( - runInferenceSet( - { provider: "nvidia-nim", model: "nvidia/model-b", sandboxName: "gamma" }, - deps, - ), - ).rejects.toMatchObject({ - message: expect.stringMatching(/recovered-live.*lacks durable provider or model metadata/s), - exitCode: 2, - }); - - expect(deps.calls.rewriteConfigUrlsWithDnsPinning).not.toHaveBeenCalled(); - expect(deps.calls.captureOpenshell).not.toHaveBeenCalled(); - expect(deps.calls.readSandboxConfig).not.toHaveBeenCalled(); - expect(deps.calls.writeSandboxConfig).not.toHaveBeenCalled(); - expect(deps.calls.updateSandbox).not.toHaveBeenCalled(); - expect(deps.calls.appendAuditEntry).not.toHaveBeenCalled(); - expect(deps.calls.restartSandboxGateway).not.toHaveBeenCalled(); + const before = structuredClone(mockRegistryState); + const callsBefore = vi.mocked(captureOpenshell).mock.calls.slice(); + expect(() => + prepareInferenceSetRoute({ + entry: mockRegistryState.sandboxes.gamma!, + sandboxName: "gamma", + provider: "registered-provider", + model: "nvidia/model-b", + customRoute: {}, + session: null, + sandboxes: Object.values(mockRegistryState.sandboxes), + }), + ).toThrowError( + expect.objectContaining({ + message: expect.stringMatching(/recovered-live.*lacks durable provider or model metadata/s), + exitCode: 2, + }), + ); + expect(mockRegistryState).toEqual(before); + expect(vi.mocked(captureOpenshell).mock.calls).toEqual(callsBefore); }); it("recovers only the targeted gateway's sandboxes, not a sibling gateway's (#7105)", async () => { diff --git a/src/lib/sandbox/build-context.ts b/src/lib/sandbox/build-context.ts index 760d10b1875..f00da63bfb7 100644 --- a/src/lib/sandbox/build-context.ts +++ b/src/lib/sandbox/build-context.ts @@ -176,7 +176,7 @@ function stageSecurityInventorySources(rootDir: string, buildCtx: string): void function stageNativeInferenceConfig(rootDir: string, buildCtx: string): void { const relativeDir = path.join("src", "lib", "inference", "native-provider"); fs.mkdirSync(path.join(buildCtx, relativeDir), { recursive: true }); - for (const fileName of ["agent-config.ts", "hosted.ts", "contract.ts"]) { + for (const fileName of ["agent-config.ts", "catalog.ts"]) { fs.copyFileSync( path.join(rootDir, relativeDir, fileName), path.join(buildCtx, relativeDir, fileName), @@ -402,6 +402,10 @@ function stageOptimizedSandboxBuildContext( path.join(rootDir, "scripts", "patch-openclaw-tool-catalog.mts"), path.join(stagedScriptsDir, "patch-openclaw-tool-catalog.mts"), ); + fs.copyFileSync( + path.join(rootDir, "scripts", "lib", "patch-openclaw-native-network.mts"), + path.join(stagedScriptsDir, "lib", "patch-openclaw-native-network.mts"), + ); fs.copyFileSync( path.join(rootDir, "scripts", "lib", "patch-openclaw-dependency-overrides.mts"), path.join(stagedScriptsDir, "lib", "patch-openclaw-dependency-overrides.mts"), diff --git a/src/lib/sandbox/compose-sandbox-config-body.test.ts b/src/lib/sandbox/compose-sandbox-config-body.test.ts index 1cd86899f78..421390fd5dc 100644 --- a/src/lib/sandbox/compose-sandbox-config-body.test.ts +++ b/src/lib/sandbox/compose-sandbox-config-body.test.ts @@ -253,6 +253,20 @@ describe("composeSandboxConfigBody", () => { }); it.each([ + [ + "compatible-endpoint", + "inference", + "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "http://host.openshell.internal:11434/v1", + "openai-completions", + ], + [ + "ollama-local", + "inference", + "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "http://host.openshell.internal:11435/v1", + "openai-completions", + ], [ "compatible-endpoint", "inference", @@ -354,7 +368,9 @@ describe("composeSandboxConfigBody", () => { { value: "openshell:resolve:env:NVIDIA_INFERENCE_API_KEY", expectedStatus: 1 }, { value: "openshell:resolve:env:v43_OTHER_API_KEY", expectedStatus: 1 }, ].flatMap((scenario) => - ["NVIDIA_INFERENCE_API_KEY", "COMPATIBLE_API_KEY"].map((key) => ({ ...scenario, key })), + ["NVIDIA_INFERENCE_API_KEY", "COMPATIBLE_API_KEY", "NEMOCLAW_LOCAL_INFERENCE_TOKEN"].map( + (key) => ({ ...scenario, key }), + ), ), )( "refuses invalid native $key handle $value before config mutation", @@ -368,7 +384,10 @@ describe("composeSandboxConfigBody", () => { { dotpath: "models.providers.inference", value: { - baseUrl: NVIDIA_HOSTED_NATIVE_ENDPOINT, + baseUrl: + key === "NEMOCLAW_LOCAL_INFERENCE_TOKEN" + ? "http://host.openshell.internal:11434/v1" + : NVIDIA_HOSTED_NATIVE_ENDPOINT, apiKey: key === "NVIDIA_INFERENCE_API_KEY" ? NVIDIA_INFERENCE_PLACEHOLDER @@ -565,6 +584,39 @@ describe("composeSandboxConfigBody", () => { ).toThrow(); }); + it.each([ + { dotpath: "models.providers.openai", patch: {} }, + { + dotpath: "models.providers.inference", + patch: { apiKey: "openshell:resolve:env:COMPATIBLE_API_KEY" }, + }, + { + dotpath: "models.providers.inference", + patch: { baseUrl: "https://fixture.example/v1" }, + }, + { + dotpath: "models.providers.inference", + patch: { headers: { "X-NemoClaw-Upstream-Provider": "openai-api" } }, + }, + ])( + "refuses mismatched local credential projection %j before config mutation", + ({ dotpath, patch }) => { + expect(() => + buildOpenClawNativeConfigBatchInvocation("alpha", [ + { + dotpath, + value: { + baseUrl: "http://host.openshell.internal:11434/v1", + apiKey: "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + headers: { "X-NemoClaw-Upstream-Provider": "compatible-endpoint" }, + ...patch, + }, + }, + ]), + ).toThrow(); + }, + ); + it("rejects a custom native path mismatch or ambiguous batch before config mutation (#13195)", () => { const update = { dotpath: "models.providers.anthropic", diff --git a/src/lib/sandbox/config-format.ts b/src/lib/sandbox/config-format.ts index 0ef184ed026..e01bc75659c 100644 --- a/src/lib/sandbox/config-format.ts +++ b/src/lib/sandbox/config-format.ts @@ -2,6 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 import JSON5 from "json5"; +import type { AgentConfigTarget } from "./agent-config"; +import { buildHermesUpstreamHeader } from "./hermes-upstream-header"; import { assertSafeConfigStructure } from "../security/config-structure.js"; @@ -55,3 +57,15 @@ export function serializeConfig(config: ConfigObject, format: string): string { } return JSON.stringify(config, null, 2); } + +/** Include the Hermes upstream header in its serialized configuration. */ +export function composeSandboxConfigBody(config: ConfigObject, target: AgentConfigTarget): string { + const body = + target.agentName === "hermes" && target.format === "yaml" + ? require("yaml").stringify(config, { compat: "yaml-1.1" }) + : serializeConfig(config, target.format); + if (target.agentName === "hermes" && target.format === "yaml") { + return `${buildHermesUpstreamHeader(config as Record)}${body}`; + } + return body; +} diff --git a/src/lib/sandbox/config.ts b/src/lib/sandbox/config.ts index d1ad0ecd770..98f2d4bd869 100644 --- a/src/lib/sandbox/config.ts +++ b/src/lib/sandbox/config.ts @@ -12,6 +12,8 @@ // config set: Validated host mutation for Hermes. // config rotate-token: Credential rotation via stdin or env var. +import { writeHermesConfigNatively } from "./hermes-native-config"; +import { resolveSandboxConfigRuntimeSelection } from "../actions/sandbox/mcp-bridge-provider-inspection"; import type { AgentConfigTarget } from "./agent-config"; import type { OpenShellRuntimeSelection } from "../adapters/openshell/client"; import { @@ -41,7 +43,6 @@ const os = require("os"); const path = require("path"); const { promises: dnsPromises } = require("node:dns"); const { isIP } = require("node:net"); -const { isErrnoException }: typeof import("../core/errno") = require("../core/errno"); const { validateName, shellQuote } = require("../runner"); const credentialFilter: typeof import("../security/credential-filter") = require("../security/credential-filter"); const { stripCredentials, isConfigObject, isConfigValue, isCredentialField } = credentialFilter; @@ -58,12 +59,9 @@ const { capturePrivilegedSandboxCommand, withPrivilegedSandboxExecutionLease, }: typeof import("./privileged-exec") = require("./privileged-exec"); -const { - buildHermesUpstreamHeader, -}: typeof import("./hermes-upstream-header") = require("./hermes-upstream-header"); const { parseConfig, - serializeConfig, + composeSandboxConfigBody, }: typeof import("./config-format") = require("./config-format"); const { buildOpenClawNativeConfigBatchInvocation, @@ -152,12 +150,11 @@ async function restartSandboxAgentAfterConfigSet( (require("../actions/sandbox/process-recovery").restartSandboxGateway as ManagedGatewayRestart); const result = await restart(sandboxName, ...(runtimeSelection ? [{ runtimeSelection }] : [])); if (!result.ok) { - // The config write already completed, but the running agent was not - // reloaded. Say so plainly and point to the idempotent retry rather than - // leaving disk and the live gateway silently diverged. The restart layer - // has already printed its own failure detail. + // The Hermes native transaction verifies its reload before this optional restart. configFail([ - ` Config was written to disk but NOT applied to the running agent.`, + agentName === "hermes" + ? " Hermes configuration was applied and its reload verified. The additional restart was not confirmed." + : " Config was written to disk but NOT applied to the running agent.", ` The ${agentName} gateway restart did not complete for '${sandboxName}' (see the failure above).`, ` Retry the restart with: nemoclaw ${shellQuote(sandboxName)} gateway restart`, ]); @@ -166,8 +163,8 @@ async function restartSandboxAgentAfterConfigSet( function buildConfigSetRestartGuidance(sandboxName: string): string[] { return [ - " Note: Hermes may restart its gateway when it applies this configuration.", - ` Use --restart to request and verify a restart, or run: nemoclaw ${shellQuote(sandboxName)} gateway restart`, + " Hermes verified its gateway reload after applying this configuration.", + ` For another verified restart, use --restart or run: nemoclaw ${shellQuote(sandboxName)} gateway restart`, ]; } @@ -383,22 +380,6 @@ function classifyNewKeyGate(inputs: NewKeyGateInputs): NewKeyGate { return { mode: "prompt" }; } -/** - * Pure body composition for {@link writeSandboxConfig}: serialize the config - * and prepend agent-specific headers. Extracted so unit tests can assert the - * exact byte sequence that lands in the sandbox without driving the - * privileged docker exec path. - */ -function composeSandboxConfigBody(config: ConfigObject, target: AgentConfigTarget): string { - if (target.agentName === "hermes" && target.format === "yaml") { - // Hermes/PyYAML reads YAML 1.1. Preserve strings such as tool_search's "on" - // when a credential-only restore writes a config sanitized as JSON. - const body = require("yaml").stringify(config, { compat: "yaml-1.1" }); - return `${buildHermesUpstreamHeader(config as Record)}${body}`; - } - return serializeConfig(config, target.format); -} - /** * Parse a CLI-provided config value as JSON when possible, otherwise keep it * as a string literal. @@ -539,8 +520,19 @@ function writeSandboxConfig( sandboxName: string, target: AgentConfigTarget, config: ConfigObject, - options: { restoreNativeLocalCredential?: true } = {}, + optionsOrGateway: + | { + gatewayName?: string; + workspace?: string; + localTlsDir?: string; + deferReload?: true; + restoreNativeLocalCredential?: true; + } + | string = {}, ): void { + const nativeGatewayName = + typeof optionsOrGateway === "string" ? optionsOrGateway : optionsOrGateway.gatewayName; + const options = typeof optionsOrGateway === "string" ? {} : optionsOrGateway; if (target.agentName === "openclaw") { throw new Error( "Refusing a whole-file OpenClaw config write; use OpenClaw's native config commands.", @@ -556,6 +548,29 @@ function writeSandboxConfig( "Refusing Hermes config write without the digest from the matching sandbox read.", ); } + if (nativeGatewayName) { + writeHermesConfigNatively( + getOpenshellBinary(), + sandboxName, + nativeGatewayName, + expectedConfigSha256, + content, + runOpenshellCommand, + { + ...options, + ...(options.workspace !== undefined + ? { + env: buildSelectedOpenShellSubprocessEnv({ + gatewayName: nativeGatewayName, + workspace: options.workspace, + ...(options.localTlsDir ? { localTlsDir: options.localTlsDir } : {}), + }), + } + : {}), + }, + ); + return; + } try { privilegedSandboxExec( sandboxName, @@ -1079,9 +1094,10 @@ async function configSet(sandboxName: string, opts: ConfigSetOpts = {}): Promise const target = resolveAgentConfig(sandboxName); const unsupportedAgent = configSetUnsupportedAgentMessage(target, configKey, shellQuote); if (unsupportedAgent) configFail(unsupportedAgent); + const runtimeSelection = resolveSandboxConfigRuntimeSelection(sandboxName); // Read current config console.log(` Reading ${target.agentName} config...`); - const config = readSandboxConfig(sandboxName, target); + const config = readSandboxConfig(sandboxName, target, runtimeSelection); const initialConfigSha256 = (config as ConfigObject & { [CONFIG_SOURCE_SHA256]?: string })[ CONFIG_SOURCE_SHA256 ]; @@ -1139,7 +1155,12 @@ async function configSet(sandboxName: string, opts: ConfigSetOpts = {}): Promise confirmed = await confirmYesNo(" Write this new key? [y/N] "); } catch (error) { // The shared prompt re-raises SIGINT; only EOF needs config-specific remediation here. - if (isErrnoException(error) && error.code === "EOF") { + if ( + typeof error === "object" && + error !== null && + "code" in error && + error.code === "EOF" + ) { configFail([ " No input available on stdin, so config set cannot confirm the new key.", " Re-run with --config-accept-new-path or set NEMOCLAW_CONFIG_ACCEPT_NEW_PATH=1.", @@ -1182,7 +1203,17 @@ async function configSet(sandboxName: string, opts: ConfigSetOpts = {}): Promise const telemetryMetadataErrors: import("../domain/telemetry/event").TelemetryMetadataError[] = []; // Re-read under the sandbox mutation lock and enforce the source digest. await withSandboxMutationLock(sandboxName, () => { - const currentConfig = readSandboxConfig(sandboxName, target); + const currentRuntime = resolveSandboxConfigRuntimeSelection(sandboxName); + if ( + currentRuntime.gatewayName !== runtimeSelection.gatewayName || + currentRuntime.workspace !== runtimeSelection.workspace || + currentRuntime.localTlsDir !== runtimeSelection.localTlsDir + ) { + configFail( + " Sandbox runtime target changed while validating configuration. Retry config set.", + ); + } + const currentConfig = readSandboxConfig(sandboxName, target, runtimeSelection); const currentConfigSha256 = ( currentConfig as ConfigObject & { [CONFIG_SOURCE_SHA256]?: string } )[CONFIG_SOURCE_SHA256]; @@ -1194,25 +1225,24 @@ async function configSet(sandboxName: string, opts: ConfigSetOpts = {}): Promise setDotpath(currentConfig, opts.key!, safeValue); console.log(` Writing config to sandbox (${target.configPath})...`); - writeSandboxConfig(sandboxName, target, currentConfig); + writeSandboxConfig(sandboxName, target, currentConfig, runtimeSelection); telemetryPersistenceVerified = persistConfigurationTelemetry( sandboxName, configKey, currentConfig, - Object.is(oldValue, safeValue) && !opts.restart ? undefined : true, + false, !Object.is(oldValue, safeValue), ); if (!telemetryPersistenceVerified) { telemetryMetadataErrors.push({ category: "native_model_source" }); - if (!Object.is(oldValue, safeValue) || opts.restart) - telemetryMetadataErrors.push({ category: "configuration_apply_state" }); + telemetryMetadataErrors.push({ category: "configuration_apply_state" }); } if (isTelemetryConfigurationKey(configKey)) { recordTelemetryTarget({ scope: "configuration", sandboxName, outcome: Object.is(oldValue, safeValue) && !opts.restart ? "no_change" : "completed", - state: Object.is(oldValue, safeValue) && !opts.restart ? "unchanged" : "pending", + state: Object.is(oldValue, safeValue) && !opts.restart ? "unchanged" : "applied", ...(!telemetryPersistenceVerified ? { verificationStatus: "collection_error" as const, @@ -1234,7 +1264,12 @@ async function configSet(sandboxName: string, opts: ConfigSetOpts = {}): Promise // Restart if requested if (opts.restart) { try { - await restartSandboxAgentAfterConfigSet(sandboxName, target.agentName); + await restartSandboxAgentAfterConfigSet( + sandboxName, + target.agentName, + undefined, + runtimeSelection, + ); } catch (error) { if (isTelemetryConfigurationKey(configKey)) { recordTelemetryTarget({ diff --git a/src/lib/sandbox/hermes-native-config.test.ts b/src/lib/sandbox/hermes-native-config.test.ts new file mode 100644 index 00000000000..4c4b8c693c1 --- /dev/null +++ b/src/lib/sandbox/hermes-native-config.test.ts @@ -0,0 +1,86 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { writeHermesConfigNatively } from "./hermes-native-config"; + +const execute = vi.fn(); +const content = "model:\n api_key: private-input\n"; +const digest = "a".repeat(64); + +describe("native Hermes configuration transport", () => { + beforeEach(() => { + execute.mockReturnValue({ + status: 0, + stdout: "updated=1\n", + stderr: "", + signal: null, + pid: 1, + output: [], + }); + }); + + it("sends private configuration only through stdin to the selected gateway", () => { + writeHermesConfigNatively( + "/pinned/openshell", + "hermes", + "selected-gateway", + digest, + content, + execute, + ); + const [binary, argv, options] = execute.mock.calls[0]!; + expect(binary).toBe("/pinned/openshell"); + expect(argv.slice(0, 6)).toEqual([ + "-g", + "selected-gateway", + "sandbox", + "exec", + "--name", + "hermes", + ]); + expect(argv.at(-1)).toBe(digest); + expect(argv.join(" ")).not.toContain("private-input"); + expect(options).toMatchObject({ input: content, stdio: "pipe", timeout: 650_000 }); + }); + + it("uses the supplied runtime environment without merging ambient selectors", () => { + vi.stubEnv("OPENSHELL_GATEWAY", "ambient-gateway"); + vi.stubEnv("OPENSHELL_WORKSPACE", "ambient-workspace"); + vi.stubEnv("OPENSHELL_LOCAL_TLS_DIR", "/ambient/tls"); + vi.stubEnv("OPENSHELL_TOKEN", "ambient-token"); + writeHermesConfigNatively( + "/pinned/openshell", + "hermes", + "recorded-gateway", + digest, + content, + execute, + { env: { OPENSHELL_GATEWAY: "recorded-gateway", OPENSHELL_WORKSPACE: "recorded-workspace" } }, + ); + const [, argv, options] = execute.mock.calls[0]!; + expect(argv.slice(0, 2)).toEqual(["-g", "recorded-gateway"]); + expect(options).toMatchObject({ + replaceEnv: true, + env: { + OPENSHELL_GATEWAY: "recorded-gateway", + OPENSHELL_WORKSPACE: "recorded-workspace", + }, + }); + expect(options?.env).not.toHaveProperty("OPENSHELL_LOCAL_TLS_DIR"); + expect(options?.env).not.toHaveProperty("OPENSHELL_TOKEN"); + }); + + it.each([ + { status: 1, stdout: "", stderr: "private-input" }, + { status: 0, stdout: "unexpected", stderr: "private-input" }, + { status: null, signal: "SIGTERM" as const, stdout: "", stderr: "private-input" }, + ])("rejects an unconfirmed write without exposing private diagnostics %j", (result) => { + execute.mockReturnValue({ signal: null, pid: 1, output: [], ...result }); + expect(() => + writeHermesConfigNatively("openshell", "hermes", "gateway", digest, content, execute), + ).toThrow( + "Native Hermes configuration transaction failed; configuration application was not confirmed.", + ); + }); +}); diff --git a/src/lib/sandbox/hermes-native-config.ts b/src/lib/sandbox/hermes-native-config.ts new file mode 100644 index 00000000000..92e9510571e --- /dev/null +++ b/src/lib/sandbox/hermes-native-config.ts @@ -0,0 +1,174 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Use the image's existing same-UID lifecycle, transaction lock, descriptor +// checks and hash owner. Configuration travels through stdin, never argv. +export const HERMES_NATIVE_CONFIG_WRITE = String.raw` +import hashlib, importlib.util, os, re, secrets, sys + +def update(): + expected = sys.argv[1] + mode = sys.argv[2] if len(sys.argv) > 2 else "reload" + if mode not in ("reload", "deferred", "restore-native-local"): + raise RuntimeError("Invalid native Hermes configuration mode") + if not re.fullmatch(r"[0-9a-f]{64}", expected) or os.geteuid() == 0: + raise RuntimeError("Invalid native Hermes configuration transaction") + spec = importlib.util.spec_from_file_location("nemoclaw_mcp", "/usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py") + helper = importlib.util.module_from_spec(spec) + spec.loader.exec_module(helper) + helper._assert_non_root_lifecycle_identity() + helper._configure_gateway_public_port() + guard = helper._load_guard() + candidate = sys.stdin.buffer.read(guard.MAX_CONFIG_INPUT_BYTES + 1) + if len(candidate) > guard.MAX_CONFIG_INPUT_BYTES: + raise RuntimeError("Hermes configuration input is too large") + guard._validate_hermes_config_candidate(candidate) + replacement = candidate.decode("utf-8") + with helper._mcp_transaction_lock(): + helper._assert_non_root_lifecycle_identity() + original, original_snapshot = guard._read_text(helper.CONFIG_PATH) + helper._assert_mutable_snapshot(original_snapshot) + if not secrets.compare_digest(hashlib.sha256(original.encode("utf-8")).hexdigest(), expected): + raise RuntimeError("Hermes configuration changed after it was read") + if guard._canonical_mcp_servers_digest(original) != guard._canonical_mcp_servers_digest(replacement): + raise RuntimeError("Inference configuration cannot change MCP servers") + hashes = {path: guard._read_text(path) for path in helper._managed_hash_paths(False)} + if mode == "restore-native-local": + source, source_snapshot, _env, env_snapshot = guard._validate_restored_native_local_source( + helper.HERMES_DIR, expected, candidate, + ) + if source != original or source_snapshot != original_snapshot: + raise RuntimeError("Restored Hermes configuration changed during validation") + # Archive hashes can belong to the prior generation. Adopt only the + # validated credential replacement, preserving every other input. + snapshots = [(helper.CONFIG_PATH, source_snapshot), + (helper.HERMES_DIR + "/.env", env_snapshot)] + for path, (text, snapshot) in hashes.items(): + guard._parse_config_hash(text, helper.CONFIG_PATH, helper.HERMES_DIR + "/.env") + snapshots.append((path, snapshot)) + metadata = guard._inode_metadata(os.lstat(helper.HERMES_DIR)) + file_states = {os.path.basename(path): {"original": { + "uid": snapshot.uid, "gid": snapshot.gid, "mode": snapshot.mode, + }} for path, snapshot in snapshots} + if not guard._mutable_nonroot_reconciliation_posture_is_allowed(metadata, file_states): + raise RuntimeError("Restored Hermes inputs are not owned mutable sandbox state") + for path, snapshot in snapshots: + helper._assert_mutable_snapshot(snapshot) + opened = guard._open_regular(path) + try: + if opened.snapshot != snapshot: + raise RuntimeError("Restored Hermes inputs changed before writing") + finally: + opened.close() + else: + integrity = guard.inspect_mcp_integrity_snapshot(helper.HERMES_DIR, helper.HERMES_DIR + "/.config-hash") + guard.assert_mcp_integrity_snapshot_current(integrity) + env_snapshot = integrity.env_snapshot + replacement_snapshot = None + def verify_written_inputs(): + for path, snapshot in ((helper.CONFIG_PATH, replacement_snapshot), + (helper.HERMES_DIR + "/.env", env_snapshot)): + opened = guard._open_regular(path) + try: + if opened.snapshot != snapshot: + raise RuntimeError("Hermes inputs changed during configuration commit") + finally: + opened.close() + try: + guard._write_existing(helper.CONFIG_PATH, replacement, original_snapshot) + written, observed_snapshot = guard._read_text(helper.CONFIG_PATH) + if written != replacement: + raise RuntimeError("Hermes configuration changed during the transaction") + replacement_snapshot = observed_snapshot + environment = guard._open_regular(helper.HERMES_DIR + "/.env") + try: + if environment.snapshot != env_snapshot: + raise RuntimeError("Hermes environment changed during configuration write") + finally: + environment.close() + helper._refresh_and_verify_hashes(guard, False) + verify_written_inputs() + if mode == "reload" and not helper.reload_gateway(): + raise RuntimeError("Hermes gateway stopped before configuration reload") + current, current_snapshot = guard._read_text(helper.CONFIG_PATH) + if current != replacement: + raise RuntimeError("Hermes configuration changed during reload") + replacement_snapshot = current_snapshot + helper._refresh_and_verify_hashes(guard, False) + verify_written_inputs() + except Exception: + if replacement_snapshot is not None: + # The guarded replacement refuses to overwrite a concurrent writer. + guard._write_existing(helper.CONFIG_PATH, original, replacement_snapshot) + helper._restore_hash_snapshots(guard, hashes) + if mode == "reload" and not helper.reload_gateway(): + raise RuntimeError("Hermes configuration rollback reload was not confirmed") + raise + print("updated=1") + +try: + update() +except Exception: + # Config parser and filesystem errors can include private values. + sys.stderr.write("Native Hermes configuration transaction failed; configuration application was not confirmed.\n") + sys.exit(1) +`; + +export function writeHermesConfigNatively( + binary: string, + sandboxName: string, + gateway: string, + expectedConfigSha256: string, + content: string, + execute: typeof import("../adapters/openshell/client").runOpenshellCommand, + options: { + deferReload?: true; + restoreNativeLocalCredential?: true; + env?: Record; + } = {}, +): void { + const result = execute( + binary, + [ + "-g", + gateway, + "sandbox", + "exec", + "--name", + sandboxName, + "--no-tty", + "--no-login-shell", + "--timeout", + "620", + "--", + "/opt/hermes/.venv/bin/python", + "-I", + "-c", + HERMES_NATIVE_CONFIG_WRITE, + expectedConfigSha256, + ...(options.restoreNativeLocalCredential + ? ["restore-native-local"] + : options.deferReload + ? ["deferred"] + : []), + ], + { + ...(options.env ? { env: options.env, replaceEnv: true } : {}), + input: content, + timeout: 650_000, + maxBuffer: 64 * 1024, + ignoreError: true, + stdio: "pipe", + }, + ); + if ( + result.error || + result.signal || + result.status !== 0 || + result.stdout?.trim() !== "updated=1" + ) { + throw new Error( + "Native Hermes configuration transaction failed; configuration application was not confirmed.", + ); + } +} diff --git a/src/lib/sandbox/mutable-config-perms.test.ts b/src/lib/sandbox/mutable-config-perms.test.ts index ed2f6a52826..7452b76d650 100644 --- a/src/lib/sandbox/mutable-config-perms.test.ts +++ b/src/lib/sandbox/mutable-config-perms.test.ts @@ -8,25 +8,49 @@ import path from "node:path"; import { beforeEach, describe, expect, it, vi } from "vitest"; -const { capturePrivilegedSandboxCommand, resolveAgentConfig, withMcpLifecycleLockSync } = - vi.hoisted(() => ({ - capturePrivilegedSandboxCommand: vi.fn(), - resolveAgentConfig: vi.fn(), - withMcpLifecycleLockSync: vi.fn((_name, action) => action()), - })); - -vi.mock("./privileged-exec", () => ({ - capturePrivilegedSandboxCommand, +const { + captureSanitizedResolvedOpenshell, + getSandbox, + resolveAgentConfig, + withMcpLifecycleLockSync, +} = vi.hoisted(() => ({ + captureSanitizedResolvedOpenshell: vi.fn(), + getSandbox: vi.fn(), + resolveAgentConfig: vi.fn(), + withMcpLifecycleLockSync: vi.fn((_name, action) => action()), +})); + +vi.mock("../adapters/openshell/sanitized-capture", () => ({ + captureSanitizedResolvedOpenshell, })); vi.mock("./agent-config", () => ({ resolveAgentConfig })); vi.mock("../state/mcp-lifecycle-lock-acquisition", () => ({ withMcpLifecycleLockSync })); import type { AgentConfigTarget } from "./agent-config"; import { + inspectMutableHermesConfigPerms, mutableHermesConfigProbeCommand, verifyMutableHermesConfigForTarget, } from "./mutable-config-perms"; +// The host has no sandbox account; retain real file ownership and execution identity. +const localSandboxIdentity = ` +import os, pwd, types, runpy +sandbox_uid, sandbox_gid = os.geteuid(), os.getegid() +pwd.getpwnam = lambda name: types.SimpleNamespace(pw_uid=sandbox_uid, pw_gid=sandbox_gid) +startup_fixture = {"requires_completion": lambda: True, "wait_for_completion": lambda: "fixture", "read_completion": lambda: "fixture"} +def topology_fixture(): + return None +def load_runtime_helper(path): + if path.endswith("hermes-startup-completion.py"): + if startup_fixture is None: + raise FileNotFoundError(2, "missing helper", path) + return startup_fixture + assert path == "/usr/local/lib/nemoclaw/hermes-runtime-config-guard.py" + return {"inspect_private_mutable_topology": lambda: topology_fixture()} +runpy.run_path = load_runtime_helper +`; + const hermesTarget: AgentConfigTarget = { agentName: "hermes", configDir: "/sandbox/.hermes", @@ -39,8 +63,86 @@ describe("mutable Hermes config permissions", () => { beforeEach(() => { vi.clearAllMocks(); resolveAgentConfig.mockReturnValue(hermesTarget); + getSandbox.mockReturnValue({ name: "hermes-demo", agent: "hermes", gatewayPort: 40808 }); + captureSanitizedResolvedOpenshell.mockReturnValue({ status: 0, output: "" }); + }); + + it("inspects the registered gateway as its native user while holding the lifecycle lock", () => { + withMcpLifecycleLockSync.mockImplementationOnce((_name, action) => { + expect(captureSanitizedResolvedOpenshell).not.toHaveBeenCalled(); + const result = action(); + expect(captureSanitizedResolvedOpenshell).toHaveBeenCalledOnce(); + return result; + }); + expect(inspectMutableHermesConfigPerms("hermes-demo", getSandbox)).toEqual({ + verified: true, + errors: [], + }); + expect(withMcpLifecycleLockSync).toHaveBeenCalledWith("hermes-demo", expect.any(Function)); + expect(captureSanitizedResolvedOpenshell).toHaveBeenCalledWith( + [ + "sandbox", + "exec", + "--name", + "hermes-demo", + "-g", + "nemoclaw-40808", + "--no-tty", + "--no-login-shell", + "--timeout", + "30", + "--", + ...mutableHermesConfigProbeCommand(hermesTarget), + ], + expect.objectContaining({ timeout: 35_000, maxBuffer: 64 * 1024 }), + ); + expect(mutableHermesConfigProbeCommand(hermesTarget)[0]).toBe("/opt/hermes/.venv/bin/python3"); + }); + + it.each([ + { status: 1, output: "" }, + { status: null, output: "", signal: "SIGTERM" }, + { status: 0, output: "", error: new Error("transport failed") }, + ])("rejects an incomplete native permission probe: %j", (result) => { + captureSanitizedResolvedOpenshell.mockReturnValue(result); + expect(inspectMutableHermesConfigPerms("hermes-demo", getSandbox).verified).toBe(false); + expect(captureSanitizedResolvedOpenshell).toHaveBeenCalledOnce(); + }); + + it("retains bounded probe failure details with credentials redacted and controls escaped", () => { + captureSanitizedResolvedOpenshell.mockReturnValue({ + status: 1, + output: "unused stdout", + stdout: "OPENAI_API_KEY=stdout-secret", + stderr: + "x".repeat(5000) + + "\nPermissionError: config.yaml remains read-only\n" + + "MSTEAMS_APP_PASSWORD=diagnostic-test-secret\n" + + "https://diagnostic-user:diagnostic-password@example.test/?token=diagnostic-token\n" + + "\u001b[2J\u009b\u202euntrusted output", + }); + + const verification = inspectMutableHermesConfigPerms("hermes-demo", getSandbox); + const message = verification.errors[0]!; + + expect(verification.verified).toBe(false); + expect(message).toContain("PermissionError: config.yaml remains read-only"); + expect(message).toContain(""); + expect(message).not.toMatch(/diagnostic-(?:test-secret|user|password|token)|stdout-secret/); + expect(message).not.toMatch(/[\p{Cc}\p{Cf}]/u); + expect(message).toContain("\\u{001b}[2J"); + expect(message.length).toBeLessThanOrEqual(4096 + "Hermes config probe failed ()".length); }); + it.each([null, { gatewayName: "unregistered-gateway" }])( + "rejects missing or invalid registry authority: %j", + (entry) => { + getSandbox.mockReturnValue(entry); + expect(inspectMutableHermesConfigPerms("hermes-demo", getSandbox).verified).toBe(false); + expect(captureSanitizedResolvedOpenshell).not.toHaveBeenCalled(); + }, + ); + it("claims mutable Hermes posture only after the exact probe succeeds", () => { const execute = vi.fn(); @@ -59,11 +161,11 @@ describe("mutable Hermes config permissions", () => { it("executes the Hermes probe and rejects read-only or linked config artifacts", () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-mutable-probe-")); + fs.chownSync(root, process.getuid!(), process.getgid!()); const configDir = path.join(root, ".hermes"); const configPath = path.join(configDir, "config.yaml"); const hashPath = path.join(configDir, ".config-hash"); const envPath = path.join(configDir, ".env"); - const commandShim = path.join(root, "run-privileged-command.sh"); fs.mkdirSync(configDir, { mode: 0o700 }); fs.chmodSync(configDir, 0o3770); fs.writeFileSync(configPath, "fixture\n", { mode: 0o640 }); @@ -72,19 +174,6 @@ describe("mutable Hermes config permissions", () => { fs.chmodSync(configPath, 0o640); fs.chmodSync(hashPath, 0o640); fs.chmodSync(envPath, 0o640); - fs.writeFileSync( - commandShim, - [ - "#!/usr/bin/env bash", - "set -euo pipefail", - "shift", - 'while [ "$#" -gt 0 ] && [ "$1" != "--" ]; do shift; done', - '[ "${1:-}" = "--" ]', - "shift", - 'exec "$@"', - ].join("\n"), - { mode: 0o700 }, - ); const fixtureTarget: AgentConfigTarget = { ...hermesTarget, @@ -94,9 +183,14 @@ describe("mutable Hermes config permissions", () => { }; const runProbe = () => verifyMutableHermesConfigForTarget(fixtureTarget, (command) => { - const result = spawnSync(commandShim, [...command], { - encoding: "utf8", - }); + const codeIndex = command.indexOf("-c") + 1; + const result = spawnSync( + "python3", + ["-I", "-c", localSandboxIdentity + command[codeIndex], ...command.slice(codeIndex + 1)], + { + encoding: "utf8", + }, + ); expect(result.error).toBeUndefined(); expect(result.status, result.stderr).toBe(0); }); @@ -135,19 +229,157 @@ describe("mutable Hermes config permissions", () => { }); it.each([ - [0o3770, "unavailable", true], - [0o700, "verified", true], - [0o700, "unavailable", false], - [0o700, "changed", false], - [0o700, "directory-race", false], - [0o3770, "access-denied", false], - [0o3770, "append-only-root", false], - [0o3770, "immutable-artifact", false], - [0o750, "verified", false], + ["created", false, "shared-state.db-shm=created"], + ["removed", false, "shared-state.db-wal=removed"], + ["replaced", false, "shared-state.db=replaced"], + ["metadata", false, "shared-state.db=metadata-changed"], + ["symlink", false, "gateway_state.json=replaced"], + ["unavailable", false, "gateway_state.json=unavailable"], + ["unlisted", false, "known entries in inspected directory: none observed"], + ["stable", true, ""], + ] as const)( + "reports fixed runtime entry metadata for %s without reading contents or retrying", + (scenario, expected, diagnostic) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-entry-metadata-")); + fs.chownSync(root, process.getuid!(), process.getgid!()); + const configDir = path.join(root, ".hermes"); + fs.mkdirSync(configDir, { mode: 0o700 }); + fs.writeFileSync(path.join(configDir, "config.yaml"), "config-fixture", { mode: 0o640 }); + fs.writeFileSync(path.join(configDir, "shared-state.db"), "private-state-canary"); + fs.writeFileSync(path.join(configDir, "shared-state.db-wal"), "private-state-canary"); + const outside = path.join(root, "private-target-canary"); + fs.writeFileSync(outside, "private-target-content"); + fs.symlinkSync(outside, path.join(configDir, "gateway_state.json")); + const command = mutableHermesConfigProbeCommand({ + ...hermesTarget, + configDir, + configPath: path.join(configDir, "config.yaml"), + sensitiveFiles: [], + }); + const codeIndex = command.indexOf("-c") + 1; + const prelude = ` +${localSandboxIdentity} +import pathlib +scenario = ${JSON.stringify(scenario)} +calls = 0 +original_stat = os.stat +known = ("gateway_state.json", "shared-state.db", "shared-state.db-wal", "shared-state.db-shm") +def observed_stat(name, *args, **kwargs): + if name in known: + assert kwargs.get("follow_symlinks") is False + assert isinstance(kwargs.get("dir_fd"), int) + if scenario == "unavailable" and name == "gateway_state.json" and calls == 2: + raise PermissionError("private-error-canary") + return original_stat(name, *args, **kwargs) +os.stat = observed_stat +def replace_entry(): + temporary = pathlib.Path(config_dir) / ".private-entry-canary" + temporary.write_text("private-state-canary") + os.replace(temporary, pathlib.Path(config_dir) / ("gateway_state.json" if scenario == "symlink" else "shared-state.db")) +def transient_entry(): + temporary = pathlib.Path(config_dir) / ".private-entry-canary" + temporary.write_text("private-state-canary") + temporary.unlink() +def topology_fixture(): + global calls + calls += 1 + assert calls <= 2 + if calls == 2: + mutations = { + "created": lambda: (pathlib.Path(config_dir) / "shared-state.db-shm").write_text("private-state-canary"), + "removed": lambda: (pathlib.Path(config_dir) / "shared-state.db-wal").unlink(), + "replaced": replace_entry, + "metadata": lambda: os.chmod(pathlib.Path(config_dir) / "shared-state.db", 0o600), + "symlink": replace_entry, + "unavailable": lambda: None, + "unlisted": transient_entry, + "stable": lambda: None, + } + mutations[scenario]() + if scenario != "stable": + info = original_stat(config_dir) + os.utime(config_dir, ns=(info.st_atime_ns, info.st_mtime_ns + 1000000)) + return None +probe = ${JSON.stringify(command[codeIndex])} +try: + exec(probe) +finally: + assert calls == 2 +`; + try { + const result = spawnSync( + "python3", + ["-I", "-c", prelude, ...command.slice(codeIndex + 1)], + { + encoding: "utf8", + timeout: 15_000, + }, + ); + expect(result.error).toBeUndefined(); + expect(result.status === 0, result.stderr).toBe(expected); + expect(result.stderr).toContain(diagnostic); + expect(result.stderr === "").toBe(expected); + expect(result.stderr.includes("Hermes config root changed during inspection")).toBe( + !expected, + ); + expect(result.stderr).not.toMatch( + /private-(?:state|target|entry|error)-(?:canary|content)/u, + ); + expect(result.stderr).not.toContain(root); + expect(result.stdout).toBe(""); + expect(fs.readFileSync(outside, "utf8")).toBe("private-target-content"); + expect(fs.readFileSync(path.join(configDir, "config.yaml"), "utf8")).toBe("config-fixture"); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, + ); + + it.each([ + [0o3770, "unavailable", true, ""], + [0o700, "verified", true, ""], + [0o700, "guard-import", true, ""], + [0o700, "guard-timeout", false, "Private Hermes config requires an attested same-UID runtime"], + [0o700, "retained-image", true, ""], + [0o3770, "root-separated", true, ""], + [0o700, "unavailable", false, "Private Hermes config requires an attested same-UID runtime"], + [0o700, "changed", false, "Private Hermes config requires an attested same-UID runtime"], + [ + 0o700, + "directory-race", + false, + "Hermes config root changed during inspection: st_mode, st_ctime_ns; directory mtime unchanged", + ], + [ + 0o700, + "directory-timestamp", + false, + "Hermes config root changed during inspection: st_ctime_ns; directory mtime unchanged", + ], + [ + 0o700, + "directory-entry", + false, + "Hermes config root changed during inspection: st_ctime_ns; directory mtime changed", + ], + [0o3770, "access-denied", false, "Hermes config root is not writable by the sandbox identity"], + [0o3770, "append-only-root", false, "Hermes config has immutable or append-only inode flags"], + [0o3770, "immutable-artifact", false, "Hermes config has immutable or append-only inode flags"], + [0o750, "verified", false, "Hermes config root does not have an allowed mutable mode"], + [0o3770, "root", false, "Hermes config probe requires the non-root sandbox identity"], + [0o3770, "wrong-user", false, "Hermes config probe requires the non-root sandbox identity"], + [0o3770, "wrong-group", false, "Hermes config probe requires the non-root sandbox identity"], + [ + 0o3770, + "startup-generation", + false, + "Hermes startup generation changed during config inspection", + ], ] as const)( "checks mode %s with %s topology without changing state", - (mode, topology, expected) => { + (mode, topology, expected, diagnostic) => { const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-private-hermes-probe-")); + fs.chownSync(root, process.getuid!(), process.getgid!()); const configDir = path.join(root, ".hermes"); fs.mkdirSync(configDir, { mode }); fs.chmodSync(configDir, mode); @@ -169,8 +401,27 @@ describe("mutable Hermes config permissions", () => { // Guard tests exercise its real procfs proof separately. // Supply blocking inode flags as external metadata; keep file I/O real. const prelude = ` -import fcntl, os, stat, struct, subprocess, types +${localSandboxIdentity} +import fcntl, os, stat, struct, signal, sys, types topology = ${JSON.stringify(topology)} +original_load_runtime_helper = runpy.run_path +def load_before_snapshot(path): + if topology == "guard-import" and path.endswith("hermes-runtime-config-guard.py"): + directory = sys.argv[1] + before = os.stat(directory) + transient = os.path.join(directory, ".runtime-import-fixture") + with open(transient, "x", encoding="utf-8") as output: + output.write("fixture") + os.unlink(transient) + os.utime(directory, ns=(before.st_atime_ns, before.st_mtime_ns + 1000000)) + return original_load_runtime_helper(path) +runpy.run_path = load_before_snapshot +if topology == "startup-generation": + startup_fixture = {"requires_completion": lambda: True, "wait_for_completion": lambda: "old", "read_completion": lambda: "new"} +elif topology == "retained-image": + startup_fixture = None +elif topology == "root-separated": + startup_fixture = {"requires_completion": lambda: False} original_fstat = os.fstat original_ioctl = fcntl.ioctl def blocking_flags(fd, append_flag, immutable_flag): @@ -190,16 +441,36 @@ def native_flags(fd): return types.SimpleNamespace(**fields) fcntl.ioctl = inode_flags os.fstat = native_flags +if topology == "root": + os.geteuid = lambda: 0 +elif topology == "wrong-user": + os.geteuid = lambda: sandbox_uid + 1 +elif topology == "wrong-group": + os.getegid = lambda: sandbox_gid + 1 calls = 0 -def topology_response(args, **kwargs): +def topology_fixture(): global calls calls += 1 - assert args[3] == "inspect-private-mutable-topology" if topology == "directory-race" and calls == 2: os.chmod(config_dir, 0o750) + if topology == "directory-timestamp" and calls == 2: + info = os.stat(config_dir) + os.utime(config_dir, ns=(info.st_atime_ns, info.st_mtime_ns)) + if topology == "directory-entry" and calls == 2: + info = os.stat(config_dir) + transient = os.path.join(config_dir, ".private-entry-canary") + with open(transient, "x", encoding="utf-8") as output: + output.write("private-state-canary") + os.unlink(transient) + os.utime(config_dir, ns=(info.st_atime_ns, info.st_mtime_ns + 1000000)) + if topology == "guard-timeout": + try: + signal.raise_signal(signal.SIGALRM) + except Exception: + pass success = topology != "unavailable" and not (topology == "changed" and calls == 2) - return types.SimpleNamespace(returncode=0 if success else 1, stdout=b"same-uid-nonroot\\n") -subprocess.run = topology_response + if not success: + raise RuntimeError("topology unavailable") if topology == "access-denied": os.access = lambda *args, **kwargs: False os.mkdir = lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("probe must not create state")) @@ -216,8 +487,15 @@ os.mkdir = lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("probe m ); expect(result.error).toBeUndefined(); expect(result.status === 0, result.stderr).toBe(expected); + expect(result.stderr).toContain(diagnostic); + expect(result.stdout).toBe(""); + expect(result.stderr).not.toContain(configDir); + expect(result.stderr).not.toContain("private-entry-canary"); + expect(result.stderr).not.toContain("private-state-canary"); expect(fs.readdirSync(configDir).sort()).toEqual([...names].sort()); - expect(fs.statSync(configDir).mtimeMs).toBe(before); + expect(fs.statSync(configDir).mtimeMs === before).toBe( + !["directory-entry", "guard-import"].includes(topology), + ); expect(names.map((name) => fs.readFileSync(path.join(configDir, name), "utf8"))).toEqual([ "fixture\n", "fixture\n", diff --git a/src/lib/sandbox/mutable-config-perms.ts b/src/lib/sandbox/mutable-config-perms.ts index 144126b6eef..7a8ac900053 100644 --- a/src/lib/sandbox/mutable-config-perms.ts +++ b/src/lib/sandbox/mutable-config-perms.ts @@ -2,45 +2,92 @@ // SPDX-License-Identifier: Apache-2.0 import { validateName } from "../runner"; +import { captureSanitizedResolvedOpenshell } from "../adapters/openshell/sanitized-capture"; +import { + resolveSandboxGatewayName, + type SandboxGatewayBinding, +} from "../onboard/gateway-binding/identity"; import { withMcpLifecycleLockSync } from "../state/mcp-lifecycle-lock-acquisition"; +import { renderTerminalText } from "../policy/preset-scope-render"; import { resolveAgentConfig, type AgentConfigTarget } from "./agent-config"; -import { capturePrivilegedSandboxCommand } from "./privileged-exec"; export interface MutableHermesConfigVerification { readonly verified: boolean; readonly errors: readonly string[]; } -const MUTABLE_HERMES_CONFIG_PROBE_TIMEOUT_MS = 20_000; +const MUTABLE_HERMES_CONFIG_PROBE_TIMEOUT_MS = 35_000; const MUTABLE_HERMES_CONFIG_PROBE = String.raw` import errno import fcntl import os +import pwd +import runpy +import signal import stat import struct -import subprocess import sys +sandbox_identity = pwd.getpwnam("sandbox") +if (os.geteuid() == 0 or os.geteuid() != sandbox_identity.pw_uid or + os.getegid() != sandbox_identity.pw_gid): + raise RuntimeError("Hermes config probe requires the non-root sandbox identity") + config_dir = os.path.normpath(sys.argv[1]) config_paths = [os.path.normpath(value) for value in sys.argv[2:]] if not os.path.isabs(config_dir) or not config_paths: raise RuntimeError("Hermes mutable config probe requires absolute paths") +# Health can precede the supervisor's final startup writes. Wait before taking +# the integrity snapshot; never retry an inspection that observes a mutation. +startup_helper = "/usr/local/lib/nemoclaw/hermes-startup-completion.py" +try: + startup = runpy.run_path(startup_helper) +except FileNotFoundError as error: + if error.filename != startup_helper: + raise + # Retained/custom images predate the scheduling helper. Their existing + # integrity contract remains required, without a startup-completion claim. + startup = None +if startup is not None and not startup["requires_completion"](): + # The root-separated entrypoint retains its existing startup protocol. + startup = None +startup_receipt = startup["wait_for_completion"]() if startup is not None else None + directory_flags = os.O_RDONLY | os.O_CLOEXEC | os.O_NOFOLLOW | os.O_DIRECTORY file_flags = os.O_RDONLY | os.O_NONBLOCK | os.O_CLOEXEC | os.O_NOFOLLOW +# Load the existing checker before the integrity snapshot. Starting another +# interpreter inside each check lets background runtime writes overlap imports. +private_topology = None +if stat.S_IMODE(os.stat(config_dir, follow_symlinks=False).st_mode) == 0o700: + private_topology = runpy.run_path( + "/usr/local/lib/nemoclaw/hermes-runtime-config-guard.py" + )["inspect_private_mutable_topology"] + +class TopologyDeadline(BaseException): + pass + +def topology_deadline(_signal, _frame): + raise TopologyDeadline("Hermes topology inspection timed out") + def require_private_topology(): - result = subprocess.run( - ["/opt/hermes/.venv/bin/python", "-I", - "/usr/local/lib/nemoclaw/hermes-runtime-config-guard.py", - "inspect-private-mutable-topology", "--hermes-dir", config_dir], - capture_output=True, timeout=5, check=False, - ) - if result.returncode != 0 or result.stdout != b"same-uid-nonroot\n": - raise RuntimeError("Private Hermes config requires an attested same-UID runtime; rebuild with the current Hermes image") + previous_handler = signal.signal(signal.SIGALRM, topology_deadline) + signal.alarm(5) + try: + if private_topology is None: + raise RuntimeError("Hermes config root changed before inspection") + private_topology() + except (Exception, TopologyDeadline) as error: + raise RuntimeError("Private Hermes config requires an attested same-UID runtime; rebuild with the current Hermes image") from error + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, previous_handler) + +metadata_fields = ("st_dev", "st_ino", "st_uid", "st_gid", "st_mode", "st_nlink", "st_ctime_ns") def metadata(st): - return (st.st_dev, st.st_ino, st.st_uid, st.st_gid, st.st_mode, st.st_nlink, st.st_ctime_ns) + return tuple(getattr(st, name) for name in metadata_fields) def require_mutable_flags(descriptor, info): if hasattr(info, "st_flags"): @@ -58,9 +105,46 @@ def require_mutable_flags(descriptor, info): if flags & blocked: raise PermissionError("Hermes config has immutable or append-only inode flags") +# Metadata only for fixed runtime-state entries; never follow their targets. +known_runtime_names = ("gateway_state.json", "shared-state.db", "shared-state.db-wal", "shared-state.db-shm") + +def known_runtime_state(): + snapshot = {} + for name in known_runtime_names: + try: + info = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) + snapshot[name] = ("present", info.st_dev, info.st_ino, info.st_mode, + info.st_nlink, info.st_ctime_ns, info.st_mtime_ns) + except FileNotFoundError: + snapshot[name] = ("absent",) + except OSError: + snapshot[name] = ("unavailable",) + return snapshot + +def known_runtime_changes(initial): + current = known_runtime_state() + changes = [] + for name in known_runtime_names: + before, after = initial[name], current[name] + if "unavailable" in (before[0], after[0]): + change = "unavailable" + elif before == after: + continue + elif before[0] == "absent": + change = "created" + elif after[0] == "absent": + change = "removed" + elif before[1:3] != after[1:3]: + change = "replaced" + else: + change = "metadata-changed" + changes.append(name + "=" + change) + return ", ".join(changes) if changes else "none observed" + directory_fd = os.open(config_dir, directory_flags) try: directory = os.fstat(directory_fd) + initial_known_runtime_state = known_runtime_state() if not stat.S_ISDIR(directory.st_mode): raise RuntimeError("Hermes config root is not a directory") if directory.st_uid != os.geteuid() or directory.st_gid != os.getegid(): @@ -98,8 +182,22 @@ try: if private_root: require_private_topology() - if metadata(os.stat(config_dir, follow_symlinks=False)) != metadata(directory): - raise RuntimeError("Hermes config root changed during inspection") + current_directory_info = os.stat(config_dir, follow_symlinks=False) + current_directory = metadata(current_directory_info) + initial_directory = metadata(directory) + if current_directory != initial_directory: + changed_fields = [name for name, before, after in + zip(metadata_fields, initial_directory, current_directory) + if before != after] + # Report timestamp and fixed-entry metadata comparisons without state contents. + # These observations do not identify the writer or permit another inspection. + mtime_change = "changed" if current_directory_info.st_mtime_ns != directory.st_mtime_ns else "unchanged" + raise RuntimeError("Hermes config root changed during inspection: " + ", ".join(changed_fields) + + "; directory mtime " + mtime_change + + "; known entries in inspected directory: " + + known_runtime_changes(initial_known_runtime_state)) + if startup is not None and startup["read_completion"]() != startup_receipt: + raise RuntimeError("Hermes startup generation changed during config inspection") finally: os.close(directory_fd) `; @@ -109,12 +207,7 @@ export function mutableHermesConfigProbeCommand(target: AgentConfigTarget): read throw new Error(`agent ${target.agentName} does not use the mutable Hermes config contract`); } return [ - "/usr/bin/setpriv", - "--reuid=sandbox", - "--regid=sandbox", - "--init-groups", - "--", - "/usr/bin/python3", + "/opt/hermes/.venv/bin/python3", "-I", "-c", MUTABLE_HERMES_CONFIG_PROBE, @@ -141,15 +234,48 @@ export function verifyMutableHermesConfigForTarget( export function inspectMutableHermesConfigPerms( sandboxName: string, + getSandbox: (name: string) => SandboxGatewayBinding | null, ): MutableHermesConfigVerification { validateName(sandboxName, "sandbox name"); return withMcpLifecycleLockSync(sandboxName, () => { const target = resolveAgentConfig(sandboxName); return verifyMutableHermesConfigForTarget(target, (command) => { - capturePrivilegedSandboxCommand(sandboxName, command, { - sanitizeEnvironment: true, - timeout: MUTABLE_HERMES_CONFIG_PROBE_TIMEOUT_MS, - }); + const sandbox = getSandbox(sandboxName); + if (!sandbox) throw new Error("Hermes config probe requires a registered sandbox"); + const result = captureSanitizedResolvedOpenshell( + [ + "sandbox", + "exec", + "--name", + sandboxName, + "-g", + resolveSandboxGatewayName(sandbox), + "--no-tty", + "--no-login-shell", + "--timeout", + "30", + "--", + ...command, + ], + { + ignoreError: true, + includeStderr: true, + includeStreams: true, + maxBuffer: 64 * 1024, + timeout: MUTABLE_HERMES_CONFIG_PROBE_TIMEOUT_MS, + }, + ); + if (result.error || result.signal || result.status !== 0) { + const detail = [ + result.error?.message ?? result.signal ?? `exit ${result.status}`, + result.stderr, + ] + .filter(Boolean) + .join(": "); + throw new Error( + `Hermes config probe failed (${renderTerminalText(detail.trim()).slice(-4096)})`, + ); + } }); }); } diff --git a/src/lib/sandbox/openclaw-native-config.ts b/src/lib/sandbox/openclaw-native-config.ts index 58eee502240..4d471f9641b 100644 --- a/src/lib/sandbox/openclaw-native-config.ts +++ b/src/lib/sandbox/openclaw-native-config.ts @@ -4,6 +4,9 @@ import type { OpenShellRuntimeSelection } from "../adapters/openshell/client"; import type { ConfigValue } from "../security/credential-filter"; +import { nativeLocalCredentialReference } from "../inference/native-local/model-router"; +import { usesNativeLocalInference } from "../inference/native-local/eligibility"; +import { NATIVE_LOCAL_CREDENTIAL_ENV } from "../inference/native-local/agent-config"; const { buildSelectedOpenShellSubprocessEnv, captureOpenshellCommand, @@ -12,7 +15,6 @@ const { const { OPENSHELL_OPERATION_TIMEOUT_MS, }: typeof import("../adapters/openshell/timeouts") = require("../adapters/openshell/timeouts"); -const { shellQuote }: typeof import("../core/shell-quote") = require("../core/shell-quote"); const { NVIDIA_INFERENCE_PLACEHOLDER, }: typeof import("../inference-credential") = require("../inference-credential"); @@ -137,7 +139,9 @@ function nativeProviderHandleUpdate(updates: readonly OpenClawConfigUpdate[]) { typeof provider === "string" && (Boolean(fixedNativeProvider(provider)) || (typeof value.baseUrl === "string" && - Boolean(nativeHostedAgentConfig(provider, value.baseUrl)))) + (Boolean(nativeHostedAgentConfig(provider, value.baseUrl)) || + (usesNativeLocalInference(provider, value.baseUrl) && + Boolean(nativeLocalCredentialReference(provider, value.baseUrl)))))) ); }); if (matching.length === 0) return undefined; @@ -146,15 +150,23 @@ function nativeProviderHandleUpdate(updates: readonly OpenClawConfigUpdate[]) { if (!isConfigObject(value)) throw new Error("Invalid native provider config update"); const headers = isConfigObject(value.headers) ? value.headers : {}; const provider = headers["X-NemoClaw-Upstream-Provider"]; + const local = + typeof provider === "string" && + typeof value.baseUrl === "string" && + usesNativeLocalInference(provider, value.baseUrl) + ? nativeLocalCredentialReference(provider, value.baseUrl) + : null; const native = value.apiKey === NVIDIA_INFERENCE_PLACEHOLDER ? { credentialEnv: NVIDIA_HOSTED_CREDENTIAL_ENV, apiKey: NVIDIA_INFERENCE_PLACEHOLDER, } - : typeof provider === "string" && typeof value.baseUrl === "string" - ? nativeHostedAgentConfig(provider, value.baseUrl) - : undefined; + : local + ? { credentialEnv: NATIVE_LOCAL_CREDENTIAL_ENV, apiKey: local } + : typeof provider === "string" && typeof value.baseUrl === "string" + ? nativeHostedAgentConfig(provider, value.baseUrl) + : undefined; if (!native) { throw new Error("Native provider identity does not match its credential placeholder"); } @@ -180,10 +192,9 @@ function nativeProviderHandleUpdate(updates: readonly OpenClawConfigUpdate[]) { : typeof provider === "string" ? provider : null; - const providerKey = nativeOpenClawProviderKey( - logicalProvider, - typeof value.api === "string" ? value.api : null, - ); + const providerKey = local + ? "inference" + : nativeOpenClawProviderKey(logicalProvider, typeof value.api === "string" ? value.api : null); if (!providerKey) throw new Error("Native provider identity does not match its credential placeholder"); const expectedPath = `models.providers.${providerKey}`; @@ -205,7 +216,7 @@ export function buildOpenClawNativeConfigBatchInvocation( ) { const nativeHandleUpdate = nativeProviderHandleUpdate(updates); const materializeCommand = nativeHandleUpdate - ? `python3 -c ${shellQuote(materializeNativeProviderHandle)} "$file" ${shellQuote(nativeHandleUpdate.credentialEnv)} ${shellQuote(nativeHandleUpdate.endpoint)} ${shellQuote(nativeHandleUpdate.placeholder)} ${shellQuote(nativeHandleUpdate.providerPath)} || exit $?; ` + ? 'python3 -c "$1" "$file" "$2" "$3" "$4" "$5" || exit $?; ' : ""; return { nativeHandleUpdate: Boolean(nativeHandleUpdate), @@ -228,6 +239,15 @@ export function buildOpenClawNativeConfigBatchInvocation( "-c", `umask 077; file=$(mktemp /tmp/nemoclaw-openclaw-config.XXXXXX) || exit $?; trap 'rm -f "$file"' EXIT; cat >"$file" || exit $?; ${materializeCommand}openclaw config set --batch-file "$file"`, "nemoclaw-openclaw-config-set-batch", + ...(nativeHandleUpdate + ? [ + materializeNativeProviderHandle, + nativeHandleUpdate.credentialEnv, + nativeHandleUpdate.endpoint, + nativeHandleUpdate.placeholder, + nativeHandleUpdate.providerPath, + ] + : []), ], input: JSON.stringify(updates.map(({ dotpath, value }) => ({ path: dotpath, value }))), }; diff --git a/src/lib/sandbox/privileged-exec.test.ts b/src/lib/sandbox/privileged-exec.test.ts index 1999c842e4d..316963a5edf 100644 --- a/src/lib/sandbox/privileged-exec.test.ts +++ b/src/lib/sandbox/privileged-exec.test.ts @@ -8,6 +8,7 @@ import { createRequire } from "node:module"; import os from "node:os"; import path from "node:path"; import { describe, expect, it, vi } from "vitest"; +import type { RuntimeProviderPrivilegedSandboxControl } from "../onboard/runtime-provider/contract"; import { dockerContainerNameMatchesSandbox as containerNameMatchesSandbox, selectDockerPrivilegedSandboxTarget as selectDirectSandboxContainer, @@ -25,6 +26,7 @@ const dockerControlPath = const dockerOperationAuthorityPath = require.resolve("../onboard/runtime-provider/docker-operation-authority"); const dockerRunPath = require.resolve("../adapters/docker/run"); +const dockerExecPath = require.resolve("../adapters/docker/exec"); const portableLifecyclePath = require.resolve("../onboard/experimental/portable-demo-lifecycle"); const registryPath = require.resolve("../state/registry"); const lifecycleGenerationPath = require.resolve("../state/registry/lifecycle-generation"); @@ -44,6 +46,8 @@ function restoreRequireCacheEntry(modulePath: string, priorEntry: unknown): void function withPrivilegedExecMocks( deps: { + control?: RuntimeProviderPrivilegedSandboxControl; + dockerSpawnSync?: typeof import("../adapters/docker/exec").dockerSpawnSync; dockerCapture: (args: readonly string[], options?: { timeout?: number }) => string; dockerRun?: ( args: readonly string[], @@ -79,6 +83,7 @@ function withPrivilegedExecMocks( const priorDockerControl = require.cache[dockerControlPath]; const priorDockerOperationAuthority = require.cache[dockerOperationAuthorityPath]; const priorDockerRun = require.cache[dockerRunPath]; + const priorDockerExec = require.cache[dockerExecPath]; const priorPortableLifecycle = require.cache[portableLifecyclePath]; const priorRegistry = require.cache[registryPath]; const priorLifecycleGeneration = require.cache[lifecycleGenerationPath]; @@ -87,6 +92,18 @@ function withPrivilegedExecMocks( delete require.cache[helperPath]; delete require.cache[dockerControlPath]; delete require.cache[runtimeProviderSelectionPath]; + requireCache[dockerExecPath] = { + id: dockerExecPath, + filename: dockerExecPath, + loaded: true, + exports: { + dockerSpawnSync: + deps.dockerSpawnSync ?? + (() => { + throw new Error("Unexpected Docker execution in test"); + }), + }, + } as any; requireCache[dockerOperationAuthorityPath] = { id: dockerOperationAuthorityPath, filename: dockerOperationAuthorityPath, @@ -157,7 +174,8 @@ function withPrivilegedExecMocks( deps.compareAndSetLegacySandboxLifecycleGeneration ?? (() => false), }, } as any; - const dockerControl = require(dockerControlPath).createDockerPrivilegedSandboxControl(); + const dockerControl = + deps.control ?? require(dockerControlPath).createDockerPrivilegedSandboxControl(); requireCache[currentRuntimeProvidersPath] = { id: currentRuntimeProvidersPath, filename: currentRuntimeProvidersPath, @@ -199,6 +217,7 @@ function withPrivilegedExecMocks( restoreRequireCacheEntry(dockerControlPath, priorDockerControl); restoreRequireCacheEntry(dockerOperationAuthorityPath, priorDockerOperationAuthority); restoreRequireCacheEntry(dockerRunPath, priorDockerRun); + restoreRequireCacheEntry(dockerExecPath, priorDockerExec); restoreRequireCacheEntry(portableLifecyclePath, priorPortableLifecycle); restoreRequireCacheEntry(registryPath, priorRegistry); restoreRequireCacheEntry(lifecycleGenerationPath, priorLifecycleGeneration); @@ -207,6 +226,114 @@ function withPrivilegedExecMocks( } describe("privileged sandbox exec routing", () => { + it.each([undefined, "998:999"])( + "executes with the requested owner %s and retains the pinned target", + (sandboxUser) => { + const target = "a".repeat(64); + const dockerSpawnSync = vi.fn((_args: readonly string[], _options?: unknown) => ({ + status: 0, + signal: null, + stdout: Buffer.from("observed"), + stderr: Buffer.alloc(0), + pid: 1, + output: [], + })); + const capture = vi.fn( + (_args: readonly string[]) => + `${target}\t/openshell-demo\ttrue\t${JSON.stringify({ "openshell.ai/managed-by": "openshell", "openshell.ai/sandbox-name": "demo" })}`, + ); + withPrivilegedExecMocks( + { + dockerCapture: capture, + dockerSpawnSync, + getSandbox: () => ({ name: "demo", openshellDriver: "docker" }), + listSandboxes: () => ({ sandboxes: [{ name: "demo" }] }), + }, + ({ executePrivilegedSandboxCommand }) => { + const result = executePrivilegedSandboxCommand("demo", ["/usr/bin/python3", "-I"], { + expectedResourceHandle: target, + sandboxUser, + sanitizeEnvironment: true, + input: "probe", + timeout: 30000, + maxOutputBytes: 4096, + }); + expect(result.stdout.toString()).toBe("observed"); + }, + ); + expect(capture.mock.calls[0]?.[0]).toEqual(expect.arrayContaining(["inspect", target])); + expect(dockerSpawnSync).toHaveBeenCalledWith( + expect.arrayContaining(["--user", sandboxUser ?? "root", target, "/usr/bin/python3", "-I"]), + expect.objectContaining({ input: Buffer.from("probe"), timeout: 30000, maxBuffer: 4096 }), + ); + const argv = dockerSpawnSync.mock.calls[0]?.[0]; + expect(argv).toContain("--env"); + expect(argv).not.toContain("--privileged"); + }, + ); + + it.each(["0:0", "998:0", "--privileged", "998"])( + "rejects invalid owner %s before discovery or execution", + (sandboxUser) => { + const dockerCapture = vi.fn(() => ""); + const dockerSpawnSync = vi.fn(); + withPrivilegedExecMocks( + { + dockerCapture, + dockerSpawnSync, + getSandbox: () => ({ name: "demo", openshellDriver: "docker" }), + listSandboxes: () => ({ sandboxes: [] }), + }, + ({ executePrivilegedSandboxCommand }) => { + expect(() => executePrivilegedSandboxCommand("demo", ["id"], { sandboxUser })).toThrow( + /non-root numeric UID and GID/u, + ); + }, + ); + expect(dockerCapture).not.toHaveBeenCalled(); + expect(dockerSpawnSync).not.toHaveBeenCalled(); + }, + ); + + it("does not fall back to root when owner execution is unavailable", () => { + const execute = vi.fn(); + withPrivilegedExecMocks( + { + control: { resolveTarget: vi.fn(), execute }, + dockerCapture: vi.fn(), + getSandbox: () => ({ name: "demo", openshellDriver: "docker" }), + listSandboxes: () => ({ sandboxes: [] }), + }, + ({ executePrivilegedSandboxCommand }) => { + expect(() => + executePrivilegedSandboxCommand("demo", ["id"], { sandboxUser: "998:999" }), + ).toThrow(/cannot execute as the sandbox owner/u); + }, + ); + expect(execute).not.toHaveBeenCalled(); + }); + + it("refuses owner execution when the pinned container changed", () => { + const dockerSpawnSync = vi.fn(); + withPrivilegedExecMocks( + { + dockerCapture: () => "container changed", + dockerSpawnSync, + getSandbox: () => ({ name: "demo", openshellDriver: "docker" }), + listSandboxes: () => ({ sandboxes: [] }), + }, + ({ executePrivilegedSandboxCommand }) => { + expect(() => + executePrivilegedSandboxCommand("demo", ["id"], { + sandboxUser: "998:998", + expectedResourceHandle: "a".repeat(64), + }), + ).toThrow(/identity/u); + }, + ); + expect(dockerSpawnSync).not.toHaveBeenCalled(); + }); + it("matches only the requested OpenShell sandbox container name pattern", () => { expect(containerNameMatchesSandbox("openshell-demo", "demo")).toBe(true); expect(containerNameMatchesSandbox("openshell-demo-abc123", "demo")).toBe(true); @@ -752,12 +879,24 @@ describe("privileged sandbox exec routing", () => { ); }); - it("rejects malformed Docker metadata", () => { - expect(() => selectDirectSandboxContainer("demo", "openshell-demo\n", ["demo"])).toThrow( + it.each([ + "openshell-demo\n", + "abc123\topenshell-demo\tunknown\n", + "abc123\topenshell-demo\tsandbox\textra\n", + ])("rejects malformed Docker metadata %j", (output) => { + expect(() => selectDirectSandboxContainer("demo", output, ["demo"])).toThrow( /malformed OpenShell sandbox container metadata/, ); }); + it("does not return a supervisor when no workload remains", () => { + expect( + selectDirectSandboxContainer("demo", "abc123\topenshell-demo-supervisor\tsupervisor\n", [ + "demo", + ]), + ).toBeNull(); + }); + it("rejects an authoritative label and container-name mismatch", () => { expect(() => selectDirectSandboxContainer("alpha", "gateway-id\topenshell-gateway-nemoclaw\n", ["alpha"]), @@ -846,6 +985,8 @@ describe("privileged sandbox exec routing", () => { "unix:///run/user/1001/podman/podman.sock", "exec", "--env", + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "--env", "BASH_ENV=", "--env", "ENV=", diff --git a/src/lib/sandbox/privileged-exec.ts b/src/lib/sandbox/privileged-exec.ts index e41c4d1afc2..54486c34ccd 100644 --- a/src/lib/sandbox/privileged-exec.ts +++ b/src/lib/sandbox/privileged-exec.ts @@ -25,6 +25,7 @@ type SandboxEntry = import("../state/registry").SandboxEntry; export interface PrivilegedSandboxCommandOptions { readonly input?: string | Buffer; + readonly sandboxUser?: string; readonly sanitizeEnvironment?: boolean; readonly expectedResourceHandle?: string; readonly timeout?: number; @@ -124,7 +125,7 @@ export function executePrivilegedSandboxCommand( : Buffer.isBuffer(options.input) ? Buffer.from(options.input) : Buffer.from(options.input, "utf8"); - return control.execute({ + const request = { registeredSandboxNames: registeredSandboxNames(sandboxName), sandbox, sandboxName, @@ -136,7 +137,14 @@ export function executePrivilegedSandboxCommand( ? { expectedResourceHandle: options.expectedResourceHandle } : {}), ...(options.maxOutputBytes ? { maxOutputBytes: options.maxOutputBytes } : {}), - }); + }; + if (options.sandboxUser !== undefined) { + if (!control.executeAsSandboxUser) { + throw new Error("The selected runtime provider cannot execute as the sandbox owner."); + } + return control.executeAsSandboxUser({ ...request, sandboxUser: options.sandboxUser }); + } + return control.execute(request); } /** Retained Docker CLI compatibility for portable and Docker-specific probes. */ diff --git a/src/lib/state/registry-registered-provider-attachment.test.ts b/src/lib/state/registry-registered-provider-attachment.test.ts new file mode 100644 index 00000000000..858593a99f4 --- /dev/null +++ b/src/lib/state/registry-registered-provider-attachment.test.ts @@ -0,0 +1,140 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; +import type { SandboxEntry } from "./registry/types"; +import type { RegisteredInferenceProviderAttachment } from "../inference/native-provider/registered"; + +let home: string; +beforeEach(async () => { + home = await fs.mkdtemp(path.join(os.tmpdir(), "nemoclaw-registered-provider-")); + vi.stubEnv("HOME", home); + vi.resetModules(); +}); +afterEach(async () => { + await fs.rm(home, { recursive: true, force: true }); +}); + +function selectedProvider(attachedBySelection = false): SandboxEntry { + const binding: RegisteredInferenceProviderAttachment = { + schemaVersion: 1, + gatewayName: "nemoclaw", + sandboxName: "alpha", + providerName: "operator-provider", + providerId: "operator-provider-id", + profileId: "operator-profile", + profileDigest: "a".repeat(64), + endpointUrl: "https://api.example/v1", + inferenceApi: "openai-completions", + credentialEnv: "OPERATOR_API_KEY", + attachedBySelection, + }; + return { + name: binding.sandboxName, + gatewayName: binding.gatewayName, + provider: binding.providerName, + model: "model-a", + endpointUrl: binding.endpointUrl, + preferredInferenceApi: binding.inferenceApi, + credentialEnv: binding.credentialEnv, + registeredInferenceProviderAttachment: binding, + }; +} + +it.each([false, true])( + "persists an operator binding without adopting its provider (attachment created: %s)", + async (created) => { + const registry = await import("./registry"); + const { REGISTRY_FILE } = await import("./registry/persistence"); + const entry = selectedProvider(created); + registry.registerSandbox(entry); + expect(registry.getSandbox("alpha")?.registeredInferenceProviderAttachment).toEqual( + entry.registeredInferenceProviderAttachment, + ); + const disk = JSON.parse(await fs.readFile(REGISTRY_FILE, "utf8")); + expect(disk.nativeHostedProviderAuthorities).toBeUndefined(); + expect(disk.nativeNvidiaProviderAuthorities).toBeUndefined(); + expect(disk.sandboxes.alpha.nativeHostedProviderAttachment).toBeUndefined(); + }, +); + +it.each([ + ["gateway", { gatewayName: "other" }], + ["sandbox", { sandboxName: "other" }], + ["provider", { providerName: "other" }], + ["endpoint", { endpointUrl: "https://other.example/v1" }], + ["API", { inferenceApi: "openai-responses" }], + ["credential", { credentialEnv: "OTHER_KEY" }], +] as const)("rejects a persisted operator binding with a different %s", async (_label, change) => { + const registry = await import("./registry"); + const { REGISTRY_FILE } = await import("./registry/persistence"); + registry.registerSandbox(selectedProvider()); + const disk = JSON.parse(await fs.readFile(REGISTRY_FILE, "utf8")); + Object.assign(disk.sandboxes.alpha.registeredInferenceProviderAttachment, change); + await fs.writeFile(REGISTRY_FILE, JSON.stringify(disk)); + expect(() => registry.getSandbox("alpha")).toThrow( + "Recorded registered inference selection does not match its sandbox route", + ); +}); + +it("rejects invalid binding updates before replacing durable state", async () => { + const registry = await import("./registry"); + const { REGISTRY_FILE } = await import("./registry/persistence"); + const entry = selectedProvider(); + registry.registerSandbox(entry); + const before = await fs.readFile(REGISTRY_FILE, "utf8"); + expect(() => + registry.updateSandbox("alpha", { + registeredInferenceProviderAttachment: { + ...entry.registeredInferenceProviderAttachment!, + providerId: "", + }, + }), + ).toThrow(/registered inference provider/i); + expect(await fs.readFile(REGISTRY_FILE, "utf8")).toBe(before); +}); + +it.each([ + ["model", { model: "model-b" }, true], + ["endpoint", { endpointUrl: "https://other.example/v1" }, false], + ["provider", { provider: "other-provider" }, false], +] as const)( + "retains an operator binding only for the same reservation identity: %s", + async (_label, change, retained) => { + const registry = await import("./registry"); + const entry = selectedProvider(); + registry.registerSandbox(entry); + registry.reserveSandboxInferenceRoute("alpha", { + provider: entry.provider ?? null, + model: entry.model ?? null, + endpointUrl: entry.endpointUrl ?? null, + credentialEnv: entry.credentialEnv ?? null, + preferredInferenceApi: entry.preferredInferenceApi ?? null, + gatewayName: entry.gatewayName!, + ...change, + }); + expect(registry.getSandbox("alpha")?.registeredInferenceProviderAttachment).toEqual( + retained ? entry.registeredInferenceProviderAttachment : undefined, + ); + }, +); + +it("excludes only a coherent operator binding from shared-route checks", async () => { + const { isSharedGatewayRouteParticipant } = await import("./registry/route-reservation"); + const entry = selectedProvider(); + expect(isSharedGatewayRouteParticipant(entry)).toBe(false); + expect(isSharedGatewayRouteParticipant({ ...entry, gatewayName: "other" })).toBe(true); +}); + +it("lists selected operator providers only for the requested gateway and provider", async () => { + const registry = await import("./registry"); + const { listRegisteredInferenceProviderAttachmentSandboxNames: names } = + await import("./registry/native-provider-authority"); + registry.registerSandbox(selectedProvider()); + expect(names("nemoclaw", "operator-provider")).toEqual(["alpha"]); + expect(names("other", "operator-provider")).toEqual([]); + expect(names("nemoclaw", "other-provider")).toEqual([]); +}); diff --git a/src/lib/state/registry.ts b/src/lib/state/registry.ts index e2ffac82313..93982dc4e6c 100644 --- a/src/lib/state/registry.ts +++ b/src/lib/state/registry.ts @@ -10,6 +10,7 @@ import { isDeepStrictEqual } from "node:util"; import { isDeferredN1xManagedVllmAcceptanceRoute } from "../domain/sandbox/n1x-managed-vllm-rebuild"; import type { InferenceSelection } from "../inference/selection"; import { + recordedRegisteredInferenceProviderAttachment, inferenceSelectionRegistryFields, normalizeInferenceSelection, } from "../inference/selection"; @@ -29,6 +30,7 @@ import { load, save } from "./registry/persistence"; import { normalizeNativeLocalProviderAttachment, retainedNativeLocalProviderAttachment, + retainedRegisteredInferenceProviderAttachment, nativeHostedReservationAttachment, isCurrentSandboxInferenceRouteReservation, isCurrentPendingSandboxCreateReservation, @@ -517,6 +519,8 @@ export function registerSandbox( entry.nativeHostedProviderAttachment, entry.provider, ); + const registeredInferenceProviderAttachment = + recordedRegisteredInferenceProviderAttachment(entry); const registered: SandboxEntry = { name: entry.name, createdAt: entry.createdAt || new Date().toISOString(), @@ -577,6 +581,7 @@ export function registerSandbox( ...(nativeLocalProviderAttachment ? { nativeLocalProviderAttachment } : {}), ...(nativeNvidiaProviderAttachment ? { nativeNvidiaProviderAttachment } : {}), ...(nativeHostedProviderAttachment ? { nativeHostedProviderAttachment } : {}), + ...(registeredInferenceProviderAttachment ? { registeredInferenceProviderAttachment } : {}), lifecycleGeneration: entry.lifecycleGeneration, lifecycleLiveIdentityFingerprint: entry.lifecycleLiveIdentityFingerprint, messaging: cloneSandboxMessagingState(entry.messaging), @@ -814,6 +819,10 @@ export function reserveSandboxInferenceRoute( nativeNvidiaProviderAttachment: isNativeNvidiaProvider(normalized.provider) ? (nativeNvidiaProviderAttachment ?? existing?.nativeNvidiaProviderAttachment) : undefined, + registeredInferenceProviderAttachment: retainedRegisteredInferenceProviderAttachment( + existing ?? undefined, + { ...normalized, gatewayName: route.gatewayName }, + ), ...(route.hostLocalInferenceReceipt !== undefined ? { hostLocalInferenceReceipt: route.hostLocalInferenceReceipt } : {}), diff --git a/src/lib/state/registry/native-provider-authority-state.ts b/src/lib/state/registry/native-provider-authority-state.ts index f4bfb13b15a..45fa9178c74 100644 --- a/src/lib/state/registry/native-provider-authority-state.ts +++ b/src/lib/state/registry/native-provider-authority-state.ts @@ -4,7 +4,7 @@ import { normalizeNativeNvidiaProviderAttachment, type NativeNvidiaProviderAttachment, -} from "../../inference/native-nvidia"; +} from "../../inference/native-nvidia/contract"; import type { NativeProviderAttachment } from "../../inference/native-provider/contract"; import { hostedNativeProvider } from "../../inference/native-provider/hosted"; import { diff --git a/src/lib/state/registry/native-provider-authority.ts b/src/lib/state/registry/native-provider-authority.ts index 006b458545a..cdacb83bb0b 100644 --- a/src/lib/state/registry/native-provider-authority.ts +++ b/src/lib/state/registry/native-provider-authority.ts @@ -145,3 +145,19 @@ export function listNativeHostedProviderAttachmentSandboxNames( .map((sandbox) => sandbox.name) .sort(); } + +/** Recorded selections prevent reset from deleting an operator provider still in use. */ +export function listRegisteredInferenceProviderAttachmentSandboxNames( + gatewayName: string, + providerName: string, +): readonly string[] { + // Registry loading rejects malformed or mismatched external bindings. + return Object.values(load().sandboxes) + .filter( + (sandbox) => + sandbox.gatewayName === gatewayName && + sandbox.registeredInferenceProviderAttachment?.providerName === providerName, + ) + .map((sandbox) => sandbox.name) + .sort(); +} diff --git a/src/lib/state/registry/persistence.ts b/src/lib/state/registry/persistence.ts index 08f9adfe59e..cbdf6e1001e 100644 --- a/src/lib/state/registry/persistence.ts +++ b/src/lib/state/registry/persistence.ts @@ -7,6 +7,7 @@ import { } from "../../inference/native-local/contract"; import path from "node:path"; +import { recordedRegisteredInferenceProviderAttachment } from "../../inference/native-provider/registered"; import { requireHostedProviderAttachment } from "../../inference/native-provider/hosted-attachment"; import { normalizeNativeHostedProviderAuthorities } from "./native-provider-authority-state"; import { isObjectRecord } from "../../core/json-types"; @@ -263,6 +264,8 @@ function normalizeSandboxEntryForRuntime(entry: SandboxEntry): SandboxEntry { const deferredN1xManagedVllmAccepted = normalizeDeferredN1xManagedVllmAcceptance(entry, "load"); const policyEntry = normalizeSandboxPolicyAttribution(entry); const nativeLocalProviderAttachment = requireNativeLocalProviderAttachment(policyEntry, "load"); + const registeredInferenceProviderAttachment = + recordedRegisteredInferenceProviderAttachment(policyEntry); const { cuaRuntimeReadiness: _legacyCuaRuntimeReadiness, messaging: _messaging, @@ -272,6 +275,7 @@ function normalizeSandboxEntryForRuntime(entry: SandboxEntry): SandboxEntry { servingProfileProvenance: _servingProfileProvenance, deferredN1xManagedVllmAccepted: _deferredN1xManagedVllmAccepted, nativeLocalProviderAttachment: _nativeLocalProviderAttachment, + registeredInferenceProviderAttachment: _registeredInferenceProviderAttachment, nativeHostedProviderAttachment: hostedAttachment, nativeNvidiaProviderAuthority: _legacyNativeNvidiaProviderAuthority, mcp: _legacyMcp, @@ -288,6 +292,7 @@ function normalizeSandboxEntryForRuntime(entry: SandboxEntry): SandboxEntry { return { ...rest, ...(nativeLocalProviderAttachment ? { nativeLocalProviderAttachment } : {}), + ...(registeredInferenceProviderAttachment ? { registeredInferenceProviderAttachment } : {}), ...(nativeHostedProviderAttachment ? { nativeHostedProviderAttachment } : {}), ...(workload ? { workload } : {}), ...(hostLocalInferenceReceipt !== undefined ? { hostLocalInferenceReceipt } : {}), @@ -336,6 +341,8 @@ function serializeSandboxEntryForDisk(entry: SandboxEntry): SandboxEntry { const deferredN1xManagedVllmAccepted = normalizeDeferredN1xManagedVllmAcceptance(durable, "save"); const policyEntry = normalizeSandboxPolicyAttribution(durable); const nativeLocalProviderAttachment = requireNativeLocalProviderAttachment(policyEntry, "save"); + const registeredInferenceProviderAttachment = + recordedRegisteredInferenceProviderAttachment(policyEntry); const { cuaRuntimeReadiness: _legacyCuaRuntimeReadiness, messaging: _messaging, @@ -345,6 +352,7 @@ function serializeSandboxEntryForDisk(entry: SandboxEntry): SandboxEntry { servingProfileProvenance: _servingProfileProvenance, deferredN1xManagedVllmAccepted: _deferredN1xManagedVllmAccepted, nativeLocalProviderAttachment: _nativeLocalProviderAttachment, + registeredInferenceProviderAttachment: _registeredInferenceProviderAttachment, nativeHostedProviderAttachment: hostedAttachment, nativeNvidiaProviderAuthority: _legacyNativeNvidiaProviderAuthority, mcp: _legacyMcp, @@ -361,6 +369,7 @@ function serializeSandboxEntryForDisk(entry: SandboxEntry): SandboxEntry { return { ...rest, ...(nativeLocalProviderAttachment ? { nativeLocalProviderAttachment } : {}), + ...(registeredInferenceProviderAttachment ? { registeredInferenceProviderAttachment } : {}), ...(nativeHostedProviderAttachment ? { nativeHostedProviderAttachment } : {}), ...(rest.dashboardPort === 0 ? { dashboardPort: null } : {}), ...(workload ? { workload } : {}), diff --git a/src/lib/state/registry/route-reservation.ts b/src/lib/state/registry/route-reservation.ts index b0b63653c90..f1ac3df412e 100644 --- a/src/lib/state/registry/route-reservation.ts +++ b/src/lib/state/registry/route-reservation.ts @@ -11,6 +11,7 @@ import { isDeepStrictEqual } from "node:util"; import { normalizeInferenceSelection, type InferenceSelection } from "../../inference/selection"; import { recordedNativeProviderAttachment } from "../../inference/native-provider/recorded-selection"; +import { recordedRegisteredInferenceProviderAttachment } from "../../inference/native-provider/registered"; import { hostedNativeProvider } from "../../inference/native-provider/hosted"; import { requireHostedProviderAttachment } from "../../inference/native-provider/hosted-attachment"; import { normalizeNativeNvidiaProviderAttachment } from "../../inference/native-nvidia"; @@ -34,6 +35,7 @@ const ROUTE_RESERVATION_KEYS = new Set([ "nativeLocalProviderAttachment", "nativeNvidiaProviderAttachment", "nativeHostedProviderAttachment", + "registeredInferenceProviderAttachment", "openshellDriver", "pendingRouteReservation", "pendingCreateIdentity", @@ -83,6 +85,7 @@ function withVerifiedCreateCheckpoint( function validCarriedRouteMetadata(entry: SandboxEntry): boolean { try { requireHostedProviderAttachment(entry.nativeHostedProviderAttachment, entry.provider); + recordedRegisteredInferenceProviderAttachment(entry); } catch { return false; } @@ -217,12 +220,24 @@ export function isPublishedSandboxRegistration(entry: { pendingRouteReservation? /** True when an entry participates in the inference route shared by its gateway. */ export function isSharedGatewayRouteParticipant(entry: { nativeLocalProviderAttachment?: unknown; + registeredInferenceProviderAttachment?: unknown; + name?: string; + gatewayName?: string | null; + endpointUrl?: string | null; + preferredInferenceApi?: string | null; + credentialEnv?: string | null; nimContainer?: string | null; pendingRouteReservation?: true; createdAt?: string; provider?: string | null; model?: string | null; }): boolean { + try { + if (recordedRegisteredInferenceProviderAttachment(entry)) return false; + } catch { + // Unproven native identity must not bypass shared-route conflict checks. + return true; + } const ownedLocalNim = entry.provider === "vllm-local" && Boolean(entry.nimContainer?.trim()); if (!ownedLocalNim && normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment)) return false; @@ -399,6 +414,22 @@ export function sandboxRegistrationMatchesInferenceRouteReservation( export { normalizeNativeLocalProviderAttachment }; +/** A model-only reservation keeps the selected operator provider's binding. */ +export function retainedRegisteredInferenceProviderAttachment( + existing: SandboxEntry | undefined, + selection: Pick< + SandboxEntry, + "provider" | "endpointUrl" | "credentialEnv" | "preferredInferenceApi" | "gatewayName" + >, +): SandboxEntry["registeredInferenceProviderAttachment"] { + if (!existing?.registeredInferenceProviderAttachment) return undefined; + try { + return recordedRegisteredInferenceProviderAttachment({ ...existing, ...selection }); + } catch { + return undefined; + } +} + /** Keep a native attachment only when reusing its recorded route and gateway. */ export function retainedNativeLocalProviderAttachment( existing: SandboxEntry | undefined, diff --git a/src/lib/state/registry/types.ts b/src/lib/state/registry/types.ts index 61fa6f0735a..48080a0b21a 100644 --- a/src/lib/state/registry/types.ts +++ b/src/lib/state/registry/types.ts @@ -3,6 +3,7 @@ import type { NativeLocalProviderAttachment } from "../../inference/native-local/contract"; import type { NativeProviderAttachment } from "../../inference/native-provider/contract"; +import type { RegisteredInferenceProviderAttachment } from "../../inference/native-provider/registered"; import type { NativeHostedProviderAuthorities } from "./native-provider-authority-state"; import type { InferenceSelection } from "../../inference/selection"; import type { NativeNvidiaProviderAttachment } from "../../inference/native-nvidia"; @@ -154,6 +155,8 @@ export interface SandboxEntry extends Partial { nativeNvidiaProviderAttachment?: NativeNvidiaProviderAttachment; /** Exact selected fixed hosted provider identity; separate from gateway ownership. */ nativeHostedProviderAttachment?: NativeProviderAttachment; + /** Selected operator provider; this receipt grants no provider ownership. */ + registeredInferenceProviderAttachment?: RegisteredInferenceProviderAttachment; /** Explicit hidden-lifecycle provenance; absence keeps llama.cpp on its legacy path. */ hostLocalInferenceProvenance?: SandboxHostLocalInferenceProvenance; /** Explicit Deferred N1x managed-vLLM choice retained after successful onboarding. */ diff --git a/src/lib/state/sandbox.ts b/src/lib/state/sandbox.ts index 8baabc3c0af..341bb492367 100644 --- a/src/lib/state/sandbox.ts +++ b/src/lib/state/sandbox.ts @@ -2907,8 +2907,9 @@ function backupNativeSandboxState( // owned by that user before reading the complete native tree. Materialize // any OpenClaw database plus committed WAL state into a private standalone // copy, then stream that copy with the other process-quiescent files. - // Keep the SSH ancestry live so the archive can stream, and always resume - // processes through the EXIT trap, including tar failures and signals. + // Keep the SSH ancestry, including the native PID 1 launcher, live so + // the archive can stream. Always resume processes through the EXIT trap, + // including tar failures and signals. const command = [ "set -eu", `root=${shellQuote(rootResult.root)}`, @@ -2918,7 +2919,7 @@ function backupNativeSandboxState( "self=$$", 'ancestors=" $self "', "cursor=$PPID", - 'while [ "$cursor" -gt 1 ] 2>/dev/null; do ancestors="$ancestors$cursor "; parent=""; { while IFS=":" read -r key value; do if [ "$key" = "PPid" ]; then set -- $value; parent=${1:-}; break; fi; done; } 2>/dev/null < "/proc/$cursor/status" || break; cursor=$parent; [ -n "$cursor" ] || break; done', + 'while [ "$cursor" -gt 0 ] 2>/dev/null; do ancestors="$ancestors$cursor "; parent=""; { while IFS=":" read -r key value; do if [ "$key" = "PPid" ]; then set -- $value; parent=${1:-}; break; fi; done; } 2>/dev/null < "/proc/$cursor/status" || break; cursor=$parent; [ -n "$cursor" ] || break; done', 'collect_candidates() { candidates=""; for proc in /proc/[0-9]*; do pid=${proc##*/}; case "$ancestors" in *" $pid "*) continue ;; esac; owner=""; while IFS=":" read -r key value; do if [ "$key" = "Uid" ]; then set -- $value; owner=${1:-}; break; fi; done 2>/dev/null < "$proc/status" || :; [ "$owner" = "$uid" ] && candidates="$candidates $pid"; done; }', 'stopped=""', 'resume() { [ -z "$stopped" ] || kill -CONT $stopped 2>/dev/null || :; rm -rf -- "$db_stage"; }', @@ -3383,6 +3384,13 @@ export async function restoreRecreatedSandboxState( }); } +export function nativeRestoreTransportError(): string | null { + const result = spawnSync("sftp", ["-h"], { stdio: "ignore", timeout: 10_000 }); + return result.error || result.signal + ? "Native state restore requires the OpenSSH sftp client. Install the OpenSSH client tools on this host, then retry." + : null; +} + async function restoreNativeSandboxState( sandboxName: string, backupPath: string, @@ -3411,6 +3419,8 @@ async function restoreNativeSandboxState( `Backup agent '${manifest.agentType}' does not match target agent '${options.targetAgentType}'`, ); } + const transportError = nativeRestoreTransportError(); + if (transportError) return failure(transportError); if (!options.authority || !options.validateBeforeMutation) { if (manifest.workload?.kind === "managed-image") { return failure(MANAGED_REBUILD_RESTORE_AUTHORITY_ERROR); @@ -3447,89 +3457,186 @@ async function restoreNativeSandboxState( const mutationError = await validateSnapshotRestoreMutation(backupPath, options); if (mutationError) return failure(mutationError); - const restoreScript = [ - "root=$1", - '{ [ -d "$root" ] && [ ! -L "$root" ]; } || exit 20', - 'stage="$(mktemp -d "$root/.nemoclaw-native-restore.XXXXXX")"', - "trap 'rm -rf -- \"$stage\"' EXIT HUP INT TERM", - 'tar --no-same-owner -xf - -C "$stage"', - 'if find "$stage" -type f -links +1 -print -quit | grep -q .; then echo "native restore archive contains a hard link" >&2; exit 21; fi', - 'mkdir -p -- "$stage/.nemoclaw" "$stage/.openclaw/agents/main/sessions" "$stage/.hermes/runtime"', - 'uid="$(id -u)"', - "preserve_replacement_path() {", - ' case "$1" in', - " .nemoclaw/config.json|.nemoclaw/blueprints|.openclaw/.nemoclaw-post-upgrade-doctor|.openclaw/agents/main/sessions/nemoclaw-onboard-warmup-*|.hermes/gateway.pid|.hermes/runtime/gateway.pid|.hermes/runtime/gateway.lock) return 0 ;;", - " esac", - " return 1", - "}", - "restore_dir() {", - ' local source_dir="$1" target_dir="$2" target_item source_item name owner relative', - ' for target_item in "$target_dir"/* "$target_dir"/.[!.]* "$target_dir"/..?*; do', - ' { [ -e "$target_item" ] || [ -L "$target_item" ]; } || continue', - ' [ "$target_item" = "$stage" ] && continue', - ' name="${target_item##*/}"', - ' source_item="$source_dir/$name"', - ' relative="${target_item#"$root"/}"', - ' if { [ ! -e "$source_item" ] && [ ! -L "$source_item" ]; }; then', - ' preserve_replacement_path "$relative" && continue', - ' owner="$(stat -c %u -- "$target_item")"', - ' if [ "$owner" = "$uid" ] && [ -w "$target_dir" ]; then rm -rf -- "$target_item"; continue; fi', - ' echo "native restore could not remove replacement-only state at: $target_item" >&2', - " exit 22", - " fi", - ' owner="$(stat -c %u -- "$target_item")"', - ' if [ -d "$target_item" ] && [ ! -L "$target_item" ] && [ -d "$source_item" ] && [ ! -L "$source_item" ]; then', - ' restore_dir "$source_item" "$target_item"', - ' rmdir -- "$source_item"', - ' elif [ "$owner" = "$uid" ] && [ -w "$target_dir" ]; then', - ' rm -rf -- "$target_item"', - ' elif [ -f "$target_item" ] && [ ! -L "$target_item" ] && [ -f "$source_item" ] && [ ! -L "$source_item" ] && cmp -s -- "$source_item" "$target_item"; then', - ' rm -f -- "$source_item"', - ' elif [ -L "$target_item" ] && [ -L "$source_item" ] && [ "$(readlink -- "$source_item")" = "$(readlink -- "$target_item")" ]; then', - ' rm -f -- "$source_item"', - " else", - ' echo "native restore could not preserve archived state at: $target_item" >&2', - " exit 22", - " fi", - " done", - ' for source_item in "$source_dir"/* "$source_dir"/.[!.]* "$source_dir"/..?*; do', - ' { [ -e "$source_item" ] || [ -L "$source_item" ]; } || continue', - ' if [ -w "$target_dir" ]; then', - ' mv -- "$source_item" "$target_dir"/', - " else", - ' echo "native restore could not preserve archived state below: $target_dir" >&2', - " exit 22", - " fi", - " done", - "}", - 'restore_dir "$stage" "$root"', - ].join("\n"); - // Restore the archive as the single agent-state source of truth. Keep - // only the control-plane and generation-local paths excluded at capture; - // replacement-only agent state is removed, while unsafe ownership or - // writability conflicts fail instead of producing a mixed state tree. - const command = `bash -ceu ${shellQuote(restoreScript)} -- ${shellQuote(rootResult.root)}`; - const result = spawnSync("ssh", [...sshArgs(temporary.file, sandboxName), command], { - ...(selectedEnv ? { env: selectedEnv } : {}), - stdio: [archiveFd, "pipe", "pipe"], - timeout: NATIVE_STATE_CAPTURE_TIMEOUT_MS, - maxBuffer: 1024 * 1024, - }); - if (result.status !== 0 || result.error || result.signal) { - const detail = - result.error?.message ?? - (result.signal - ? `signal ${result.signal}` - : result.stderr?.toString().trim() || `exit ${String(result.status)}`); - return failure(`Native home/workspace restore failed: ${detail.substring(0, 240)}`); + const connection = sshArgs(temporary.file, sandboxName); + const transfer = path.posix.join( + rootResult.root, + `.nemoclaw-native-restore-${path.basename(temporary.dir)}`, + ); + const runRemote = (script: string, args: string[]) => + spawnSync( + "ssh", + [...connection, `bash -ceu ${shellQuote(script)} -- ${args.map(shellQuote).join(" ")}`], + { + ...(selectedEnv ? { env: selectedEnv } : {}), + stdio: ["ignore", "pipe", "pipe"], + timeout: NATIVE_STATE_CAPTURE_TIMEOUT_MS, + maxBuffer: 1024 * 1024, + }, + ); + const commandError = (result: ReturnType): string | null => + result.status === 0 && !result.error && !result.signal + ? null + : ( + result.error?.message ?? + (result.signal + ? `signal ${result.signal}` + : result.stderr?.toString().trim() || `exit ${String(result.status)}`) + ).substring(0, 240); + const created = runRemote( + 'root=$1; transfer=$2; { [ -d "$root" ] && [ ! -L "$root" ]; } || exit 20; mkdir -m 700 -- "$transfer"; stat -c %d:%i -- "$transfer"', + [rootResult.root, transfer], + ); + const transferIdentity = created.stdout?.toString().trim() ?? ""; + const createError = commandError(created); + if (createError || !/^\d+:\d+$/.test(transferIdentity)) { + return failure( + `Native restore staging failed: ${createError ?? "invalid directory identity"}. Inspect ${JSON.stringify(transfer)} before retrying.`, + ); } - return { - success: true, - restoredDirs: ["."], - failedDirs: [], - restoredFiles: [], - failedFiles: [], + const verifyTransfer = [ + "transfer=$1; identity=$2", + '[ -d "$transfer" ] && [ ! -L "$transfer" ]', + '[ "$(stat -c %d:%i -- "$transfer")" = "$identity" ]', + '[ "$(stat -c %u -- "$transfer")" = "$(id -u)" ]', + ].join("\n"); + const performRestore = async (): Promise => { + const batch = path.join(temporary.dir, "upload.batch"); + const remoteArchive = `${transfer}/archive.tar`; + writeFileSync(batch, `put /dev/stdin "${remoteArchive.replace(/[\\"]/g, "\\$&")}"\n`, { + mode: 0o600, + flag: "wx", + }); + // SFTP acknowledges bounded requests before accepting more archive bytes. + // The input is the unlinked, validated copy, never the original pathname. + const uploaded = spawnSync( + "sftp", + ["-b", batch, "-R", "16", "-B", "32768", ...connection], + { + ...(selectedEnv ? { env: selectedEnv } : {}), + stdio: [archiveFd, "pipe", "pipe"], + timeout: NATIVE_STATE_CAPTURE_TIMEOUT_MS, + maxBuffer: 1024 * 1024, + }, + ); + const uploadError = commandError(uploaded); + if (uploadError) return failure(`Native home/workspace upload failed: ${uploadError}`); + const authorityError = await validateSnapshotRestoreMutation(backupPath, options); + if (authorityError) return failure(authorityError); + const restoreScript = [ + verifyTransfer, + "root=$3", + '{ [ -d "$root" ] && [ ! -L "$root" ]; } || exit 20', + 'archive="$transfer/archive.tar"', + '[ -f "$archive" ] && [ ! -L "$archive" ]', + 'exec 3< "$archive"', + '[ "$(stat -Lc %u:%h -- /proc/self/fd/3)" = "$(id -u):1" ]', + '[ "$(stat -c %d:%i -- "$archive")" = "$(stat -Lc %d:%i -- /proc/self/fd/3)" ]', + 'rm -- "$archive"', + '[ "$(stat -Lc %h -- /proc/self/fd/3)" = 0 ]', + 'digest="$(sha256sum /proc/self/fd/3)"', + '[ "${digest%% *}" = "$4" ] || { echo "native restore archive checksum mismatch" >&2; exit 21; }', + 'stage="$transfer/state"; mkdir -m 700 -- "$stage"', + 'tar --no-same-owner -xf /proc/self/fd/3 -C "$stage"', + "exec 3<&-", + 'if find "$stage" -type f -links +1 -print -quit | grep -q .; then echo "native restore archive contains a hard link" >&2; exit 21; fi', + 'mkdir -p -- "$stage/.nemoclaw" "$stage/.openclaw/agents/main/sessions" "$stage/.hermes/runtime"', + 'uid="$(id -u)"', + "preserve_replacement_path() {", + ' case "$1" in', + " .nemoclaw/config.json|.nemoclaw/blueprints|.openclaw/.nemoclaw-post-upgrade-doctor|.openclaw/agents/main/sessions/nemoclaw-onboard-warmup-*|.hermes/gateway.pid|.hermes/runtime/gateway.pid|.hermes/runtime/gateway.lock) return 0 ;;", + " esac", + " return 1", + "}", + "restore_dir() {", + ' local source_dir="$1" target_dir="$2" target_item source_item name owner relative', + ' for target_item in "$target_dir"/* "$target_dir"/.[!.]* "$target_dir"/..?*; do', + ' { [ -e "$target_item" ] || [ -L "$target_item" ]; } || continue', + ' [ "$target_item" = "$transfer" ] && continue', + ' name="${target_item##*/}"', + ' source_item="$source_dir/$name"', + ' relative="${target_item#"$root"/}"', + ' if { [ ! -e "$source_item" ] && [ ! -L "$source_item" ]; }; then', + ' preserve_replacement_path "$relative" && continue', + ' owner="$(stat -c %u -- "$target_item")"', + ' if [ "$owner" = "$uid" ] && [ -w "$target_dir" ]; then rm -rf -- "$target_item"; continue; fi', + ' echo "native restore could not remove replacement-only state at: $target_item" >&2', + " exit 22", + " fi", + ' owner="$(stat -c %u -- "$target_item")"', + ' if [ -d "$target_item" ] && [ ! -L "$target_item" ] && [ -d "$source_item" ] && [ ! -L "$source_item" ]; then', + ' restore_dir "$source_item" "$target_item"', + ' rmdir -- "$source_item"', + ' elif [ "$owner" = "$uid" ] && [ -w "$target_dir" ]; then', + ' rm -rf -- "$target_item"', + ' elif [ -f "$target_item" ] && [ ! -L "$target_item" ] && [ -f "$source_item" ] && [ ! -L "$source_item" ] && cmp -s -- "$source_item" "$target_item"; then', + ' rm -f -- "$source_item"', + ' elif [ -L "$target_item" ] && [ -L "$source_item" ] && [ "$(readlink -- "$source_item")" = "$(readlink -- "$target_item")" ]; then', + ' rm -f -- "$source_item"', + " else", + ' echo "native restore could not preserve archived state at: $target_item" >&2', + " exit 22", + " fi", + " done", + ' for source_item in "$source_dir"/* "$source_dir"/.[!.]* "$source_dir"/..?*; do', + ' { [ -e "$source_item" ] || [ -L "$source_item" ]; } || continue', + ' if [ -w "$target_dir" ]; then', + ' mv -- "$source_item" "$target_dir"/', + " else", + ' echo "native restore could not preserve archived state below: $target_dir" >&2', + " exit 22", + " fi", + " done", + "}", + 'restore_dir "$stage" "$root"', + ].join("\n"); + // Restore the archive as the single agent-state source of truth. Keep + // only the control-plane and generation-local paths excluded at capture; + // replacement-only agent state is removed, while unsafe ownership or + // writability conflicts fail instead of producing a mixed state tree. + const result = runRemote(restoreScript, [ + transfer, + transferIdentity, + rootResult.root, + manifest.nativeState!.sha256, + ]); + const restoreError = commandError(result); + return restoreError + ? failure(`Native home/workspace restore failed: ${restoreError}`) + : { + success: true, + restoredDirs: ["."], + failedDirs: [], + restoredFiles: [], + failedFiles: [], + }; }; + let outcome = failure("Native home/workspace restore did not complete"); + try { + outcome = await performRestore(); + } catch (error) { + outcome = failure( + `Native home/workspace restore failed: ${error instanceof Error ? error.message : String(error)}`, + ); + } finally { + const removed = runRemote(verifyTransfer + '\nrm -rf -- "$transfer"', [ + transfer, + transferIdentity, + ]); + const cleanupError = commandError(removed); + if (cleanupError) { + outcome = { + ...outcome, + success: false, + error: [ + outcome.error, + `Native restore cleanup failed for ${JSON.stringify(transfer)}: ${cleanupError}`, + ] + .filter(Boolean) + .join("; "), + }; + } + } + return outcome; }; return withSshCleanupResult(await runWithTempSshConfigCleanupAsync(temporary, restore)); } finally { diff --git a/src/lib/verify-deployment-inference-route.test.ts b/src/lib/verify-deployment-inference-route.test.ts index 7e75d45886f..d7a5e9cf5b2 100644 --- a/src/lib/verify-deployment-inference-route.test.ts +++ b/src/lib/verify-deployment-inference-route.test.ts @@ -60,7 +60,7 @@ function makeModelsRouteDeps(code: string, overrides: Record = describe("verifyDeployment inference route model-catalog validation", () => { it.each([ - ["openclaw", false], + ["openclaw", true], [DCODE_AGENT, true], ["pi", true], ["hermes", true], @@ -76,35 +76,53 @@ describe("verifyDeployment inference route model-catalog validation", () => { expect(context.invocation.nativeLocalExpected).toBe(expected); }); - it("verifies owned Local NIM through the shared route while expecting native standalone vLLM", () => { - const flow = { - provider: "vllm-local", - model: "test-model", - endpointUrl: "http://host.openshell.internal:8000/v1", - preferredInferenceApi: "openai-completions", - }; - const binding = { - ...localBinding, - provider: "vllm-local" as const, - endpointUrl: "http://host.openshell.internal:8000/v1", - }; - const staleAttachment = { - ...binding, - ...nativeLocalIdentity(binding), - schemaVersion: 1 as const, - providerId: "stale-provider", - }; - const nim = onboardInferenceVerificationContext("my-sandbox", "my-gateway", DCODE_AGENT, flow, { - nimContainer: "nemoclaw-nim-owned", - nativeLocalProviderAttachment: staleAttachment, - }); - const vllm = onboardInferenceVerificationContext("my-sandbox", "my-gateway", DCODE_AGENT, flow); - expect(nim.route.nativeLocalExpected).toBe(false); - expect(nim.invocation.nativeLocalExpected).toBe(false); - expect(nim.route.nativeLocalProviderAttachment).toBeUndefined(); - expect(nim.invocation.nativeLocalProviderAttachment).toBeUndefined(); - expect(vllm.route.nativeLocalExpected).toBe(true); - }); + it.each(["sentinel", "authenticated", "missing"] as const)( + "verifies Local NIM only with its sentinel attachment: %s", + async (mode) => { + const flow = { + provider: "vllm-local", + model: "test-model", + nimContainer: "nemoclaw-nim-owned", + endpointUrl: "http://host.openshell.internal:8000/v1", + preferredInferenceApi: "openai-completions", + }; + const binding = { + ...localBinding, + provider: "vllm-local" as const, + endpointUrl: flow.endpointUrl, + authMode: mode === "authenticated" ? ("authenticated" as const) : ("sentinel" as const), + }; + const receipt = { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1 as const, + providerId: "selected-provider", + }; + const context = onboardInferenceVerificationContext( + "my-sandbox", + "my-gateway", + DCODE_AGENT, + flow, + { + nimContainer: flow.nimContainer, + nativeLocalProviderAttachment: mode === "missing" ? undefined : receipt, + }, + ); + const probe = vi.fn(async () => ({ ok: true }) as const); + const expected = mode === "sentinel" ? receipt : undefined; + expect(context.route.nativeLocalExpected).toBe(true); + expect(context.invocation.nativeLocalExpected).toBe(true); + expect(context.route.nativeLocalProviderAttachment).toEqual(expected); + expect(context.invocation.nativeLocalProviderAttachment).toEqual(expected); + const result = await probeOnboardInferenceInvocation(context.invocation, probe); + expect(result.ok).toBe(mode === "sentinel"); + expect(probe.mock.calls).toEqual( + mode === "sentinel" + ? [[expect.objectContaining({ nativeLocalProviderAttachment: receipt })]] + : [], + ); + }, + ); it("uses the recorded native local attachment instead of the shared models route", async () => { const scripts: string[] = []; diff --git a/src/lib/verify-deployment-native-hosted.test.ts b/src/lib/verify-deployment-native-hosted.test.ts index c9261a27592..ecf1298be0a 100644 --- a/src/lib/verify-deployment-native-hosted.test.ts +++ b/src/lib/verify-deployment-native-hosted.test.ts @@ -4,15 +4,23 @@ import { beforeEach, expect, it, vi } from "vitest"; import { buildChain } from "./dashboard/contract"; import { hostedNativeProvider } from "./inference/native-provider/hosted"; -import { probeOnboardInferenceInvocation, verifyDeployment } from "./verify-deployment"; +import { + onboardInferenceVerificationContext, + probeOnboardInferenceInvocation, + verifyDeployment, +} from "./verify-deployment"; +import { registeredInferenceFixture } from "../../test/helpers/registered-inference"; +import { verifyRegisteredInferenceProviderAttachment } from "./inference/config"; import { runSandboxInferenceInvocationProbe, verifyNativeHostedStatusAttachment, + verifyRegisteredStatusAttachment, } from "./actions/sandbox/inference-route-health"; vi.mock("./actions/sandbox/inference-route-health", () => ({ runSandboxInferenceInvocationProbe: vi.fn(async () => ({ ok: true })), verifyNativeHostedStatusAttachment: vi.fn(async () => undefined), + verifyRegisteredStatusAttachment: vi.fn(async () => undefined), })); const definition = hostedNativeProvider( @@ -37,7 +45,10 @@ const context = { nativeHostedProviderAttachment: attachment, }; -beforeEach(() => vi.clearAllMocks()); +beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(verifyRegisteredStatusAttachment).mockReset(); +}); it.each([true, false])( "qualifies a native hosted route only when its invocation succeeds: %s", @@ -102,3 +113,121 @@ it("preserves the legacy route when no native attachment was recorded", async () expect.not.objectContaining({ nativeProvider: true }), ); }); + +it.each(["openclaw", "hermes"])( + "preserves a registered provider through %s onboarding verification", + async (agentName) => { + const fixture = registeredInferenceFixture(true); + const flow = { + provider: "native-extra", + model: "selected-model", + endpointUrl: fixture.binding.endpointUrl, + preferredInferenceApi: fixture.binding.inferenceApi, + credentialEnv: fixture.binding.credentialEnv, + }; + const selected = onboardInferenceVerificationContext( + "alpha", + fixture.binding.gatewayName, + agentName, + flow, + { + ...flow, + registeredInferenceProviderAttachment: fixture.binding, + }, + ); + expect(selected.route.registeredInferenceProviderAttachment).toEqual(fixture.binding); + expect(selected.invocation.registeredInferenceProviderAttachment).toEqual(fixture.binding); + vi.mocked(verifyRegisteredStatusAttachment).mockImplementationOnce(async ({ expected }) => { + await verifyRegisteredInferenceProviderAttachment({ adapter: fixture.adapter, expected }); + }); + await expect(probeOnboardInferenceInvocation(selected.invocation)).resolves.toEqual({ + ok: true, + }); + expect(verifyRegisteredStatusAttachment).toHaveBeenCalledExactlyOnceWith({ + sandboxName: "alpha", + gatewayName: fixture.binding.gatewayName, + expected: fixture.binding, + }); + expect(runSandboxInferenceInvocationProbe).toHaveBeenCalledExactlyOnceWith( + expect.objectContaining({ + provider: "native-extra", + model: "selected-model", + nativeProvider: true, + registeredInferenceProviderAttachment: fixture.binding, + preferredInferenceApi: fixture.binding.inferenceApi, + }), + ); + expect(verifyNativeHostedStatusAttachment).not.toHaveBeenCalled(); + }, +); + +it.each(["served", "failed", "missing-probe"])( + "requires an actual registered-provider invocation for deployment: %s", + async (scenario) => { + const { binding } = registeredInferenceFixture(true); + const executeSandboxCommand = vi.fn(async () => ({ status: 0, stdout: "200", stderr: "" })); + const probeInferenceInvocation = vi.fn(async () => ({ ok: scenario === "served" })); + const result = await verifyDeployment( + "alpha", + buildChain(), + { + executeSandboxCommand, + probeHostPort: () => 200, + getMessagingChannels: () => [], + providerExistsInGateway: () => true, + ...(scenario === "missing-probe" ? {} : { probeInferenceInvocation }), + }, + { + retryDelaysMs: [], + sleep: async () => {}, + inferenceRouteContext: { + gatewayName: binding.gatewayName, + provider: binding.providerName, + endpointUrl: binding.endpointUrl, + preferredInferenceApi: binding.inferenceApi, + credentialEnv: binding.credentialEnv, + registeredInferenceProviderAttachment: binding, + }, + }, + ); + expect(result.verification.inferenceRouteWorking).toBe(scenario === "served"); + expect(executeSandboxCommand.mock.calls.flat().join("\n")).not.toContain("inference.local"); + }, +); + +it.each([ + "replaced", + "detached", + "wrong-gateway", + "wrong-endpoint", + "wrong-api", + "wrong-key", + "malformed", +])("refuses a registered invocation after its selected binding changes: %s", async (scenario) => { + const fixture = registeredInferenceFixture(true); + const changes: Record> = { + replaced: { id: "replacement-id" }, + detached: { attached: false }, + }; + Object.assign(fixture.state, changes[scenario]); + vi.mocked(verifyRegisteredStatusAttachment).mockImplementationOnce(async ({ expected }) => { + await verifyRegisteredInferenceProviderAttachment({ adapter: fixture.adapter, expected }); + }); + const result = await probeOnboardInferenceInvocation({ + sandboxName: "alpha", + agentName: "hermes", + provider: "native-extra", + model: "selected-model", + gatewayName: scenario === "wrong-gateway" ? "other" : fixture.binding.gatewayName, + endpointUrl: + scenario === "wrong-endpoint" ? "https://other.example/v1" : fixture.binding.endpointUrl, + preferredInferenceApi: + scenario === "wrong-api" ? "anthropic-messages" : fixture.binding.inferenceApi, + credentialEnv: scenario === "wrong-key" ? "OTHER_KEY" : fixture.binding.credentialEnv, + registeredInferenceProviderAttachment: scenario === "malformed" ? {} : fixture.binding, + }); + expect(result.ok).toBe(false); + expect(runSandboxInferenceInvocationProbe).not.toHaveBeenCalled(); + expect(fixture.adapter.attachProvider).not.toHaveBeenCalled(); + expect(fixture.adapter.detachProvider).not.toHaveBeenCalled(); +}); diff --git a/src/lib/verify-deployment.ts b/src/lib/verify-deployment.ts index 60d4cfef4bb..c7b74f1d794 100644 --- a/src/lib/verify-deployment.ts +++ b/src/lib/verify-deployment.ts @@ -25,6 +25,7 @@ import { parseVersionFromText } from "./adapters/openshell/client"; import { runSandboxInferenceInvocationProbe, verifyNativeHostedStatusAttachment, + verifyRegisteredStatusAttachment, type SandboxInferenceRouteHealthContext, } from "./actions/sandbox/inference-route-health"; import { compareChannelSets, type RuntimeChannelStatus } from "./channel-runtime-status"; @@ -35,6 +36,7 @@ import { listMessagingChannelsWithoutCredentials } from "./messaging/channels"; import { retryUntilAsync } from "./core/retry"; import { isNativeNvidiaProvider } from "./inference/native-nvidia"; import { requireHostedProviderAttachment } from "./inference/native-provider/hosted-attachment"; +import { recordedRegisteredInferenceProviderAttachment } from "./inference/native-provider/registered"; import { isPersistedPiSharedLocalInferenceRoute, normalizeNativeLocalBinding, @@ -293,8 +295,15 @@ export type InferenceRouteContext = { nativeLocalExpected?: boolean; nativeLocalProviderAttachment?: NativeLocalProviderAttachment; nativeHostedProviderAttachment?: unknown; + registeredInferenceProviderAttachment?: unknown; + preferredInferenceApi?: string | null; + credentialEnv?: string | null; }; +function registeredVerificationAttachment(context: InferenceRouteContext, sandboxName: string) { + return recordedRegisteredInferenceProviderAttachment({ ...context, name: sandboxName }); +} + function toRouteHealthContext(context: InferenceRouteContext): SandboxInferenceRouteHealthContext { return { provider: context.provider ?? null }; } @@ -457,7 +466,9 @@ async function verifyInferenceRoute( const nativeLocal = context.nativeLocalExpected || context.nativeLocalProviderAttachment !== undefined; let hostedAttachment; + let registeredAttachment; try { + registeredAttachment = registeredVerificationAttachment(context, sandboxName); hostedAttachment = requireHostedProviderAttachment( context.nativeHostedProviderAttachment, context.provider, @@ -465,9 +476,9 @@ async function verifyInferenceRoute( } catch { return { status: "unhealthy", - detail: "Invalid native hosted provider attachment", + detail: "Invalid native provider attachment", httpCode: 0, - hint: "Restore the recorded provider ownership before retrying onboarding.", + hint: "Restore the recorded provider binding before retrying onboarding.", }; } if (nativeLocal && !matchesNativeLocalVerificationRoute(context, sandboxName)) { @@ -478,12 +489,19 @@ async function verifyInferenceRoute( hint: "Reconcile the recorded native local provider before retrying onboarding.", }; } - if (isNativeNvidiaProvider(context.provider) || hostedAttachment || nativeLocal) { - const label = nativeLocal - ? "local" - : hostedAttachment - ? (context.provider?.trim() ?? "hosted") - : "NVIDIA"; + if ( + isNativeNvidiaProvider(context.provider) || + hostedAttachment || + nativeLocal || + registeredAttachment + ) { + const label = registeredAttachment + ? "registered provider" + : nativeLocal + ? "local" + : hostedAttachment + ? (context.provider?.trim() ?? "hosted") + : "NVIDIA"; const invocation = await retryUntilAsync( async () => (await deps.probeInferenceInvocation?.()) ?? null, { @@ -1006,11 +1024,14 @@ export function onboardInferenceVerificationContext( flow: Pick & { endpointUrl?: string | null; nimContainer?: string | null; + credentialEnv?: string | null; }, attachment?: | (Pick< InferenceRouteContext, - "nativeLocalProviderAttachment" | "nativeHostedProviderAttachment" + | "nativeLocalProviderAttachment" + | "nativeHostedProviderAttachment" + | "registeredInferenceProviderAttachment" > & { nimContainer?: string | null; agent?: string | null; @@ -1020,6 +1041,7 @@ export function onboardInferenceVerificationContext( createdAt?: string | null; pendingRouteReservation?: boolean; hostLocalInferenceReceipt?: string | null; + credentialEnv?: string | null; }) | null, ): { route: InferenceRouteContext; invocation: InferenceInvocationContext } { @@ -1031,10 +1053,14 @@ export function onboardInferenceVerificationContext( flow.provider === "vllm-local" && Boolean(attachment?.nimContainer), ) && !isPersistedPiSharedLocalInferenceRoute(attachment, flow); const nativeLocalProviderAttachment = - flow.provider === "vllm-local" && attachment?.nimContainer + flow.provider === "vllm-local" && + (flow.nimContainer || attachment?.nimContainer) && + attachment?.nativeLocalProviderAttachment?.authMode !== "sentinel" ? undefined : attachment?.nativeLocalProviderAttachment; const nativeHostedProviderAttachment = attachment?.nativeHostedProviderAttachment; + const registeredInferenceProviderAttachment = attachment?.registeredInferenceProviderAttachment; + const credentialEnv = flow.credentialEnv ?? attachment?.credentialEnv; const endpointUrl = agentName === "hermes" && nativeLocalExpected && @@ -1056,6 +1082,9 @@ export function onboardInferenceVerificationContext( nativeLocalExpected, nativeLocalProviderAttachment, nativeHostedProviderAttachment, + registeredInferenceProviderAttachment, + preferredInferenceApi: flow.preferredInferenceApi, + credentialEnv, }, invocation: { sandboxName, @@ -1068,6 +1097,8 @@ export function onboardInferenceVerificationContext( nativeLocalExpected, nativeLocalProviderAttachment, nativeHostedProviderAttachment, + registeredInferenceProviderAttachment, + credentialEnv, }, }; } @@ -1100,7 +1131,15 @@ export async function probeOnboardInferenceInvocation( }; } let hostedAttachment; + let registeredAttachment; try { + registeredAttachment = registeredVerificationAttachment(context, context.sandboxName); + if (registeredAttachment) + await verifyRegisteredStatusAttachment({ + gatewayName: context.gatewayName, + sandboxName: context.sandboxName, + expected: registeredAttachment, + }); hostedAttachment = requireHostedProviderAttachment( context.nativeHostedProviderAttachment, provider, @@ -1114,7 +1153,7 @@ export async function probeOnboardInferenceInvocation( } catch { return { ok: false, - detail: "the recorded native hosted provider attachment could not be verified", + detail: "the recorded native provider attachment could not be verified", }; } const result = await probe({ @@ -1124,11 +1163,15 @@ export async function probeOnboardInferenceInvocation( provider, model, preferredInferenceApi: context.preferredInferenceApi, + ...(registeredAttachment + ? { registeredInferenceProviderAttachment: registeredAttachment } + : {}), ...(context.nativeLocalProviderAttachment ? { nativeLocalProviderAttachment: context.nativeLocalProviderAttachment } : {}), ...(isNativeNvidiaProvider(provider) || hostedAttachment || + registeredAttachment || context.nativeLocalProviderAttachment ? { nativeProvider: true } : {}), diff --git a/test/agents/deepagents/dcode-managed-exec.test.ts b/test/agents/deepagents/dcode-managed-exec.test.ts index c10f108d62b..ad2eb92d5b7 100644 --- a/test/agents/deepagents/dcode-managed-exec.test.ts +++ b/test/agents/deepagents/dcode-managed-exec.test.ts @@ -13,7 +13,6 @@ const LAUNCHER_PATH = path.join( "langchain-deepagents-code", "dcode-launcher.sh", ); -const TEST_OWNER_UID = process.getuid?.() ?? 0; type RlimitHelperInstaller = (helperPath: string, markerPath: string) => void; @@ -60,8 +59,6 @@ function makeLauncherFixture( "dcode-managed-exec", ); const markerPath = path.join(tempDir, "observability-enabled"); - const hostPath = path.join(tempDir, "trusted-proxy-host"); - const portPath = path.join(tempDir, "trusted-proxy-port"); const wrapperPath = path.join(tempDir, "dcode-wrapper.sh"); const wrapperMarkerPath = path.join(tempDir, "wrapper-ran"); const rlimitPath = path.join(tempDir, "sandbox-rlimits.sh"); @@ -80,22 +77,8 @@ function makeLauncherFixture( .replace( 'readonly MANAGED_OBSERVABILITY_MARKER="/sandbox/.deepagents/.nemoclaw-observability-enabled"', `readonly MANAGED_OBSERVABILITY_MARKER="${markerPath}"`, - ) - .replace( - 'readonly MANAGED_PROXY_HOST_FILE="/usr/local/share/nemoclaw/dcode-proxy-host"', - `readonly MANAGED_PROXY_HOST_FILE="${hostPath}"`, - ) - .replace( - 'readonly MANAGED_PROXY_PORT_FILE="/usr/local/share/nemoclaw/dcode-proxy-port"', - `readonly MANAGED_PROXY_PORT_FILE="${portPath}"`, - ) - .replace( - "readonly MANAGED_PROXY_OWNER_UID=0", - `readonly MANAGED_PROXY_OWNER_UID=${TEST_OWNER_UID}`, ); - fs.writeFileSync(hostPath, "managed-proxy.internal\n", { mode: 0o444 }); - fs.writeFileSync(portPath, "3128\n", { mode: 0o444 }); installRlimitHelper(rlimitPath, rlimitMarkerPath); fs.writeFileSync( wrapperPath, @@ -124,7 +107,7 @@ describe("Deep Agents Code side-effect-free managed exec", () => { [ "/bin/sh", "-c", - 'printf "OBS=%s PROXY=%s" "${NEMOCLAW_OBSERVABILITY-__unset__}" "$HTTPS_PROXY"', + 'printf "OBS=%s PROXY=%s" "${NEMOCLAW_OBSERVABILITY-__unset__}" "${HTTPS_PROXY-__unset__}"', ], { env: { PATH: process.env.PATH ?? "/usr/bin:/bin" }, @@ -133,7 +116,7 @@ describe("Deep Agents Code side-effect-free managed exec", () => { ); expect(result.status, result.stderr).toBe(0); - expect(result.stdout).toBe("OBS=1 PROXY=http://managed-proxy.internal:3128"); + expect(result.stdout).toBe("OBS=1 PROXY=__unset__"); expect(fs.existsSync(wrapperMarkerPath)).toBe(false); expect(fs.readFileSync(rlimitMarkerPath, "utf8")).toBe("hardened\nverified\n"); expect(fs.readFileSync(markerPath, "utf8")).toBe("1\n"); @@ -153,7 +136,7 @@ describe("Deep Agents Code side-effect-free managed exec", () => { [ "/bin/sh", "-c", - 'printf "OBS=%s PROXY=%s" "${NEMOCLAW_OBSERVABILITY-__unset__}" "$HTTPS_PROXY"', + 'printf "OBS=%s PROXY=%s" "${NEMOCLAW_OBSERVABILITY-__unset__}" "${HTTPS_PROXY-__unset__}"', ], { env: { @@ -165,7 +148,7 @@ describe("Deep Agents Code side-effect-free managed exec", () => { ); expect(result.status, result.stderr).toBe(0); - expect(result.stdout).toBe("OBS=__unset__ PROXY=http://managed-proxy.internal:3128"); + expect(result.stdout).toBe("OBS=__unset__ PROXY=__unset__"); expect(fs.existsSync(wrapperMarkerPath)).toBe(false); expect(fs.readFileSync(rlimitMarkerPath, "utf8")).toBe("hardened\nverified\n"); expect(fs.existsSync(markerPath)).toBe(false); diff --git a/test/agents/deepagents/langchain-deepagents-code-auto-approval-image.test.ts b/test/agents/deepagents/langchain-deepagents-code-auto-approval-image.test.ts index 775af754998..30ad7e1fb2e 100644 --- a/test/agents/deepagents/langchain-deepagents-code-auto-approval-image.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-auto-approval-image.test.ts @@ -32,10 +32,10 @@ describe("LangChain Deep Agents Code auto-approval image contracts", () => { `printf '%s\\n' "$NEMOCLAW_DCODE_AUTO_APPROVAL" > /usr/local/share/nemoclaw/dcode-auto-approval`, ); expect(dockerfile).toContain( - "chown root:root /usr/local/share/nemoclaw/dcode-proxy-host /usr/local/share/nemoclaw/dcode-proxy-port /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval", + "chown root:root /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval", ); expect(dockerfile).toContain( - "chmod 0444 /usr/local/share/nemoclaw/dcode-proxy-host /usr/local/share/nemoclaw/dcode-proxy-port /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval", + "chmod 0444 /usr/local/share/nemoclaw/dcode-inference-base-url /usr/local/share/nemoclaw/dcode-upstream-provider /usr/local/share/nemoclaw/dcode-auto-approval", ); const envBlock = dockerfile.slice(dockerfile.indexOf("ENV HOME=")); expect(envBlock).not.toContain("NEMOCLAW_DCODE_AUTO_APPROVAL"); diff --git a/test/agents/deepagents/langchain-deepagents-code-config.test.ts b/test/agents/deepagents/langchain-deepagents-code-config.test.ts index a2ac27ed54a..1aea0977fff 100644 --- a/test/agents/deepagents/langchain-deepagents-code-config.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-config.test.ts @@ -70,6 +70,23 @@ function runGenerator(env: Record): string { } describe("LangChain Deep Agents Code config generator", () => { + it.each([ + ["compatible-endpoint", "COMPATIBLE_API_KEY", "http://10.20.30.40:43123/v1"], + [ + "compatible-anthropic-endpoint", + "COMPATIBLE_ANTHROPIC_API_KEY", + "http://10.20.30.40:43123/v1", + ], + ["compatible-endpoint", "COMPATIBLE_API_KEY", "https://integrate.api.nvidia.com/v1"], + ])("keeps %s hosted endpoints on their attached credential", (provider, credential, endpoint) => { + const config = runGenerator({ + NEMOCLAW_UPSTREAM_PROVIDER: provider, + NEMOCLAW_INFERENCE_BASE_URL: endpoint, + }); + expect(config).toContain(`api_key_env = "${credential}"`); + expect(config).not.toContain("NEMOCLAW_LOCAL_INFERENCE_TOKEN"); + }); + it.each([ ["compatible-endpoint", "openai", "nvidia/nvidia/nemotron-3-ultra"], ["openrouter", "openrouter", "nvidia/nemotron-3-ultra-550b-a55b"], diff --git a/test/agents/deepagents/langchain-deepagents-code-direct-module-patch.test.ts b/test/agents/deepagents/langchain-deepagents-code-direct-module-patch.test.ts index 2572f4ec38c..a3925a988da 100644 --- a/test/agents/deepagents/langchain-deepagents-code-direct-module-patch.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-direct-module-patch.test.ts @@ -144,42 +144,6 @@ describe("LangChain Deep Agents Code managed package patch", () => { expect(result.stderr).toContain("Managed package patch is partial: helper is missing or stale"); expect(fs.readFileSync(target, "utf8")).toBe(corrupted); }); - it("preserves upstream MCP JSON diagnostics around the managed descriptor loader", () => { - const tempDir = createPatchedPackageFixture(); - const invalidConfig = path.join(tempDir, "invalid-mcp.json"); - fs.writeFileSync(invalidConfig, '{"mcpServers": }\n', "utf8"); - const result = spawnSync( - "python3", - [ - "-c", - `import json -from deepagents_code.mcp_tools import load_mcp_config - -try: - load_mcp_config(${JSON.stringify(invalidConfig)}) -except json.JSONDecodeError as exc: - print(str(exc)) -else: - raise AssertionError("invalid JSON unexpectedly loaded") -`, - ], - { - encoding: "utf8", - env: { PATH: process.env.PATH, PYTHONPATH: tempDir }, - }, - ); - - expect(result.status, result.stderr).toBe(0); - expect(result.stdout).toContain("Invalid JSON in MCP config file"); - expect(result.stdout).toContain("Check the JSON syntax."); - expect(result.stdout).toContain("^"); - const patchedLoader = fs.readFileSync( - path.join(tempDir, "deepagents_code", "mcp_tools.py"), - "utf8", - ); - expect(patchedLoader).toContain("return json.loads(managed_payload)"); - expect(patchedLoader).toContain("return json.load(file_obj)"); - }); it.each([ ["update"], @@ -1207,6 +1171,15 @@ async def validate(): assert os.environ["LANGGRAPH_CLI_NO_ANALYTICS"] == "1" assert os.environ["OTEL_ENABLED"] == "false" + hostile_network_env = { + "OPENAI_PROXY": "http://attacker.example:8080", + "HTTP_PROXY": "http://attacker.example:8080", + "ALL_PROXY": "socks5://attacker.example:1080", + "OTEL_EXPORTER_OTLP_ENDPOINT": "https://attacker.example/v1/traces", + "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "https://attacker.example/v1/traces", + "OTEL_EXPORTER_OTLP_HEADERS": "authorization=attacker", + "OTEL_EXPORTER_OTLP_TRACES_HEADERS": "authorization=attacker", + } analytics_child = server.ServerProcess( [ sys.executable, @@ -1217,6 +1190,8 @@ async def validate(): "assert os.environ['LANGGRAPH_CLI_NO_ANALYTICS'] == '1'; " "assert os.environ['OTEL_ENABLED'] == 'false'; " "assert 'OPENAI_PROXY' not in os.environ; " + "assert 'HTTP_PROXY' not in os.environ; " + "assert 'ALL_PROXY' not in os.environ; " "assert 'OTEL_EXPORTER_OTLP_ENDPOINT' not in os.environ; " "assert 'OTEL_EXPORTER_OTLP_TRACES_ENDPOINT' not in os.environ; " "assert 'OTEL_EXPORTER_OTLP_HEADERS' not in os.environ; " @@ -1228,32 +1203,20 @@ async def validate(): "LANGGRAPH_NO_VERSION_CHECK": "false", "LANGGRAPH_CLI_NO_ANALYTICS": "0", "OTEL_ENABLED": "true", - "OPENAI_PROXY": "http://attacker.example:8080", - "OTEL_EXPORTER_OTLP_ENDPOINT": "https://attacker.example/v1/traces", - "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "https://attacker.example/v1/traces", - "OTEL_EXPORTER_OTLP_HEADERS": "authorization=attacker", - "OTEL_EXPORTER_OTLP_TRACES_HEADERS": "authorization=attacker", + **hostile_network_env, }, ) analytics_child._persistent_env_overrides.update({ "LANGGRAPH_NO_VERSION_CHECK": "false", "LANGGRAPH_CLI_NO_ANALYTICS": "false", "OTEL_ENABLED": "true", - "OPENAI_PROXY": "http://attacker.example:8080", - "OTEL_EXPORTER_OTLP_ENDPOINT": "https://attacker.example/v1/traces", - "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "https://attacker.example/v1/traces", - "OTEL_EXPORTER_OTLP_HEADERS": "authorization=attacker", - "OTEL_EXPORTER_OTLP_TRACES_HEADERS": "authorization=attacker", + **hostile_network_env, }) analytics_child._env_overrides.update({ "LANGGRAPH_NO_VERSION_CHECK": "0", "LANGGRAPH_CLI_NO_ANALYTICS": "0", "OTEL_ENABLED": "1", - "OPENAI_PROXY": "http://attacker.example:8080", - "OTEL_EXPORTER_OTLP_ENDPOINT": "https://attacker.example/v1/traces", - "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT": "https://attacker.example/v1/traces", - "OTEL_EXPORTER_OTLP_HEADERS": "authorization=attacker", - "OTEL_EXPORTER_OTLP_TRACES_HEADERS": "authorization=attacker", + **hostile_network_env, }) await analytics_child.start() await analytics_child.restart() @@ -1271,7 +1234,12 @@ async def validate(): assert "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL" not in os.environ assert "PROJECT_API_KEY" not in config._preview_dotenv_environ() assert "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL" not in config._preview_dotenv_environ() - assert _nemoclaw_managed.managed_fetch_proxy_url() is None + assert _nemoclaw_managed._managed_fetch_uses_openshell() is True + proxy_names = ('HTTP_PROXY', 'HTTPS_PROXY', 'NO_PROXY', 'http_proxy', 'https_proxy', 'no_proxy', 'ALL_PROXY', 'all_proxy', 'grpc_proxy', 'GRPC_PROXY', 'NODE_USE_ENV_PROXY', 'DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL', 'NEMOCLAW_PROXY_HOST', 'NEMOCLAW_PROXY_PORT', 'OPENAI_PROXY') + for name in proxy_names: + os.environ[name] = "http://attacker.example:8080" + _nemoclaw_managed.assert_safe_runtime() + assert all(name not in os.environ for name in proxy_names) for name in ( "LANGSMITH_TRACING", "LANGSMITH_TRACING_V2", @@ -1396,7 +1364,7 @@ print("managed-auto-approval-ok") }); it("fails closed when the installed version or required source shape drifts", () => { - const wrongVersion = createPackageFixture("0.1.31"); + const wrongVersion = createPackageFixture("0.1.51"); const versionResult = spawnSync("python3", [patcher], { env: { PATH: process.env.PATH, PYTHONPATH: wrongVersion }, encoding: "utf8", diff --git a/test/agents/deepagents/langchain-deepagents-code-fetch-proxy.test.ts b/test/agents/deepagents/langchain-deepagents-code-fetch-proxy.test.ts index 892f56a7c57..de46f664318 100644 --- a/test/agents/deepagents/langchain-deepagents-code-fetch-proxy.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-fetch-proxy.test.ts @@ -33,41 +33,46 @@ function readAgentFile(name: string): string { afterEach(cleanupPackageFixtures); -describe("LangChain Deep Agents Code managed fetch proxy", () => { - it("persists the root-owned proxy as the explicit fetch_url delegation", () => { +describe("LangChain Deep Agents Code managed fetch transport", () => { + it.each(["_pinned_dns", "_is_blocked_ip"])( + "refuses an upstream package missing the %s fetch guard", + (symbol) => { + const tempDir = createPackageFixture(); + const toolsPath = path.join(tempDir, "deepagents_code", "tools.py"); + fs.writeFileSync( + toolsPath, + fs.readFileSync(toolsPath, "utf8").replace(`def ${symbol}(`, `def removed${symbol}(`), + ); + expect(() => patchFixture(tempDir)).toThrow(symbol); + }, + ); + it("clears obsolete proxy delegation in startup and sourced shell environments", () => { const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-fetch-proxy-")); try { const { envFile, scriptPath } = makeStartScriptFixture(tempDir, { markerDir: dcodeStateDir(tempDir), }); const { envFileText, output } = runStartScriptProxyProbe(scriptPath, envFile, {}); - const managedProxy = "http://10.200.0.1:3128"; + const managedProxy = "__unset__"; const outputLines = output.trimEnd().split("\n"); expect(outputLines).toContain(`RUNTIME_${TRUSTED_FETCH_PROXY_ENV_NAME}=${managedProxy}`); expect(outputLines).toContain(`SOURCED_${TRUSTED_FETCH_PROXY_ENV_NAME}=${managedProxy}`); expect(envFileText.trimEnd().split("\n")).toContain( - `export ${TRUSTED_FETCH_PROXY_ENV_NAME}=${managedProxy}`, + `unset NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT ${TRUSTED_FETCH_PROXY_ENV_NAME}`, ); } finally { fs.rmSync(tempDir, { recursive: true, force: true }); } }); - it("rejects consistently forged proxy env that differs from root-owned files", () => { - const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-proxy-root-")); + it("rejects an unsafe managed provider identity before fetching", () => { + const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-fetch-identity-")); try { - const hostFile = path.join(tempDir, "dcode-proxy-host"); - const portFile = path.join(tempDir, "dcode-proxy-port"); const runtimeFile = path.join(tempDir, "managed-dcode-runtime.py"); - fs.writeFileSync(hostFile, "trusted-proxy.internal\n", { mode: 0o444 }); - fs.writeFileSync(portFile, "3129\n", { mode: 0o444 }); - fs.chmodSync(hostFile, 0o444); - fs.chmodSync(portFile, 0o444); fs.writeFileSync( runtimeFile, addDarwinFcntlSealConstants(readAgentFile("managed-dcode-runtime.py")), - "utf8", ); const result = spawnSync( "python3", @@ -77,53 +82,38 @@ describe("LangChain Deep Agents Code managed fetch proxy", () => { import importlib.util import os from pathlib import Path - -spec = importlib.util.spec_from_file_location( - "nemoclaw_managed_proxy_test", - ${JSON.stringify(runtimeFile)}, -) +spec = importlib.util.spec_from_file_location("managed", ${JSON.stringify(runtimeFile)}) runtime = importlib.util.module_from_spec(spec) spec.loader.exec_module(runtime) -runtime._MANAGED_PROXY_HOST_FILE = Path(${JSON.stringify(hostFile)}) -runtime._MANAGED_PROXY_PORT_FILE = Path(${JSON.stringify(portFile)}) +identity = Path(${JSON.stringify(tempDir)}) / "provider" +runtime._UPSTREAM_PROVIDER_FILE = identity runtime._MANAGED_FILE_OWNER_UID = os.getuid() - -for name in ( - "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL", - "HTTP_PROXY", - "HTTPS_PROXY", - "http_proxy", - "https_proxy", -): - os.environ[name] = "http://attacker.internal:4444" - +assert runtime._managed_fetch_uses_openshell() is False +identity.write_text("openai\\n") +identity.chmod(0o444) +assert runtime._managed_fetch_uses_openshell() is True +identity.chmod(0o644) +try: + runtime.managed_fetch_with_redirects("https://example.com", timeout=3, max_redirects=5, + original_fetch=lambda *_args, **_kwargs: (_ for _ in ()).throw(AssertionError("unsafe fallback")), + validation_error=ValueError, pin_dns=None, is_blocked_ip=None) +except ValueError as exc: + assert str(exc) == "managed upstream provider file has unsafe ownership or mode" +else: + raise AssertionError("writable managed identity was accepted") +identity.unlink() +identity.symlink_to(identity.with_name("missing")) try: - runtime.managed_fetch_proxy_url() + runtime._managed_fetch_uses_openshell() except RuntimeError as exc: - assert str(exc) == "managed fetch URL proxy does not match root-owned proxy" - assert "attacker.internal" not in str(exc) + assert str(exc) == "managed upstream provider file is missing or unsafe" else: - raise AssertionError("consistently forged proxy environment was accepted") - -trusted = "http://trusted-proxy.internal:3129" -for name in ( - "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL", - "HTTP_PROXY", - "HTTPS_PROXY", - "http_proxy", - "https_proxy", -): - os.environ[name] = trusted -os.environ["NO_PROXY"] = "raw.githubusercontent.com" -assert runtime.managed_fetch_proxy_url() == trusted -print("root-owned-proxy-verification-ok") + raise AssertionError("dangling identity symlink was accepted") `, ], { encoding: "utf8", env: { PATH: process.env.PATH } }, ); - expect(result.status, result.stderr).toBe(0); - expect(result.stdout).toContain("root-owned-proxy-verification-ok"); } finally { fs.rmSync(tempDir, { recursive: true, force: true }); } @@ -185,7 +175,7 @@ print("root-owned-proxy-verification-ok") expect(check).toContain("fetch_url_probe_source"); expect(check).toContain("from deepagents_code.tools import fetch_url"); - expect(check).toContain(TRUSTED_FETCH_PROXY_ENV_NAME); + expect(check).toContain("_managed_fetch_uses_openshell"); expect(check).toContain("expect_fetch_reached"); expect(check).toContain("FETCH_SUCCESS:2[0-9]{2}:[1-9][0-9]*"); expect(check).toContain("https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/README.md"); @@ -245,12 +235,8 @@ class Session: self.closed = True def get(self, _url, **kwargs): - global pinned_transport_count - with state_lock: - pinned_transport_count += 1 - if pinned_transport_count == PINNED_FETCH_COUNT: - pinned_ready.set() - assert release_pinned.wait(10), "timed out waiting for concurrent CA bundle mutation" + target = urlparse(_url) + urllib3_connection.create_connection((target.hostname, target.port or 443)) ca_contents = Path(kwargs["verify"]).read_text(encoding="utf-8") with state_lock: successful_ca_contents.append(ca_contents) @@ -261,25 +247,62 @@ requests.Session = Session requests.exceptions = types.SimpleNamespace(TooManyRedirects=RuntimeError) sys.modules["requests"] = requests +import ipaddress +import socket +from urllib.parse import urlparse +socket_calls = [] +dns_answers = {} +def managed_dns(host, _port, **_kwargs): + values = dns_answers.get(host, ["198.18.0.2"]) + if isinstance(values, Exception): + raise values + return [(socket.AF_INET6 if ":" in ip else socket.AF_INET, + socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (ip, 0)) for ip in values] +socket.getaddrinfo = managed_dns +urllib3_connection = types.ModuleType("urllib3.util.connection") +def socket_boundary(address, *_args, **_kwargs): + # The pinned upstream owner must pass a vetted numeric address, never a + # hostname that could resolve again between validation and connection. + ipaddress.ip_address(address[0]) + socket_calls.append(address) + return object() +urllib3_connection.create_connection = socket_boundary +urllib3_util = types.ModuleType("urllib3.util") +urllib3_util.connection = urllib3_connection +sys.modules["urllib3"] = types.ModuleType("urllib3") +sys.modules["urllib3.util"] = urllib3_util +sys.modules["urllib3.util.connection"] = urllib3_connection + from deepagents_code import _nemoclaw_managed, tools root = Path(${JSON.stringify(tempDir)}) -proxy_host_file = root / "managed-proxy-host" -proxy_port_file = root / "managed-proxy-port" +provider_file = root / "managed-provider" managed_ca_file = root / "managed-ca.pem" attacker_ca_file = root / "attacker-ca.pem" -proxy_host_file.write_text("managed-proxy.internal\\n", encoding="utf-8") -proxy_port_file.write_text("3128\\n", encoding="utf-8") +provider_file.write_text("openai\\n", encoding="utf-8") managed_ca_file.write_text("trusted CA bundle\\n", encoding="utf-8") attacker_ca_file.write_text("attacker CA bundle\\n", encoding="utf-8") -for trusted_file in (proxy_host_file, proxy_port_file, managed_ca_file, attacker_ca_file): +for trusted_file in (provider_file, managed_ca_file, attacker_ca_file): trusted_file.chmod(0o444) -_nemoclaw_managed._MANAGED_PROXY_HOST_FILE = proxy_host_file -_nemoclaw_managed._MANAGED_PROXY_PORT_FILE = proxy_port_file -_nemoclaw_managed._MANAGED_FETCH_CA_BUNDLE_FILE = managed_ca_file +_nemoclaw_managed._UPSTREAM_PROVIDER_FILE = provider_file _nemoclaw_managed._MANAGED_FILE_OWNER_UID = os.getuid() +# Exercise the installed path without replacing the runtime's trust selector. +# Redirect only that projected runtime path into this test's isolated filesystem. +original_open = os.open +def open_projected_trust(path, flags, *args, **kwargs): + assert Path(path) == Path("/run/openshell-supervisor-ca/material/ca-bundle.crt") + global pinned_transport_count + descriptor = original_open(managed_ca_file, flags, *args, **kwargs) + with state_lock: + pinned_transport_count += 1 + if pinned_transport_count == PINNED_FETCH_COUNT: + pinned_ready.set() + assert release_pinned.wait(10), "timed out waiting for concurrent CA bundle mutation" + return descriptor +os.open = open_projected_trust + for name in ( "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL", "HTTP_PROXY", @@ -364,7 +387,7 @@ print("concurrent-ca-bundle-mutation-ok") expect(result.stdout).toContain("concurrent-ca-bundle-mutation-ok"); }); - it("keeps redirects and concurrent fetches behind the explicit proxy without direct DNS", () => { + it("keeps redirects and concurrent fetches on the mediated network without ambient proxies", () => { const tempDir = createPackageFixture(); patchFixture(tempDir); const proxyUrl = "http://managed-proxy.internal:3128"; @@ -416,6 +439,8 @@ class Session: self.closed = True def get(self, url, **kwargs): + target = urlparse(url) + urllib3_connection.create_connection((target.hostname, target.port or 443)) global ca_swap_thread if ca_swap_thread is not None: thread = ca_swap_thread @@ -434,37 +459,60 @@ requests.Session = Session requests.exceptions = types.SimpleNamespace(HTTPError=ProxyPolicyDenied, TooManyRedirects=RuntimeError) sys.modules["requests"] = requests +import ipaddress +import socket +from urllib.parse import urlparse +socket_calls = [] +dns_answers = {} +def managed_dns(host, _port, **_kwargs): + values = dns_answers.get(host, ["198.18.0.2"]) + if isinstance(values, Exception): + raise values + return [(socket.AF_INET6 if ":" in ip else socket.AF_INET, + socket.SOCK_STREAM, socket.IPPROTO_TCP, "", (ip, 0)) for ip in values] +socket.getaddrinfo = managed_dns +urllib3_connection = types.ModuleType("urllib3.util.connection") +def socket_boundary(address, *_args, **_kwargs): + # The pinned upstream owner must pass a vetted numeric address, never a + # hostname that could resolve again between validation and connection. + ipaddress.ip_address(address[0]) + socket_calls.append(address) + return object() +urllib3_connection.create_connection = socket_boundary +urllib3_util = types.ModuleType("urllib3.util") +urllib3_util.connection = urllib3_connection +sys.modules["urllib3"] = types.ModuleType("urllib3") +sys.modules["urllib3.util"] = urllib3_util +sys.modules["urllib3.util.connection"] = urllib3_connection + from deepagents_code import _nemoclaw_managed, tools -from deepagents_code._nemoclaw_managed import managed_fetch_proxy_url -proxy_host_file = Path(${JSON.stringify(tempDir)}) / "managed-proxy-host" -proxy_port_file = Path(${JSON.stringify(tempDir)}) / "managed-proxy-port" + +provider_file = Path(${JSON.stringify(tempDir)}) / "managed-provider" managed_ca_file = Path(${JSON.stringify(tempDir)}) / "managed-ca.pem" attacker_ca_file = Path(${JSON.stringify(tempDir)}) / "attacker-ca.pem" writable_ca_file = Path(${JSON.stringify(tempDir)}) / "writable-sensitive-ca.pem" symlink_ca_file = Path(${JSON.stringify(tempDir)}) / "symlink-sensitive-ca.pem" -proxy_host_file.write_text("managed-proxy.internal\\n", encoding="utf-8") -proxy_port_file.write_text("3128\\n", encoding="utf-8") +provider_file.write_text("openai\\n", encoding="utf-8") managed_ca_file.write_text("test CA bundle\\n", encoding="utf-8") +managed_ca_file.chmod(0o444) attacker_ca_file.write_text("attacker CA bundle\\n", encoding="utf-8") writable_ca_file.write_text("unsafe CA bundle\\n", encoding="utf-8") writable_ca_file.chmod(0o666) symlink_ca_file.symlink_to(managed_ca_file) -proxy_host_file.chmod(0o444) -proxy_port_file.chmod(0o444) -_nemoclaw_managed._MANAGED_PROXY_HOST_FILE = proxy_host_file -_nemoclaw_managed._MANAGED_PROXY_PORT_FILE = proxy_port_file +provider_file.chmod(0o444) +_nemoclaw_managed._UPSTREAM_PROVIDER_FILE = provider_file _nemoclaw_managed._MANAGED_FETCH_CA_BUNDLE_FILE = managed_ca_file _nemoclaw_managed._MANAGED_FILE_OWNER_UID = os.getuid() os.environ["REQUESTS_CA_BUNDLE"] = "relative/../hostile-requests-ca.pem" os.environ["CURL_CA_BUNDLE"] = "/missing/hostile-curl-ca.pem" os.environ["SSL_CERT_FILE"] = "/missing/hostile-ssl-ca.pem" -def forbidden_direct_dns(*_args, **_kwargs): - raise AssertionError("managed fetch attempted direct DNS validation") +def forbidden_public_only_dns(*_args, **_kwargs): + raise AssertionError("managed fetch used the public-only upstream DNS guard") -tools._validate_url = forbidden_direct_dns -expected_proxies = {"http": ${JSON.stringify(proxyUrl)}, "https": ${JSON.stringify(proxyUrl)}} +tools._validate_url = forbidden_public_only_dns +expected_proxies = {} def assert_fd_ca_path(candidate): assert candidate.startswith(("/proc/self/fd/", "/dev/fd/")) @@ -505,7 +553,7 @@ assert { "timeout": 8, "headers": {"User-Agent": "Mozilla/5.0 (compatible; DeepAgents/1.0)"}, "allow_redirects": False, - "proxies": {"http": ${JSON.stringify(proxyUrl)}, "https": ${JSON.stringify(proxyUrl)}}, + "proxies": {}, } assert_fd_ca_path(call_kwargs["verify"]) assert sessions[0].ca_contents == ["test CA bundle\\n"] @@ -544,32 +592,119 @@ assert_managed_hops([ "https://raw.githubusercontent.com/example/main/README.md", ]) -# Cross-host redirects remain behind the same explicit proxy. The adapter does -# not locally authorize IMDS; it propagates the policy proxy's denial. The live -# egress check separately proves that OpenShell denies the IMDS destination. -metadata_url = "https://169.254.169.254/latest/meta-data/" +# Literal non-public destinations and resolved local sockets stop before a +# request. Each redirect receives the same admission and connection pinning. +for denied_url in ( + "https://127.0.0.1/", "https://127.1/", "https://2130706433/", + "https://0x7f000001/", "https://0177.0.0.1/", "https://127.0.0.1./", + "https://[::1]/", "https://[::ffff:127.0.0.1]/", "https://[2002:7f00:1::]/", + "http://0.0.0.0/", "http://[::]/", "https://10.0.0.1/", + "https://169.254.169.254/latest/meta-data/", "https://[fe80::1]/", + "https://[fc00::1]/", "https://198.18.0.2/", +): + for redirect in (False, True): + calls.clear() + socket_calls.clear() + responses.clear() + initial = "https://raw.githubusercontent.com/redirect" if redirect else denied_url + if redirect: + responses.append(Response(302, denied_url)) + try: + tools._fetch_with_redirects(initial, timeout=8) + except tools._UrlValidationError as exc: + assert str(exc) == "URL destination is blocked" + else: + raise AssertionError("non-public literal reached transport") + assert len(calls) == int(redirect) + assert len(socket_calls) == int(redirect) + assert all(address[0] == "198.18.0.2" for address in socket_calls) + +for answers in (["127.0.0.1"], ["::1"], ["::ffff:127.0.0.1"], ["0.0.0.0"], + ["::"], ["198.18.0.2", "127.0.0.1"]): + dns_answers["loopback-alias.example"] = answers + for redirect in (False, True): + calls.clear() + socket_calls.clear() + responses.clear() + denied_url = "https://loopback-alias.example/private" + initial = "https://raw.githubusercontent.com/redirect" if redirect else denied_url + if redirect: + responses.append(Response(302, denied_url)) + try: + tools._fetch_with_redirects(initial, timeout=8) + except tools._UrlValidationError as exc: + assert str(exc) == "URL destination is blocked" + else: + raise AssertionError("resolved local address reached transport") + assert len(calls) == int(redirect) + assert len(socket_calls) == int(redirect) + +# A synthetic address is not an upstream destination. The runtime still owns +# its policy decision, including rejection after a hostname redirect. expect_redirect_policy_denial( - "https://raw.githubusercontent.com/example/redirect-to-imds", - metadata_url, - "cross-host metadata", + "https://raw.githubusercontent.com/example/redirect-to-rebind", + "https://rebind.internal/private", "DNS-rebinding hostname", ) - -# DNS and resolved-IP policy belong to OpenShell. A rebinding candidate must be -# passed by hostname through the explicit proxy without local DNS, and the -# proxy's denial must propagate rather than triggering a direct retry. -rebind_url = "https://rebind.internal/private" -original_getaddrinfo = socket.getaddrinfo -def forbidden_local_dns(*_args, **_kwargs): - raise AssertionError("managed redirect attempted local DNS") -socket.getaddrinfo = forbidden_local_dns +assert all(address[0] == "198.18.0.2" for address in socket_calls) + +# A second hostname lookup would return loopback. The actual connection owner +# receives the first vetted address and cannot perform that second lookup. +original_dns = socket.getaddrinfo +dns_lookups = [] +def rebind_dns(host, port, **kwargs): + dns_lookups.append(host) + return [(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", + ("198.18.0.2" if len(dns_lookups) == 1 else "127.0.0.1", 0))] +socket.getaddrinfo = rebind_dns +calls.clear() +socket_calls.clear() +responses.clear() try: - expect_redirect_policy_denial( - "https://raw.githubusercontent.com/example/redirect-to-rebind", - rebind_url, - "DNS-rebinding hostname", - ) + assert tools._fetch_with_redirects("https://rebind.example/", timeout=8).status_code == 200 + assert dns_lookups == ["rebind.example"] + assert socket_calls == [("198.18.0.2", 443)] finally: - socket.getaddrinfo = original_getaddrinfo + socket.getaddrinfo = original_dns + +# Valid public literals and synthetic IPv6 answers retain their supported route. +for candidate, answer, expected_host in ( + ("https://8.8.8.8/", "8.8.8.8", "8.8.8.8"), + ("https://[2606:4700:4700::1111]/", "2606:4700:4700::1111", "2606:4700:4700::1111"), + ("https://synthetic.example/", "fd00::1234", "synthetic.example"), + ("https://täst.example/", "198.18.0.3", "xn--tst-qla.example"), + ("https://raw.githubusercontent.com./", "198.18.0.4", "raw.githubusercontent.com."), +): + dns_answers[expected_host] = [answer] + socket_calls.clear() + assert tools._fetch_with_redirects(candidate, timeout=8).status_code == 200 + assert socket_calls == [(answer, 443)] + del dns_answers[expected_host] + +for answer, error in ( + ([], "URL hostname resolved to no addresses"), + (["not-an-address"], "URL hostname returned an invalid address"), + (socket.gaierror("sensitive resolver detail"), "URL hostname could not be resolved"), +): + dns_answers["unresolved.example"] = answer + calls.clear() + socket_calls.clear() + try: + tools._fetch_with_redirects("https://unresolved.example/", timeout=8) + except tools._UrlValidationError as exc: + assert str(exc) == error + else: + raise AssertionError("invalid DNS result reached transport") + assert calls == [] and socket_calls == [] + +calls.clear() +socket_calls.clear() +try: + tools._fetch_with_redirects("https://127%2e0%2e0%2e1/", timeout=8) +except tools._UrlValidationError as exc: + assert str(exc) == "URL hostname is invalid" +else: + raise AssertionError("encoded authority escaped admission") +assert calls == [] and socket_calls == [] calls.clear() responses.append(Response(302)) @@ -640,45 +775,14 @@ for malformed in ("https://[broken", "https://example.com:not-a-port"): else: raise AssertionError(f"malformed URL escaped validation: {malformed}") +calls.clear() os.environ["HTTP_PROXY"] = "http://attacker.internal:4444" -try: - tools._fetch_with_redirects("https://raw.githubusercontent.com/example", timeout=8) -except tools._UrlValidationError as exc: - assert str(exc) == "managed fetch URL proxy does not match runtime proxy" - assert "attacker.internal" not in str(exc) -else: - raise AssertionError("proxy-integrity error escaped fetch_url validation") - -for invalid_proxy in ( - "http://user:proxy-secret@proxy.internal:3128", - "http://proxy.internal", - "http://proxy.internal:0", - "http://proxy.internal:70000", - "http://proxy.internal:3128/unexpected", - "http://proxy.internal:3128?route=unsafe", - "http://proxy.internal:3128#fragment", - " http://proxy.internal:3128", - "http://proxy.internal:3128\\n", -): - os.environ["DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL"] = invalid_proxy - for proxy_name in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"): - os.environ[proxy_name] = invalid_proxy - try: - managed_fetch_proxy_url() - except RuntimeError as exc: - assert str(exc) == "managed fetch URL proxy is invalid" - assert "proxy-secret" not in str(exc) - else: - raise AssertionError(f"invalid managed proxy was accepted: {invalid_proxy!r}") - -for proxy_name in ( - "DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL", - "HTTP_PROXY", - "HTTPS_PROXY", - "http_proxy", - "https_proxy", -): - os.environ[proxy_name] = ${JSON.stringify(proxyUrl)} +os.environ["HTTPS_PROXY"] = "http://user:secret@attacker.internal:4444" +os.environ["ALL_PROXY"] = "socks5://attacker.internal:1080" +os.environ["NO_PROXY"] = "*" +os.environ["DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL"] = "http://attacker.internal:4444" +assert tools._fetch_with_redirects("https://raw.githubusercontent.com/example", timeout=8).status_code == 200 +assert_managed_hops(["https://raw.githubusercontent.com/example"]) original_import = builtins.__import__ def block_requests_import(name, *args, **kwargs): @@ -702,7 +806,7 @@ finally: # Atomically replace the validated pathname after the managed helper opens it, # but before the transport consumes the CA input. The fd-backed verify path -# must remain pinned to the original root-owned trust bytes. +# must remain pinned to the original projected trust bytes. def replace_managed_ca_path(): replacement = managed_ca_file.with_name("managed-ca-replacement") if replacement.exists() or replacement.is_symlink(): @@ -723,6 +827,7 @@ assert sessions[-1].ca_contents == ["test CA bundle\\n"] assert_fd_ca_path(calls[0][2]["verify"]) managed_ca_file.unlink() managed_ca_file.write_text("test CA bundle\\n", encoding="utf-8") +managed_ca_file.chmod(0o444) for invalid_ca_bundle, expected_error in ( ( @@ -743,18 +848,38 @@ for invalid_ca_bundle, expected_error in ( raise AssertionError(f"invalid CA bundle was accepted: {invalid_ca_bundle!r}") _nemoclaw_managed._MANAGED_FETCH_CA_BUNDLE_FILE = managed_ca_file -_nemoclaw_managed._MANAGED_FILE_OWNER_UID = os.getuid() + 1 +# Image-file ownership does not supply authority for runtime-projected trust. +_nemoclaw_managed._MANAGED_FILE_OWNER_UID = 0 +descriptor, _ = _nemoclaw_managed._managed_fetch_ca_bundle() +_nemoclaw_managed._close_managed_fetch_ca_bundle(descriptor) +original_fstat = os.fstat +def foreign_owner_fstat(descriptor): + metadata = list(original_fstat(descriptor)) + metadata[4] = os.geteuid() + 1 + return os.stat_result(metadata) +os.fstat = foreign_owner_fstat try: _nemoclaw_managed._managed_fetch_ca_bundle() except RuntimeError as exc: assert str(exc) == "managed fetch CA bundle is invalid" else: raise AssertionError("wrong-owner CA bundle was accepted") +finally: + os.fstat = original_fstat _nemoclaw_managed._MANAGED_FILE_OWNER_UID = os.getuid() -# The proxy and CA are immutable process-wide image inputs. Per-call isolation -# therefore means a fresh Session and explicit proxy mapping for each fetch, -# including concurrent calls, rather than unsupported mutable configurations. +for mode in (0o644, 0o544, 0o400): + managed_ca_file.chmod(mode) + try: + _nemoclaw_managed._managed_fetch_ca_bundle() + except RuntimeError as exc: + assert str(exc) == "managed fetch CA bundle is invalid" + else: + raise AssertionError("CA bundle with a non-projected mode was accepted") +managed_ca_file.chmod(0o444) + +# Each fetch keeps its own Session and CA descriptor. The upstream connection +# lock serializes transport without sharing environment-derived credentials. calls.clear() session_start = len(sessions) concurrent_urls = [ @@ -813,7 +938,7 @@ print("managed-fetch-proxy-ok") expect(result.stdout).toContain("managed-fetch-proxy-ok"); }); - it("preserves unmanaged fallback when proxy env is absent", () => { + it("preserves upstream transport outside the managed image", () => { const tempDir = createPackageFixture(); patchFixture(tempDir); @@ -822,7 +947,9 @@ print("managed-fetch-proxy-ok") [ "-c", [ - "from deepagents_code import tools", + "from deepagents_code import tools, _nemoclaw_managed", + "from pathlib import Path", + `_nemoclaw_managed._UPSTREAM_PROVIDER_FILE = Path(${JSON.stringify(path.join(tempDir, "no-managed-provider"))})`, 'result = tools._fetch_with_redirects("https://example.com", timeout=3)', 'assert result == {"transport": "direct", "url": "https://example.com", "timeout": 3}', ].join("; "), diff --git a/test/agents/deepagents/langchain-deepagents-code-headless-runtime.test.ts b/test/agents/deepagents/langchain-deepagents-code-headless-runtime.test.ts index ce52fd5e95f..d5e140c4aee 100644 --- a/test/agents/deepagents/langchain-deepagents-code-headless-runtime.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-headless-runtime.test.ts @@ -235,73 +235,4 @@ describe("LangChain Deep Agents Code headless runtime contracts", () => { ); expect(classify("1", "something happened\nDCODE_EXIT:1")).toBe("fail:nonzero-exit"); }); - - it("accepts only the normalized login-shell proxy contract (#6191)", () => { - const validate = (proxyUrl: string, noProxy: string, lowerProxy = proxyUrl) => { - const loginHome = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-login-")); - const hostFile = path.join(loginHome, "trusted-proxy-host"); - const portFile = path.join(loginHome, "trusted-proxy-port"); - const proxyEnvFile = path.join(loginHome, "proxy-env.sh"); - const checkFixture = path.join(loginHome, "headless-check.sh"); - const runtimeUid = process.getuid?.() ?? 0; - fs.writeFileSync(hostFile, "10.200.0.1\n", "utf8"); - fs.writeFileSync(portFile, "3128\n", "utf8"); - fs.chmodSync(hostFile, 0o444); - fs.chmodSync(portFile, 0o444); - fs.writeFileSync( - checkFixture, - fs - .readFileSync(headlessCheckPath, "utf8") - .replaceAll("/usr/local/share/nemoclaw/dcode-proxy-host", hostFile) - .replaceAll("/usr/local/share/nemoclaw/dcode-proxy-port", portFile) - .replaceAll("/tmp/nemoclaw-proxy-env.sh", proxyEnvFile) - .replace('= "0:444"', `= "${runtimeUid}:444"`) - .replace( - 'runtime_uid="$(id -u)" || contract_fail runtime-user; sandbox_uid="$(id -u sandbox)" || contract_fail runtime-user;', - `runtime_uid=${runtimeUid}; sandbox_uid=${runtimeUid};`, - ), - "utf8", - ); - fs.writeFileSync( - proxyEnvFile, - [ - `export HTTP_PROXY=${JSON.stringify(proxyUrl)}`, - `export HTTPS_PROXY=${JSON.stringify(proxyUrl)}`, - `export http_proxy=${JSON.stringify(lowerProxy)}`, - `export https_proxy=${JSON.stringify(lowerProxy)}`, - `export NO_PROXY=${JSON.stringify(noProxy)}`, - `export no_proxy=${JSON.stringify(noProxy)}`, - "unset ALL_PROXY all_proxy", - "", - ].join("\n"), - "utf8", - ); - fs.chmodSync(proxyEnvFile, 0o444); - fs.writeFileSync( - path.join(loginHome, ".profile"), - `export HOME=/sandbox\n. ${JSON.stringify(proxyEnvFile)}\n`, - "utf8", - ); - return runHeadlessCheckSnippet( - [ - "sandbox_login_exec() {", - " case \"$1\" in *$'\\n'*|*$'\\r'*) return 97 ;; esac", - ' env -u HTTP_PROXY -u HTTPS_PROXY -u NO_PROXY -u http_proxy -u https_proxy -u no_proxy -u ALL_PROXY -u all_proxy HOME="$TEST_LOGIN_HOME" bash -lc "$1"', - "}", - "if sandbox_login_proxy_contract >/dev/null 2>&1; then printf pass; else printf fail; fi", - ].join("\n"), - { TEST_LOGIN_HOME: loginHome }, - checkFixture, - ); - }; - - const managedProxy = "http://10.200.0.1:3128"; - const managedNoProxy = "localhost,127.0.0.1,::1,10.200.0.1"; - expect(validate(managedProxy, managedNoProxy)).toBe("pass"); - expect(validate(managedProxy, `${managedNoProxy},inference.local`)).toBe("fail"); - expect(validate("http://corp-user:corp-password@proxy.example:8080", managedNoProxy)).toBe( - "fail", - ); - expect(validate(managedProxy, managedNoProxy, "http://other-proxy.example:3128")).toBe("fail"); - }); }); diff --git a/test/agents/deepagents/langchain-deepagents-code-image.test.ts b/test/agents/deepagents/langchain-deepagents-code-image.test.ts index 06b75733f71..1cee79eefc9 100644 --- a/test/agents/deepagents/langchain-deepagents-code-image.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-image.test.ts @@ -328,7 +328,7 @@ describe("LangChain Deep Agents Code image contracts", () => { } }); - function verifyManagedRuntimeProxyReplacement() { + function verifyManagedRuntimeProxyRemoval() { const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-start-")); const { envFile, scriptPath } = makeStartScriptFixture(tempDir, { markerDir: dcodeStateDir(tempDir), @@ -361,8 +361,8 @@ describe("LangChain Deep Agents Code image contracts", () => { ...inheritedTracingFlags, ...inheritedAnalyticsFlags, }); - const managedProxy = "http://10.200.0.1:3128"; - const managedNoProxy = "localhost,127.0.0.1,::1,10.200.0.1"; + const managedProxy = "__unset__"; + const managedNoProxy = "__unset__"; const outputLines = output.trimEnd().split("\n"); const envFileLines = envFileText.trimEnd().split("\n"); expect(fs.statSync(envFile).mode & 0o777).toBe(0o444); @@ -370,12 +370,12 @@ describe("LangChain Deep Agents Code image contracts", () => { for (const name of PROXY_URL_ENV_NAMES) { expect(outputLines).toContain(`RUNTIME_${name}=${managedProxy}`); expect(outputLines).toContain(`SOURCED_${name}=${managedProxy}`); - expect(envFileLines).toContain(`export ${name}=${managedProxy}`); + expect(envFileText).not.toContain(`export ${name}=`); } for (const name of NO_PROXY_ENV_NAMES) { expect(outputLines).toContain(`RUNTIME_${name}=${managedNoProxy}`); expect(outputLines).toContain(`SOURCED_${name}=${managedNoProxy}`); - expect(envFileLines).toContain(`export ${name}=${managedNoProxy.replaceAll(",", "\\,")}`); + expect(envFileText).not.toContain(`export ${name}=`); } for (const name of TRACING_ENABLE_ENV_NAMES) { expect(outputLines).toContain(`RUNTIME_${name}=false`); @@ -388,7 +388,7 @@ describe("LangChain Deep Agents Code image contracts", () => { expect(envFileLines).toContain(`export ${name}=1`); } expect(envFileLines).toContain("unset NEMOCLAW_ATTACHED_PROVIDER_API_KEY"); - expect(envFileLines).toContain("unset ALL_PROXY all_proxy OPENAI_PROXY"); + expect(envFileText).toContain("unset ALL_PROXY all_proxy OPENAI_PROXY"); expect( outputLines.filter((line) => /^(?:RUNTIME|SOURCED)_(?:NO_PROXY|no_proxy)=/.test(line)), ).not.toEqual(expect.arrayContaining([expect.stringContaining("inference.local")])); @@ -408,8 +408,8 @@ describe("LangChain Deep Agents Code image contracts", () => { } it( - "replaces inherited host proxy values with the managed runtime proxy (#6191)", - verifyManagedRuntimeProxyReplacement, + "clears inherited host proxy values from the managed runtime", + verifyManagedRuntimeProxyRemoval, ); it("keeps all Deep Agents Code entry points behind the managed wrapper boundary", () => { @@ -702,7 +702,7 @@ describe("LangChain Deep Agents Code image contracts", () => { quickjsImagePath?.startsWith(`${root}/`), ), ).toBe(true); - expect(defaultPolicy.landlock).toMatchObject({ compatibility: "strict" }); + expect(defaultPolicy.landlock).toMatchObject({ compatibility: "hard_requirement" }); const githubBinaries = policyBinaryPaths(defaultPolicy, "github"); expect(githubBinaries).toEqual( @@ -797,7 +797,7 @@ describe("LangChain Deep Agents Code image contracts", () => { expect(pythonEgressCheck).toContain(`DCODE_CANONICAL_PATH="${DCODE_CANONICAL_PATH}"`); expect(pythonEgressCheck).not.toContain("mktemp"); for (const expected of [ - 'grep -Fxq "PATH=${DCODE_CANONICAL_PATH}"', + `grep -Fxq "PATH=\${DCODE_CANONICAL_PATH}"`, 'printf "PYTHON_REAL=%s\\n"', "^PYTHON=/opt/venv/bin/python3$", "^PIP=/opt/venv/bin/pip3$", diff --git a/test/agents/deepagents/langchain-deepagents-code-managed-entrypoints.test.ts b/test/agents/deepagents/langchain-deepagents-code-managed-entrypoints.test.ts index 3df6d62fba5..d47820765ab 100644 --- a/test/agents/deepagents/langchain-deepagents-code-managed-entrypoints.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-managed-entrypoints.test.ts @@ -255,7 +255,9 @@ describe.concurrent("LangChain Deep Agents Code managed entrypoints", () => { it("does not serialize provider or optional-service secrets into the shell env file", () => { const start = readAgentFile("start.sh"); expect(start).toContain('chmod 444 "$tmp"'); - expect(start).toContain("write_export_if_set HTTPS_PROXY"); + expect(start).toContain( + "unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy", + ); expect(start).not.toContain("write_proxy_export_pair"); expect(start).toContain("export DEEPAGENTS_CODE_OFFLINE=1"); expect(start).toContain("export DEEPAGENTS_CODE_RIPGREP_INSTALLER=system"); diff --git a/test/agents/deepagents/langchain-deepagents-code-managed-mcp-hardening.test.ts b/test/agents/deepagents/langchain-deepagents-code-managed-mcp-hardening.test.ts index eed98932faa..884cd55ed15 100644 --- a/test/agents/deepagents/langchain-deepagents-code-managed-mcp-hardening.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-managed-mcp-hardening.test.ts @@ -5,7 +5,15 @@ import { spawnSync } from "node:child_process"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { describe, expect, it } from "vitest"; +import { afterAll, afterEach, describe, expect, it } from "vitest"; +import { + cleanupCachedPatchedFixture, + cleanupPackageFixtures, + createPatchedPackageFixture, +} from "../../helpers/langchain-deepagents-code-patch-fixture"; + +afterEach(cleanupPackageFixtures); +afterAll(cleanupCachedPatchedFixture); const managedRuntimePath = path.join( process.cwd(), @@ -30,6 +38,43 @@ function runManagedHelper(source: string) { } describe("Deep Agents managed MCP runtime hardening", () => { + it("preserves upstream MCP JSON diagnostics around the managed descriptor loader", () => { + const tempDir = createPatchedPackageFixture(); + const invalidConfig = path.join(tempDir, "invalid-mcp.json"); + fs.writeFileSync(invalidConfig, '{"mcpServers": }\n', "utf8"); + const result = spawnSync( + "python3", + [ + "-c", + `import json +from deepagents_code.mcp_tools import load_mcp_config + +try: + load_mcp_config(${JSON.stringify(invalidConfig)}) +except json.JSONDecodeError as exc: + print(str(exc)) +else: + raise AssertionError("invalid JSON unexpectedly loaded") +`, + ], + { + encoding: "utf8", + env: { PATH: process.env.PATH, PYTHONPATH: tempDir }, + }, + ); + + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain("Invalid JSON in MCP config file"); + expect(result.stdout).toContain("Check the JSON syntax."); + expect(result.stdout).toContain("^"); + const patchedLoader = fs.readFileSync( + path.join(tempDir, "deepagents_code", "mcp_tools.py"), + "utf8", + ); + expect(patchedLoader).toContain("return json.loads(managed_payload)"); + expect(patchedLoader).toContain("return json.load(file_obj)"); + }); + it.runIf(process.platform === "linux")( "rejects OpenShell supervisor TLS identity from the managed child runtime", () => { diff --git a/test/agents/deepagents/langchain-deepagents-code-managed-model-params.test.ts b/test/agents/deepagents/langchain-deepagents-code-managed-model-params.test.ts index 17b3e80b994..d6579ddbf74 100644 --- a/test/agents/deepagents/langchain-deepagents-code-managed-model-params.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-managed-model-params.test.ts @@ -92,6 +92,54 @@ print("managed-ultra-template-argument-ok") expect(output).toContain("managed-ultra-template-argument-ok"); }); + it.each([ + ["compatible-endpoint", "COMPATIBLE_API_KEY", "http://10.20.30.40:43123/v1"], + [ + "compatible-anthropic-endpoint", + "COMPATIBLE_ANTHROPIC_API_KEY", + "http://10.20.30.40:43123/v1", + ], + ["compatible-endpoint", "COMPATIBLE_API_KEY", "https://integrate.api.nvidia.com/v1"], + ])( + "supplies the attached %s credential in managed model options", + (provider, credential, endpoint) => { + const directory = createPackageFixture(); + patchFixture(directory); + const validation = ` +import os +from pathlib import Path +from deepagents_code import config +for file, value in ( + (${JSON.stringify(path.join(directory, "managed-inference-base-url"))}, ${JSON.stringify(endpoint)}), + (${JSON.stringify(managedUpstreamProviderPath(directory))}, ${JSON.stringify(provider)}), +): + p = Path(file) + p.chmod(0o644) + p.write_text(value + "\\n") + p.chmod(0o444) +name = ${JSON.stringify(credential)} +reference = "openshell:resolve:env:s" + "a" * 64 + "_" + name +os.environ[name] = reference +os.environ["NEMOCLAW_LOCAL_INFERENCE_TOKEN"] = "openshell:resolve:env:v42_NEMOCLAW_LOCAL_INFERENCE_TOKEN" +assert config._get_provider_kwargs("openai")["api_key"] == reference +for invalid in ("", "raw-secret", "openshell:resolve:env:" + name, "openshell:resolve:env:v42_OTHER_KEY"): + os.environ[name] = invalid + try: + config._get_provider_kwargs("openai") + except RuntimeError as error: + assert not invalid or invalid not in str(error) + else: + raise AssertionError("invalid custom reference accepted") +print("custom-hosted-constructor-reference-ok") +`; + const output = execFileSync("python3", ["-c", validation], { + env: { PATH: process.env.PATH, PYTHONPATH: directory }, + encoding: "utf8", + }); + expect(output).toContain("custom-hosted-constructor-reference-ok"); + }, + ); + it("passes the issued native-local reference to the actual model constructor (#12558)", () => { const tempDir = createPackageFixture(); patchFixture(tempDir); @@ -112,6 +160,8 @@ name = "NEMOCLAW_LOCAL_INFERENCE_TOKEN" for upstream in ("ollama-local", "vllm-local", "llama-cpp-local", "compatible-endpoint"): write_owned(${JSON.stringify(providerFile)}, upstream) for endpoint in ("http://host.openshell.internal:8000/v1", "http://host.docker.internal:11434/v1", "http://10.2.3.4:8000/v1", "http://172.16.2.3:8000/v1", "http://192.168.2.3:8000/v1"): + if upstream == "compatible-endpoint" and not endpoint.startswith(("http://host.openshell.internal:", "http://host.docker.internal:")): + continue write_owned(${JSON.stringify(routeFile)}, endpoint) for identity in ("v42", "s" + "a" * 64): reference = "openshell:resolve:env:" + identity + "_" + name @@ -132,18 +182,12 @@ for upstream in ("ollama-local", "vllm-local", "llama-cpp-local", "compatible-en else: raise AssertionError("invalid native-local reference accepted") -# Ambient local credentials cannot replace a hosted compatible reference. +# Ambient local credentials cannot replace the managed hosted-route token. +write_owned(${JSON.stringify(providerFile)}, "ollama-local") os.environ[name] = "openshell:resolve:env:v42_" + name -write_owned(${JSON.stringify(routeFile)}, "https://inference.local/v1") -assert config._get_provider_kwargs("openai")["api_key"] == "nemoclaw-managed-inference" -for endpoint in ("https://host.openshell.internal:8000/v1", "http://172.32.2.3:8000/v1"): +for endpoint in ("https://inference.local/v1", "https://host.openshell.internal:8000/v1", "http://172.32.2.3:8000/v1"): write_owned(${JSON.stringify(routeFile)}, endpoint) - try: - config._get_provider_kwargs("openai") - except RuntimeError: - pass - else: - raise AssertionError("local token replaced a hosted compatible reference") + assert config._get_provider_kwargs("openai")["api_key"] == "nemoclaw-managed-inference" write_owned(${JSON.stringify(routeFile)}, "http://host.openshell.internal:8000/v1") write_owned(${JSON.stringify(providerFile)}, "nvidia-prod") assert config._get_provider_kwargs("openai")["api_key"] == "nemoclaw-managed-inference" @@ -211,66 +255,6 @@ print("native-managed-constructor-placeholder-ok") expect(output).toContain("native-managed-constructor-placeholder-ok"); }); - it("passes the issued compatible placeholder to the hosted DCode constructor (#13164)", () => { - const tempDir = createPackageFixture(); - patchFixture(tempDir); - const routeFile = path.join(tempDir, "managed-inference-base-url"); - const providerFile = managedUpstreamProviderPath(tempDir); - const validation = ` -import os -from pathlib import Path -from deepagents_code import config - -def write_owned(file, value): - path = Path(file) - path.chmod(0o644) - path.write_text(value + "\\n") - path.chmod(0o444) - -write_owned(${JSON.stringify(providerFile)}, "compatible-endpoint") -name = "COMPATIBLE_API_KEY" -os.environ["NEMOCLAW_LOCAL_INFERENCE_TOKEN"] = "openshell:resolve:env:v42_NEMOCLAW_LOCAL_INFERENCE_TOKEN" -for endpoint in ("https://inference-api.nvidia.com/v1", "https://integrate.api.nvidia.com/v1", "https://compatible.example/v1"): - write_owned(${JSON.stringify(routeFile)}, endpoint) - for reference in ( - "openshell:resolve:env:" + name, - "openshell:resolve:env:v12_" + name, - "openshell:resolve:env:s" + "a" * 64 + "_" + name, - ): - os.environ[name] = reference - for adapter in ("openai", "openrouter"): - resolved = config._get_provider_kwargs(adapter) - assert resolved["api_key"] == reference - assert resolved["base_url"] == endpoint - - for invalid in ("", "raw-secret", "nemoclaw-managed-inference", "openshell:resolve:env:v12_OTHER_KEY"): - os.environ[name] = invalid - try: - config._get_provider_kwargs("openai") - except RuntimeError as error: - assert not invalid or invalid not in str(error) - else: - raise AssertionError("hosted compatible route accepted an invalid reference") - del os.environ[name] - try: - config._get_provider_kwargs("openai") - except RuntimeError: - pass - else: - raise AssertionError("hosted compatible route accepted a missing reference") - -for endpoint in ("https://inference.local/v1", "http://localhost:8000/v1"): - write_owned(${JSON.stringify(routeFile)}, endpoint) - assert config._get_provider_kwargs("openai")["api_key"] == "nemoclaw-managed-inference" -print("native-compatible-constructor-placeholder-ok") -`; - const output = execFileSync("python3", ["-c", validation], { - env: { PATH: process.env.PATH, PYTHONPATH: tempDir }, - encoding: "utf8", - }); - expect(output).toContain("native-compatible-constructor-placeholder-ok"); - }); - it("uses only the native OpenRouter resolver handle for Deep Code requests (#13045)", () => { const tempDir = createPackageFixture(); patchFixture(tempDir); diff --git a/test/agents/deepagents/langchain-deepagents-code-progressive-tool-disclosure.test.ts b/test/agents/deepagents/langchain-deepagents-code-progressive-tool-disclosure.test.ts index d09bab8aa52..260aedcee3c 100644 --- a/test/agents/deepagents/langchain-deepagents-code-progressive-tool-disclosure.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-progressive-tool-disclosure.test.ts @@ -122,6 +122,13 @@ _MAX_FETCH_REDIRECTS = 5 class _UrlValidationError(ValueError): pass +# Package validation requires these upstream signatures; disclosure tests never fetch URLs. +def _is_blocked_ip(ip): + raise AssertionError("disclosure fixture must not classify network destinations") + +def _pinned_dns(hostname, allowed_ips): + raise AssertionError("disclosure fixture must not open network connections") + def _fetch_with_redirects(url, *, timeout): return url, timeout `, @@ -879,7 +886,7 @@ describe("Deep Agents 0.1.87 progressive-disclosure build patch", () => { }); it("fails closed on the pinned package version before changing source", () => { - const fixture = makePatchFixture("0.1.31"); + const fixture = makePatchFixture("0.1.51"); const before = snapshot(fixture.sourcePaths); const result = runPatcher(fixture); diff --git a/test/agents/deepagents/langchain-deepagents-code-proxy-launcher.test.ts b/test/agents/deepagents/langchain-deepagents-code-proxy-launcher.test.ts index ff56137b5f7..28e356ceda8 100644 --- a/test/agents/deepagents/langchain-deepagents-code-proxy-launcher.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-proxy-launcher.test.ts @@ -8,25 +8,14 @@ import path from "node:path"; import { promisify } from "node:util"; import { describe, expect, it, vi } from "vitest"; -import { isValidProxyHost, isValidProxyPort } from "../../../src/lib/onboard/dockerfile-patch.ts"; import { TRUSTED_FETCH_PROXY_ENV_NAME } from "../../helpers/langchain-deepagents-code-headless.ts"; import { - DEFAULT_MANAGED_PROXY, dcodeStateDir, - type ManagedProxyEndpoint, makeStartScriptFixture, - prepareManagedProxyFixture, + prepareManagedScriptFixture, } from "../../support/dcode-start-script-fixture.ts"; const agentDir = path.join(process.cwd(), "agents", "langchain-deepagents-code"); -const headlessCheckPath = path.join( - process.cwd(), - "test", - "e2e", - "e2e-cloud-experimental", - "checks", - "07-deepagents-code-headless-inference.sh", -); const PROXY_URL_ENV_NAMES = ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"] as const; const NO_PROXY_ENV_NAMES = ["NO_PROXY", "no_proxy"] as const; const CLEARED_PROXY_ENV_NAMES = ["ALL_PROXY", "all_proxy", "OPENAI_PROXY"] as const; @@ -44,25 +33,17 @@ function readAgentFile(name: string): string { return fs.readFileSync(path.join(agentDir, name), "utf8"); } -function makeLauncherFixtureSource( - source: string, - tempDir: string, - managedProxy: ManagedProxyEndpoint = DEFAULT_MANAGED_PROXY, -): string { - return prepareManagedProxyFixture( +function makeLauncherFixtureSource(source: string, tempDir: string): string { + return prepareManagedScriptFixture( source.replace( 'exec /opt/venv/bin/python3 -I "$MANAGED_SESSION_SUPERVISOR" "$MANAGED_DCODE_WRAPPER" "$@"', 'exec "$MANAGED_DCODE_WRAPPER" "$@"', ), tempDir, - { managedProxy }, ); } -function makeLauncherProxyProbeFixture( - tempDir: string, - managedProxy: ManagedProxyEndpoint = DEFAULT_MANAGED_PROXY, -): string { +function makeLauncherProxyProbeFixture(tempDir: string): string { const launcherPath = path.join(tempDir, "dcode-launcher.sh"); const probePath = path.join(tempDir, "managed-dcode-probe.sh"); const markerFile = observabilityMarkerPath(tempDir); @@ -84,7 +65,6 @@ function makeLauncherProxyProbeFixture( `readonly MANAGED_OBSERVABILITY_MARKER="${markerFile}"`, ), tempDir, - managedProxy, ); fs.mkdirSync(path.dirname(markerFile), { recursive: true }); fs.writeFileSync(probePath, probe, "utf8"); @@ -94,15 +74,16 @@ function makeLauncherProxyProbeFixture( return launcherPath; } -function makeStartProxyProbeFixture( - tempDir: string, - managedProxy: ManagedProxyEndpoint = DEFAULT_MANAGED_PROXY, -): { envFile: string; ephemeralDir: string; markerFile: string; scriptPath: string } { +function makeStartProxyProbeFixture(tempDir: string): { + envFile: string; + ephemeralDir: string; + markerFile: string; + scriptPath: string; +} { const ephemeralDir = path.join(tempDir, "ephemeral-tmp"); const markerFile = observabilityMarkerPath(tempDir); const { envFile, scriptPath } = makeStartScriptFixture(tempDir, { envDir: ephemeralDir, - managedProxy, markerDir: path.dirname(markerFile), }); return { envFile, ephemeralDir, markerFile, scriptPath }; @@ -139,30 +120,14 @@ function runLauncher( }); } -async function shellValidatorAccepts( - source: string, - name: string, - value: string, -): Promise { - const match = source.match(new RegExp(`${name}\\(\\) \\{[\\s\\S]*?\\n\\}`)); - expect(match, `${name} must exist`).not.toBeNull(); - const definition = match?.[0] ?? ""; - return ( - (await runCommand("bash", ["-c", `${definition}\n${name} "$1"`, "bash", value])).status === 0 - ); -} - describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { - it("uses the live OpenShell CA bundle before the pre-resume fallback (#9360)", async () => { + it("exports OpenShell projected trust for startup and sourced shells", async () => { const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-live-ca-")); try { - const liveCaFile = path.join(tempDir, "openshell-live-ca.pem"); - const fallbackCaFile = path.join(tempDir, "managed-startup-ca.pem"); - fs.writeFileSync(liveCaFile, "live OpenShell CA\n", { mode: 0o444 }); - fs.writeFileSync(fallbackCaFile, "pre-resume fallback CA\n", { mode: 0o444 }); + const caFile = path.join(tempDir, "openshell-ca-bundle.crt"); + fs.writeFileSync(caFile, "OpenShell mediation CA and system roots\n", { mode: 0o444 }); const { envFile, scriptPath } = makeStartScriptFixture(tempDir, { - liveCaFile, - fallbackCaFile, + caFile, }); const liveResult = await runCommand("bash", [scriptPath, "true"], { @@ -175,21 +140,20 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { }); expect(liveResult.status, liveResult.stderr).toBe(0); const liveEnvironment = fs.readFileSync(envFile, "utf8"); - expect(liveEnvironment).toContain(`_nemoclaw_dcode_ca_bundle=${liveCaFile}`); + expect(liveEnvironment).toContain(`_nemoclaw_dcode_ca_bundle=${caFile}`); expect(liveEnvironment).toContain("export SSL_CERT_FILE=/ambient-live-ca.pem"); expect(liveEnvironment).toContain("export REQUESTS_CA_BUNDLE=/ambient-live-ca.pem"); expect(liveEnvironment).toContain("export NODE_EXTRA_CA_CERTS=/ambient-live-ca.pem"); - fs.rmSync(liveCaFile); - const fallbackResult = await runCommand("bash", [scriptPath, "true"], { + const projectedResult = await runCommand("bash", [scriptPath, "true"], { env: { PATH: DEFAULT_TEST_PATH }, }); - expect(fallbackResult.status, fallbackResult.stderr).toBe(0); - const fallbackEnvironment = fs.readFileSync(envFile, "utf8"); - expect(fallbackEnvironment).toContain(`_nemoclaw_dcode_ca_bundle=${fallbackCaFile}`); - expect(fallbackEnvironment).toContain(`export SSL_CERT_FILE=${fallbackCaFile}`); - expect(fallbackEnvironment).toContain(`export REQUESTS_CA_BUNDLE=${fallbackCaFile}`); - expect(fallbackEnvironment).toContain(`export NODE_EXTRA_CA_CERTS=${fallbackCaFile}`); + expect(projectedResult.status, projectedResult.stderr).toBe(0); + const projectedEnvironment = fs.readFileSync(envFile, "utf8"); + expect(projectedEnvironment).toContain(`_nemoclaw_dcode_ca_bundle=${caFile}`); + expect(projectedEnvironment).toContain(`export SSL_CERT_FILE=${caFile}`); + expect(projectedEnvironment).toContain(`export REQUESTS_CA_BUNDLE=${caFile}`); + expect(projectedEnvironment).toContain(`export NODE_EXTRA_CA_CERTS=${caFile}`); } finally { fs.rmSync(tempDir, { recursive: true, force: true }); } @@ -276,12 +240,9 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { expect(fs.existsSync(bashEnvMarker)).toBe(false); }); - it("normalizes proxy state for direct dcode launcher execution (#6191)", async () => { + it("clears ambient proxies before direct dcode execution", async () => { const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-direct-proxy-")); - const launcherPath = makeLauncherProxyProbeFixture(tempDir, { - host: "managed-proxy.internal", - port: "65535", - }); + const launcherPath = makeLauncherProxyProbeFixture(tempDir); const result = await runLauncher(launcherPath, ["-n", "PONG"], { HTTP_PROXY: "http://corp-user:corp-password@corp-proxy.example:8080", HTTPS_PROXY: "http://corp-user:corp-password@corp-proxy.example:8080", @@ -296,8 +257,8 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { expect(result.status, result.stderr).toBe(0); const lines = result.stdout.trimEnd().split("\n"); - const managedProxy = "http://managed-proxy.internal:65535"; - const managedNoProxy = "localhost,127.0.0.1,::1,managed-proxy.internal"; + const managedProxy = "__unset__"; + const managedNoProxy = "__unset__"; expect( PROXY_URL_ENV_NAMES.every((name) => lines.includes(`LAUNCHER_${name}=${managedProxy}`)), ).toBe(true); @@ -426,35 +387,10 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { expect(symlinkedStart.stderr).toContain("Unsafe managed observability marker target"); }); - it("pins validated proxy overrides into direct dcode execution paths (#6191)", () => { - const dockerfile = readAgentFile("Dockerfile"); - const launcher = readAgentFile("dcode-launcher.sh"); - - expect(dockerfile).toContain("ARG NEMOCLAW_PROXY_HOST=10.200.0.1"); - expect(dockerfile).toContain("ARG NEMOCLAW_PROXY_PORT=3128"); - expect(dockerfile).toContain("printf '%s\\n' \"$NEMOCLAW_PROXY_HOST\""); - expect(dockerfile).toContain("printf '%s\\n' \"$NEMOCLAW_PROXY_PORT\""); - expect(dockerfile).toContain("chmod 0444 /usr/local/share/nemoclaw/dcode-proxy-host"); - expect(dockerfile).toContain("chown root:root /usr/local/share/nemoclaw/dcode-proxy-host"); - expect(dockerfile).not.toContain(" NEMOCLAW_PROXY_HOST=${NEMOCLAW_PROXY_HOST}"); - expect(dockerfile).not.toContain(" NEMOCLAW_PROXY_PORT=${NEMOCLAW_PROXY_PORT}"); - expect(launcher).toContain('readonly MANAGED_PROXY_HOST_FILE="/usr/local/share/nemoclaw'); - expect(launcher).toContain("Runtime env is untrusted and cannot override"); - expect(launcher).toContain('"${MANAGED_PROXY_OWNER_UID}:444"'); - expect(launcher).toContain( - 'export PATH="/usr/local/bin:/opt/venv/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin"', - ); - expect(launcher).toContain('export HTTPS_PROXY="$_PROXY_URL"'); - expect(launcher).toContain('export DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL="$_PROXY_URL"'); - expect(launcher).toContain('export no_proxy="$_NO_PROXY_VAL"'); - expect(launcher).toContain("unset ALL_PROXY all_proxy OPENAI_PROXY"); - }); - - it("does not let runtime config override the image-baked dcode proxy (#6191)", async () => { + it("clears ambient proxies in startup, sourced shells, and direct execution", async () => { const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-trusted-proxy-")); - const trustedProxy = { host: "trusted-proxy.internal", port: "3129" }; - const launcherPath = makeLauncherProxyProbeFixture(tempDir, trustedProxy); - const { envFile, scriptPath } = makeStartProxyProbeFixture(tempDir, trustedProxy); + const launcherPath = makeLauncherProxyProbeFixture(tempDir); + const { envFile, scriptPath } = makeStartProxyProbeFixture(tempDir); const untrustedEnv = { HTTP_PROXY: "http://corp-user:corp-password@corp-proxy.example:8080", NO_PROXY: "corp.internal,inference.local", @@ -472,7 +408,7 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { scriptPath, "bash", "-c", - 'printf \'START_PROXY=%s|%s|%s|%s|%s|%s\\n\' "$HTTPS_PROXY" "$NO_PROXY" "${NEMOCLAW_PROXY_HOST-__unset__}" "${NEMOCLAW_PROXY_PORT-__unset__}" "${ALL_PROXY-__unset__}" "${all_proxy-__unset__}"', + 'printf \'START_PROXY=%s|%s|%s|%s|%s|%s\\n\' "${HTTPS_PROXY-__unset__}" "${NO_PROXY-__unset__}" "${NEMOCLAW_PROXY_HOST-__unset__}" "${NEMOCLAW_PROXY_PORT-__unset__}" "${ALL_PROXY-__unset__}" "${all_proxy-__unset__}"', ], { env: { PATH: DEFAULT_TEST_PATH, ...untrustedEnv }, @@ -490,28 +426,17 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { const startNoProxy = startResult.stdout.match(/^START_PROXY=[^|]*\|([^|]*)\|/m)?.[1]; expect(fs.statSync(envFile).mode & 0o777).toBe(0o444); expect(startResult.stdout).toContain( - "START_PROXY=http://trusted-proxy.internal:3129|localhost,127.0.0.1,::1,trusted-proxy.internal|__unset__|__unset__|__unset__|__unset__", + "START_PROXY=__unset__|__unset__|__unset__|__unset__|__unset__|__unset__", ); - expect(envFileText).toContain("export HTTPS_PROXY=http://trusted-proxy.internal:3129"); expect(envFileText).toContain( - "export DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL=http://trusted-proxy.internal:3129", + "unset HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy", ); expect(launcherResult.stdout).toContain( - "LAUNCHER_DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL=http://trusted-proxy.internal:3129", + "LAUNCHER_DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL=__unset__", ); - expect(envFileText).toContain( - "export NO_PROXY=localhost\\,127.0.0.1\\,::1\\,trusted-proxy.internal", - ); - expect(envFileText).toContain("unset ALL_PROXY all_proxy OPENAI_PROXY"); - expect(envFileText).not.toMatch(/^export (?:ALL_PROXY|all_proxy|OPENAI_PROXY)=/m); - // The two standalone shell boundaries construct the same exclusion list. - // TypeScript does not reconstruct NO_PROXY; its connect probe deliberately - // sources this persisted value from /tmp/nemoclaw-proxy-env.sh. - expect(launcherNoProxy).toBe("localhost,127.0.0.1,::1,trusted-proxy.internal"); + expect(launcherNoProxy).toBe("__unset__"); expect(startNoProxy).toBe(launcherNoProxy); const combined = `${launcherResult.stdout}\n${launcherResult.stderr}\n${startResult.stdout}\n${startResult.stderr}\n${envFileText}`; - expect(combined).toContain("http://trusted-proxy.internal:3129"); - expect(combined).toContain("localhost,127.0.0.1,::1,trusted-proxy.internal"); expect(combined).not.toContain("attacker-proxy.internal"); expect(combined).not.toContain("corp-proxy.example"); expect(combined).not.toContain("corp-password"); @@ -519,56 +444,12 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { expect(combined).not.toContain("all-password"); }); - it.each(["trusted-proxy-host", "trusted-proxy-port"])( - "fails closed when the image-baked dcode proxy contract is missing [case %#] (#6191)", - async (missingFile) => { - const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-missing-proxy-")); - const launcherPath = makeLauncherProxyProbeFixture(tempDir); - const { scriptPath } = makeStartProxyProbeFixture(tempDir); - fs.unlinkSync(path.join(tempDir, missingFile)); - const launcherResult = await runLauncher(launcherPath, ["-n", "PONG"], { - NEMOCLAW_PROXY_HOST: "attacker-proxy.internal", - NEMOCLAW_PROXY_PORT: "4444", - }); - const startResult = await runCommand("bash", [scriptPath, "true"], { - env: { - PATH: DEFAULT_TEST_PATH, - NEMOCLAW_PROXY_HOST: "attacker-proxy.internal", - NEMOCLAW_PROXY_PORT: "4444", - }, - }); - - expect(launcherResult.status).not.toBe(0); - expect(startResult.status).not.toBe(0); - const combined = `${launcherResult.stdout}\n${launcherResult.stderr}\n${startResult.stdout}\n${startResult.stderr}`; - expect(combined).toContain("trusted managed proxy"); - expect(combined).not.toContain("attacker-proxy.internal"); - }, - ); - - it("rejects writable image-baked dcode proxy files (#6191)", async () => { - const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-proxy-mode-")); - const launcherPath = makeLauncherProxyProbeFixture(tempDir); - const { scriptPath } = makeStartProxyProbeFixture(tempDir); - fs.chmodSync(path.join(tempDir, "trusted-proxy-host"), 0o644); - const launcherResult = await runLauncher(launcherPath, ["-n", "PONG"], {}); - const startResult = await runCommand("bash", [scriptPath, "true"], { - env: { PATH: DEFAULT_TEST_PATH }, - }); - - expect(launcherResult.status).not.toBe(0); - expect(startResult.status).not.toBe(0); - expect(`${launcherResult.stderr}\n${startResult.stderr}`).toContain( - "Unsafe ownership or mode on trusted managed proxy host file", - ); - }); - const expectManagedCaBundleRejection = async ({ expected, mutate, }: { expected: string; - mutate: (caFile: string) => void; + mutate: (caFile: string, scripts: string[]) => void; }): Promise => { const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-ca-bundle-")); const launcherPath = makeLauncherProxyProbeFixture(tempDir); @@ -581,13 +462,14 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { expect(safeStart.status, safeStart.stderr).toBe(0); expect(fs.existsSync(envFile)).toBe(true); - mutate(caFile); - const launcherResult = await runLauncher(launcherPath, ["-n", "PONG"], {}); + mutate(caFile, [launcherPath, scriptPath, envFile]); + const env = { PATH: DEFAULT_TEST_PATH }; + const launcherResult = await runLauncher(launcherPath, ["-n", "PONG"], env); const startResult = await runCommand("bash", [scriptPath, "true"], { - env: { PATH: process.env.PATH ?? "/usr/bin:/bin" }, + env, }); const connectSourceResult = await runCommand("sh", ["-c", '. "$1"', "sh", envFile], { - env: { PATH: process.env.PATH ?? "/usr/bin:/bin" }, + env, }); expect(launcherResult.status).not.toBe(0); @@ -601,6 +483,36 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { }; it.each([ + { + condition: "foreign-owned", + expected: "Unsafe ownership or mode on managed fetch CA bundle file", + mutate: (caFile: string, scripts: string[]) => { + const statPath = path.join(path.dirname(caFile), "stat"); + fs.writeFileSync( + statPath, + `#!/bin/sh\nprintf '%s\\n' '${(process.getuid?.() ?? 0) + 1}:444:18'\n`, + { mode: 0o755 }, + ); + const replaceStatObservation = (script: string): void => { + const source = fs.readFileSync(script, "utf8"); + fs.chmodSync(script, 0o600); + fs.writeFileSync(script, source.replaceAll("stat -", `${JSON.stringify(statPath)} -`)); + }; + replaceStatObservation(scripts[0]); + replaceStatObservation(scripts[1]); + replaceStatObservation(scripts[2]); + }, + }, + { + condition: "owner-writable", + expected: "Unsafe ownership or mode on managed fetch CA bundle file", + mutate: (caFile: string) => fs.chmodSync(caFile, 0o644), + }, + { + condition: "executable", + expected: "Unsafe ownership or mode on managed fetch CA bundle file", + mutate: (caFile: string) => fs.chmodSync(caFile, 0o544), + }, { condition: "writable", expected: "Unsafe ownership or mode on managed fetch CA bundle file", @@ -656,92 +568,4 @@ describe.concurrent("Deep Agents Code direct-exec proxy launcher", () => { }); }, ); - - it.each([ - "10.200.0.1", - "managed-proxy.internal", - "proxy_name", - "http://proxy.internal", - "user:password@proxy.internal", - "proxy.internal/path", - "proxy internal", - "proxy.internal\ninjected", - "", - ])("keeps dcode shell proxy host validation aligned [%s] (#6191)", async (value) => { - const start = readAgentFile("start.sh"); - const launcher = readAgentFile("dcode-launcher.sh"); - const expected = isValidProxyHost(value); - expect(await shellValidatorAccepts(start, "is_valid_proxy_host", value), value).toBe(expected); - expect(await shellValidatorAccepts(launcher, "is_valid_proxy_host", value), value).toBe( - expected, - ); - }); - - it.each(["1", "3128", "65535", "00001", "0", "65536", "000001", "12a", ""])( - "keeps dcode shell proxy port validation aligned [%s] (#6191)", - async (value) => { - const start = readAgentFile("start.sh"); - const launcher = readAgentFile("dcode-launcher.sh"); - const expected = isValidProxyPort(value); - expect(await shellValidatorAccepts(start, "is_valid_proxy_port", value), value).toBe( - expected, - ); - expect(await shellValidatorAccepts(launcher, "is_valid_proxy_port", value), value).toBe( - expected, - ); - }, - ); - - it.each([ - "# Invalid state:", - "# Source boundary:", - "# Source-fix constraint:", - "# Regression:", - "# Removal condition:", - ])("documents the proxy-only source boundary and removal condition [%s] (#6191)", (marker) => { - const start = readAgentFile("start.sh"); - const launcher = readAgentFile("dcode-launcher.sh"); - const headlessCheck = fs.readFileSync(headlessCheckPath, "utf8"); - - expect(start).toContain(marker); - - expect(start).toContain("Direct DNS/hosts resolution is not required"); - expect(launcher).toContain("Remove it only when OpenShell normalizes every sandbox exec/login"); - expect(headlessCheck).toContain("getent hosts inference.local >/dev/null 2>&1"); - expect(headlessCheck).toContain("direct inference.local DNS/hosts is absent"); - expect(headlessCheck).toContain('stat -c "%u:%a"'); - expect(headlessCheck).toContain( - "direct-exec dcode -n reached ${route_contract:-unknown} inference", - ); - expect(headlessCheck).toContain( - "connect --probe-only accepted the ${route_contract:-unknown} inference route", - ); - }); - - it.each([ - { host: "corp-user:corp-password@proxy.example", port: "3128" }, - { host: "proxy.example/path", port: "3128" }, - { host: "10.200.0.1", port: "0" }, - { host: "10.200.0.1", port: "65536" }, - ])( - "rejects unsafe direct dcode proxy overrides before managed code runs [case %#] (#6191)", - async (managedProxy) => { - const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-launch-invalid-")); - const launcherPath = makeLauncherProxyProbeFixture(tempDir, managedProxy); - const { scriptPath } = makeStartProxyProbeFixture(tempDir, managedProxy); - const result = await runLauncher(launcherPath, ["-n", "PONG"], {}); - const startResult = await runCommand("bash", [scriptPath, "true"], { - env: { PATH: DEFAULT_TEST_PATH }, - }); - - expect(result.status).not.toBe(0); - expect(startResult.status).not.toBe(0); - expect(result.stdout).not.toContain("LAUNCHER_"); - expect( - Object.values(managedProxy).every( - (value) => !`${result.stdout}\n${result.stderr}\n${startResult.stderr}`.includes(value), - ), - ).toBe(true); - }, - ); }); diff --git a/test/agents/deepagents/langchain-deepagents-code-proxy-runtime-contract.test.ts b/test/agents/deepagents/langchain-deepagents-code-proxy-runtime-contract.test.ts index 0235bb6cd6b..9053b500a10 100644 --- a/test/agents/deepagents/langchain-deepagents-code-proxy-runtime-contract.test.ts +++ b/test/agents/deepagents/langchain-deepagents-code-proxy-runtime-contract.test.ts @@ -49,22 +49,16 @@ function validateLoginProxyContract( allProxy: string | null = null, ): string { const loginHome = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-login-")); - const hostFile = path.join(loginHome, "trusted-proxy-host"); - const portFile = path.join(loginHome, "trusted-proxy-port"); const runtimeEnvFile = path.join(loginHome, "proxy-env.sh"); const checkFixture = path.join(loginHome, "headless-check.sh"); - fs.writeFileSync(hostFile, "10.200.0.1\n", "utf8"); - fs.writeFileSync(portFile, "3128\n", "utf8"); - fs.chmodSync(hostFile, 0o444); - fs.chmodSync(portFile, 0o444); const runtimeEnvText = [ "export HOME=/sandbox", - `export HTTP_PROXY=${JSON.stringify(proxyUrl)}`, - `export HTTPS_PROXY=${JSON.stringify(proxyUrl)}`, - `export http_proxy=${JSON.stringify(lowerProxy)}`, - `export https_proxy=${JSON.stringify(lowerProxy)}`, - `export NO_PROXY=${JSON.stringify(noProxy)}`, - `export no_proxy=${JSON.stringify(noProxy)}`, + proxyUrl ? `export HTTP_PROXY=${JSON.stringify(proxyUrl)}` : "unset HTTP_PROXY", + proxyUrl ? `export HTTPS_PROXY=${JSON.stringify(proxyUrl)}` : "unset HTTPS_PROXY", + lowerProxy ? `export http_proxy=${JSON.stringify(lowerProxy)}` : "unset http_proxy", + lowerProxy ? `export https_proxy=${JSON.stringify(lowerProxy)}` : "unset https_proxy", + noProxy ? `export NO_PROXY=${JSON.stringify(noProxy)}` : "unset NO_PROXY", + noProxy ? `export no_proxy=${JSON.stringify(noProxy)}` : "unset no_proxy", ...(allProxy === null ? ["unset ALL_PROXY all_proxy"] : [ @@ -86,10 +80,7 @@ function validateLoginProxyContract( fs.chmodSync(runtimeEnvFile, runtimeEnvMetadata === "writable" ? 0o644 : 0o444); } let checkSource = fs.readFileSync(headlessCheckPath, "utf8"); - checkSource = mustReplace(checkSource, "/usr/local/share/nemoclaw/dcode-proxy-host", hostFile); - checkSource = mustReplace(checkSource, "/usr/local/share/nemoclaw/dcode-proxy-port", portFile); checkSource = mustReplace(checkSource, "/tmp/nemoclaw-proxy-env.sh", runtimeEnvFile); - checkSource = mustReplace(checkSource, '= "0:444"', `= "${process.getuid?.() ?? 0}:444"`); checkSource = mustReplace( checkSource, 'sandbox_uid="$(id -u sandbox)"', @@ -137,10 +128,10 @@ function validateLoginProxyContract( describe("Deep Agents Code login-shell proxy contract", () => { it.each(["symlink", "writable", "wrong-user", "wrong-owner", "root-user"] as const)( - "sources normalized proxy values and rejects runtime metadata drift [%s] (#6191)", + "rejects ambient proxies and unsafe runtime environment files [%s] (#6191)", (runtimeEnvMetadata) => { - const managedProxy = "http://10.200.0.1:3128"; - const managedNoProxy = "localhost,127.0.0.1,::1,10.200.0.1"; + const managedProxy = ""; + const managedNoProxy = ""; expect(validateLoginProxyContract(managedProxy, managedNoProxy)).toBe("pass"); expect( diff --git a/test/agents/hermes/hermes-capability-free-supervisor.test.ts b/test/agents/hermes/hermes-capability-free-supervisor.test.ts new file mode 100644 index 00000000000..c3b33104967 --- /dev/null +++ b/test/agents/hermes/hermes-capability-free-supervisor.test.ts @@ -0,0 +1,243 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; + +const guardPath = path.resolve( + import.meta.dirname, + "../../../agents/hermes/runtime-config-guard.py", +); +const harness = String.raw` +import importlib.util, json, os, pathlib, stat, sys, tempfile, types +spec = importlib.util.spec_from_file_location("guard", sys.argv[1]) +guard = importlib.util.module_from_spec(spec) +sys.modules[spec.name] = guard +spec.loader.exec_module(guard) +scenario = sys.argv[2] +with tempfile.TemporaryDirectory() as tmp: + root = pathlib.Path(tmp) + proc = root / "proc" + proc.mkdir() + guard.PROC_ROOT = str(proc) + guard.__file__ = guard.INSTALLED_RUNTIME_CONFIG_GUARD + guard.HERMES_ROOT_LIFECYCLE_MARKER = str(root / "root-marker") + guard.HERMES_STARTUP_READY_FILE = str(root / "ready-marker") + guard.pwd.getpwnam = lambda _name: types.SimpleNamespace(pw_uid=1000, pw_gid=1000) + guard.os.getppid = lambda: 20 + dev = root / "dev" + dev.mkdir() + (dev / "dri").mkdir() + sysfs = root / "drm" + sysfs.mkdir() + real_open, real_fstat, real_scandir = guard.os.open, guard.os.fstat, guard.os.scandir + real_path_flag = getattr(os, "O_PATH", None) + guard.os.O_PATH = real_path_flag or 0x200000 + sysfs_inode = sysfs.stat().st_ino + dev_inode, dri_inode = dev.stat().st_ino, (dev / "dri").stat().st_ino + def open_device(path, flags, *args, **kwargs): + if path == "/dev": + if scenario == "bootstrap-drm-parent-symlink": + raise OSError("symlink refused") + path = str(dev) + if path == "dri" and scenario == "bootstrap-drm-directory-symlink": + raise OSError("symlink refused") + if path in (str(dev), "dri"): + if not flags & guard.os.O_PATH: + raise PermissionError("Landlock denies device directory reads") + if real_path_flag is None: + flags &= ~guard.os.O_PATH + if path == "/sys/class/drm": + if scenario == "bootstrap-drm-sysfs-unreadable": + raise PermissionError("sysfs denied") + if scenario == "bootstrap-drm-sysfs-symlink": + raise OSError("symlink refused") + path = str(sysfs) + return real_open(path, flags, *args, **kwargs) + def directory_metadata(fd): + metadata = real_fstat(fd) + if metadata.st_ino in (dev_inode, dri_inode, sysfs_inode): + values = list(metadata) + values[4] = 1000 if scenario == "bootstrap-drm-directory-owner" else 0 + values[5] = 1000 if scenario == "bootstrap-drm-directory-group" else 0 + values[0] = stat.S_IFDIR | (0o777 if scenario == "bootstrap-drm-directory-writable" else 0o755) + if metadata.st_ino == sysfs_inode: + if scenario == "bootstrap-drm-sysfs-owner": values[4] = 1000 + if scenario == "bootstrap-drm-sysfs-group": values[5] = 1000 + if scenario == "bootstrap-drm-sysfs-writable": values[0] = stat.S_IFDIR | 0o777 + return os.stat_result(values) + return metadata + devices = {"card0": (44, 0), "renderD128": (992, 128)} + if scenario == "bootstrap-drm-extended-minors": + devices = {"card256": (44, 256), "renderD257": (992, 257)} + if scenario == "bootstrap-drm-too-many": + devices = {f"card{n}": (44, n) for n in range(257)} + class Devices: + def __enter__(self): + names = list(devices) + if scenario == "bootstrap-drm-unassigned-device": + names += ["card9999", "card0-HDMI-A-1"] + if scenario == "bootstrap-drm-sysfs-too-many": + names += ["connector"] * 32768 + return iter(types.SimpleNamespace(name=name) for name in names) + def __exit__(self, *_args): pass + def device_metadata(name): + if name not in devices: raise FileNotFoundError(name) + gid, minor = devices[name] + mode = stat.S_IFCHR | 0o660 + if scenario == "bootstrap-drm-node-symlink": mode = stat.S_IFLNK | 0o777 + if scenario == "bootstrap-drm-regular-file": mode = stat.S_IFREG | 0o660 + if scenario == "bootstrap-drm-world-rw": mode = stat.S_IFCHR | 0o666 + return types.SimpleNamespace( + st_mode=mode, + st_uid=1001 if scenario == "bootstrap-drm-device-owner" else 1000, + st_gid=0 if scenario == "bootstrap-drm-device-root-group" else gid, + st_rdev=os.makedev(1 if scenario == "bootstrap-drm-wrong-major" else 226, + minor + 1 if scenario == "bootstrap-drm-wrong-minor" else minor), + ) + def scan_devices(fd): + return Devices() if isinstance(fd, int) and real_fstat(fd).st_ino == sysfs_inode else real_scandir(fd) + guard.os.open, guard.os.fstat, guard.os.scandir = open_device, directory_metadata, scan_devices + fields = { + "Uid": "1000 1000 1000 1000", "Gid": "1000 1000 1000 1000", + "Groups": "", "NoNewPrivs": "1", "NSpid": "1", + **{key: "0000000000000000" for key in ("CapInh", "CapPrm", "CapEff", "CapBnd", "CapAmb")}, + } + if scenario in fields: + fields[scenario] = "1" if scenario.startswith("Cap") else { + "Uid": "1000 1000 0 1000", "Gid": "1000 1000 0 1000", + "Groups": "1000", "NoNewPrivs": "0", "NSpid": "2", + }[scenario] + if scenario in ("bootstrap-primary-group", "runtime-bootstrap-primary-group"): + fields["Groups"] = "1000" + if scenario == "bootstrap-extra-group": + fields["Groups"] = "1000 1001" + if scenario == "bootstrap-root-group": + fields["Groups"] = "0" + if scenario == "bootstrap-repeated-group": + fields["Groups"] = "1000 1000" + if "drm" in scenario: + fields["Groups"] = "44 992 1000" + if scenario == "bootstrap-drm-unexplained": + fields["Groups"] += " 1001" + if scenario == "bootstrap-drm-repeated": + fields["Groups"] += " 44" + status = "".join(f"{key}:\t{value}\n" for key, value in fields.items()) + if scenario == "duplicate-field": + status += "NoNewPrivs:\t1\n" + if scenario == "missing-field": + status = status.replace("CapAmb:\t0000000000000000\n", "") + bootstrap = b"/.openshell/channel/sandbox/bootstrap.json" + argv = b"\0".join((guard.OPENSHELL_SUPERVISOR_ARGV0, b"launch-capability-free", b"1000", b"1000", bootstrap, b"/sandbox")) + b"\0" + if scenario.startswith("bootstrap"): + argv = guard.OPENSHELL_SUPERVISOR_ARGV0 + b"\0--bootstrap\0" + bootstrap + b"\0" + if scenario.startswith("runtime-bootstrap"): + argv = b"/.openshell/runtime/openshell-sandbox\0--bootstrap\0" + bootstrap + b"\0" + if scenario == "argv": + argv += b"foreign\0" + def process(pid, parent, status, argv): + directory = proc / str(pid) + directory.mkdir() + values = ["S", str(parent)] + ["0"] * 17 + [str(100 + pid)] + (directory / "stat").write_text(f"{pid} (fixture) " + " ".join(values)) + (directory / "status").write_text(status) + (directory / "cmdline").write_bytes(argv) + process(1, 0, status, argv) + process(20, 1, "Uid:\t1000 1000 1000 1000\nNSpid:\t20\n", b"/usr/bin/bash\0/usr/local/bin/nemoclaw-start\0") + # Model Linux nondumpable proc status ownership without requiring root in unit tests. + real_stat = guard.os.stat + pid1_inode = real_stat(proc / "1").st_ino + def metadata(name, **kwargs): + if isinstance(name, str) and name.startswith(("card", "renderD")): + assert kwargs == {"dir_fd": kwargs["dir_fd"], "follow_symlinks": False} + assert real_fstat(kwargs["dir_fd"]).st_ino == dri_inode + return device_metadata(name) + observed = real_stat(name, **kwargs) + if name == "status" and os.fstat(kwargs["dir_fd"]).st_ino == pid1_inode: + values = list(observed) + values[4] = 1000 if scenario == "status-uid" else 0 + values[5] = 1000 if scenario == "status-gid" else 0 + if scenario == "status-type": + values[0] = stat.S_IFLNK | 0o777 + return os.stat_result(values) + return observed + guard.os.stat = metadata + outcomes = {} + for action, startup_owner in (("ensure-api-key", True), ("write-config", False)): + try: + guard._validate_action_readiness(action, startup_owner) + outcomes[action] = True + except guard.UnsafePathError: + outcomes[action] = False + print(json.dumps(outcomes)) + guard.os.open, guard.os.fstat, guard.os.scandir = real_open, real_fstat, real_scandir + guard.os.stat = real_stat + if real_path_flag is None: del guard.os.O_PATH + else: guard.os.O_PATH = real_path_flag +`; + +describe("Hermes capability-free supervisor authority", () => { + it.each([ + ["rootless", true], + ["bootstrap", true], + ["runtime-bootstrap", true], + ["bootstrap-primary-group", true], + ["runtime-bootstrap-primary-group", true], + ["bootstrap-drm", true], + ["runtime-bootstrap-drm", true], + ["rootless-drm", false], + ["bootstrap-drm-parent-symlink", false], + ["bootstrap-drm-directory-symlink", false], + ["bootstrap-drm-directory-owner", false], + ["bootstrap-drm-directory-group", false], + ["bootstrap-drm-directory-writable", false], + ["bootstrap-drm-node-symlink", false], + ["bootstrap-drm-regular-file", false], + ["bootstrap-drm-world-rw", false], + ["bootstrap-drm-device-owner", false], + ["bootstrap-drm-device-root-group", false], + ["bootstrap-drm-wrong-major", false], + ["bootstrap-drm-wrong-minor", false], + ["bootstrap-drm-too-many", false], + ["bootstrap-drm-extended-minors", true], + ["bootstrap-drm-unassigned-device", true], + ["bootstrap-drm-sysfs-unreadable", false], + ["bootstrap-drm-sysfs-symlink", false], + ["bootstrap-drm-sysfs-owner", false], + ["bootstrap-drm-sysfs-group", false], + ["bootstrap-drm-sysfs-writable", false], + ["bootstrap-drm-sysfs-too-many", false], + ["bootstrap-drm-unexplained", false], + ["bootstrap-drm-repeated", false], + ["bootstrap-extra-group", false], + ["bootstrap-root-group", false], + ["bootstrap-repeated-group", false], + ["argv", false], + ["status-uid", false], + ["status-gid", false], + ["status-type", false], + ["Uid", false], + ["Gid", false], + ["Groups", false], + ["NoNewPrivs", false], + ["NSpid", false], + ["CapInh", false], + ["CapPrm", false], + ["CapEff", false], + ["CapBnd", false], + ["CapAmb", false], + ["duplicate-field", false], + ["missing-field", false], + ] as const)("gates startup and host config actions for %s", (scenario, allowed) => { + const result = spawnSync("python3", ["-c", harness, guardPath, scenario], { + encoding: "utf8", + timeout: 5000, + }); + expect(result.status, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toEqual({ + "ensure-api-key": allowed, + "write-config": allowed, + }); + }); +}); diff --git a/test/agents/hermes/hermes-config-transaction-wiring.test.ts b/test/agents/hermes/hermes-config-transaction-wiring.test.ts index 34b543560e8..16ce3cdad6d 100644 --- a/test/agents/hermes/hermes-config-transaction-wiring.test.ts +++ b/test/agents/hermes/hermes-config-transaction-wiring.test.ts @@ -13,32 +13,13 @@ const SOURCE_REQUIRE_HOOK = path.join( "onboard-script-mocks.cjs", ); -describe("Hermes host config transaction wiring", () => { - const handle = "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; - it.each([ - { update: "model", expected: { model: { default: "trusted-model-v2" } } }, - { - update: "restored-native-local", - expected: { - model: { default: "trusted-model-v2", api_key: handle }, - providers: { "vllm-local": { api_key: handle } }, - custom_providers: [{ api_key: handle }], - }, - }, - { - update: "native-local", - expected: { - model: { default: "trusted-model-v2", api_key: handle }, - providers: { "vllm-local": { api_key: handle } }, - custom_providers: [{ api_key: handle }], - }, - }, - ])( - "passes the read digest and $update update to the sanitized write transaction", - ({ update, expected }) => { - const rawConfig = "model:\n default: trusted-model\n"; - const expectedDigest = createHash("sha256").update(rawConfig).digest("hex"); - const script = String.raw` +function runConfigWrite( + mode: "sealed" | "native" | "native-restored" | "public" | "missing-gateway" | "changed-target", + update = "model", +) { + const rawConfig = "model:\n default: trusted-model\n"; + const expectedDigest = createHash("sha256").update(rawConfig).digest("hex"); + const script = String.raw` const Module = require("node:module"); const path = require("node:path"); const root = process.cwd(); @@ -55,9 +36,23 @@ function installMock(filename, exports) { const rawConfig = ${JSON.stringify(rawConfig)}; let captured = null; let capturedPrivilegedExec = null; +const reads = []; +let selectionReads = 0; +console.log = () => undefined; installMock(source("runner.js"), { validateName: () => undefined, ROOT: root, shellQuote: require(source("core", "shell-quote.js")).shellQuote }); installMock(source("state", "registry.js"), { - getSandbox: () => ({ name: "alpha", agent: "hermes" }), + getSandbox: () => ({ name: "alpha", agent: "hermes", gatewayName: ${mode === "missing-gateway" ? "undefined" : JSON.stringify("selected-gateway")} }), +}); +installMock(source("state", "audit", "operational.js"), { appendAuditEntry: () => undefined }); +installMock(source("state", "mcp-lifecycle-lock.js"), { + withSandboxMutationLock: (_name, callback) => callback(), +}); +installMock(source("actions", "sandbox", "mcp-bridge-provider-inspection.js"), { + resolveSandboxConfigRuntimeSelection: () => { + if (${mode === "missing-gateway"}) throw new Error("Sandbox has no registered gateway."); + selectionReads += 1; + return { gatewayName: ${mode === "changed-target"} && selectionReads > 1 ? "changed-gateway" : "selected-gateway", workspace: "default" }; + }, }); installMock(source("agent", "defs.js"), { loadAgent: () => ({ @@ -69,9 +64,15 @@ installMock(source("agent", "defs.js"), { }, }), }); +const { buildSelectedOpenShellSubprocessEnv } = require(source("adapters", "openshell", "command-argv.js")); +process.env.OPENSHELL_GATEWAY = "ambient-gateway"; +process.env.OPENSHELL_WORKSPACE = "ambient-workspace"; +process.env.OPENSHELL_LOCAL_TLS_DIR = "/ambient/tls"; +process.env.OPENSHELL_TOKEN = "ambient-token"; installMock(source("adapters", "openshell", "client.js"), { - captureOpenshellCommand: () => ({ status: 0, output: rawConfig }), - runOpenshellCommand: () => ({ status: 0 }), + buildSelectedOpenShellSubprocessEnv, + captureOpenshellCommand: (_binary, argv) => { reads.push(argv); return { status: 0, stdout: rawConfig, output: rawConfig }; }, + runOpenshellCommand: (_binary, argv, options) => { captured = { argv, options }; return { status: 0, stdout: "updated=1\\n".replace("\\n", "\n") }; }, }); installMock(source("sandbox", "privileged-exec.js"), { capturePrivilegedSandboxCommand: (sandboxName, command, options) => { @@ -83,40 +84,137 @@ installMock(source("sandbox", "privileged-exec.js"), { }); const config = require(source("sandbox", "config.js")); +async function run() { +if (${mode === "public" || mode === "missing-gateway" || mode === "changed-target"}) { + let error = null; + try { + await config.configSet("alpha", { key: "model.max_tokens", value: "24577", acceptNewPath: true }); + } catch (failure) { error = failure.message; } + process.stdout.write(JSON.stringify({ ...captured, reads, error, privilegedExec: capturedPrivilegedExec })); + return; +} const target = config.resolveAgentConfig("alpha"); const parsed = config.readSandboxConfig("alpha", target); parsed.model.default = "trusted-model-v2"; if (${JSON.stringify(update)} !== "model") { const { applyHermesManagedRoute } = require(source("hermes-managed-route.js")); applyHermesManagedRoute(parsed, { - model: "trusted-model-v2", - upstreamProvider: "vllm-local", - baseUrl: "http://host.openshell.internal:8000/v1", - inferenceApi: "openai-completions", + model: "trusted-model-v2", upstreamProvider: "vllm-local", + baseUrl: "http://host.openshell.internal:8000/v1", inferenceApi: "openai-completions", nativeCredentialHandle: "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN", }); } -config.writeSandboxConfig("alpha", target, parsed, ${JSON.stringify(update === "restored-native-local" ? { restoreNativeLocalCredential: true } : {})}); +config.writeSandboxConfig("alpha", target, parsed, ${JSON.stringify(mode === "native" ? "selected-gateway" : mode === "native-restored" ? { gatewayName: "selected-gateway", deferReload: true, restoreNativeLocalCredential: true } : update === "restored-native-local" ? { restoreNativeLocalCredential: true } : {})}); process.stdout.write(JSON.stringify({ ...captured, privilegedExec: capturedPrivilegedExec })); +} +run().catch(error => { console.error(error); process.exitCode = 1; }); `; - const result = spawnSync(process.execPath, ["--require", SOURCE_REQUIRE_HOOK, "-e", script], { - cwd: path.join(import.meta.dirname, "../../.."), - encoding: "utf-8", - env: { ...process.env, NODE_OPTIONS: "" }, - timeout: 5000, - }); + const result = spawnSync(process.execPath, ["--require", SOURCE_REQUIRE_HOOK, "-e", script], { + cwd: path.join(import.meta.dirname, "../../.."), + encoding: "utf-8", + env: { ...process.env, NODE_OPTIONS: "" }, + timeout: 5000, + }); - expect(result.status, result.stderr).toBe(0); - const captured = JSON.parse(result.stdout) as { - argv: string[]; - options: { input: string; timeout: number; sanitizeEnvironment: boolean }; - privilegedExec: { command: string[]; sanitizeEnvironment: boolean }; - }; - const digestFlag = captured.argv.indexOf("--expected-config-sha256"); + expect(result.status, result.stderr).toBe(0); + const captured = JSON.parse(result.stdout) as { + argv: string[]; + options: { + input: string; + timeout: number; + sanitizeEnvironment: boolean; + env?: Record; + replaceEnv?: boolean; + }; + privilegedExec: { command: string[]; sanitizeEnvironment: boolean }; + reads: string[][]; + error: string | null; + }; + return { captured, expectedDigest }; +} + +describe("Hermes host config transaction wiring", () => { + it("writes public config through the sandbox owner on its registered gateway (#12603)", () => { + const { captured, expectedDigest } = runConfigWrite("public"); + expect(captured.error).toBeNull(); + expect(captured.privilegedExec).toBeNull(); + expect(captured.reads).toHaveLength(2); + const target = ["-g", "selected-gateway", "sandbox", "exec", "--name", "alpha"]; + expect([...captured.reads, captured.argv].map((argv) => argv.slice(0, 6))).toEqual([ + target, + target, + target, + ]); + expect(captured.argv.at(-1)).toBe(expectedDigest); + expect(captured.options.input).toContain("max_tokens: 24577"); + expect(captured.options).toMatchObject({ + replaceEnv: true, + env: { + OPENSHELL_GATEWAY: "selected-gateway", + OPENSHELL_WORKSPACE: "default", + }, + }); + expect(captured.options.env).not.toHaveProperty("OPENSHELL_LOCAL_TLS_DIR"); + expect(captured.options.env).not.toHaveProperty("OPENSHELL_TOKEN"); + }); + + it("rejects public config writes without a registered gateway before reading or mutating (#12603)", () => { + const { captured } = runConfigWrite("missing-gateway"); + expect(captured.error).toMatch(/registered gateway/); + expect(captured.reads).toEqual([]); + expect(captured.privilegedExec).toBeNull(); + expect(captured.argv).toBeUndefined(); + }); + it("rejects a changed runtime target before the locked read or write (#12603)", () => { + const { captured } = runConfigWrite("changed-target"); + expect(captured.error).toMatch(/runtime target changed/); + expect(captured.reads).toHaveLength(1); + expect(captured.privilegedExec).toBeNull(); + expect(captured.argv).toBeUndefined(); + }); + + it("passes the read digest and serialized update to the native transaction", () => { + const { captured, expectedDigest } = runConfigWrite("native"); + + expect(captured.argv.slice(0, 6)).toEqual([ + "-g", + "selected-gateway", + "sandbox", + "exec", + "--name", + "alpha", + ]); + expect(captured.argv.at(-1)).toBe(expectedDigest); + expect(captured.options.input).toContain("default: trusted-model-v2"); + expect(captured.privilegedExec).toBeNull(); + }); + it("routes restored credentials through the selected native runtime without reloading", () => { + const { captured, expectedDigest } = runConfigWrite("native-restored", "restored-native-local"); + expect(captured.argv.slice(0, 2)).toEqual(["-g", "selected-gateway"]); + expect(captured.argv.slice(-2)).toEqual([expectedDigest, "restore-native-local"]); + expect(captured.privilegedExec).toBeNull(); + expect(captured.argv.join(" ")).not.toContain("openshell:resolve:env:v7_"); + }); + it.each(["model", "native-local", "restored-native-local"])( + "passes the read digest and %s update to the sealed transaction", + (update) => { + const { captured, expectedDigest } = runConfigWrite("sealed", update); + const handle = "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; expect(captured.argv.includes("--restore-native-local-credential")).toBe( update === "restored-native-local", ); + expect(yaml.load(captured.options.input)).toMatchObject( + update === "model" + ? { model: { default: "trusted-model-v2" } } + : { + model: { default: "trusted-model-v2", api_key: handle }, + providers: { "vllm-local": { api_key: handle } }, + custom_providers: [{ api_key: handle }], + }, + ); + expect(captured.argv.join(" ")).not.toContain(handle); + const digestFlag = captured.argv.indexOf("--expected-config-sha256"); expect(captured.argv).toEqual( expect.arrayContaining([ "timeout", @@ -138,8 +236,6 @@ process.stdout.write(JSON.stringify({ ...captured, privilegedExec: capturedPrivi expect(digestFlag).toBeGreaterThanOrEqual(0); expect(captured.argv[digestFlag + 1]).toBe(expectedDigest); expect(captured.options.input).toContain("default: trusted-model-v2"); - expect(yaml.load(captured.options.input)).toMatchObject(expected); - expect(captured.argv.join(" ")).not.toContain(handle); expect(captured.options.timeout).toBe(150000); expect(captured.options.sanitizeEnvironment).toBe(true); expect(captured.privilegedExec.sanitizeEnvironment).toBe(true); diff --git a/test/agents/hermes/hermes-cron-restore-control.test.ts b/test/agents/hermes/hermes-cron-restore-control.test.ts index 157623da9a5..e001cf4687c 100644 --- a/test/agents/hermes/hermes-cron-restore-control.test.ts +++ b/test/agents/hermes/hermes-cron-restore-control.test.ts @@ -5,11 +5,14 @@ import assert from "node:assert/strict"; import { spawnSync } from "node:child_process"; import { chmodSync, + chownSync, existsSync, + linkSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, + readdirSync, rmSync, symlinkSync, writeFileSync, @@ -17,7 +20,10 @@ import { import os from "node:os"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { HERMES_CRON_RESTORE_DRAIN_MARKER_ROLLBACK_FAILED_CODE } from "../../../src/lib/actions/sandbox/rebuild-hermes-post-restore"; +import { + HERMES_CRON_RESTORE_DRAIN_MARKER_ROLLBACK_FAILED_CODE, + HERMES_RECOVERY_GENERATION_OBSERVER, +} from "../../../src/lib/actions/sandbox/rebuild-hermes-post-restore"; import { validateHermesCronRestoreBackup } from "../../../src/lib/state/rebuild/hermes-cron-restore-backup"; const HELPER = path.resolve("agents/hermes/cron-restore-control.py"); @@ -26,6 +32,9 @@ const RECEIPT_PREFIX = "NEMOCLAW_HERMES_CRON_RESTORE_V1:"; const CONTROL_ERROR_PREFIX = "NEMOCLAW_HERMES_CRON_RESTORE_ERROR_V1:"; const LIFECYCLE_HARNESS = String.raw` import importlib.util +import contextlib +import io +import json import os import sys import threading @@ -387,6 +396,48 @@ try: module._wait_for_release_disposition = fail_release fail_directory_sync_on(3) module.complete_replacement(41, 902, 77, 903, token) + elif scenario.startswith("owner-observation-"): + original_open = module.os.open + def deny_generation(path, *args, **kwargs): + if Path(path) == module.GATEWAY_RECOVERY_WAITING_PATH: + raise PermissionError(13, "simulated root without DAC override") + return original_open(path, *args, **kwargs) + module.os.open = deny_generation + try: + module.prepare_recovery() + raise AssertionError("root read unexpectedly succeeded") + except module.ControlError as error: + assert error.code == "gateway-generation-owner-read-required" + finally: + module.os.open = original_open + output = io.StringIO() + with contextlib.redirect_stdout(output): + observer = sys.argv[4].replace( + 'Path("/sandbox/.hermes")', "Path(" + repr(str(module.HERMES_HOME)) + ")" + ).replace( + 'Path("/tmp/nemoclaw-hermes-gateway-recovery-waiting")', + "Path(" + repr(str(module.GATEWAY_RECOVERY_WAITING_PATH)) + ")", + ) + exec(observer, {}) + assert not module.GATEWAY_RECOVERY_REQUEST_PATH.exists() + observation = output.getvalue() + def replace_generation(): + staged = module.GATEWAY_RECOVERY_WAITING_PATH.with_suffix(".new") + staged.write_text("v1 " + "c" * 64 + "\n") + os.chmod(staged, 0o600) + staged.replace(module.GATEWAY_RECOVERY_WAITING_PATH) + mutations = { + "owner-observation-valid": lambda: None, + "owner-observation-replaced": replace_generation, + "owner-observation-modified": lambda: module.GATEWAY_RECOVERY_WAITING_PATH.write_text("v1 " + "d" * 64 + "\n"), + "owner-observation-mode": lambda: os.chmod(module.GATEWAY_RECOVERY_WAITING_PATH, 0o644), + "owner-observation-missing": lambda: module.GATEWAY_RECOVERY_WAITING_PATH.unlink(), + "owner-observation-invalid-nonce": lambda: None, + } + mutations[scenario]() + observation = observation.replace('"generation":"c', '"generation":"Z') if scenario == "owner-observation-invalid-nonce" else observation + module.os.open = deny_generation + module.prepare_recovery(observation) elif scenario == "prepare-handoff-world-writable": module._write_release_recovery("a" * 32, RECOVERY_STARTED_AT_NS) handoff = module.GATEWAY_RECOVERY_REQUEST_PATH.parent @@ -594,6 +645,8 @@ describe("Hermes in-sandbox cron restore validator", () => { beforeEach(() => { root = mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-cron-helper-")); + // macOS inherits the temp parent's group; controller fixtures require the executing identity. + chownSync(root, process.getuid!(), process.getgid!()); hermesHome = path.join(root, ".hermes"); mkdirSync(hermesHome); }); @@ -648,6 +701,12 @@ describe("Hermes in-sandbox cron restore validator", () => { | "drain-unlink-sync-failure" | "recovery-unlink-sync-failure" | "rollback-publication-sync-failure" + | "owner-observation-valid" + | "owner-observation-replaced" + | "owner-observation-modified" + | "owner-observation-mode" + | "owner-observation-missing" + | "owner-observation-invalid-nonce" | "prepare-handoff-world-writable" | "prepare-handoff-symlink" | "prepare-recovery-only" @@ -671,7 +730,15 @@ describe("Hermes in-sandbox cron restore validator", () => { ) { return spawnSync( process.env.PYTHON || "python3", - ["-I", "-c", LIFECYCLE_HARNESS, HELPER, scenario, hermesHome], + [ + "-I", + "-c", + LIFECYCLE_HARNESS, + HELPER, + scenario, + hermesHome, + HERMES_RECOVERY_GENERATION_OBSERVER, + ], { encoding: "utf8" }, ); } @@ -1157,6 +1224,99 @@ describe("Hermes in-sandbox cron restore validator", () => { expect(readFileSync(requestPath, "utf8")).toBe(`v2 ${"e".repeat(64)} 321 654\n`); }); + it("reads recovery as its owner while keeping request publication with the root controller", () => { + const result = runLifecycle("owner-observation-valid"); + expect(result.status, result.stderr).toBe(0); + expect( + readFileSync(path.join(root, "recovery-run", "hermes-gateway-recovery-request"), "utf8"), + ).toBe(`v2 ${"c".repeat(64)} 321 654\n`); + expect(lstatSync(path.join(root, "run", "hermes-gateway-recovery-waiting")).mode & 0o777).toBe( + 0o600, + ); + }); + + it.skipIf(process.getuid?.() === 0)( + "observes an owned generation without reading the inaccessible cron controller", + () => { + const controller = path.join(root, "root-only-controller.py"); + const generation = path.join(root, "generation"); + writeFileSync(controller, readFileSync(HELPER), { mode: 0o000 }); + writeFileSync(generation, `v1 ${"a".repeat(64)}\n`, { mode: 0o600 }); + const before = readdirSync(root).sort(); + const observer = HERMES_RECOVERY_GENERATION_OBSERVER.replace( + 'Path("/sandbox/.hermes")', + `Path(${JSON.stringify(hermesHome)})`, + ).replace( + 'Path("/tmp/nemoclaw-hermes-gateway-recovery-waiting")', + `Path(${JSON.stringify(generation)})`, + ); + try { + // Mode 000 gives this unprivileged test UID the same lack of access + // that sandbox UID 998 has to the image's root:root 0700 controller. + const inaccessible = spawnSync( + process.env.PYTHON || "python3", + ["-I", controller, "observe-recovery-generation"], + { encoding: "utf8" }, + ); + expect(inaccessible.status).not.toBe(0); + expect(inaccessible.stderr).toContain("Permission denied"); + const observed = spawnSync(process.env.PYTHON || "python3", ["-I", "-c", observer], { + encoding: "utf8", + }); + expect(observed.status, observed.stderr).toBe(0); + expect(JSON.parse(observed.stdout)).toMatchObject({ + version: 1, + generation: "a".repeat(64), + identity: expect.arrayContaining([process.getuid!(), 68]), + }); + expect(readdirSync(root).sort()).toEqual(before); + expect(readFileSync(generation, "utf8")).toBe(`v1 ${"a".repeat(64)}\n`); + expect(lstatSync(generation).mode & 0o777).toBe(0o600); + expect(lstatSync(controller).mode & 0o777).toBe(0o000); + } finally { + chmodSync(controller, 0o700); + } + }, + ); + + it.each([ + ["public mode", (file: string) => chmodSync(file, 0o644)], + ["hard link", (file: string) => linkSync(file, `${file}.link`)], + ["symlink", (file: string) => symlinkSync(file, `${file}.symlink`)], + ["invalid nonce", (file: string) => writeFileSync(file, `v1 ${"Z".repeat(64)}\n`)], + ["oversized payload", (file: string) => writeFileSync(file, `v1 ${"a".repeat(65)}\n`)], + ] as const)("rejects an owner generation with %s", (kind, mutate) => { + const generation = path.join(root, "generation"); + writeFileSync(generation, `v1 ${"a".repeat(64)}\n`, { mode: 0o600 }); + mutate(generation); + const observedPath = kind === "symlink" ? `${generation}.symlink` : generation; + const observer = HERMES_RECOVERY_GENERATION_OBSERVER.replace( + 'Path("/sandbox/.hermes")', + `Path(${JSON.stringify(hermesHome)})`, + ).replace( + 'Path("/tmp/nemoclaw-hermes-gateway-recovery-waiting")', + `Path(${JSON.stringify(observedPath)})`, + ); + const result = spawnSync(process.env.PYTHON || "python3", ["-I", "-c", observer], { + encoding: "utf8", + }); + expect(result.status).not.toBe(0); + expect(result.stdout).toBe(""); + }); + + it.each([ + "owner-observation-replaced", + "owner-observation-modified", + "owner-observation-mode", + "owner-observation-missing", + "owner-observation-invalid-nonce", + ] as const)("refuses %s before publishing recovery authority", (scenario) => { + const result = runLifecycle(scenario); + expect(result.status).toBe(1); + expect(result.stderr).toContain("generation changed after observation"); + expect(result.stdout).toContain("RECOVERY_REQUEST:absent"); + }); + it("reports false when no gateway recovery generation appears", () => { const result = runLifecycle("prepare-without-generation"); diff --git a/test/agents/hermes/hermes-env-secret-boundary-hardening.test.ts b/test/agents/hermes/hermes-env-secret-boundary-hardening.test.ts index 4c365d38f93..4911a0f545a 100644 --- a/test/agents/hermes/hermes-env-secret-boundary-hardening.test.ts +++ b/test/agents/hermes/hermes-env-secret-boundary-hardening.test.ts @@ -354,13 +354,13 @@ except module.UnsafeEnvInputError: }); describe("Hermes env secret-boundary namespace pinning", () => { - const validateHermesRootMode = (mode: number) => { + const validateHermesRootMode = (mode: number, sandboxMode = 0o770) => { const root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-env-legacy-"))); const sandbox = path.join(root, "sandbox"); const hermes = path.join(sandbox, ".hermes"); const envPath = path.join(hermes, ".env"); fs.mkdirSync(hermes, { recursive: true }); - fs.chmodSync(sandbox, 0o770); + fs.chmodSync(sandbox, sandboxMode); fs.chmodSync(hermes, mode); fs.writeFileSync(envPath, "SAFE=1\n", { mode: 0o640 }); fs.chmodSync(envPath, 0o640); @@ -403,6 +403,19 @@ raise SystemExit(module.validate_env_file(sys.argv[3]))`, ); }); + it("accepts the sandbox-owned 0700 workspace used by rootless Podman", () => { + const result = validateHermesRootMode(0o750, 0o700); + expect(result.status, result.stderr).toBe(0); + }); + + it("reports the rejected sandbox owner, group, and mode", () => { + const result = validateHermesRootMode(0o750, 0o701); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain( + `/sandbox does not match a trusted owner/group/mode posture (uid=${String(os.userInfo().uid)}, gid=${String(os.userInfo().gid)}, mode=0701)`, + ); + }); + it("anchors installed validation at sandbox when Landlock denies opening root", () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-env-landlock-root-")); const sandbox = path.join(root, "sandbox"); diff --git a/test/agents/hermes/hermes-image-build-probes.test.ts b/test/agents/hermes/hermes-image-build-probes.test.ts index 58ca98341a7..638845d870b 100644 --- a/test/agents/hermes/hermes-image-build-probes.test.ts +++ b/test/agents/hermes/hermes-image-build-probes.test.ts @@ -593,6 +593,15 @@ describe("Hermes image build probes", () => { ), ).toEqual([digest, digest]); const normalizedDockerfile = imageDockerfile.replace(/\\\n/gu, " "); + expect(normalizedDockerfile).toContain( + "chown root:root /usr/local/lib/nemoclaw/hermes-cron-restore-control.py", + ); + expect(normalizedDockerfile).toContain( + "chmod 700 /usr/local/lib/nemoclaw/hermes-cron-restore-control.py", + ); + expect(normalizedDockerfile).toContain( + "check_metadata /usr/local/lib/nemoclaw/hermes-cron-restore-control.py 'root:root 700'", + ); expect(normalizedDockerfile).toContain("install -d -o root -g root -m 0755 /etc/nemoclaw"); expect(normalizedDockerfile).toMatch( /touch \/etc\/nemoclaw\/hermes-mcp-transaction\.lock[\s\S]*chown root:root[\s\S]*\/etc\/nemoclaw\/hermes-mcp-transaction\.lock[\s\S]*chmod 444[\s\S]*\/etc\/nemoclaw\/hermes-mcp-transaction\.lock/u, diff --git a/test/agents/hermes/hermes-mcp-config-transaction.test.ts b/test/agents/hermes/hermes-mcp-config-transaction.test.ts index 06d1cfde643..c9dbaf09962 100644 --- a/test/agents/hermes/hermes-mcp-config-transaction.test.ts +++ b/test/agents/hermes/hermes-mcp-config-transaction.test.ts @@ -165,9 +165,9 @@ if len(errors) != 3: expect(result.status, result.stderr).toBe(0); expect(JSON.parse(result.stdout)).toEqual([ - "Authenticated MCP OpenShell host aliases are unavailable with OpenShell v0.0.116", - "Authenticated MCP OpenShell host aliases are unavailable with OpenShell v0.0.116", - "Authenticated MCP OpenShell host aliases are unavailable with OpenShell v0.0.116", + "Authenticated MCP OpenShell host aliases require an attested driver gateway address", + "Authenticated MCP OpenShell host aliases require an attested driver gateway address", + "Authenticated MCP OpenShell host aliases require an attested driver gateway address", ]); }); diff --git a/test/agents/hermes/hermes-mcp-credential-boundary-manifest.test.ts b/test/agents/hermes/hermes-mcp-credential-boundary-manifest.test.ts index 02baa2be6ba..0ee4d93a493 100644 --- a/test/agents/hermes/hermes-mcp-credential-boundary-manifest.test.ts +++ b/test/agents/hermes/hermes-mcp-credential-boundary-manifest.test.ts @@ -14,9 +14,9 @@ const TRANSACTION = path.join( "hermes", "mcp-config-transaction.py", ); -const MANIFEST_NAME = "openshell-child-visible-credentials.v0.0.116.json"; +const MANIFEST_NAME = "openshell-child-visible-credentials.v0.1.7.json"; const validManifest: Record = { - openshellVersion: "0.0.116", + openshellVersion: "0.1.7", rawChildValueKeys: ["RAW_CHILD_VALUE"], rewrittenChildValueKeys: ["REWRITTEN_CHILD_VALUE"], runtimeControlKeys: ["RUNTIME_CONTROL"], @@ -68,6 +68,39 @@ function runEmbeddedTransactionImportWithManifest(manifest: Record { + it("loads the reviewed manifest from an isolated installed-helper layout", () => { + expect(runEmbeddedTransactionImportWithManifest(validManifest)).toEqual({ + loaded: true, + type: "", + message: "", + }); + }); + + it("rejects the previous manifest version at the current installed path", () => { + expect( + runEmbeddedTransactionImportWithManifest({ ...validManifest, openshellVersion: "0.1.2" }), + ).toEqual({ + loaded: false, + type: "RuntimeError", + message: "Hermes MCP credential boundary manifest is invalid", + }); + }); + + it("does not load the old manifest filename from an installed helper", () => { + expect( + runEmbeddedTransactionImport((helperDir) => { + fs.writeFileSync( + path.join(helperDir, "openshell-child-visible-credentials.v0.1.2.json"), + JSON.stringify(validManifest), + ); + }), + ).toEqual({ + loaded: false, + type: "RuntimeError", + message: "Hermes MCP credential boundary manifest is missing", + }); + }); + it("fails closed when the manifest is missing", () => { expect(runEmbeddedTransactionImport()).toEqual({ loaded: false, diff --git a/test/agents/hermes/hermes-mcp-startup-probe.test.ts b/test/agents/hermes/hermes-mcp-startup-probe.test.ts index 75a2661a4a4..e9dc6f7f57f 100644 --- a/test/agents/hermes/hermes-mcp-startup-probe.test.ts +++ b/test/agents/hermes/hermes-mcp-startup-probe.test.ts @@ -7,6 +7,7 @@ const mocks = vi.hoisted(() => ({ executeGatewaySupervisorAction: vi.fn(), runOpenshellProviderCommand: vi.fn(), sleepMs: vi.fn(), + sleepSeconds: vi.fn(), waitUntil: vi.fn(), waitUntilAsync: vi.fn(), })); @@ -26,6 +27,7 @@ vi.mock("../../../src/lib/adapters/sandbox/command-transport", async (importOrig vi.mock("../../../src/lib/core/wait", () => ({ sleepMs: mocks.sleepMs, + sleepSeconds: mocks.sleepSeconds, waitUntil: mocks.waitUntil, waitUntilAsync: mocks.waitUntilAsync, })); diff --git a/test/agents/hermes/hermes-native-config-transaction.test.ts b/test/agents/hermes/hermes-native-config-transaction.test.ts new file mode 100644 index 00000000000..e9cf4492074 --- /dev/null +++ b/test/agents/hermes/hermes-native-config-transaction.test.ts @@ -0,0 +1,301 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { HERMES_NATIVE_CONFIG_WRITE } from "../../../src/lib/sandbox/hermes-native-config"; + +const original = "model:\n default: old-model\nmcp_servers: {}\ncustom: retained\n"; +const updated = original.replace("old-model", "selected-model"); + +function transaction(scenario: string, mode = "reload", source = original, replacement = updated) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "hermes-native-config-")); + try { + const home = path.join(root, "home"); + fs.mkdirSync(home, { mode: 0o700 }); + fs.writeFileSync(path.join(home, "config.yaml"), source, { mode: 0o600 }); + fs.writeFileSync(path.join(home, ".env"), "SAFE_SETTING=true\n", { mode: 0o600 }); + const hash = (text: string) => createHash("sha256").update(text).digest("hex"); + const hashes = `${hash(source)} ${home}/config.yaml\n${hash("SAFE_SETTING=true\n")} ${home}/.env\n`; + fs.writeFileSync(path.join(home, ".config-hash"), hashes, { mode: 0o600 }); + const strictHashPath = path.join(root, "strict-hash"); + fs.writeFileSync(strictHashPath, hashes, { mode: 0o444 }); + const lockDir = path.join(root, "lock"); + fs.mkdirSync(lockDir, { mode: 0o755 }); + fs.writeFileSync(path.join(lockDir, "transaction"), "", { mode: 0o444 }); + fs.chmodSync(lockDir, 0o755); + fs.chmodSync(path.join(lockDir, "transaction"), 0o444); + const shim = path.join(root, "helper.py"); + // Exercise the real filesystem guards and transaction lock. Runtime identity + // and the installed Hermes schema are exercised by the Brev reproduction. + fs.writeFileSync( + shim, + ` +import importlib.util, os, sys, types +from pathlib import Path + +def load(name, filename): + spec = importlib.util.spec_from_file_location(name, filename) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + +guard = load("native_guard", ${JSON.stringify(path.resolve("agents/hermes/runtime-config-guard.py"))}) +helper = load("native_helper", ${JSON.stringify(path.resolve("agents/hermes/mcp-config-transaction.py"))}) +helper.HERMES_DIR = ${JSON.stringify(home)} +helper.CONFIG_PATH = helper.HERMES_DIR + "/config.yaml" +helper.STRICT_HASH_PATH = ${JSON.stringify(strictHashPath)} +helper.MCP_TRANSACTION_LOCK_PATH = ${JSON.stringify(path.join(lockDir, "transaction"))} +helper.MCP_TRANSACTION_LOCK_EXPECTED_UID = os.geteuid() +helper.MCP_TRANSACTION_LOCK_EXPECTED_GID = os.stat(helper.MCP_TRANSACTION_LOCK_PATH).st_gid +helper._load_guard = lambda: guard +guard._sandbox_identity = lambda: (os.geteuid(), os.getegid()) +helper._assert_non_root_lifecycle_identity = lambda: None +helper._configure_gateway_public_port = lambda: None +def reload_gateway(): + with open(helper.HERMES_DIR + "/reloads", "a") as output: output.write("reload\\n") + return True +helper.reload_gateway = reload_gateway +schema = types.ModuleType("gateway.config") +schema.GatewayConfig = type("GatewayConfig", (), {"from_dict": staticmethod(lambda data: data)}) +sys.modules["gateway.config"] = schema +scenario = ${JSON.stringify(scenario)} +if scenario == "lifecycle": + def reject(): raise PermissionError("wrong lifecycle") + helper._assert_non_root_lifecycle_identity = reject +if scenario == "reload-failure": + helper.reload_gateway = lambda: False +if scenario == "stale-archive-hash": + Path(helper.HERMES_DIR + "/.config-hash").write_text("0" * 64 + " " + helper.CONFIG_PATH + "\\n" + "0" * 64 + " " + helper.HERMES_DIR + "/.env\\n") +if scenario == "unsafe-mode": + os.chmod(helper.CONFIG_PATH, 0o644) +if scenario == "raw-secret": + Path(helper.HERMES_DIR + "/.env").write_text("NVIDIA_API_KEY=raw-fixture-secret\\n") +if scenario == "hash-drift": + Path(helper.HERMES_DIR + "/.env").write_text("changed=true\\n") +if scenario == "symlink": + current = Path(helper.CONFIG_PATH) + current.rename(current.with_suffix(".original")) + current.symlink_to(current.with_suffix(".original")) +if scenario == "hardlink": + os.link(helper.CONFIG_PATH, helper.CONFIG_PATH + ".link") +if scenario == "hash-failure": + def fail_hash(*args): raise RuntimeError("injected hash failure") + helper._refresh_and_verify_hashes = fail_hash +if scenario in ("late-environment-race", "final-environment-race", "final-config-race"): + refresh = helper._refresh_and_verify_hashes + refresh_count = 0 + def raced_refresh(*args): + global refresh_count + refresh_count += 1 + trigger = 1 if scenario == "late-environment-race" else 2 + if refresh_count == trigger: + if scenario == "final-config-race": + Path(helper.CONFIG_PATH).write_text("model: concurrent-writer\\n") + else: + Path(helper.HERMES_DIR + "/.env").write_text("NVIDIA_API_KEY=synthetic-race-value\\n") + return refresh(*args) + helper._refresh_and_verify_hashes = raced_refresh +if scenario == "environment-race": + write = guard._write_existing + def raced_environment(filename, text, snapshot, mode=None): + write(filename, text, snapshot, mode) + if filename == helper.CONFIG_PATH: + Path(helper.HERMES_DIR + "/.env").write_text("CHANGED=true\\n") + guard._write_existing = raced_environment +if scenario == "concurrent-write": + write = guard._write_existing + def raced_write(filename, text, snapshot, mode=None): + write(filename, text, snapshot, mode) + if filename == helper.CONFIG_PATH: + Path(filename).write_text("model: concurrent-writer\\n") + guard._write_existing = raced_write +for name, value in vars(helper).items(): + if not name.startswith("__"): globals()[name] = value +`, + ); + const candidate = + scenario === "mcp-change" + ? replacement.replace( + "mcp_servers: {}", + "mcp_servers: {other: {url: 'https://example.test/mcp'}}", + ) + : replacement; + const script = HERMES_NATIVE_CONFIG_WRITE.replace( + "except Exception:\n # Config parser", + "except Exception:\n import traceback; traceback.print_exc()\n # Config parser", + ).replace("/usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py", shim); + const result = spawnSync( + "python3", + ["-c", script, scenario === "stale" ? "0".repeat(64) : hash(source), mode], + { + input: candidate, + encoding: "utf8", + timeout: 10_000, + }, + ); + return { + result, + config: fs.readFileSync(path.join(home, "config.yaml"), "utf8"), + hash: fs.readFileSync(path.join(home, ".config-hash"), "utf8"), + originalHash: hashes, + strictHash: fs.readFileSync(strictHashPath, "utf8"), + updatedHash: hash(replacement), + reloads: fs.existsSync(path.join(home, "reloads")) + ? fs.readFileSync(path.join(home, "reloads"), "utf8") + : "", + }; + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +} + +describe("native Hermes config transaction", () => { + it("updates the selected model and compatibility hash while preserving unrelated configuration", () => { + const actual = transaction("success"); + expect(actual.result.status, actual.result.stderr).toBe(0); + expect(actual.config).toBe(updated); + expect(actual).toMatchObject({ + hash: expect.stringContaining(actual.updatedHash), + strictHash: actual.originalHash, + }); + }); + + it.each([ + "stale", + "lifecycle", + "symlink", + "hardlink", + "mcp-change", + "hash-failure", + "reload-failure", + ])("preserves configuration and its hash after %s rejection", (scenario) => { + const actual = transaction(scenario); + expect(actual.result.status).not.toBe(0); + expect(actual.config).toBe(original); + expect(actual.hash).toBe(actual.originalHash); + }); + + it("refuses mismatched environment integrity before writing configuration", () => { + const actual = transaction("hash-drift"); + expect(actual.result.status).not.toBe(0); + expect(actual.config).toBe(original); + }); + + it("retains a concurrent writer's configuration instead of rolling it back", () => { + const actual = transaction("concurrent-write"); + expect(actual.result.status).not.toBe(0); + expect(actual.config).toBe("model: concurrent-writer\n"); + }); +}); + +const restoredLocal = `model: + default: local-model + provider: custom + base_url: http://host.openshell.internal:8000/v1 + api_key: openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN +_nemoclaw_upstream: + provider: vllm-local + provider_key: vllm-local + model: local-model +providers: + vllm-local: + default_model: local-model + api: http://host.openshell.internal:8000/v1 + api_key: openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN +custom_providers: + - name: vllm-local + base_url: http://host.openshell.internal:8000/v1 + api_key: openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN +mcp_servers: {} +custom: retained +`; +const reboundLocal = restoredLocal.replaceAll(":v1_", ":v2_"); + +describe("native Hermes deferred configuration", () => { + it("commits initial configuration without reloading before the restart owner", () => { + const actual = transaction("success", "deferred"); + expect(actual.result.status, actual.result.stderr).toBe(0); + expect(actual.config).toBe(updated); + expect(actual.reloads).toBe(""); + }); + + it.each(["deferred", "restore-native-local"])( + "rejects environment changes during %s writes", + (mode) => { + const actual = transaction("environment-race", mode, restoredLocal, reboundLocal); + expect(actual.result.status).not.toBe(0); + expect(actual.config).toBe(restoredLocal); + expect(actual.hash).toBe(actual.originalHash); + expect(actual.reloads).toBe(""); + }, + ); + + it.each(["late-environment-race", "final-environment-race"])( + "refuses %s during restored hash refresh without accepting changed credentials", + (scenario) => { + const actual = transaction(scenario, "restore-native-local", restoredLocal, reboundLocal); + expect(actual.result.status).not.toBe(0); + expect(actual.config).toBe(restoredLocal); + expect(actual.hash).toBe(actual.originalHash); + expect(actual.reloads).toBe(""); + }, + ); + + it("preserves a concurrent configuration writer during final hash refresh", () => { + const actual = transaction( + "final-config-race", + "restore-native-local", + restoredLocal, + reboundLocal, + ); + expect(actual.result.status).not.toBe(0); + expect(actual.config).toBe("model: concurrent-writer\n"); + expect(actual.reloads).toBe(""); + }); + + it.each(["success", "stale-archive-hash"])( + "rebinds restored credential slots with %s", + (scenario) => { + const actual = transaction(scenario, "restore-native-local", restoredLocal, reboundLocal); + expect(actual.result.status, actual.result.stderr).toBe(0); + expect(actual.config).toBe(reboundLocal); + expect(actual.hash).toContain(actual.updatedHash); + expect(actual.reloads).toBe(""); + }, + ); + + it.each([ + { + name: "model change", + scenario: "success", + candidate: reboundLocal.replaceAll("local-model", "other-model"), + }, + { + name: "MCP change", + scenario: "success", + candidate: reboundLocal.replace( + "mcp_servers: {}", + "mcp_servers: {extra: {command: untrusted}}", + ), + }, + { name: "raw environment credential", scenario: "raw-secret", candidate: reboundLocal }, + { name: "hash failure", scenario: "hash-failure", candidate: reboundLocal }, + { name: "stale read", scenario: "stale", candidate: reboundLocal }, + { name: "wrong lifecycle", scenario: "lifecycle", candidate: reboundLocal }, + { name: "symlink", scenario: "symlink", candidate: reboundLocal }, + { name: "hard link", scenario: "hardlink", candidate: reboundLocal }, + { name: "non-private restored mode", scenario: "unsafe-mode", candidate: reboundLocal }, + ])("preserves restored config after $name", ({ scenario, candidate }) => { + const actual = transaction(scenario, "restore-native-local", restoredLocal, candidate); + expect(actual.result.status).not.toBe(0); + expect(actual.config).toBe(restoredLocal); + expect(actual.hash).toBe(actual.originalHash); + expect(actual.reloads).toBe(""); + }); +}); diff --git a/test/agents/hermes/hermes-native-network.test.ts b/test/agents/hermes/hermes-native-network.test.ts new file mode 100644 index 00000000000..bf69e671d2c --- /dev/null +++ b/test/agents/hermes/hermes-native-network.test.ts @@ -0,0 +1,199 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { execFileSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { extractShellFunction } from "../../support/hermes-shell-harness"; +import { shellQuote } from "../../../src/lib/core/shell-quote"; +import { dockerfileInstructions } from "../../../src/lib/onboard/dockerfile-tool-disclosure-contract"; + +const ROOT = path.resolve(import.meta.dirname, "../../.."); +const PROXY_NAMES = [ + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "ALL_PROXY", + "all_proxy", + "OPENAI_PROXY", + "grpc_proxy", + "GRPC_PROXY", + "NODE_USE_ENV_PROXY", + "NEMOCLAW_PROXY_HOST", + "NEMOCLAW_PROXY_PORT", +]; + +describe("Hermes native network environment", () => { + it("resolves certifi to the OpenShell 0.1.7 workload CA bundle", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "hermes-certifi-")); + try { + const dockerfile = fs.readFileSync(path.join(ROOT, "agents/hermes/Dockerfile"), "utf8"); + const instructions = dockerfileInstructions(dockerfile).map(({ text }) => text); + const target = instructions + .find((line) => line.startsWith("ARG SSL_CERT_FILE="))! + .slice("ARG SSL_CERT_FILE=".length); + // OpenShell v0.1.7 installs CA material here before it starts the workload. + const bundle = path.join(directory, "run/openshell-supervisor-ca/material/ca-bundle.crt"); + fs.mkdirSync(path.dirname(bundle), { recursive: true }); + fs.writeFileSync(bundle, "supervisor CA bundle\n"); + const certifi = path.join(directory, "cacert.pem"); + fs.writeFileSync(certifi, "packaged certifi bundle\n"); + const python = path.join(directory, "python"); + fs.writeFileSync(python, `#!/bin/sh\nprintf '%s\\n' ${shellQuote(certifi)}\n`, { + mode: 0o755, + }); + const link = instructions + .find((line) => line.startsWith("RUN _hermes_certifi="))! + .slice("RUN ".length) + .replace("/opt/hermes/.venv/bin/python", shellQuote(python)); + execFileSync("sh", ["-eu", "-c", link], { + env: { ...process.env, SSL_CERT_FILE: path.join(directory, target) }, + }); + expect(fs.readFileSync(certifi, "utf8")).toBe("supervisor CA bundle\n"); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); + + it("clears startup proxy inputs and retains TLS trust", () => { + const source = fs.readFileSync(path.join(ROOT, "agents/hermes/start.sh"), "utf8"); + const start = source.indexOf("unset HTTP_PROXY HTTPS_PROXY NO_PROXY"); + const end = source.indexOf("# Corporate proxy CA merge", start); + const output = execFileSync( + "bash", + [ + "-c", + source.slice(start, end) + + PROXY_NAMES.map((name) => `printf '%s=%s\\n' '${name}' "\${${name}+set}"`).join("\n") + + '\nprintf "CA=%s\\n" "$SSL_CERT_FILE"', + ], + { + encoding: "utf8", + env: { + ...process.env, + ...Object.fromEntries(PROXY_NAMES.map((name) => [name, "obsolete-proxy-value"])), + SSL_CERT_FILE: "/trusted/ca file.pem", + }, + }, + ); + expect(output).toBe( + PROXY_NAMES.map((name) => `${name}=\n`).join("") + "CA=/trusted/ca file.pem\n", + ); + }); + + it.each(["sh", "bash"])("clears proxy inputs in %s and retains TLS trust", (mode) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "hermes-native-env-")); + try { + const source = fs.readFileSync(path.join(ROOT, "agents/hermes/start.sh"), "utf8"); + const envFile = path.join(directory, "connect.sh"); + const caFile = path.join(directory, "trusted ca.pem"); + const probe = + PROXY_NAMES.map((name) => `printf '%s=%s\\n' '${name}' "\${${name}+set}"`).join("\n") + + '\nprintf "CA=%s\\n" "$SSL_CERT_FILE"'; + const seed = { + ...process.env, + ...Object.fromEntries(PROXY_NAMES.map((name) => [name, "obsolete-proxy-value"])), + SSL_CERT_FILE: caFile, + }; + execFileSync( + "bash", + [ + "-c", + [ + 'emit_sandbox_sourced_file() { cat >"$1"; chmod 444 "$1"; }', + `_PROXY_ENV_FILE=${shellQuote(envFile)}`, + "HERMES_DIR=/sandbox/.hermes", + "HERMES_SANDBOX_LAZY_INSTALL_TARGET=/sandbox/.hermes/lazy-packages", + extractShellFunction(source, "write_runtime_shell_env"), + "write_runtime_shell_env", + ].join("\n"), + ], + { encoding: "utf8", env: seed }, + ); + const output = execFileSync(mode, ["-c", `. ${shellQuote(envFile)}\n${probe}`], { + encoding: "utf8", + env: seed, + }); + expect(fs.statSync(envFile).mode & 0o777).toBe(0o444); + expect(output).toBe(PROXY_NAMES.map((name) => `${name}=\n`).join("") + `CA=${caFile}\n`); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); +}); + +describe("Hermes Browser Use TLS connection", () => { + it.each([false, true])("uses the remote host and propagates TLS failure=%s", (rejectTls) => { + const output = execFileSync( + "python3", + [ + "-I", + "-c", + ` +import ast +import json +import pathlib +import ssl +import sys +from unittest.mock import patch + +tree = ast.parse(pathlib.Path(sys.argv[1]).read_text()) +script = next(ast.literal_eval(node.value) for node in tree.body + if isinstance(node, ast.Assign) and any(isinstance(target, ast.Name) + and target.id == "_BROWSER_USE_CDP_TUNNEL_SCRIPT" for target in node.targets)) +reject_tls = sys.argv[2] == "true" +events = [] +class Socket: + def close(self): + events.append("close") + def sendall(self, _data): + raise AssertionError("HTTP CONNECT must not be used") +raw = Socket() +def connect(address, timeout): + events.append(["connect", list(address), timeout]) + return raw +class Context: + def wrap_socket(self, sock, server_hostname): + assert sock is raw + events.append(["tls", server_hostname]) + if reject_tls: + raise ssl.SSLCertVerificationError("untrusted peer") + return "verified stream" +def context(): + events.append("default trust") + return Context() +namespace = {} +with patch.object(sys, "argv", ["bridge", "wss://session.browser-use.com:8443/cdp?capability=fixture"]), \\ + patch("socket.create_connection", connect), patch("ssl.create_default_context", context): + exec(script.split("listener = socket.socket", 1)[0], namespace) + try: + result = namespace["_connect_remote"]() + except ssl.SSLCertVerificationError: + result = "certificate rejected" +print(json.dumps({"events": events, "result": result, "path": namespace["target_path"]})) +`, + path.join(ROOT, "agents/hermes/plugin/__init__.py"), + String(rejectTls), + ], + { + encoding: "utf8", + env: { ...process.env, HTTPS_PROXY: "http://retired.invalid:3128" }, + }, + ); + expect(JSON.parse(output)).toEqual({ + events: [ + ["connect", ["session.browser-use.com", 8443], 15], + "default trust", + ["tls", "session.browser-use.com"], + ...(rejectTls ? ["close"] : []), + ], + result: rejectTls ? "certificate rejected" : "verified stream", + path: "/cdp?capability=fixture", + }); + }); +}); diff --git a/test/agents/hermes/hermes-restart-config-seal-write-lock.test.ts b/test/agents/hermes/hermes-restart-config-seal-write-lock.test.ts index 02ffada038b..77322691408 100644 --- a/test/agents/hermes/hermes-restart-config-seal-write-lock.test.ts +++ b/test/agents/hermes/hermes-restart-config-seal-write-lock.test.ts @@ -239,6 +239,7 @@ stripped = sys.argv[2] == 'nvidia-stripped' def sha(b):return hashlib.sha256(b).hexdigest() def run(case): with tempfile.TemporaryDirectory(prefix='hermes-restored-') as t: + os.chown(t,os.geteuid(),os.getegid()) r=pathlib.Path(t);s=r/'sandbox';h=s/'.hermes';h.mkdir(parents=True) os.chmod(s,0o770);os.chmod(h,0o700) config=h/'config.yaml';env=h/'.env';compat=h/'.config-hash';strict=r/'strict';state=r/'state.json' @@ -247,8 +248,16 @@ def run(case): local_body=body def body(ref): data=local_body(ref) - return data.replace(b'compatible-endpoint',b'nvidia-prod').replace(b'http://host.openshell.internal:8000/v1',b'https://integrate.api.nvidia.com/v1').replace(b'NEMOCLAW_LOCAL_INFERENCE_TOKEN',b'NVIDIA_INFERENCE_API_KEY') if nvidia else data - old=body('[STRIPPED_BY_MIGRATION]' if stripped else 'openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN');fresh=body('openshell:resolve:env:v8_NEMOCLAW_LOCAL_INFERENCE_TOKEN') + return data.replace(b'compatible-endpoint',b'nvidia-prod').replace(b'http://host.openshell.internal:8000/v1',b'https://integrate.api.nvidia.com/v1').replace(b'NEMOCLAW_LOCAL_INFERENCE_TOKEN',b'NVIDIA_INFERENCE_API_KEY') if nvidia and not case.startswith('registered-') else data + old=body('[STRIPPED_BY_MIGRATION]' if stripped and not case.startswith('registered-') else 'openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN');fresh=body('openshell:resolve:env:v8_NEMOCLAW_LOCAL_INFERENCE_TOKEN') + if case.startswith('registered-'): + old=old.replace(b'compatible-endpoint',b'operator-provider').replace(b'NEMOCLAW_LOCAL_INFERENCE_TOKEN',b'VENDOR_API_KEY') + fresh=fresh.replace(b'compatible-endpoint',b'operator-provider').replace(b'NEMOCLAW_LOCAL_INFERENCE_TOKEN',b'VENDOR_API_KEY') + if case=='registered-env-change':fresh=fresh.replace(b'VENDOR_API_KEY',b'OTHER_KEY') + if case=='registered-raw-source':old=old.replace(b'openshell:resolve:env:v7_VENDOR_API_KEY',b'raw-secret') + if case=='registered-placeholder-source':old=old.replace(b'openshell:resolve:env:v7_VENDOR_API_KEY',b'$'+b'{VENDOR_API_KEY}') + if case=='registered-route-drift':fresh=fresh.replace(b'host.openshell.internal',b'changed.example') + oldenv=b'SAFE_SETTING=trusted\nAPI_SERVER_KEY='+b'a'*64+b'\n';freshenv=b'SAFE_SETTING=trusted\nAPI_SERVER_KEY='+b'b'*64+b'\n' config.write_bytes(old);env.write_bytes(oldenv);os.chmod(config,0o640);os.chmod(env,0o600) oldhash=guard._hash_text(str(config),str(env))[0] @@ -300,7 +309,7 @@ def run(case): except Exception as e:recovery=type(e).__name__+': '+str(e) after={p.name:sha(p.read_bytes()) for p in paths} return {'state_held_before_recovery':held,'recovery':recovery,'anchors_match':strict.read_text()==compat.read_text(),'config_source_preserved':sha(config.read_bytes())==before['config.yaml'],'config_candidate_applied':config.read_bytes()==fresh,'case':case,'error':error,'files_unchanged':before==after,'before':before,'after':after,'state_removed':not state.exists(),'lock_removed':not (r/'hermes-config-mutation.lock').exists(),'config_mode':oct(config.stat().st_mode&0o7777),'env_mode':oct(env.stat().st_mode&0o7777)} -cases=['valid-restored','ordinary-write','model-drift','route-drift','mcp-drift','user-setting-drift','invalid-new-handle','slot-disagreement','alias-side-effect','stale-read','raw-env','invalid-schema','write-failure','partial-adoption','partial-adoption-source-race'] +cases=['valid-restored','ordinary-write','model-drift','route-drift','mcp-drift','user-setting-drift','invalid-new-handle','slot-disagreement','alias-side-effect','stale-read','raw-env','invalid-schema','write-failure','partial-adoption','partial-adoption-source-race','registered-valid','registered-env-change','registered-raw-source','registered-placeholder-source','registered-route-drift'] rows=[run(x) for x in cases] print(json.dumps({'source_sha256':sha(source.read_bytes()),'seams':['GatewayConfig fixture schema surface','ordinary-write nonroot topology admission only; restore admission uses real mutable posture'],'cases':rows},indent=2)) assert rows[0]['error'] is None,rows[0] @@ -310,6 +319,8 @@ assert rows[13]['state_held_before_recovery'] and rows[13]['recovery']=='recover assert rows[14]['state_held_before_recovery'] and not rows[14]['state_removed'] and 'source bytes changed' in rows[14]['recovery'],rows[14] assert rows[12]['error'] and rows[12]['config_source_preserved'] and rows[12]['anchors_match'] and rows[12]['state_removed'],rows[12] assert rows[0]['config_candidate_applied'] +assert rows[15]['error'] is None and rows[15]['config_candidate_applied'] and rows[15]['state_removed'] and rows[15]['anchors_match'],rows[15] +for row in rows[16:]:assert row['error'] and row['files_unchanged'] and row['state_removed'] and row['lock_removed'],row `, RUNTIME_CONFIG_GUARD, provider, @@ -318,7 +329,7 @@ assert rows[0]['config_candidate_applied'] ); expect(result.status, result.stderr + result.stdout).toBe(0); const report = JSON.parse(result.stdout); - expect(report.cases).toHaveLength(15); + expect(report.cases).toHaveLength(20); expect(report.cases[0]).toMatchObject({ error: null, config_candidate_applied: true, diff --git a/test/agents/hermes/hermes-runtime-api-key.test.ts b/test/agents/hermes/hermes-runtime-api-key.test.ts index 4365d26f55c..c90d879775a 100644 --- a/test/agents/hermes/hermes-runtime-api-key.test.ts +++ b/test/agents/hermes/hermes-runtime-api-key.test.ts @@ -613,22 +613,30 @@ describe("agents/hermes/start.sh runtime API server key", () => { }); it.each([ - "DISCORD_BOT_TOKEN=openshell:resolve:env:DISCORD_BOT_TOKEN\n", - "DISCORD_BOT_TOKEN=openshell:resolve:env:v111_DISCORD_BOT_TOKEN\n", + { + envFile: "DISCORD_BOT_TOKEN=openshell:resolve:env:DISCORD_BOT_TOKEN\n", + reference: "openshell:resolve:env:v222_DISCORD_BOT_TOKEN", + }, + { + envFile: "DISCORD_BOT_TOKEN=openshell:resolve:env:v111_DISCORD_BOT_TOKEN\n", + reference: "openshell:resolve:env:v222_DISCORD_BOT_TOKEN", + }, + { + envFile: "DISCORD_BOT_TOKEN=openshell:resolve:env:v111_DISCORD_BOT_TOKEN\n", + reference: `openshell:resolve:env:s${"a".repeat(64)}_DISCORD_BOT_TOKEN`, + }, ])( "preserves the exact OpenShell provider placeholder generation in Hermes .env [case %#] (#8893)", - (envFile) => { + ({ envFile, reference }) => { const run = runHermesRuntimeProviderPlaceholderRefresh({ envFile, envOverrides: { - DISCORD_BOT_TOKEN: "openshell:resolve:env:v222_DISCORD_BOT_TOKEN", + DISCORD_BOT_TOKEN: reference, }, }); expect(run.result.status, run.result.stderr).toBe(0); - expect(run.envFileContent).toContain( - "DISCORD_BOT_TOKEN=openshell:resolve:env:v222_DISCORD_BOT_TOKEN\n", - ); + expect(run.envFileContent).toContain(`DISCORD_BOT_TOKEN=${reference}\n`); expect(run.envFileContent).not.toContain("v111_DISCORD_BOT_TOKEN"); expect(run.strictHashValid).toBe(true); }, @@ -779,6 +787,10 @@ describe("agents/hermes/start.sh runtime API server key", () => { name: "an overlong credential revision", value: `openshell:resolve:env:v${"1".repeat(21)}_DISCORD_BOT_TOKEN`, }, + { + name: "a truncated credential snapshot", + value: `openshell:resolve:env:s${"a".repeat(63)}_DISCORD_BOT_TOKEN`, + }, { name: "a raw credential", value: "raw-discord-token", diff --git a/test/agents/hermes/hermes-runtime-config-guard.test.ts b/test/agents/hermes/hermes-runtime-config-guard.test.ts index 9ebee8b67af..ba7ef042c0d 100644 --- a/test/agents/hermes/hermes-runtime-config-guard.test.ts +++ b/test/agents/hermes/hermes-runtime-config-guard.test.ts @@ -852,7 +852,7 @@ with tempfile.TemporaryDirectory() as tmp: guard.PROC_ROOT = str(proc) guard.HERMES_ROOT_LIFECYCLE_MARKER = str(root / "root-marker") guard.HERMES_STARTUP_READY_FILE = str(root / "ready-marker") - guard.pwd.getpwnam = lambda _name: types.SimpleNamespace(pw_uid=1000) + guard.pwd.getpwnam = lambda _name: types.SimpleNamespace(pw_uid=1000, pw_gid=1000) def process(pid, parent, uid, argv): directory = proc / str(pid) diff --git a/test/agents/hermes/hermes-service-restart.test.ts b/test/agents/hermes/hermes-service-restart.test.ts index 2bbc178f19b..23eec6a997f 100644 --- a/test/agents/hermes/hermes-service-restart.test.ts +++ b/test/agents/hermes/hermes-service-restart.test.ts @@ -15,7 +15,12 @@ function runSupervisor( firstExit: number, finalExit: number, repeatedFirstExits = 1, - options: { elapsedPerGateway?: number; cooldownFails?: boolean; prepareFails?: boolean } = {}, + options: { + elapsedPerGateway?: number; + cooldownFails?: boolean; + prepareFails?: boolean; + completionFails?: boolean; + } = {}, ) { const script = [ "set -uo pipefail", @@ -33,6 +38,7 @@ function runSupervisor( "auxiliary_count=0", "finalize_count=0", "refresh_count=0", + "completion_count=0", "recovery_count=0", "SECONDS=0", "sleep_count=0", @@ -49,6 +55,7 @@ function runSupervisor( "ensure_hermes_supervised_auxiliaries() { auxiliary_count=$((auxiliary_count + 1)); }", "finalize_tirith_marker_retry() { finalize_count=$((finalize_count + 1)); }", "refresh_hermes_supervised_child_pids() { refresh_count=$((refresh_count + 1)); }", + `publish_hermes_startup_completion() { [ "$finalize_count" -eq "$launch_count" ] && [ "$auxiliary_count" -eq "$launch_count" ] && [ "$refresh_count" -eq "$launch_count" ] || return 1; completion_count=$((completion_count + 1)); ${options.completionFails ? "return 1" : ":"}; }`, "wait_for_hermes_gateway_recovery_request() { recovery_count=$((recovery_count + 1)); }", extractShellFunction(source, "relaunch_hermes_gateway_current_user"), extractShellFunction(source, "supervise_hermes_service_restarts_current_user"), @@ -56,6 +63,7 @@ function runSupervisor( "supervise_hermes_service_restarts_current_user || status=$?", 'printf "%s\\n" "status=$status waits=$wait_count launches=$launch_count marks=$mark_count prepares=$prepare_count ready=$ready_count auxiliaries=$auxiliary_count finalize=$finalize_count refresh=$refresh_count recoveries=$recovery_count gateway=$GATEWAY_PID"', 'printf "cooldown:sleeps=%s seconds=%s\\n" "$sleep_count" "$sleep_seconds" >&2', + 'printf "completed:%s\\n" "$completion_count" >&2', ].join("\n"); return spawnSync("bash", ["-c", script], { encoding: "utf8", @@ -207,33 +215,36 @@ describe("Hermes native service restart supervision", () => { expect(result.stderr).not.toContain("relaunching"); }); - it("stops when no gated recovery request arrives before the deadline", () => { - const script = [ - "set -uo pipefail", - "readonly HERMES_GATEWAY_RECOVERY_REQUESTER_EXIT_ATTEMPTS=30", - "readonly HERMES_GATEWAY_RECOVERY_REQUEST_WAIT_SECONDS=2", - "readonly HERMES_GATEWAY_RECOVERY_TRANSPORT_SETTLE_SECONDS=1", - "sleep_count=0", - 'publish_hermes_gateway_recovery_generation() { HERMES_GATEWAY_RECOVERY_GENERATION="$(printf c%.0s {1..64})"; }', - 'hermes_gateway_recovery_request_value() { printf "%s\\n" absent; }', - "hermes_recovery_requester_start_time() { return 1; }", - "sleep() { sleep_count=$((sleep_count + 1)); }", - extractShellFunction(source, "wait_for_hermes_recovery_requester_exit"), - extractShellFunction(source, "wait_for_hermes_gateway_recovery_request"), - "status=0", - "wait_for_hermes_gateway_recovery_request || status=$?", - 'printf "status=%s sleeps=%s\\n" "$status" "$sleep_count"', - ].join("\n"); - - const result = spawnSync("bash", ["-c", script], { encoding: "utf8", timeout: 5_000 }); - - expect(result.status, result.stderr).toBe(0); - expect(result.stdout.trim()).toBe("status=1 sleeps=2"); - expect(result.stderr).toContain("recovery request timed out after 2 seconds"); - expect(result.stderr).toContain("nemoclaw stop"); - expect(result.stderr).toContain("nemoclaw start"); - expect(result.stderr).not.toContain("relaunching"); - }); + it.each(["absent", `v2 ${"d".repeat(64)} 321 654`])( + "stops when a current gated recovery request never arrives (%s)", + (request) => { + const script = [ + "set -uo pipefail", + "readonly HERMES_GATEWAY_RECOVERY_REQUESTER_EXIT_ATTEMPTS=30", + "readonly HERMES_GATEWAY_RECOVERY_REQUEST_WAIT_SECONDS=2", + "readonly HERMES_GATEWAY_RECOVERY_TRANSPORT_SETTLE_SECONDS=1", + "sleep_count=0", + 'publish_hermes_gateway_recovery_generation() { HERMES_GATEWAY_RECOVERY_GENERATION="$(printf c%.0s {1..64})"; }', + `hermes_gateway_recovery_request_value() { printf "%s\\n" '${request}'; }`, + "hermes_recovery_requester_start_time() { return 1; }", + "sleep() { sleep_count=$((sleep_count + 1)); }", + extractShellFunction(source, "wait_for_hermes_recovery_requester_exit"), + extractShellFunction(source, "wait_for_hermes_gateway_recovery_request"), + "status=0", + "wait_for_hermes_gateway_recovery_request || status=$?", + 'printf "status=%s sleeps=%s\\n" "$status" "$sleep_count"', + ].join("\n"); + + const result = spawnSync("bash", ["-c", script], { encoding: "utf8", timeout: 5_000 }); + + expect(result.status, result.stderr).toBe(0); + expect(result.stdout.trim()).toBe("status=1 sleeps=2"); + expect(result.stderr).toContain("recovery request timed out after 2 seconds"); + expect(result.stderr).toContain("nemoclaw stop"); + expect(result.stderr).toContain("nemoclaw start"); + expect(result.stderr).not.toContain("relaunching"); + }, + ); it("rejects an untrusted recovery request marker", () => { const script = [ @@ -293,6 +304,14 @@ describe("Hermes native service restart supervision", () => { "status=9 waits=2 launches=1 marks=1 prepares=1 ready=1 auxiliaries=1 finalize=1 refresh=1 recoveries=0 gateway=101", ); expect(result.stderr).toContain("Hermes requested a service-managed restart"); + expect(result.stderr).toContain("completed:1"); + }); + + it("stops supervision when startup completion cannot be published", () => { + const result = runSupervisor(75, 9, 1, { completionFails: true }); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain("status=1 waits=1 launches=1"); + expect(result.stderr).toContain("completed:1"); }); it("preserves the supervisor and rate limit across repeated bursts of requested restarts", () => { diff --git a/test/agents/hermes/hermes-startup-completion.test.ts b/test/agents/hermes/hermes-startup-completion.test.ts new file mode 100644 index 00000000000..73192fd99b7 --- /dev/null +++ b/test/agents/hermes/hermes-startup-completion.test.ts @@ -0,0 +1,284 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { shellQuote } from "../../../src/lib/core/shell-quote"; +import { mutableHermesConfigProbeCommand } from "../../../src/lib/sandbox/mutable-config-perms"; +import { extractShellFunction } from "../../support/hermes-shell-harness"; + +const helper = path.resolve(import.meta.dirname, "../../../agents/hermes/startup-completion.py"); + +function probe(body: string) { + return spawnSync( + "python3", + [ + "-I", + "-c", + ` +import importlib.util, json, os, pathlib, tempfile +spec = importlib.util.spec_from_file_location("completion", ${JSON.stringify(helper)}) +m = importlib.util.module_from_spec(spec) +spec.loader.exec_module(m) +observe_process = m.process_identity +supervisor = {"pid": os.getppid(), "parent": 1, "start": "10"} +gateway = {"pid": 999999, "parent": os.getppid(), "start": "20"} +identities = {supervisor["pid"]: supervisor, gateway["pid"]: gateway} +m.process_identity = lambda pid: identities.get(pid) +with tempfile.TemporaryDirectory() as root: + os.chown(root, os.geteuid(), os.getegid()) + receipt = str(pathlib.Path(root) / "completion.json") + def publish(): + m.publish_completion(gateway["pid"], receipt) + def rejected(action, message): + try: + action() + except (RuntimeError, OSError) as error: + assert message in str(error), str(error) + else: + raise AssertionError("unsafe receipt was accepted") +${body + .split("\n") + .map((line) => ` ${line}`) + .join("\n")} +`, + ], + { encoding: "utf8", timeout: 5000 }, + ); +} + +describe("Hermes supervisor startup completion", () => { + it + .skipIf(process.platform !== "linux" || process.getuid?.() === 0) + .each(["live", "missing", "mismatched"])( + "binds the non-root startup probe to the %s supervisor receipt", + (condition) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "hermes-startup-handoff-")); + const configDir = path.join(root, "config"); + const configPath = path.join(configDir, "config.yaml"); + const receipt = path.join(root, "completion.json"); + const inspection = path.join(root, "inspected"); + const publisher = path.join(root, "completion.py"); + const consumer = path.join(root, "probe.py"); + fs.mkdirSync(configDir); + fs.chmodSync(configDir, 0o3770); + fs.writeFileSync(configPath, "fixture\n", { mode: 0o640 }); + fs.chmodSync(configPath, 0o640); + // Relocate installed paths only; retain real /proc identities and receipt validation. + fs.writeFileSync( + publisher, + fs + .readFileSync(helper, "utf8") + .replace( + 'RECEIPT = "/tmp/nemoclaw-hermes-startup-complete.json"', + `RECEIPT = ${JSON.stringify(receipt)}`, + ), + ); + const command = mutableHermesConfigProbeCommand({ + agentName: "hermes", + configDir, + configPath, + configFile: "config.yaml", + format: "yaml", + sensitiveFiles: [], + }); + fs.writeFileSync( + consumer, + ` +import json, os, pathlib, pwd, runpy, types +pwd.getpwnam = lambda name: types.SimpleNamespace(pw_uid=os.geteuid(), pw_gid=os.getegid()) +load_helper = runpy.run_path +completion = load_helper(${JSON.stringify(publisher)}) +receipt = ${JSON.stringify(receipt)} +if ${JSON.stringify(condition)} == "missing": + os.unlink(receipt) +elif ${JSON.stringify(condition)} == "mismatched": + value = json.loads(pathlib.Path(receipt).read_text()) + value["gateway"]["start"] = str(int(value["gateway"]["start"]) + 1) + pathlib.Path(receipt).write_text(json.dumps(value)) +else: + value = completion["read_completion"]() + assert value["supervisor"] == completion["process_identity"](os.getppid()) + assert value["gateway"]["parent"] == os.getppid() +def installed_helper(name): + assert name == "/usr/local/lib/nemoclaw/hermes-startup-completion.py" + return {**completion, + "requires_completion": lambda: completion["requires_completion"](${JSON.stringify(path.join(root, "root-lifecycle"))}), + "wait_for_completion": lambda: completion["wait_for_completion"](timeout=0.2)} +runpy.run_path = installed_helper +observe_stat = os.stat +def record_inspection(name, *args, **kwargs): + if name == ${JSON.stringify(configDir)}: + pathlib.Path(${JSON.stringify(inspection)}).touch() + return observe_stat(name, *args, **kwargs) +os.stat = record_inspection +${command[command.indexOf("-c") + 1]} +`, + ); + const source = fs.readFileSync(path.join(path.dirname(helper), "start.sh"), "utf8"); + const start = source.indexOf( + 'if [ "$(id -u)" -ne 0 ]; then\n echo "[gateway] Running as non-root', + ); + const end = source.indexOf("# ── Root path (full privilege separation", start); + const script = [ + "set -euo pipefail", + "NEMOCLAW_CMD=()", + `HERMES_DIR=${shellQuote(configDir)}`, + '_HERMES_PYTHON="$(command -v python3)"', + `_HERMES_STARTUP_COMPLETION=${shellQuote(publisher)}`, + ...[ + "prepare_hermes_nonroot_runtime", + "cleanup_stale_hermes_gateway_runtime", + "prepare_restricted_log", + "validate_tmp_permissions", + "start_gateway_log_stream", + "wait_for_hermes_gateway_internal", + "ensure_hermes_supervised_auxiliaries", + "print_dashboard_urls", + ].map((name) => `${name}() { :; }`), + "launch_hermes_gateway_current_user() { sleep 30 & GATEWAY_PID=$!; }", + `finalize_tirith_marker_retry() { printf 'startup complete\\n' >>${shellQuote(configPath)}; }`, + extractShellFunction(source, "publish_hermes_startup_completion"), + `supervise_hermes_service_restarts_current_user() { "$_HERMES_PYTHON" -I ${shellQuote(consumer)} ${command + .slice(command.indexOf("-c") + 2) + .map(shellQuote) + .join(" ")}; }`, + 'trap \'if [ -n "${GATEWAY_PID:-}" ]; then kill "$GATEWAY_PID" 2>/dev/null || true; wait "$GATEWAY_PID" 2>/dev/null || true; fi\' EXIT', + source.slice(start, end), + ].join("\n"); + try { + const result = spawnSync("bash", ["-c", script], { encoding: "utf8", timeout: 5000 }); + expect(result.error).toBeUndefined(); + expect(result.status, result.stderr).toBe(condition === "live" ? 0 : 1); + expect(fs.existsSync(inspection)).toBe(condition === "live"); + expect(result.stderr.includes("startup completion was not observed")).toBe( + condition !== "live", + ); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, + ); + + it("keeps the separate root startup protocol and rejects an untrusted root marker", () => { + const result = probe(` +import stat, types +marker = str(pathlib.Path(root) / "root-lifecycle") +assert m.requires_completion(marker) +pathlib.Path(marker).write_text("root-separated\\n") +os.chmod(marker, 0o444) +rejected(lambda: m.requires_completion(marker), "Invalid Hermes root lifecycle marker") +real_fstat = os.fstat +def root_marker_metadata(fd): + value = real_fstat(fd) + return types.SimpleNamespace(st_mode=value.st_mode, st_nlink=value.st_nlink, st_uid=0, st_gid=0) +os.fstat = root_marker_metadata +assert not m.requires_completion(marker) +os.chmod(marker, 0o644) +rejected(lambda: m.requires_completion(marker), "Invalid Hermes root lifecycle marker") +os.fstat = real_fstat +`); + expect(result.status, result.stderr).toBe(0); + }); + + it("observes process start identity and rejects dead or differently owned processes", () => { + const result = probe(` +proc = pathlib.Path(root) / "123" +proc.mkdir() +fields = ["S", "100"] + ["0"] * 17 + ["456"] +def write_stat(): + (proc / "stat").write_text("123 (process name) " + " ".join(fields)) +write_stat() +(proc / "status").write_text(f"Uid:\\t{os.geteuid()}\\t{os.geteuid()}\\t{os.geteuid()}\\t{os.geteuid()}\\n") +m.Path = lambda _path: proc +assert observe_process(123) == {"pid": 123, "parent": 100, "start": "456"} +fields[0] = "Z" +write_stat() +assert observe_process(123) is None +fields[0] = "S" +write_stat() +(proc / "status").write_text("Uid:\\t0\\t0\\t0\\t0\\n") +assert observe_process(123) is None +assert observe_process(True) is None +`); + expect(result.status, result.stderr).toBe(0); + }); + + it("waits for publication before inspection and rejects a dead or reused gateway", () => { + const result = probe(` +assert m.read_completion(receipt) is None +calls = [] +def finish_startup(_interval): + calls.append("finish startup writes") + publish() +m.time.sleep = finish_startup +observed = m.wait_for_completion(path=receipt) +assert calls == ["finish startup writes"] +assert observed["gateway"] == gateway +identities.pop(gateway["pid"]) +assert m.read_completion(receipt) is None +identities[gateway["pid"]] = {**gateway, "start": "21"} +assert m.read_completion(receipt) is None +rejected(lambda: m.wait_for_completion(timeout=0, path=receipt), "completion was not observed") +`); + expect(result.status, result.stderr).toBe(0); + }); + + it.each(["symlink", "hardlink", "writable", "malformed", "parent", "oversized"])( + "rejects a %s receipt without replacing it", + (kind) => { + const result = probe(` +publish() +kind = ${JSON.stringify(kind)} +if kind == "symlink": + os.rename(receipt, receipt + ".target") + os.symlink(receipt + ".target", receipt) +elif kind == "hardlink": + os.link(receipt, receipt + ".link") +elif kind == "writable": + os.chmod(receipt, 0o666) +elif kind == "malformed": + pathlib.Path(receipt).write_text("{broken") +elif kind == "parent": + value = json.loads(pathlib.Path(receipt).read_text()) + value["gateway"]["parent"] += 1 + pathlib.Path(receipt).write_text(json.dumps(value)) +elif kind == "oversized": + pathlib.Path(receipt).write_text("x" * 1025) +before = os.lstat(receipt) +rejected(lambda: m.read_completion(receipt), "") +rejected(publish, "") +assert os.lstat(receipt) == before +`); + expect(result.status, result.stderr).toBe(0); + }, + ); + + it("rejects a receipt replaced while its descriptor is being read", () => { + const result = probe(` +publish() +read = os.read +def raced_read(fd, count): + data = read(fd, count) + os.rename(receipt, receipt + ".old") + pathlib.Path(receipt).write_bytes(data) + os.chmod(receipt, 0o600) + return data +os.read = raced_read +rejected(lambda: m.read_completion(receipt), "changed while reading") +`); + expect(result.status, result.stderr).toBe(0); + }); + + it("refuses publication when the gateway is no longer the supervisor's child", () => { + const result = probe(` +identities[gateway["pid"]] = {**gateway, "parent": 5} +rejected(publish, "live supervised gateway") +assert not os.path.exists(receipt) +`); + expect(result.status, result.stderr).toBe(0); + }); +}); diff --git a/test/agents/openclaw/openclaw-integrity-pin-suite.ts b/test/agents/openclaw/openclaw-integrity-pin-suite.ts index 005adf7b396..53eda671952 100644 --- a/test/agents/openclaw/openclaw-integrity-pin-suite.ts +++ b/test/agents/openclaw/openclaw-integrity-pin-suite.ts @@ -847,7 +847,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -902,7 +902,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -967,7 +967,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -1172,7 +1172,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -1206,7 +1206,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -1233,7 +1233,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -1259,7 +1259,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -1308,7 +1308,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes ? extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ) : extractRunBlock( DOCKERFILE_BASE, @@ -1394,7 +1394,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes label: "production Dockerfile", file: DOCKERFILE, startMarker: "# OPENCLAW_VERSION is the NemoClaw runtime build target", - endMarker: "# Patch OpenClaw media fetch", + endMarker: "# Patch OpenClaw network boundaries", }, { label: "base Dockerfile", @@ -1433,7 +1433,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), { openclawVersion: PINNED_OPENCLAW_VERSION, @@ -1456,7 +1456,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes label: "production Dockerfile", file: DOCKERFILE, startMarker: "# OPENCLAW_VERSION is the NemoClaw runtime build target", - endMarker: "# Patch OpenClaw media fetch", + endMarker: "# Patch OpenClaw network boundaries", }, { label: "base Dockerfile", @@ -1490,7 +1490,7 @@ export function registerOpenClawIntegrityPinTests(group: OpenClawIntegrityPinTes extractRunBlock( DOCKERFILE, "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), ); diff --git a/test/agents/openclaw/openclaw-real-patched-dist-harness.test.ts b/test/agents/openclaw/openclaw-real-patched-dist-harness.test.ts index ab33581efac..f1a7ebb8e27 100644 --- a/test/agents/openclaw/openclaw-real-patched-dist-harness.test.ts +++ b/test/agents/openclaw/openclaw-real-patched-dist-harness.test.ts @@ -224,30 +224,40 @@ function shellQuote(value: string): string { return `'${value.replaceAll("'", "'\\''")}'`; } -function dockerfileRuntimePatchCommand(dist: string): string { +function dockerfileRuntimePatchCommand(dist: string, nodeExecutable: string): string { return dockerRunCommandBetween( "# Native OpenClaw restart must reload", "# Keep OpenClaw 2026.9.1 scope-upgrade", ) .replaceAll("/usr/local/lib/node_modules/openclaw/dist", shellQuote(dist)) + .replaceAll('node "', `${shellQuote(nodeExecutable)} "`) .replaceAll( "/usr/local/lib/nemoclaw/patch-openclaw-", path.join(REPO_ROOT, "scripts", "lib", "patch-openclaw-"), ); } -function dockerfilePatchCommand(dist: string): string { +function dockerfilePatchCommand(dist: string, nodeExecutable: string): string { const legacy = dockerRunCommandBetween( - "# Patch OpenClaw media fetch for proxy-only sandbox", + "# Patch OpenClaw network boundaries for OpenShell 0.1.3.", "# Native OpenClaw restart must reload", ) .replaceAll("/usr/local/lib/node_modules/openclaw/dist", dist) + .replaceAll( + "node /usr/local/lib/nemoclaw/patch-openclaw-native-network.mts", + `${shellQuote(nodeExecutable)} ${shellQuote(path.join(REPO_ROOT, "scripts/lib/patch-openclaw-native-network.mts"))}`, + ) .replaceAll("/usr/local/lib/nemoclaw/extract-semver", shellQuote(OPENCLAW_VERSION_EXTRACTOR)); - return `${legacy}\n${dockerfileRuntimePatchCommand(dist)}`; + return `${legacy}\n${dockerfileRuntimePatchCommand(dist, nodeExecutable)}`; } -function runDockerfilePatchBlock(dist: string, tmp: string, version: string) { - const command = dockerfilePatchCommand(dist); +function runDockerfilePatchBlock( + dist: string, + tmp: string, + version: string, + nodeExecutable: string, +) { + const command = dockerfilePatchCommand(dist, nodeExecutable); const scriptPath = path.join(tmp, "patch-openclaw-dist.sh"); fs.writeFileSync( scriptPath, @@ -432,11 +442,26 @@ describe("OpenClaw real patched-dist materialization guard", () => { ); try { expect(() => patchOpenClawAgentMessageInput(dist, true)).toThrow("patch is missing"); - const result = spawnSync("bash", ["-c", dockerfileRuntimePatchCommand(dist)], { - encoding: "utf8", - timeout: PATCH_COMMAND_TIMEOUT_MS, - }); + const nodeExecutable = path.join(tmp, "reviewed node"); + const invocations = path.join(tmp, "node-invocations"); + fs.writeFileSync( + nodeExecutable, + `#!/bin/sh\nprintf '%s\n' "$1" >>${shellQuote(invocations)}\nexec ${shellQuote(process.execPath)} "$@"\n`, + { mode: 0o700 }, + ); + const result = spawnSync( + "bash", + ["-c", dockerfileRuntimePatchCommand(dist, nodeExecutable)], + { + encoding: "utf8", + timeout: PATCH_COMMAND_TIMEOUT_MS, + }, + ); requireSpawnSuccess(result, "apply Dockerfile runtime patches to parser fixtures"); + expect(fs.readFileSync(invocations, "utf8").trim().split("\n")).toEqual([ + path.join(REPO_ROOT, "scripts/lib/patch-openclaw-container-restart.mts"), + path.join(REPO_ROOT, "scripts/lib/patch-openclaw-agent-message-input.mts"), + ]); expect(() => patchOpenClawAgentMessageInput(dist, true)).not.toThrow(); expect(() => patchOpenClawContainerRestart(dist, true)).not.toThrow(); } finally { @@ -445,7 +470,7 @@ describe("OpenClaw real patched-dist materialization guard", () => { }); it("maps container-only patch helpers to their repository fixtures", () => { - const command = dockerfilePatchCommand("/tmp/reviewed-openclaw-dist"); + const command = dockerfilePatchCommand("/tmp/reviewed-openclaw-dist", process.execPath); expect(command).not.toContain("/usr/local/lib/nemoclaw/extract-semver"); expect(command).toContain(shellQuote(OPENCLAW_VERSION_EXTRACTOR)); @@ -801,27 +826,12 @@ try { const dist = path.join(runtimeRoot, "node_modules", "openclaw", "dist"); fs.statSync(dist).isDirectory() || runtimeMismatch("not a directory", "directory", dist); - const dockerPatch = runDockerfilePatchBlock(dist, tmp, version); + const dockerPatch = runDockerfilePatchBlock(dist, tmp, version, nodeRuntime.executable); requireSpawnSuccess(dockerPatch, "apply Dockerfile OpenClaw patches"); requireRuntimeIncludes( dockerPatch.stdout, - `Patch 2 applied to OpenClaw ${version}`, - "Patch 2", - ); - requireRuntimeIncludes( - dockerPatch.stdout, - `Patch 2b applied to OpenClaw ${version}`, - "Patch 2b", - ); - requireRuntimeIncludes( - dockerPatch.stdout, - `Patch 4 applied to OpenClaw ${version}`, - "Patch 4", - ); - requireRuntimeIncludes( - dockerPatch.stdout, - `Patch 6 applied to OpenClaw ${version}`, - "Patch 6", + `OpenClaw ${version} native network patch applied`, + "native network patch", ); const toolCatalogPatch = spawnSync( nodeRuntime.executable, @@ -835,10 +845,8 @@ try { "managed llama.cpp compact tool catalog patch output", ); [ - "nemoclaw: env-gated bypass", - "nemoclaw: OpenShell host gateway for web_fetch trusted env proxy", - "nemoclaw: route unconfigured strict fetch through sandbox egress proxy", - 'mode: "trusted_env_proxy", auditContext: "cron-model-provider-preflight"', + "!nemoclawIsSyntheticDnsAnswer(entry.address)", + "const nativeResolved = { ...resolved, policy: hostGatewayPolicy };", "nemoclaw llama.cpp compact native tool catalog (#11105)", "nemoclaw llama.cpp JSON-string tool-call input (#11105)", ].forEach((marker) => { diff --git a/test/agents/openclaw/runtime/nemoclaw-start.test.ts b/test/agents/openclaw/runtime/nemoclaw-start.test.ts index 75e7db48396..98363e5ea89 100644 --- a/test/agents/openclaw/runtime/nemoclaw-start.test.ts +++ b/test/agents/openclaw/runtime/nemoclaw-start.test.ts @@ -324,12 +324,7 @@ describe("nemoclaw-start gateway token export (#1114)", () => { 'export OPENCLAW_OAUTH_DIR="/sandbox/.openclaw/credentials"', `PUBLIC_PORT=${JSON.stringify(port)}`, 'CHAT_UI_URL="https://remote.example.test/ui"', - 'PROXY_HOST="10.200.0.1"', - 'PROXY_PORT="3128"', - '_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}"', - '_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}"', '_SANDBOX_SAFETY_NET="/tmp/safety-net.js"', - '_PROXY_FIX_SCRIPT="/tmp/http-proxy-fix.js"', '_NEMOTRON_FIX_SCRIPT="/tmp/nemotron-fix.js"', "emit_messaging_connect_runtime_preload_exports() { :; }", "_TOOL_REDIRECTS=()", @@ -365,14 +360,16 @@ describe("nemoclaw-start gateway token export (#1114)", () => { }; } - it("reads, exports, prints, and shell-escapes the gateway token without touching rc files", () => { + it("exports gateway and state settings while redacting logs and preserving rc files", () => { const { result, envFile } = runGatewayTokenHarness( JSON.stringify({ gateway: { auth: { token: "tok'en" } } }), + "stale-token", + "18790", ); expect(result.status).toBe(0); expect(result.stdout).toContain("TOKEN=tok'en"); - expect(result.stderr).toContain("http://127.0.0.1:18789/"); + expect(result.stderr).toContain("http://127.0.0.1:18790/"); expect(result.stderr).toContain("https://remote.example.test/ui/"); expect(result.stderr).toContain("Dashboard auth token redacted from startup logs."); expect(result.stderr).not.toContain("#token="); @@ -382,28 +379,6 @@ describe("nemoclaw-start gateway token export (#1114)", () => { expect(envFile).toContain("nemoclaw-configure-guard begin"); expect(envFile).not.toContain(".bashrc"); expect(envFile).not.toContain(".profile"); - }); - - it("writes the gateway port and URL into the runtime shell env (#3256)", () => { - const { result, envFile } = runGatewayTokenHarness( - JSON.stringify({ gateway: { auth: { token: "token" } } }), - "stale-token", - "18790", - ); - - expect(result.status).toBe(0); - expect(result.stderr).toContain("http://127.0.0.1:18790/"); - expect(envFile).toContain("export OPENCLAW_GATEWAY_PORT='18790'"); - expect(envFile).not.toContain("NEMOCLAW_OPENCLAW_GATEWAY_URL"); - expect(envFile).toContain("export OPENCLAW_GATEWAY_URL='ws://127.0.0.1:18790'"); - expect(envFile).toContain("OPENCLAW_GATEWAY_TOKEN='token'"); - }); - it("writes OpenClaw state env for connect-shell pairing approval (#3730)", () => { - const { result, envFile } = runGatewayTokenHarness( - JSON.stringify({ gateway: { auth: { token: "token" } } }), - ); - - expect(result.status).toBe(0); expect(envFile).toContain("export OPENCLAW_HOME='/sandbox'"); expect(envFile).toContain("export OPENCLAW_STATE_DIR='/sandbox/.openclaw'"); expect(envFile).toContain("export OPENCLAW_CONFIG_PATH='/sandbox/.openclaw/openclaw.json'"); @@ -411,6 +386,9 @@ describe("nemoclaw-start gateway token export (#1114)", () => { expect(envFile.indexOf("export OPENCLAW_STATE_DIR=")).toBeLessThan( envFile.indexOf("OPENCLAW_GATEWAY_TOKEN='"), ); + expect(envFile).toContain("export OPENCLAW_GATEWAY_PORT='18790'"); + expect(envFile).not.toContain("NEMOCLAW_OPENCLAW_GATEWAY_URL"); + expect(envFile).toContain("export OPENCLAW_GATEWAY_URL='ws://127.0.0.1:18790'"); }); it("generates a gateway token and scrubs rejected legacy metadata (#3256)", () => { @@ -430,6 +408,37 @@ describe("nemoclaw-start gateway token export (#1114)", () => { expect(envFile).toContain(`OPENCLAW_GATEWAY_TOKEN='${configAfter.gateway.auth.token}'`); expect(envFile).not.toContain("stale-token"); }); + it("clears obsolete proxy config from restored state while preserving channel authorization", () => { + const account = { + enabled: true, + botToken: "openshell:resolve:env:TELEGRAM_BOT_TOKEN", + allowFrom: ["123"], + proxy: "http://10.200.0.1:3128", + }; + const { result, configAfter } = runGatewayTokenHarness( + JSON.stringify({ + gateway: { auth: {} }, + proxy: { enabled: true, proxyUrl: "http://10.200.0.1:3128" }, + tools: { web: { fetch: { enabled: true, useTrustedEnvProxy: true } } }, + channels: { + telegram: { + proxy: "http://10.200.0.1:3128", + accounts: { default: account, second: account }, + }, + }, + }), + "stale-token", + "18790", + true, + ); + expect(result.status, result.stderr || result.stdout).toBe(0); + expect(configAfter.proxy).toBeUndefined(); + expect(configAfter.tools.web.fetch).toEqual({ enabled: true, useTrustedEnvProxy: false }); + expect(configAfter.channels.telegram.proxy).toBeUndefined(); + const { proxy: _proxy, ...retained } = account; + expect(configAfter.channels.telegram.accounts).toEqual({ default: retained, second: retained }); + }); + it("rotates an existing gateway token before writing the runtime shell env (#4517)", () => { const oldToken = "old-token-before-rebuild"; const { result, envFile, configAfter } = runGatewayTokenHarness( @@ -569,12 +578,7 @@ describe("nemoclaw-start configure guard behavior", () => { "#!/usr/bin/env bash", "set -euo pipefail", 'emit_sandbox_sourced_file() { local target="$1"; cat > "$target"; chmod 444 "$target"; }', - 'PROXY_HOST="10.200.0.1"', - 'PROXY_PORT="3128"', - '_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}"', - '_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}"', '_SANDBOX_SAFETY_NET="/tmp/safety-net.js"', - '_PROXY_FIX_SCRIPT="/tmp/http-proxy-fix.js"', '_NEMOTRON_FIX_SCRIPT="/tmp/nemotron-fix.js"', "emit_messaging_connect_runtime_preload_exports() { :; }", 'export OPENCLAW_GATEWAY_URL="ws://127.0.0.1:18789"', @@ -2968,7 +2972,6 @@ describe("Telegram diagnostics (#2766)", () => { 'validate_tmp_permissions() { printf "VALIDATE:%s\\n" "$*"; }', "_SANDBOX_HOME=/sandbox", `_SANDBOX_SAFETY_NET=${JSON.stringify(path.join(tmpDir, "safety.js"))}`, - `_PROXY_FIX_SCRIPT=${JSON.stringify(path.join(tmpDir, "proxy-fix.js"))}`, `_NEMOTRON_FIX_SCRIPT=${JSON.stringify(path.join(tmpDir, "nemotron-fix.js"))}`, `validate_nemoclaw_tmp_permissions() { validate_tmp_permissions ${JSON.stringify(preloadPath)}; }`, "NEMOCLAW_CMD=()", @@ -3193,12 +3196,7 @@ process.stderr.write('FailoverError: token=123456:LATER\\n'); "#!/usr/bin/env bash", "set -euo pipefail", 'emit_sandbox_sourced_file() { local target="$1"; cat > "$target"; chmod 444 "$target"; }', - 'PROXY_HOST="10.200.0.1"', - 'PROXY_PORT="3128"', - '_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}"', - '_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}"', `_SANDBOX_SAFETY_NET=${JSON.stringify(path.join(tmpDir, "safety.js"))}`, - `_PROXY_FIX_SCRIPT=${JSON.stringify(path.join(tmpDir, "proxy-fix.js"))}`, `_NEMOTRON_FIX_SCRIPT=${JSON.stringify(path.join(tmpDir, "nemotron-fix.js"))}`, `_MESSAGING_CONNECT_PRELOADS_FILE=${JSON.stringify(connectPreloadsPath)}`, extractShellFunctionFromSource(src, "emit_messaging_connect_runtime_preload_exports"), @@ -3309,14 +3307,11 @@ describe("native configuration during simulated root startup", () => { 'id() { if [ "${1:-}" = "-u" ]; then printf "0"; else command id "$@"; fi; }', "NEMOCLAW_CMD=()", "_DASHBOARD_PORT=18791", - '_PROXY_URL=""', - '_NO_PROXY_VAL=""', "STEP_DOWN_PREFIX_SANDBOX=(env)", 'emit_sandbox_sourced_file() { local target="$1"; cat > "$target"; chmod 444 "$target"; }', "write_auth_profile() { :; }", "harden_auth_profiles() { :; }", '_SANDBOX_SAFETY_NET=""', - '_PROXY_FIX_SCRIPT=""', '_NEMOTRON_FIX_SCRIPT=""', "emit_messaging_connect_runtime_preload_exports() { :; }", '_TOOL_REDIRECTS=("NEMOCLAW_TEST_REDIRECT=/tmp/nemoclaw-test")', diff --git a/test/agents/openclaw/runtime/pi-candidate-runtime-artifacts.test.ts b/test/agents/openclaw/runtime/pi-candidate-runtime-artifacts.test.ts index 4a0674bb687..d92137ee819 100644 --- a/test/agents/openclaw/runtime/pi-candidate-runtime-artifacts.test.ts +++ b/test/agents/openclaw/runtime/pi-candidate-runtime-artifacts.test.ts @@ -762,7 +762,13 @@ syncBuiltinESMExports();\n`, const config = JSON.parse(fs.readFileSync(path.join(home, ".pi/agent/models.json"), "utf8")); expect(config.providers.openshell.baseUrl).toBe(endpoint); expect(config.providers.openshell.models[0].id).toBe("selected-model"); - expect(config.providers.openshell.apiKey.startsWith("!")).toBe(native); + expect(config.providers.openshell.apiKey).toEqual( + native + ? expect.stringMatching(/^!/) + : provider === "compatible-endpoint" && endpoint !== "https://inference.local/v1" + ? "${COMPATIBLE_API_KEY}" + : "nemoclaw-managed-inference", + ); expect(config.providers.openshell.headers !== undefined).toBe(native); } finally { fs.rmSync(home, { recursive: true, force: true }); @@ -930,3 +936,168 @@ syncBuiltinESMExports();\n`, expect(fs.existsSync(path.join(home, ".pi", "agent", "models.json"))).toBe(false); }); }); + +const PI_PROXY_VARIABLES = [ + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "ALL_PROXY", + "all_proxy", + "OPENAI_PROXY", + "grpc_proxy", + "GRPC_PROXY", + "NODE_USE_ENV_PROXY", + "NEMOCLAW_PROXY_HOST", + "NEMOCLAW_PROXY_PORT", +]; + +function piStartupFixture(directory: string): { script: string; runtimeEnv: string } { + const script = path.join(directory, "start.sh"); + const runtimeEnv = path.join(directory, "runtime-env.sh"); + const rlimits = path.join(directory, "sandbox-rlimits.sh"); + fs.writeFileSync( + rlimits, + "harden_resource_limits() { :; }\nverify_resource_limits_exact() { :; }\n", + ); + const source = fs + .readFileSync(path.join(root, "agents/pi/start.sh"), "utf8") + .replace( + "/usr/local/lib/nemoclaw/entrypoint-env-wrapper.sh", + path.join(root, "scripts/lib/entrypoint-env-wrapper.sh"), + ) + .replace("/usr/local/lib/nemoclaw/sandbox-rlimits.sh", rlimits) + // Exercise the sandbox-user pass without changing host ownership or limits. + .replace('if [ "$(id -u)" -eq 0 ]; then', "if false; then") + .replace( + "/usr/local/share/nemoclaw/corporate-ca.pem", + path.join(directory, "absent-corporate-ca.pem"), + ) + .replace("local target=/tmp/nemoclaw-proxy-env.sh", `local target="${runtimeEnv}"`) + .replace("/tmp/nemoclaw-proxy-env.XXXXXX", path.join(directory, "proxy-env.XXXXXX")); + fs.writeFileSync(script, source); + return { script, runtimeEnv }; +} + +describe("Pi native network mediation", () => { + it.each([ + { label: "native mediation", append: "", accepted: true }, + { + label: "an HTTP proxy", + append: "export HTTP_PROXY=http://retired-proxy.invalid:3128\n", + accepted: false, + }, + { + label: "a SOCKS proxy", + append: "export ALL_PROXY=socks5://retired-proxy.invalid:1080\n", + accepted: false, + }, + { label: "missing TLS trust", append: "unset SSL_CERT_FILE\n", accepted: false }, + ])("checks the workflow shell with $label", ({ append, accepted }) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-pi-workflow-")); + try { + const { script, runtimeEnv } = piStartupFixture(directory); + const ambient = Object.fromEntries( + PI_PROXY_VARIABLES.map((name) => [name, "http://retired-proxy.invalid:3128"]), + ); + const started = spawnSync("/bin/bash", [script, "/usr/bin/true"], { + env: { + HOME: directory, + PATH: process.env.PATH ?? "", + ...ambient, + ...Object.fromEntries( + [ + "SSL_CERT_FILE", + "CURL_CA_BUNDLE", + "REQUESTS_CA_BUNDLE", + "GIT_SSL_CAINFO", + "NODE_EXTRA_CA_CERTS", + ].map((name) => [name, "/tmp/nemoclaw-ca-bundle.pem"]), + ), + }, + encoding: "utf8", + timeout: 5_000, + }); + expect(started.status, started.stderr).toBe(0); + fs.chmodSync(runtimeEnv, 0o600); + fs.appendFileSync(runtimeEnv, append); + const job = required( + readWorkflow("managed-images.yaml").jobs?.["pi-candidate"], + "Pi job missing", + ); + const command = + step(job, "Exercise the Pi candidate through its declared entrypoint").run ?? ""; + const shell = required( + /docker exec --user 999:999 "\$entrypoint_container" \/bin\/bash -ceu '([\s\S]*?)'\s*$/u.exec( + command, + )?.[1], + "Pi independent-shell probe missing", + ).replace("/tmp/nemoclaw-proxy-env.sh", runtimeEnv); + const checked = spawnSync("/bin/bash", ["-ceu", shell], { + env: { HOME: directory, PATH: process.env.PATH ?? "", ...ambient }, + encoding: "utf8", + timeout: 5_000, + }); + expect(checked.status === 0, checked.stderr).toBe(accepted); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); + + it.each(["startup", "sourced shell"])( + "clears ambient proxy settings in %s and preserves TLS trust", + (mode) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-pi-start-")); + try { + const { script, runtimeEnv } = piStartupFixture(directory); + const trust = { + SSL_CERT_FILE: "/etc/openshell-tls/ca-bundle.pem", + CURL_CA_BUNDLE: "/etc/openshell-tls/ca-bundle.pem", + REQUESTS_CA_BUNDLE: "/etc/openshell-tls/ca-bundle.pem", + GIT_SSL_CAINFO: "/etc/openshell-tls/ca-bundle.pem", + NODE_EXTRA_CA_CERTS: "/etc/openshell-tls/ca-bundle.pem", + }; + const ambient = { + PATH: process.env.PATH ?? "", + ...trust, + ...Object.fromEntries( + PI_PROXY_VARIABLES.map((name) => [name, "http://retired-proxy.invalid:3128"]), + ), + }; + const started = spawnSync("/bin/bash", [script, "/usr/bin/env"], { + env: ambient, + encoding: "utf8", + timeout: 5_000, + }); + expect(started.status, started.stderr).toBe(0); + const sourced = spawnSync( + "/bin/bash", + ["-c", 'source "$1"; exec /usr/bin/env', "pi-shell", runtimeEnv], + { + env: ambient, + encoding: "utf8", + timeout: 5_000, + }, + ); + expect(sourced.status, sourced.stderr).toBe(0); + const output = mode === "startup" ? started.stdout : sourced.stdout; + const environment = Object.fromEntries( + output + .trim() + .split("\n") + .map((line) => { + const separator = line.indexOf("="); + return [line.slice(0, separator), line.slice(separator + 1)]; + }), + ); + expect(PI_PROXY_VARIABLES.filter((name) => Object.hasOwn(environment, name))).toEqual([]); + expect(environment).toMatchObject({ ...trust, PI_OFFLINE: "1", PI_TELEMETRY: "0" }); + expect(fs.statSync(runtimeEnv).mode & 0o777).toBe(0o444); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }, + ); +}); diff --git a/test/agents/openclaw/runtime/provider-placeholder-refresh.test.ts b/test/agents/openclaw/runtime/provider-placeholder-refresh.test.ts index 3846a950bfa..42fe38d397e 100644 --- a/test/agents/openclaw/runtime/provider-placeholder-refresh.test.ts +++ b/test/agents/openclaw/runtime/provider-placeholder-refresh.test.ts @@ -10,6 +10,8 @@ import JSON5 from "json5"; import { hostedNativeProvider } from "../../../../src/lib/inference/native-provider/hosted.ts"; import { describe, expect, it, vi } from "vitest"; import { extractShellFunctionFromSource } from "../../../helpers/shell-source"; +import { buildConfig } from "../../../../scripts/generate-openclaw-config.mts"; +import { baseOpenClawGenerationEnv } from "../../../helpers/openclaw-env-fixture"; const START_SCRIPT = path.resolve(import.meta.dirname, "../../../../scripts/nemoclaw-start.sh"); const JSON5_MODULE = path.resolve(import.meta.dirname, "../../../../nemoclaw/node_modules/json5"); @@ -74,7 +76,7 @@ describe.concurrent("native provider handle refresh", () => { fs.writeFileSync( providerPath, `const fixture = require("./providers.json"); -exports.hostedNativeProvider = () => fixture.openrouter; +exports.fixedHostedNativeProvider = () => fixture.openrouter; exports.NVIDIA_HOSTED_NATIVE_ENDPOINT = fixture.NVIDIA_HOSTED_NATIVE_ENDPOINT; exports.NVIDIA_HOSTED_CREDENTIAL_ENV = fixture.NVIDIA_HOSTED_CREDENTIAL_ENV;`, ); @@ -147,6 +149,29 @@ os.fsync = sync_or_fail` } } + it.each(["v7", `s${"a".repeat(64)}`])( + "materializes the local compatible credential generated without a handle, with scope %s", + async (scope) => { + const handle = `openshell:resolve:env:${scope}_NEMOCLAW_LOCAL_INFERENCE_TOKEN`; + const generated = buildConfig({ + ...baseOpenClawGenerationEnv(), + NEMOCLAW_UPSTREAM_PROVIDER: "compatible-endpoint", + NEMOCLAW_INFERENCE_PROVIDER_ID: "inference", + NEMOCLAW_INFERENCE_BASE_URL: "http://host.openshell.internal:45173/v1", + NEMOCLAW_INFERENCE_API: "openai-completions", + }); + const run = await runRefresh(generated, { NEMOCLAW_LOCAL_INFERENCE_TOKEN: handle }); + expect(run.result.status, run.result.stderr).toBe(0); + expect(run.config.models.providers.inference.apiKey).toBe(handle); + expect(run.config.models.providers.inference.baseUrl).toBe( + "http://host.openshell.internal:45173/v1", + ); + expect(run.config.models.providers.inference.api).toBe("openai-completions"); + expect(JSON.stringify(run.config)).not.toContain("COMPATIBLE_API_KEY"); + expect(run.result.stdout + run.result.stderr).not.toContain(handle); + }, + ); + it.each([false, true])( "refreshes a persisted native NVIDIA handle after provider revision and supervisor restart (root=%s)", async (rootMode) => { diff --git a/test/automation/e2e/brev-launchable-ci-cpu-checksum.test.ts b/test/automation/e2e/brev-launchable-ci-cpu-checksum.test.ts index 0d33cc5cc31..b3ea88ad448 100644 --- a/test/automation/e2e/brev-launchable-ci-cpu-checksum.test.ts +++ b/test/automation/e2e/brev-launchable-ci-cpu-checksum.test.ts @@ -27,7 +27,7 @@ const REVIEWED_NODE_VERSION = REVIEWED_RUNTIME.nodeVersion; const REVIEWED_NPM_VERSION = REVIEWED_RUNTIME.npmVersion; const BREV_LIFECYCLE_SCRIPT_MAX_BYTES = 16 * 1024; const ASSET = "openshell-x86_64-unknown-linux-musl.tar.gz"; -const PINNED_ASSET_SHA256 = "4fb4476d80a1875a0b83547ec3aba999cf0a2e2d75f95f2f709b622e2103520e"; +const PINNED_ASSET_SHA256 = "5c1adfcae797eac9b2e8b9280c66c1f75e4e84d9e31ce6938c5b994314ef6098"; const CALLER_UMASK = process.umask().toString(8).padStart(4, "0"); type FakeSystemOptions = { @@ -96,9 +96,13 @@ function makeFakeSystem(options: FakeSystemOptions): { linkSystemCommands( fakeBin, options.nodeSourceChecksumTool === false - ? ["bash", "basename", "cut", "date", "dirname", "head", "mkdir", "mktemp", "rm", "tee"] + ? ["bash", "basename", "cut", "date", "dirname", "head", "mkdir", "rm", "tee"] : [], ); + writeExecutable( + path.join(fakeBin, "mktemp"), + '#!/bin/sh\nif [ "$#" -eq 0 ] || [ "$*" = "-d" ]; then set -- "$@" "$TMPDIR/brev.XXXXXXXX"; fi\nexec /usr/bin/mktemp "$@"\n', + ); writeExecutable( path.join(fakeBin, "uname"), @@ -293,7 +297,7 @@ done case "$(basename "$out")" in ${ASSET}) tmp="$(mktemp -d)" - printf '#!/usr/bin/env bash\\nprintf "openshell 0.0.116\\\\n"\\n' > "$tmp/openshell" + printf '#!/usr/bin/env bash\\nprintf "openshell 0.1.7\\\\n"\\n' > "$tmp/openshell" chmod +x "$tmp/openshell" /usr/bin/tar -czf "$out" -C "$tmp" openshell rm -rf "$tmp" @@ -360,7 +364,7 @@ function runLaunchable(options: FakeSystemOptions) { ...process.env, LAUNCH_LOG: fake.launchLog, NEMOCLAW_CLONE_DIR: fake.cloneDir, - OPENSHELL_VERSION: options.openshellVersion ?? "v0.0.116", + OPENSHELL_VERSION: options.openshellVersion ?? "v0.1.7", PATH: options.nodeSourceChecksumTool === false ? fake.fakeBin : `${fake.fakeBin}:/usr/bin:/bin`, COMPATIBLE_API_KEY: "host-compatible-key", @@ -676,7 +680,7 @@ describe("brev-launchable-ci-cpu.sh OpenShell checksum gate", { timeout: 30_000 const out = combinedLaunchableOutput(result, fake.launchLog); expect(result.status, out).toBe(1); expect(out).toContain( - `OpenShell release checksum for ${ASSET} does not match NemoClaw-pinned v0.0.116 digest`, + `OpenShell release checksum for ${ASSET} does not match NemoClaw-pinned v0.1.7 digest`, ); expect(fs.existsSync(fake.tarLog) ? fs.readFileSync(fake.tarLog, "utf-8") : "").toBe(""); expect(fs.existsSync(fake.sudoLog) ? fs.readFileSync(fake.sudoLog, "utf-8") : "").not.toMatch( @@ -712,7 +716,7 @@ describe("brev-launchable-ci-cpu.sh OpenShell checksum gate", { timeout: 30_000 try { const out = combinedLaunchableOutput(result, fake.launchLog); expect(result.status, out).toBe(0); - expect(out).toContain("OpenShell CLI installed: openshell 0.0.116"); + expect(out).toContain("OpenShell CLI installed: openshell 0.1.7"); expect(fs.readFileSync(fake.tarLog, "utf-8")).toContain(`xzf`); const sudoLog = fs.readFileSync(fake.sudoLog, "utf-8"); expect(sudoLog).toMatch(/^install -m 755 .*openshell/m); diff --git a/test/automation/e2e/openshell-channel-workflow.test.ts b/test/automation/e2e/openshell-channel-workflow.test.ts index f0a37595475..e16ba46f998 100644 --- a/test/automation/e2e/openshell-channel-workflow.test.ts +++ b/test/automation/e2e/openshell-channel-workflow.test.ts @@ -42,9 +42,9 @@ function resolveLaunchableVersion(options: { channel: string; explicit?: string describe("OpenShell channel workflow boundary", () => { it.each([ - { channel: "stable", expected: "v0.0.116" }, - { channel: "auto", expected: "v0.0.116" }, - { channel: "stable", explicit: "0.0.116", expected: "v0.0.116" }, + { channel: "stable", expected: "v0.1.7" }, + { channel: "auto", expected: "v0.1.7" }, + { channel: "stable", explicit: "0.1.7", expected: "v0.1.7" }, ])("resolves launchable channel $channel to $expected", ({ channel, explicit, expected }) => { const result = resolveLaunchableVersion({ channel, explicit }); expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0); @@ -61,11 +61,11 @@ describe("OpenShell channel workflow boundary", () => { it.each([ { channel: "dev" }, - { channel: "dev", explicit: "v0.0.116" }, + { channel: "dev", explicit: "v0.1.7" }, { channel: "stable", explicit: "v9.9.9" }, ])("rejects unsupported launchable OpenShell selection %#", ({ channel, explicit }) => { const result = resolveLaunchableVersion({ channel, explicit }); expect(result.status).not.toBe(0); - expect(`${result.stdout}${result.stderr}`).toContain("requires exact stable OpenShell 0.0.116"); + expect(`${result.stdout}${result.stderr}`).toContain("requires exact stable OpenShell 0.1.7"); }); }); diff --git a/test/cli/config-rotate-token-update-only.test.ts b/test/cli/config-rotate-token-update-only.test.ts index 13ed8916adc..64b879b0da2 100644 --- a/test/cli/config-rotate-token-update-only.test.ts +++ b/test/cli/config-rotate-token-update-only.test.ts @@ -37,7 +37,7 @@ describe("config rotate-token CLI", () => { "#!/bin/sh", 'printf \'%s\\n\' "$*" >> "$OPENSHELL_TEST_COMMAND_LOG"', 'if [ "$1" = "--version" ]; then', - " printf 'openshell 0.0.116\\n'", + " printf 'openshell 0.1.7\\n'", " exit 0", "fi", 'if [ "$1" = "provider" ] && [ "$2" = "update" ]; then', diff --git a/test/cli/destroy-gateway-cleanup.test.ts b/test/cli/destroy-gateway-cleanup.test.ts index 66ff0756fe6..08b2eb93aef 100644 --- a/test/cli/destroy-gateway-cleanup.test.ts +++ b/test/cli/destroy-gateway-cleanup.test.ts @@ -22,7 +22,7 @@ case "$1" in for arg do last_arg="$arg"; done case "$last_arg" in '{{.ID}}') printf 'aaaaaaaaaaaa\n' ;; - *) printf 'aaaaaaaaaaaa\topenshell\tdefault\tsb-alpha\n' ;; + *) printf 'aaaaaaaaaaaa\topenshell\tdefault\tsb-alpha\t\t\n' ;; esac fi ;; diff --git a/test/cli/exit-code-user-error-surfaces.test.ts b/test/cli/exit-code-user-error-surfaces.test.ts index fa04cf99a53..64f87f5cd1a 100644 --- a/test/cli/exit-code-user-error-surfaces.test.ts +++ b/test/cli/exit-code-user-error-surfaces.test.ts @@ -269,7 +269,7 @@ describe("onboard dashboard-port exhaustion exits non-zero (#5974)", () => { 'if [ "${1:-}" = gateway ] && [ "${2:-}" = info ]; then printf "No gateway metadata found\\n"; exit 1; fi', 'if [ "${1:-}" = gateway ] && [ "${2:-}" = list ]; then printf "[]\\n"; exit 0; fi', 'case "${1:-}" in', - ' -V|--version) printf "%s 0.0.116\\n" "${0##*/}"; exit 0;;', + ' -V|--version) printf "%s 0.1.7\\n" "${0##*/}"; exit 0;;', "esac", "exit 1", ].join("\n"), @@ -337,6 +337,8 @@ describe("onboard dashboard-port exhaustion exits non-zero (#5974)", () => { NEMOCLAW_TEST_NO_SLEEP: "1", NEMOCLAW_STATUS_PROBE_TIMEOUT_MS: "2000", NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1", + // This fixture tests occupied ports without contacting a public DNS resolver. + NEMOCLAW_SKIP_HOST_DNS_PREFLIGHT: "1", NEMOCLAW_GATEWAY_PORT: String(gatewayPort), NEMOCLAW_OPENSHELL_BIN: path.join(binDir, "openshell"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", diff --git a/test/cli/helpers.test.ts b/test/cli/helpers.test.ts index feb3329b28e..ca100349475 100644 --- a/test/cli/helpers.test.ts +++ b/test/cli/helpers.test.ts @@ -250,6 +250,64 @@ describe("source-loader Node options", () => { ).rejects.toBe(cleanupError); }); + it("preserves output and status for an ordinary async CLI failure", async () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-cli-exit-status-")); + tempDirs.add(directory); + const script = path.join(directory, "exit.cjs"); + fs.writeFileSync(script, 'process.stderr.write("expected failure\\n"); process.exitCode = 3;'); + + const result = await runCliScriptAsync(script, "", { env: { NODE_OPTIONS: "" } }); + + expect(result).toEqual({ code: 3, out: "expected failure\n" }); + }); + + it("reports signal termination without treating emitted output as success", async () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-cli-signal-")); + tempDirs.add(directory); + const script = path.join(directory, "signal.cjs"); + fs.writeFileSync( + script, + 'process.stdout.write("before signal\\n", () => process.kill(process.pid, "SIGTERM"));', + ); + + const result = await runCliScriptAsync(script, "", { env: { NODE_OPTIONS: "" } }); + + expect(result.code).not.toBe(0); + expect(result.out).toContain("before signal\n"); + expect(result.out).toContain( + 'Child termination: {"code":null,"signal":"SIGTERM","killed":false}', + ); + }); + + it("reports timeout termination after output and removes the implicit HOME", async () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-cli-async-timeout-")); + tempDirs.add(directory); + const script = path.join(directory, "wait.cjs"); + const marker = path.join(directory, "home.txt"); + fs.writeFileSync( + script, + [ + `require("node:fs").writeFileSync(${JSON.stringify(marker)}, process.env.HOME);`, + 'process.stderr.write("expected diagnostic\\n");', + "setInterval(() => {}, 1000);", + ].join("\n"), + ); + + const result = await runCliScriptAsync(script, "", { + env: { NODE_OPTIONS: "" }, + timeout: 2_000, + }); + + expect(result.code).not.toBe(0); + expect(result.out).toContain("expected diagnostic\n"); + expect(result.out).toContain( + 'Child termination: {"code":null,"signal":"SIGTERM","killed":true}', + ); + const implicitHome = fs.readFileSync(marker, "utf8"); + expect(path.isAbsolute(implicitHome)).toBe(true); + expect(fs.existsSync(implicitHome)).toBe(false); + }); + it("quotes preload paths that contain spaces for Node (#6245)", () => { const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw node options ")); tempDirs.add(directory); diff --git a/test/cli/helpers.ts b/test/cli/helpers.ts index 85fd55b13b9..f1094b91d5e 100644 --- a/test/cli/helpers.ts +++ b/test/cli/helpers.ts @@ -297,7 +297,14 @@ async function runWithEnvInternalAsync( resolve({ code, out: mergeStderrOnSuccess ? `${stdout}${stderr}` : stdout }); return; } - const errorOutput = error && typeof error.code !== "number" ? String(error) : ""; + const errorOutput = + error && typeof error.code !== "number" + ? `${String(error)}\nChild termination: ${JSON.stringify({ + code: error.code == null ? null : String(error.code).slice(0, 64), + signal: error.signal == null ? null : String(error.signal).slice(0, 64), + killed: error.killed === true, + })}` + : ""; resolve({ code, out: `${stdout}${stderr}${errorOutput}` }); }, ); diff --git a/test/credentials/credential-migration-reconciliation.test.ts b/test/credentials/credential-migration-reconciliation.test.ts index 39491d72d84..7c41858616d 100644 --- a/test/credentials/credential-migration-reconciliation.test.ts +++ b/test/credentials/credential-migration-reconciliation.test.ts @@ -6,6 +6,7 @@ import os from "node:os"; import path from "node:path"; import { describe, expect, it, vi } from "vitest"; +import { mockManagedEndpointlessProviderProfileRun } from "../helpers/onboard-script-mocks.cjs"; import { removeLegacyCredentialsFile, resolveProviderCredential, @@ -23,13 +24,6 @@ import type { Session } from "../../src/lib/state/onboard-session.js"; import { withProcessEnv } from "../support/setup-inference-test-harness.js"; const LEGACY_SECRET = "sk-TEST-NOT-A-REAL-STORED-KEY"; -const MESSAGING_PROFILE = { - id: MESSAGING_CREDENTIAL_PROVIDER_TYPE, - credentials: [], - endpoints: [], - binaries: [], - inference_capable: false, -}; type RegistrationAttempt = ReturnType< ReturnType["stageSandboxCredentialProviders"] @@ -192,7 +186,7 @@ describe("legacy credential reconciliation", () => { const isCreate = args[0] === "provider" && args[1] === "create"; const result = args.includes("profile") && args.includes("export") - ? { ...success, stdout: JSON.stringify(MESSAGING_PROFILE) } + ? mockManagedEndpointlessProviderProfileRun(args.join(" "))! : args.join(" ") === providerGetArgs.join(" ") ? providerCreated ? { @@ -351,7 +345,7 @@ describe("legacy credential reconciliation", () => { const runOpenshell = vi.fn((args: string[]) => { const result = args.includes("profile") && args.includes("export") - ? { status: 0, stdout: JSON.stringify(MESSAGING_PROFILE), stderr: "" } + ? mockManagedEndpointlessProviderProfileRun(args.join(" "))! : args[1] === "get" ? providerExists ? metadata diff --git a/test/credentials/openshell-credential-generation-window.test.ts b/test/credentials/openshell-credential-generation-window.test.ts index 77a71f7e08f..f3399788df2 100644 --- a/test/credentials/openshell-credential-generation-window.test.ts +++ b/test/credentials/openshell-credential-generation-window.test.ts @@ -5,7 +5,10 @@ import fs from "node:fs"; import { runInNewContext } from "node:vm"; import { describe, expect, it } from "vitest"; -import { credentialWindowHttpFixture } from "../helpers/credential-window-http-fixture.ts"; +import { + credentialWindowHttpFixture, + runCredentialWindowSessionFixture, +} from "../helpers/credential-window-http-fixture.ts"; import { MCP_BRIDGE_TEST_CREDENTIALS } from "../e2e/fixtures/mcp-bridge-credentials.ts"; import { @@ -23,12 +26,46 @@ import { } from "../e2e/live/openshell-credential-generation-window.ts"; describe("OpenShell exact-main credential generation-window proof", () => { + it.each(["retained", "fresh"] as const)( + "negotiates the real MCP fixture session for the %s credential probe", + async (scenario) => { + const result = await runCredentialWindowSessionFixture(scenario); + expect(result.exitCode).toBe(0); + expect(result.requests.map((request) => JSON.parse(request.body).method)).toEqual([ + "initialize", + "notifications/initialized", + "tools/list", + ]); + expect(result.requests.slice(1)).toMatchObject([ + { sessionId: "fake-session-1", protocolVersion: "2025-11-25", responseStatus: 202 }, + { sessionId: "fake-session-1", protocolVersion: "2025-11-25", responseStatus: 200 }, + ]); + expect(JSON.parse(result.summary)).toMatchObject( + scenario === "fresh" + ? { revision: "v42", status: 200 } + : { revision: "v42", outcomes: [{ step: "allowed-before-expiry", outcome: "allowed" }] }, + ); + }, + ); + + it("does not signal credential readiness when MCP initialization rejects authentication", async () => { + const result = await runCredentialWindowSessionFixture("retained", "wrong"); + expect(result.exitCode).toBe(1); + expect(result.writes.has(CREDENTIAL_WINDOW_PATHS.ready)).toBe(false); + expect(result.summary).toBe(""); + expect(result.diagnostic).toContain("MCP session negotiation failed"); + expect(result.requests.map((request) => JSON.parse(request.body).method)).toEqual([ + "initialize", + ]); + }); + it.each([200, 401, "ECONNRESET", "private-error-secret"])( "records bounded request evidence for %s without changing the acknowledgement", async (result) => { const writes = new Map(); let diagnostic = ""; let summary = ""; + const http = credentialWindowHttpFixture(result); const script = buildCredentialWindowChildScript({ mcpUrl: "https://credential-window.example.test/mcp", maxRuntimeMs: 3000, @@ -63,11 +100,22 @@ describe("OpenShell exact-main credential generation-window proof", () => { writeFileSync: (file: string, value: string) => writes.set(file, value), }, ], - ["node:https", credentialWindowHttpFixture(result)], + ["node:https", http], ]).get(name), }); const outcome = result === 200 ? "allowed" : "denied"; const expected = { step: CREDENTIAL_WINDOW_STEPS.allowedBeforeExpiry, outcome }; + expect(http.requests).toMatchObject([ + { headers: { authorization: "Bearer openshell:resolve:env:v42_FAKE_MCP_SECRET" } }, + { headers: { "mcp-session-id": "fixture-session", "mcp-protocol-version": "2025-11-25" } }, + { + headers: { + "mcp-session-id": "fixture-session", + "mcp-protocol-version": "2025-11-25", + authorization: "Bearer openshell:resolve:env:v42_FAKE_MCP_SECRET", + }, + }, + ]); expect(JSON.parse(writes.get(CREDENTIAL_WINDOW_PATHS.acknowledgement)!)).toEqual(expected); expect(JSON.parse(summary)).toEqual({ revision: "v42", outcomes: [expected] }); expect(JSON.parse(diagnostic)).toEqual({ @@ -102,22 +150,21 @@ describe("OpenShell exact-main credential generation-window proof", () => { expect(script.split(snapshot)).toHaveLength(2); expect(script.indexOf(snapshot)).toBeLessThan(script.indexOf("while (Date.now() < deadline")); expect(script).toContain('"^openshell:resolve:env:(v[0-9]{1,20})_" + config.envName + "$"'); - expect(script).toContain('authorization: "Bearer " + credentialPlaceholder'); - expect(script).toContain('response.statusCode === 200 ? "allowed" : "denied"'); + expect(script).toContain('response.status === 200 ? "allowed" : "denied"'); expect(script).toContain('transportCode: knownCodes.has(error.code) ? error.code : "OTHER"'); - expect(script).toContain("outbound.setTimeout(30_000"); + expect(script).toContain("request.setTimeout(30_000"); expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_PATHS.acknowledgement)); expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.allowedBeforeExpiry)); expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.deniedAfterExpiry)); expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.fallbackAfterEviction)); - expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.deniedAfterKeyRemoval)); + expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.deniedAfterCurrentExpiry)); expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.deniedAfterDetach)); expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.deniedAfterReadd)); expect(script).toContain(JSON.stringify(CREDENTIAL_WINDOW_STEPS.stop)); expect(script).not.toContain(MCP_BRIDGE_TEST_CREDENTIALS.generationWindow); }); - it("builds explicit bounded expiry and attached-key-removal updates", () => { + it("builds explicit credential expiry and expiry-clearing updates", () => { expect(CREDENTIAL_WINDOW_EXPIRY_DELAY_MS).toBe(3 * 60_000); expect(buildCredentialWindowProviderUpdateArgs("owned-provider", 123_456)).toEqual([ "provider", @@ -128,25 +175,53 @@ describe("OpenShell exact-main credential generation-window proof", () => { "--credential-expires-at", "FAKE_MCP_SECRET=123456", ]); - expect(buildCredentialWindowProviderUpdateArgs("owned-provider", 0, true)).toEqual([ + expect(buildCredentialWindowProviderUpdateArgs("owned-provider", 0)).toEqual([ "provider", "update", "owned-provider", "--credential", - "FAKE_MCP_SECRET=", + "FAKE_MCP_SECRET", "--credential-expires-at", "FAKE_MCP_SECRET=0", ]); }); - it("keeps fresh-exec requests revision-scoped and independently identifiable", () => { + it("keeps fresh-exec requests revision-scoped and independently identifiable", async () => { const script = buildCredentialWindowOneShotScript(); + const http = credentialWindowHttpFixture(200); + let summary = ""; + await runInNewContext(script, { + Buffer, + URL, + process: { + argv: ["node", "https://credential-window.example.test/mcp", "fresh-request"], + env: { FAKE_MCP_SECRET: "openshell:resolve:env:v43_FAKE_MCP_SECRET" }, + stderr: { write: () => {} }, + stdout: { + write: (value: string) => { + summary += value; + }, + }, + }, + require: (name: string) => (name === "node:https" ? http : undefined), + }); + expect(http.requests).toMatchObject([ + { headers: { authorization: "Bearer openshell:resolve:env:v43_FAKE_MCP_SECRET" } }, + { headers: { "mcp-session-id": "fixture-session", "mcp-protocol-version": "2025-11-25" } }, + { + headers: { + "mcp-session-id": "fixture-session", + "mcp-protocol-version": "2025-11-25", + authorization: "Bearer openshell:resolve:env:v43_FAKE_MCP_SECRET", + }, + }, + ]); + expect(JSON.parse(summary)).toEqual({ revision: "v43", status: 200 }); expect(CREDENTIAL_WINDOW_ENV_NAME).toBe("FAKE_MCP_SECRET"); expect(script).toContain("process.argv[1]"); expect(script).toContain("process.argv[2]"); expect(script).toContain('"^openshell:resolve:env:(v[0-9]{1,20})_" + config.envName + "$"'); - expect(script).toContain('authorization: "Bearer " + credentialPlaceholder'); expect(script).toContain("request.setTimeout(30_000"); expect(credentialWindowRequestId(CREDENTIAL_WINDOW_STEPS.fallbackAfterEviction)).toBe( "nemoclaw-credential-window:fallback-after-eviction", @@ -154,6 +229,42 @@ describe("OpenShell exact-main credential generation-window proof", () => { expect(script).not.toContain(MCP_BRIDGE_TEST_CREDENTIALS.generationWindow); }); + it.each([401, "private-error-secret"])( + "fails the fresh request with bounded diagnostics for %s", + async (result) => { + const http = credentialWindowHttpFixture(result); + let summary = ""; + let diagnostic = ""; + const child = { + argv: ["node", "https://credential-window.example.test/mcp", "fresh-request"], + env: { FAKE_MCP_SECRET: "openshell:resolve:env:v43_FAKE_MCP_SECRET" }, + stdout: { + write: (value: string) => { + summary += value; + }, + }, + stderr: { + write: (value: string) => { + diagnostic += value; + }, + }, + exitCode: 0, + }; + await runInNewContext(buildCredentialWindowOneShotScript(), { + Buffer, + URL, + process: child, + require: () => http, + }); + expect(child.exitCode).toBe(1); + expect(http.requests).toHaveLength(3); + expect(summary).toBe(typeof result === "number" ? '{"revision":"v43","status":401}\n' : ""); + expect(diagnostic).toBe(typeof result === "number" ? "" : "MCP credential request failed\n"); + expect(diagnostic).not.toContain("private-error-secret"); + expect(diagnostic).not.toContain("openshell:resolve:"); + }, + ); + it("keeps the live target on the reviewed agent and mutation boundaries", () => { const liveTarget = fs.readFileSync( "test/e2e/live/openshell-credential-generation-window.test.ts", @@ -163,19 +274,19 @@ describe("OpenShell exact-main credential generation-window proof", () => { expect(liveTarget).toContain('NEMOCLAW_AGENT: "openclaw"'); expect(liveTarget).toContain('["nemoclaw-start", "node", "-e"'); expect(liveTarget).toContain("CREDENTIAL_WINDOW_STEPS.deniedAfterExpiry"); - expect(liveTarget).toContain("CREDENTIAL_WINDOW_STEPS.deniedAfterKeyRemoval"); + expect(liveTarget).toContain("CREDENTIAL_WINDOW_STEPS.deniedAfterCurrentExpiry"); expect(liveTarget).toContain('[SANDBOX_NAME, "mcp", "remove", SERVER_NAME]'); expect(liveTarget).toMatch(/\[\s*SANDBOX_NAME,\s*"mcp",\s*"add",\s*SERVER_NAME,/u); expect(liveTarget).not.toContain('["sandbox", "provider", "detach"'); expect(liveTarget).toContain('[SANDBOX_NAME, "mcp", "restart", SERVER_NAME]'); expect(liveTarget).toContain('[SANDBOX_NAME, "rebuild", "--yes"]'); expect(liveTarget).toContain('[SANDBOX_NAME, "rebuild", "--yes", "--force"]'); - expect(liveTarget).toContain("HOSTED_INFERENCE_PROVIDER_NAME"); - expect(liveTarget).toContain("HOSTED_INFERENCE_CREDENTIAL_ENV"); + expect(liveTarget).toContain("normalizeNativeLocalProviderAttachment("); + expect(liveTarget).toContain("inferenceAttachment!.providerName"); + expect(liveTarget).toContain("inferenceAttachment!.credentialEnv"); expect(liveTarget).toContain("credential-window-source-sandbox-after-expired-rebuild"); expect(liveTarget).toContain("parseOpenShellSandboxId"); - expect(liveTarget).toContain("Backing up sandbox state|Deleting old sandbox"); - expect(liveTarget).toContain('expect(providerName).toBe("e2e-cred-window-mcp-fake")'); + expect(liveTarget).toContain('providerName: "e2e-cred-window-mcp-fake"'); expect(liveTarget).toContain('!request.auth.includes("openshell:resolve:env")'); }); }); diff --git a/test/credentials/rebuild-credential-preflight.test.ts b/test/credentials/rebuild-credential-preflight.test.ts index eb6eadae7c6..daa06095f90 100644 --- a/test/credentials/rebuild-credential-preflight.test.ts +++ b/test/credentials/rebuild-credential-preflight.test.ts @@ -15,6 +15,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it } from "vitest"; +import { customProviderRebuildEntry } from "../../src/lib/actions/sandbox/rebuild-flow-test-fixtures"; import { execTimeout, testTimeoutOptions } from "../helpers/timeouts"; const REPO_ROOT = path.join(import.meta.dirname, "../.."); @@ -51,6 +52,17 @@ function createFixture(opts: { providerRegistered = true, inferenceProbeHttpStatus = null, } = opts; + const nativeSelection = + provider === "compatible-endpoint" ? customProviderRebuildEntry() : undefined; + const attachment = nativeSelection?.nativeHostedProviderAttachment; + const providerName = attachment?.providerName ?? provider; + const providerMetadata = [ + `Name: ${providerName}`, + `Type: ${attachment?.profileId ?? "openai"}`, + `Credential keys: ${credentialEnv}`, + `Config keys: ${attachment ? "" : "OPENAI_BASE_URL"}`, + ...(attachment ? [`Id: ${attachment.providerId}`, "Resource version: 1"] : []), + ].join("\n"); const sandboxName = "my-assistant"; const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-2273-")); tmpFixtures.push(tmpDir); @@ -100,6 +112,7 @@ wait();`, dashboardPort: agent === "langchain-deepagents-code" ? 0 : 18789, fromDockerfile: null, agent, + ...nativeSelection, ...(agent === "langchain-deepagents-code" ? { credentialEnv, @@ -179,7 +192,7 @@ wait();`, const fs = require("fs"); const a = process.argv.slice(2); const requiredFeatures = "request-body-credential-rewrite websocket-credential-rewrite allow_all_known_mcp_methods"; -if (a[0] === "-V" || a[0] === "--version") { process.stdout.write("openshell 0.0.116\\n"); process.exit(0); } +if (a[0] === "-V" || a[0] === "--version") { process.stdout.write("openshell 0.1.7\\n"); process.exit(0); } if (a[0] === "sandbox" && a[1] === "list") { process.stdout.write("${sandboxName} Ready\\n"); process.exit(0); } if (a[0] === "sandbox" && a[1] === "ssh-config") { process.stdout.write("${sshConfig}\\n"); process.exit(0); } if (a[0] === "sandbox" && a[1] === "get") { @@ -197,7 +210,7 @@ if (a[0] === "sandbox" && a[1] === "exec") { process.stdout.write(fs.readFileSync(${JSON.stringify(atomicityMarker)}, "utf-8")); process.exit(0); } - if (command.includes("https://inference.local/")) { + if (command.includes(${JSON.stringify(nativeSelection?.endpointUrl ?? "https://inference.local/")})) { const probeStatus = ${String(inferenceProbeHttpStatus ?? 200)}; const isManagedDcodeProbe = command.includes("/usr/local/lib/nemoclaw/dcode-managed-exec"); process.stdout.write((isManagedDcodeProbe ? "" : "__NEMOCLAW_SANDBOX_EXEC_STARTED__\\n") + probeStatus + "\\n"); @@ -221,11 +234,11 @@ if (a[0] === "provider" && a[1] === "get") { process.stderr.write("Error: provider '${provider}' not found\\n"); process.exit(1); } - process.stdout.write("Name: ${provider}\\nType: openai\\nCredential keys: ${credentialEnv}\\nConfig keys: OPENAI_BASE_URL\\n"); + process.stdout.write(${JSON.stringify(providerMetadata + "\n")}); process.exit(0); } if (a[0] === "provider" && a[1] === "list") { - process.stdout.write(${JSON.stringify(JSON.stringify(providerRegistered ? [{ name: provider, credential_keys: [credentialEnv], credential_expires_at_ms: { [credentialEnv]: opts.credentialExpiresAtMs } }] : []) + "\n")}); + process.stdout.write(${JSON.stringify(JSON.stringify({ providers: providerRegistered ? [{ name: providerName, credential_keys: [credentialEnv], credential_expires_at_ms: { [credentialEnv]: opts.credentialExpiresAtMs } }] : [], next_page_token: "" }) + "\n")}); process.exit(0); } if (a[0] === "provider") process.exit(0); @@ -241,7 +254,7 @@ process.exit(0); path.join(tmpDir, component), `#!/usr/bin/env node const requiredFeatures = "request-body-credential-rewrite websocket-credential-rewrite allow_all_known_mcp_methods"; -if (process.argv[2] === "-V" || process.argv[2] === "--version") process.stdout.write("${component} 0.0.116\\n"); +if (process.argv[2] === "-V" || process.argv[2] === "--version") process.stdout.write("${component} 0.1.7\\n"); process.exit(0); `, { mode: 0o755 }, @@ -310,7 +323,7 @@ if (a[0] === "inspect") { const format = formatIndex >= 0 ? a[formatIndex + 1] : ""; if (format === "{{.State.Running}}") process.stdout.write("true\\n"); if (format === "{{json .NetworkSettings.Ports}}") process.stdout.write(JSON.stringify({"${gatewayPort}/tcp":[{HostPort:"${gatewayPort}"}]}) + "\\n"); - if (format === "{{.Config.Image}}") process.stdout.write("nvcr.io/nvidia/openshell/cluster:0.0.116\\n"); + if (format === "{{.Config.Image}}") process.stdout.write("nvcr.io/nvidia/openshell/cluster:0.1.7\\n"); process.exit(0); } if (a[0] === "ps") process.exit(0); diff --git a/test/e2e-runtime/brev-launchable-gateway-diagnostics.test.ts b/test/e2e-runtime/brev-launchable-gateway-diagnostics.test.ts index 977cd49256a..225f4de9a04 100644 --- a/test/e2e-runtime/brev-launchable-gateway-diagnostics.test.ts +++ b/test/e2e-runtime/brev-launchable-gateway-diagnostics.test.ts @@ -483,4 +483,37 @@ describe("focused staging Brev Launchable failure diagnostics", () => { status: "ABSENT", }); }); + + it("keeps the E2E failure and cleanup when the diagnostic budget expires (#6409)", () => { + const { calls, env, state, workDir } = fixture({ + e2eDiagnosticTimesOut: true, + e2eFails: true, + }); + const result = run({ + ...env, + FULL_E2E_FAILURE_DIAGNOSTIC_TIMEOUT_SECONDS: "1", + }); + + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("full E2E failed"); + const laneLog = fs.readFileSync(path.join(workDir, "lane.log"), "utf8"); + expect(laneLog).toContain( + "Full E2E failure diagnostic gateway state: status 124; output: probe timed out", + ); + expect(laneLog).toContain( + "Full E2E failure diagnostic platform state: not run; output: diagnostic budget exhausted", + ); + expect(laneLog).toContain( + "Full E2E failure diagnostic declared gateway listener: not run; output: diagnostic budget exhausted", + ); + const commands = fs.readFileSync(calls, "utf8"); + expect(commands).not.toContain("ssh full-e2e diagnostic platform state"); + expect(commands.indexOf("ExecMainCode")).toBeLessThan( + commands.indexOf("brev delete nclaw-e2e-test-1"), + ); + expect(fs.existsSync(state)).toBe(false); + expect(JSON.parse(fs.readFileSync(path.join(workDir, "cleanup.json"), "utf8"))).toMatchObject({ + status: "ABSENT", + }); + }); }); diff --git a/test/e2e-runtime/candidate-compat.test.ts b/test/e2e-runtime/candidate-compat.test.ts index d175fbe9c56..590e311489a 100644 --- a/test/e2e-runtime/candidate-compat.test.ts +++ b/test/e2e-runtime/candidate-compat.test.ts @@ -3,7 +3,15 @@ import { spawnSync } from "node:child_process"; import { createHash } from "node:crypto"; -import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { + chmodSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + symlinkSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { basename, join, resolve } from "node:path"; @@ -25,7 +33,7 @@ import { } from "../../tools/candidate-compat.mts"; const SHA = "a".repeat(40); -const VERSION = "0.0.116"; +const VERSION = "0.1.7"; const E2E_SOURCES = readFileSync(resolve(".github/workflows/e2e.yaml"), "utf8"); const ASSETS = [ { name: "openshell-x86_64-unknown-linux-musl.tar.gz", role: "cli" as const }, @@ -383,8 +391,15 @@ describe("OpenShell candidate compatibility contract", () => { async (url) => new Response(archives.get(url)), ); const log = join(directory, "invocations.log"); + const tools = join(directory, "tools"); + mkdirSync(tools); + symlinkSync(process.execPath, join(tools, "node")); const env = { ...process.env, + HOME: join(directory, "home"), + XDG_CONFIG_HOME: join(directory, "config"), + XDG_BIN_HOME: join(directory, "install-bin"), + OPENSHELL_SYSTEM_GATEWAY_DIR: join(directory, "system-gateways"), NEMOCLAW_CANDIDATE_INVOCATION_CONTEXT: `installer:${candidateReceipt.resolutionId}`, NEMOCLAW_CANDIDATE_INVOCATION_LOG: log, NEMOCLAW_OPENSHELL_CHANNEL: "stable", @@ -393,7 +408,7 @@ describe("OpenShell candidate compatibility contract", () => { NEMOCLAW_OPENSHELL_MIN_VERSION: VERSION, NEMOCLAW_OPENSHELL_PIN_VERSION: VERSION, NEMOCLAW_OPENSHELL_SANDBOX_BIN: join(observed.binDirectory, "openshell-sandbox"), - PATH: `${observed.binDirectory}:/usr/bin:/bin`, + PATH: `${observed.binDirectory}:${tools}:/usr/bin:/bin`, }; const installer = spawnSync("bash", [resolve("scripts/install-openshell.sh")], { encoding: "utf8", diff --git a/test/e2e-runtime/runner.test.ts b/test/e2e-runtime/runner.test.ts index 2dffc49c49c..2f11c270c8f 100644 --- a/test/e2e-runtime/runner.test.ts +++ b/test/e2e-runtime/runner.test.ts @@ -10,20 +10,12 @@ import path from "node:path"; import { describe, expect, it, vi } from "vitest"; import { redact, run, runCapture, runCaptureEx } from "../../src/lib/runner"; +import { PINNED_OPEN_SHELL_SHA256 } from "../helpers/openshell-release-fixtures"; +import { prepareInstallerBin } from "../support/openshell-installed-version"; const require = createRequire(import.meta.url); const runnerPath = path.join(import.meta.dirname, "..", "..", "src", "lib", "runner.ts"); const platformPath = path.join(import.meta.dirname, "..", "..", "src", "lib", "platform.ts"); -const PINNED_OPEN_SHELL_SHA256 = { - cliDarwinArm64: "e582f2374053bebac8e6aaeb4a369931b7d4bb97bd55055e2c02e85502627e22", - cliLinuxArm64: "7a949c48d1e000cd280869eea1e203e24816b9cfefc575b68a8b72b939cb3f43", - cliLinuxX64: "4fb4476d80a1875a0b83547ec3aba999cf0a2e2d75f95f2f709b622e2103520e", - gatewayDarwinArm64: "f192d3d737c125264e13ef73458541df2ca6a9eb2fa599736a7f2587d5d2ce8d", - gatewayLinuxArm64: "292c379193a339220234ffea585350901468bb8f4076e2076bc074e8ed18974b", - gatewayLinuxX64: "59c6da724eae7a00c28826f9191efbdf4fbaa5c768afdc8dea6a80a949ebcc89", - sandboxLinuxArm64: "959d9a88270e0336f04342560df750591da603424d0a9bfb481ee29670342557", - sandboxLinuxX64: "0bb160f73e5007338b94e3c868f66f50c71cd65c27c932ed9a4fa67c49e6d423", -}; type SpawnCallOptions = { stdio?: StdioOptions; @@ -1098,7 +1090,9 @@ describe("regression guards", () => { "scripts", "install-openshell.sh", ); - const tmpBin = fs.mkdtempSync(path.join(os.tmpdir(), "gh-absent-")); + const root = fs.mkdtempSync(path.join(os.tmpdir(), "gh-absent-")); + const tmpBin = path.join(root, "bin"); + prepareInstallerBin(tmpBin); const stub = ` #!/usr/bin/env bash printf '%s\n' '#!/bin/sh' 'echo "openshell 0.0.1"' > "${tmpBin}/openshell" @@ -1171,7 +1165,7 @@ describe("regression guards", () => { shift || true done [ -n "$destination" ] || return 2 - printf '%s\n' '#!/bin/sh' 'echo "0.0.116"' > "$destination/$expected" + printf '%s\n' '#!/bin/sh' 'echo "0.1.7"' > "$destination/$expected" chmod +x "$destination/$expected" ;; *) return 2 ;; @@ -1183,6 +1177,13 @@ describe("regression guards", () => { try { const result = spawnSync("bash", ["-c", stub], { encoding: "utf-8", + env: { + ...process.env, + HOME: path.join(tmpBin, "home"), + XDG_CONFIG_HOME: path.join(tmpBin, "config"), + XDG_BIN_HOME: path.join(tmpBin, "install-bin"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmpBin, "system-gateways"), + }, timeout: 5000, }); const out = (result.stdout || "") + (result.stderr || ""); @@ -1190,7 +1191,7 @@ describe("regression guards", () => { expect(out).toContain("CURL_DIRECT"); expect(out).not.toContain("gh CLI download failed"); } finally { - fs.rmSync(tmpBin, { recursive: true, force: true }); + fs.rmSync(root, { recursive: true, force: true }); } }); @@ -1202,7 +1203,9 @@ describe("regression guards", () => { "scripts", "install-openshell.sh", ); - const tmpBin = fs.mkdtempSync(path.join(os.tmpdir(), "gh-stub-")); + const root = fs.mkdtempSync(path.join(os.tmpdir(), "gh-stub-")); + const tmpBin = path.join(root, "bin"); + prepareInstallerBin(tmpBin); const checksumLog = path.join(tmpBin, "sha256sum.log"); const ghStub = path.join(tmpBin, "gh"); fs.writeFileSync(ghStub, "#!/bin/sh\nexit 4\n"); @@ -1279,7 +1282,7 @@ describe("regression guards", () => { shift || true done [ -n "$destination" ] || return 2 - printf '%s\n' '#!/bin/sh' 'echo "0.0.116"' > "$destination/$expected" + printf '%s\n' '#!/bin/sh' 'echo "0.1.7"' > "$destination/$expected" chmod +x "$destination/$expected" ;; *) return 2 ;; @@ -1291,6 +1294,13 @@ describe("regression guards", () => { try { const result = spawnSync("bash", ["-c", stub], { encoding: "utf-8", + env: { + ...process.env, + HOME: path.join(tmpBin, "home"), + XDG_CONFIG_HOME: path.join(tmpBin, "config"), + XDG_BIN_HOME: path.join(tmpBin, "install-bin"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmpBin, "system-gateways"), + }, timeout: 5000, }); const out = (result.stdout || "") + (result.stderr || ""); @@ -1298,7 +1308,7 @@ describe("regression guards", () => { expect(out).toContain("CURL_FALLBACK"); expect(fs.readFileSync(checksumLog, "utf-8")).toContain("SHA256SUM -c -"); } finally { - fs.rmSync(tmpBin, { recursive: true, force: true }); + fs.rmSync(root, { recursive: true, force: true }); } }); }); diff --git a/test/e2e/README.md b/test/e2e/README.md index 2e1269a343e..86ea8d96192 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -80,7 +80,7 @@ Earlier setup failures preserve existing policy while disposable-provider remova Removal preserves shared gateway credentials and unrelated sandbox attachments. Adding the preset alone restores its direct network rule; provider access requires an explicit attachment. A raw OpenShell attachment is temporary: it leaves the saved opt-out unchanged, so rebuild omits Tavily again. -Linux x86_64 qualification uses the existing target. WSL ARM qualification remains manual; +The existing Linux x86_64 target runs the credential-free check. WSL ARM qualification remains manual; there is no supported WSL ARM selector for this scenario. The policy live-state tests own attachment discovery, identity checks, partial failures, and operator-rule preservation. The extra-provider planning and recreate tests own rebuild exclusion and explicit re-enable behavior. @@ -408,6 +408,11 @@ PR image publication builds each shipped agent on native AMD64 and ARM64 runners The workflow retains PR all-agent image contracts for seven days so manual PR E2E can bind them after review. An expired contract blocks live E2E before candidate execution. Each platform retains its own digest contract and runs the direct startup checks. +If native startup exhausts its command-boundary probes, the error includes a JSON diagnostic summary in the CI log. +Two observations, each limited to five seconds, read container state and the final 80 log lines before cleanup. +The summary records available state fields, read results, and fixed error flags while excluding raw logs, configuration, and credentials. +A false flag does not exclude an error missing from the retained log tail. +Use the summary to diagnose startup; container removal still requires separate verification. For OpenClaw, those checks also run the shared agent-reload proof against the image's generated gateway policy and installed distribution. The proof checks live named-agent discovery, gateway health, and a nonzero exit after a restart-class bind failure. Google Chat's reload-off exception @@ -432,8 +437,7 @@ selecting a shipped-agent catalog does not qualify Pi. ### Hermes NVIDIA Stripped-State Recovery The native NVIDIA branch of `live/hermes-inference-switch.test.ts` owns the real registered rebuild boundary for #14094. -Its catalogue entry provides a 75-minute test budget and an 85-minute workflow budget. -The extra test time covers the second rebuild and its probes; ten minutes remain for workflow setup, diagnostics, and cleanup. +Its catalogue entry and workflow define the test and job deadlines. It keeps the healthy Discord-state rebuild, then tests a separate damaged-state rebuild in the same disposable sandbox. The missing coverage dimension is recovery when all three managed NVIDIA credential fields already contain `[STRIPPED_BY_MIGRATION]`. @@ -442,6 +446,9 @@ It checks the route, regular-file ownership, links, and size, compares file meta These snapshot checks detect observed changes; they do not exclude concurrent writers. It does not alter config permissions, guard hashes, supervisor state, or credential stores. The public rebuild must recover three agreeing issued references, retain the noncredential config fingerprint and a test-owned user-state file, preserve Discord settings, report healthy NVIDIA inference, and return `PROBE_OK` from a real Hermes request. +Recovery requires the registered native NVIDIA provider attachment and verified sandbox identity. +The guarded restore checks the source digest and permits only the three credential-field replacements. +Ordinary configuration writes and readiness checks must still reject stripped credentials. Neither rebuild sets `NVIDIA_INFERENCE_API_KEY` or `DISCORD_BOT_TOKEN`. The test-only enrollment transport receives the synthetic token through `NEMOCLAW_E2E_DISCORD_TOKEN`. The synthetic Discord enrollment transport does not establish real Discord delivery. @@ -452,7 +459,10 @@ Source caller tests and the integration tests in `test/agents/hermes/hermes-nvid `support/hermes-discord-rebuild-contract.test.ts` owns evidence sequencing and rejection. These deterministic tests do not qualify the installed guard. -The live assertion count decreases by two. The Discord-state reader must exit successfully before its exact settings comparison; support tests reject valid output followed by a failed read after either rebuild. Each removed check has a separate disposition: +Upstream #14153 recorded a decrease of two live assertion points against its base. +The OpenShell `0.1.7` integration has a separate assertion audit; that historical count does not describe its delta. +The Discord-state reader must exit successfully before its exact settings comparison; support tests reject valid output followed by a failed read after either rebuild. +The upstream removals retain these dispositions: | Removed check | Retained owner or reason | | --- | --- | @@ -564,7 +574,12 @@ approve arbitrary pending devices or silently skip the main-session cases. The feature tests exercise the CLI instead of separately asserting that its source and compiled files exist. The full onboarding test relies on its existing status poll, which fails when status never succeeds. MCP tests retain live allow/deny enforcement and credential-rotation checks without asserting policy -serialization or the provider-update success message. Approval phase labels live with the existing +serialization or the provider-update success message. +MCP credential scans check each supplied host credential against every selected sandbox state path. +A credential match or scan error fails the check. +An absent path is accepted only when its parent directory is readable and searchable. +The scan treats credentials as literal strings and suppresses scan output before artifact capture. +Approval phase labels live with the existing fixture evidence helpers; moving these labels does not move live assertions. The concurrent-add probe retries only the rejected command after status proves that the other command committed one coherent bridge. The rejected command must report the exact portable @@ -580,6 +595,18 @@ reject mixed cohorts, another run, a future attempt, and another candidate revis The managed-image scope does not claim trusted-private DNS-rebinding coverage: host and sandbox `/etc/hosts` fixtures do not control the OpenShell supervisor's egress resolver. Full MCP bridge E2E coverage retains that assertion for environments with supervisor-authoritative DNS. +On OpenShell 0.1.7, the full-suite DNS fixture selects the workload's supervisor by sandbox ID and namespace, then verifies its cgroup and upstream resolver. +A loopback DNS forwarder overrides only the selected fixture hostname with an IPv4 address and returns no AAAA answer for it. +Other queries use the supervisor's existing upstream resolver over UDP or TCP. +Temporary owned `iptables` or `ip6tables` rules redirect queries to the loopback forwarder. +They match only the verified supervisor cgroup, selected upstream address, and TCP or UDP port `53`. +The host `/etc/hosts` mapping remains for host-side admission checks. +Denial checks require generated override answers as well as the existing policy-denial and no-arrival evidence; IP pins remain unchanged. +Cleanup verifies that owned rules are absent, restores the host's hosts file, and closes the forwarder. +Unconfirmed cleanup fails the fixture. +The trusted-private fixture uses its existing host HTTPS listener for the default OpenShell 0.1.7 +host-network supervisor. Its discovered interface address must be routed-private, +not the Podman gateway's link-local alias. Legacy bridged workloads retain the relay. The trusted-private HTTPS fixture records bounded TLS, request-header, and completed-body counters before the status assertion and during final cleanup. TLS errors use fixed code buckets; these diagnostics contain no raw errors, request data, or credentials and do not establish successful @@ -719,9 +746,14 @@ The historical fixtures retain these version boundaries: | Fixture | Required boundary | | --- | --- | -| `openshell-gateway-upgrade` | Retain the reviewed v0.0.89 and v0.0.123 x86-64 fixtures with pinned installer commits, digests, and OpenClaw archives. Prove credential custody and authenticated agent turns before and after upgrade. Require the survivor to preserve workspace state. Create each fixture-declared stopped sandbox, preserve its workspace state, and restore its stopped phase after a lifecycle check. | +| `openshell-gateway-upgrade` | Install the reviewed v0.0.89 and v0.0.123 x86-64 fixtures with pinned installer commits, digests, and OpenClaw archives. Attempt the 0.1.x cutover with registered sandboxes and require refusal. Verify the dashboard forward, sandbox phases, workspace marker, credential custody, and authenticated agent turns remain intact. Verify the old gateway service when the fixture provides it. | | `rebuild-openclaw` | Retain the reviewed old-base build in the target. Build and create the old sandbox before testing the candidate rebuild path. | +The gateway-upgrade target observes the v0.0.89 forward through its legacy SSH registry. +The v0.0.123 fixture uses a direct `ForwardTcp` service outside that registry. +For that service, the target compares executable, complete arguments (gateway, workspace, sandbox, and ports), +PID, start time, and listener before and after refusal to require the same process. + These targets may restore the shared artifact for the candidate CLI. They must not replace a historical installer, package, image, or version boundary with that artifact. The gateway fixture already binds its remote historical inputs to immutable commits and cryptographic digests. @@ -893,7 +925,7 @@ The support test owns the shell fixture's two-attempt output. OpenShell target work in issue #9872. The trusted workflow downloads and verifies the exact OpenShell SDK archive with package-read permission. The candidate job receives the archive but no package credential. It calls a local -OpenShell 0.0.116 gateway over HTTPS with an explicit CA. The target confirms +OpenShell 0.1.7 gateway over HTTPS with an explicit CA. The target confirms that its configured authentication file path does not exist before and after the health request. A successful request proves that public health does not require a credential read or make an authenticated gateway call. @@ -912,6 +944,10 @@ This preserves the locked dependency versions and avoids npm resolving a new pee Both jobs verify that the SDK connection API loads before running tests. This keeps the private optional dependency available for SDK-backed commands such as configuration export. +For Docker, the shared fake messaging API fixture publishes its credential-free proxy on +`127.0.0.1` for the default OpenShell 0.1.7 host networking. Podman retains `0.0.0.0` publication. +The API container remains on an internal network without published ports. + The `network-policy` target also owns live configuration-export evidence for #10938, #11854, and PR #11065. After restricted OpenClaw onboarding with two read-only agents, it invokes the candidate `config export` command through the real SDK connection. It requires the command to reject the secondary-agent roster without producing a document. @@ -1064,7 +1100,7 @@ The local HTTP hop stays on the runner; host and sandbox probes use the same pub Its mock rejects a missing or incorrect credential, and the existing sandbox request and OpenClaw model run must both succeed. The config probe validates the scoped Anthropic credential reference inside the sandbox before redaction. The native NVIDIA variant uses its existing public credential before and after switching from the custom provider to NVIDIA Endpoints. -The interactive policy-preset sequencing fixture uses the same public HTTPS transport and retains its Personal shared-route assertions. +The interactive policy-preset sequencing fixture uses the same public HTTPS transport and retains its Personal native-provider assertions. Route assertions use JSON fields so terminal column alignment does not affect the result. Hermes inference-switch installation uses an authenticated local fixture before its model-only switch. The baseline model expectation comes from the environment passed to installation. @@ -1135,13 +1171,17 @@ OpenShell SSH configuration stays in a private temporary file outside the artifa The assertion attempts removal after it finishes and fails if the directory remains. A failed agent attempt retains only schema metadata, the attempt number, failure class, exit code, signal, timeout state, and one allowlisted diagnostic summary. Raw OpenClaw session and trajectory JSONL stay inside the sandbox. Reduced agent evidence artifacts retain schema metadata, the source hostname and protocol, bounded aggregate counts, and status booleans while omitting fetched content, complete URLs and queries, tool names, and provider values. +For OpenClaw 2026.9.2, the reducer reads nested tool records from the session transcript. Each record must match the trajectory run and an allowed `web_fetch` wrapper. A failed, missing, duplicate, or unrelated result cannot establish fetch success. When a redacted agent turn fails with tool-evidence validation enabled, the helper also attempts a read-only session reduction before cleanup. The failure artifact records fixed collection status and the existing aggregate tool-evidence projection, without raw session data or command errors. This diagnostic read cannot accept the failed turn or change its retry classification. Both scenarios collect bounded session tool-result diagnostics even when the agent reply fails its assertion. At most 64 tool-result records retain numeric HTTP status, a tool-error flag, and a call-pairing flag. HTTP status comes from a numeric tool-result field or the fixed `Web fetch failed (NNN):` prefix in an OpenClaw `web_fetch` error result. -These records exclude raw content, URLs, and credentials. +Failed tool results also retain an allowlisted `errorCategory`: `http`, `invalid-url`, `invalid-max-chars`, `disabled`, `extraction`, `fetch-failed`, `other`, or `unknown`. +Results without a tool error record `null`; unrecognized or missing categories become `unknown`. +`fetch-failed` identifies only the generic `fetch failed` message; it does not establish a network cause. +These records exclude raw error messages, nested causes, tool parameters, content, URLs, and credentials. An agent's final text alone does not establish the HTTP result. Missing tool-result evidence remains unavailable; collection does not add retries or change the fetch and reply assertions. After an unsuccessful agent assertion, both scenarios make one separate sandbox curl request to the same expected HTTPS URL. @@ -1191,6 +1231,17 @@ On an exact-revision manual PR run, `NEMOCLAW_E2E_RISK_SIGNAL_EXPECTED_SHA` carr The standard layout writes product evidence and `evidence-manifest.json` under `e2e-artifacts/live/`. When `shard` is not `default`, the standard layout adds the shard directory. The security-posture matrix uses the reviewed flat-shard layout to preserve its existing artifact names. +Its native OpenClaw configuration probe requires loading the proxy environment, validating the configuration, and reading `gateway.mode` to succeed in sequence. +The returned mode must be `local`; a failure in any step fails the probe. +For OpenShell 0.1.7, its process report identifies the non-root workload boundary at PID 1 separately from the `nemoclaw-start` children. +The probe runs as the managed image's declared sandbox owner and requires zero capabilities and `NoNewPrivs=1`. +It records denied PID-1 executable-link access as `null`; it does not infer an executable path from the command line. +Child executable links must remain readable and identify system Bash. Missing processes and denied child inspection fail the probe. +The report's `processScope: workload-container` identifies the process census; it does not attest the PID-1 executable binary. +Separate `containerPair` evidence binds engine launch metadata to that workload and its uniquely labelled supervisor before and after the census. +It checks running identities, entrypoints, non-root users, capability drops, no-new-privileges, and provider-specific network settings. +The supervisor must use the pinned 0.1.7 image and pass its native health command. Docker also requires its read-only root filesystem. +The inspection selects metadata fields and excludes environments, supervisor arguments, credentials, and health logs. The `gpu-double-onboard`, `gpu-e2e`, and `llama-cpp-generic-gpu` targets keep the standard layout and select `linux-amd64-gpu-rtxpro6000-latest-1` through the catalogue. The llama.cpp target compares the selected model's authenticated `/v1/models` `meta.n_ctx` with the generated OpenClaw primary model's context window, with no explicit context override. @@ -1254,7 +1305,13 @@ The coverage disposition is: The common-egress targets retain their live network policy and agent-fetch boundaries. They disable optional search explicitly and use the `nvidia-inference` profile. -The weather case checks five initial presets, then the added weather preset and its verified agent fetch. +The weather case checks five initial presets, then adds the weather preset and runs an agent curl to Open-Meteo. +The agent saves the JSON response. +The weather turn requires both the parsed payload and final visible answer to contain only `WEATHER_AGENT_OK`, apart from surrounding whitespace. +Printed commands or tool-call markup containing that token fail this check and use the existing failed-turn diagnostic collection. +The host independently checks the saved response's location, temperature, time, and Celsius unit, then records its SHA-256 hash. +When session reduction succeeds, a failed weather agent assertion records aggregate session error, final-status, tool-call, execution, and result counts, including exec call and result counts. +The artifact excludes raw session content and credentials; these counts do not establish why the command failed. Brave preset inclusion belongs to the existing onboarding policy tests, independent of weather coverage. ### Catalogue Execution Evidence @@ -1406,6 +1463,9 @@ existing final provider marker authorizes a retry after successful cleanup. Baseline and PTY cleanup calls run independently so a fatal shell error in one cannot skip the other. Failed calls retain their last 2 KiB of error output through the fixture's normal redaction boundary, and any cleanup failure prevents retry. +The PTY cleanup wait checks the monitor socket inside the sandbox with a five-second deadline. +It rejects a persistent socket or an entry with invalid type, ownership, permissions, or link count without removing it. +The separate removal step still requires socket absence, an unchanged directory identity during removal, and no unknown entries. The host launch harness uses a non-login Bash shell with its supplied environment. It does not source user login or logout files. A failing logout file can disrupt Bash 5.1 exit-trap cleanup. Deterministic unit tests separately prove selection of the complete preflight @@ -1658,7 +1718,40 @@ On Docker and rootless Podman, managed-image activation also adopts the published OpenClaw and Hermes digests through `--from-image`. It confirms OpenShell readiness, the durable external-image receipt, identity-drift rejection before replacement, rebuild from the recorded digest, NemoClaw -destruction, and shared image retention. +destruction, and shared image retention. If the rebuild fails or times out, the +target attempts to capture bounded, redacted lifecycle diagnostics before cleanup. + +Hermes images with the startup-completion helper wait up to 10 seconds for the non-root supervisor before configuration integrity inspection. +The supervisor publishes completion after gateway startup, auxiliary startup, and Tirith finalization. +The receipt binds the supervisor and gateway process generations; inspection rechecks it afterward. +An unsafe receipt, missing completion at the deadline, or a changed process generation fails verification. +All existing topology, ownership, permission, and metadata checks, including `st_ctime_ns`, remain required. +Integrity failures are not retried. +The receipt schedules inspection; it grants no security authority and does not prove that all native writers have stopped. +Retained images without the helper and root-separated startup retain their existing integrity contracts. + +The same Docker and rootless Podman jobs run a separate legacy-volume refusal +case before agent activation. It creates a disposable owned volume without +OpenShell attachment approval. The production preflight must reject that volume. +The test then reads the original marker, ownership, mode, and symlink to verify +that they remain unchanged. Cleanup removes only the fixture volume and helpers. +This case needs no inference service, gateway, GPU, or sudo. + +The maintainer decision for #12603 removes automatic copying of legacy volumes. +The live copy-failure, copy-retry, destination-selection, and migrated-state +attachment checks are therefore removed. Legacy data preservation remains covered +by the refusal case. Fresh-volume activation, agent turns, lifecycle, security +refusal, and cleanup remain covered by the activation case. + +These earlier removals retain their existing evidence owners: + +| Removed live check | Retained evidence owner | +| --- | --- | +| Four checks repeated after cleanup polling | The same poll's terminal errors and two-inventory absence condition; negative cases in `support/managed-image-activation-diagnostics.test.ts` | +| Separate architecture non-null assertion | Per-contract platform parsing still rejects mismatches; architecture cases in `src/lib/onboard/sandbox-workload-preparation.test.ts` | +| Catalog cohort and extra-key fixture guards | Production catalog validation and positive/negative source tests in `sandbox-workload-preparation.test.ts`; live per-agent digest selection remains | +| Engine `info` exit assertion | Keep the diagnostic artifact; real onboarding, lifecycle and engine inventory operations establish availability | +| Constant managed-receipt `shared` field | Receipt construction in `src/lib/onboard/managed-workload/onboard-orchestration.test.ts`; live digest and provenance checks remain | ## Device-auth health classification @@ -2336,7 +2429,7 @@ The live target refuses to replace a pre-existing sandbox or `nemoclaw-vllm` con It preserves the shared Hugging Face cache, records the created sandbox and container identities, and revalidates each identity before cleanup. If onboarding exits nonzero, the target captures the managed-container log tail and sandbox details before cleanup. The standard E2E artifacts retain bounded command output. -After onboarding, both physical-host cases request vLLM health on loopback and the private OpenShell bridge. +After onboarding, both physical-host cases request vLLM health on loopback. Each case also checks connection refusal on every discovered unintended IPv4 and IPv6 address. Spark and Station retain their own probe targets and request results. @@ -2346,10 +2439,10 @@ Run the target from a clean candidate checkout on the Spark host: E2E_JOB=1 E2E_TARGET_ID=spark-express-vllm NEMOCLAW_RUN_LIVE_E2E=1 NEMOCLAW_SANDBOX_NAME=e2e-spark-vllm npx tsx tools/e2e/live-vitest-invocation.mts run --test-path test/e2e/live/dgx-express.test.ts --selector '^spark-express-vllm:' ``` -A passing Spark target establishes that option 2 selects the fixed vLLM preset and recipe. +A passing Spark target establishes that the source-checkout option-2 path selects the fixed vLLM preset and recipe. The managed container carries catalog provenance and the catalog-derived serve command. -Health requests succeed on loopback and the private OpenShell bridge; connections to every other discovered host address are refused. -The target also requires a chat response through `inference.local` and an HTTP `403` response for unrelated sandbox egress. +Health requests succeed on loopback; connections to every other discovered host address are refused. +The authenticated native local provider completes a sandbox chat request, and unrelated sandbox egress receives an HTTP `403` response. The checker preserves coverage for every file under `test/e2e/live/` and adds workflow-selected integration files from the authoritative shared-job planner. @@ -2781,14 +2874,14 @@ test, and makes `full-e2e` the source of truth for the hard cold-path contract. ## DGX Station Express The explicit `dgx-station-express` target runs the local Station Express installer with cached Nemotron 3.5 Super VL weights, checks routed sandbox inference, and uninstalls the job runtime. See [Station dispatch](docs/dgx-station-dispatch.md) for prerequisites, workflow selection, and evidence boundaries. -After installation, this target requires vLLM health requests to succeed on loopback and the private OpenShell gateway and refuse every other assigned IPv4/IPv6 host address. +After installation, this target requires vLLM health requests to succeed on loopback and refuse every other assigned IPv4/IPv6 host address. The existing controller retains the inspected bindings, probe targets and per-request results for the candidate revision. Source tests own port and authentication permutations; these runtime requests prove Docker and host network enforcement. ## Bearerless vLLM Provider Validation The `inference-routing` target runs TC-INF-09 compatible-endpoint routing on Docker and Podman. -Its Docker-only bearerless companion validates the production provider against private bindings on port 18080 with distinct default/OpenShell gateways. +Its Docker-only bearerless companion validates the production provider against a loopback binding on port 18080. Podman skips only that companion. Artifacts retain inspected bindings, the probe command, validation result, container requests and ID-based cleanup. Readiness retries cover only refused or reset connections during the fixture's idempotent startup GET. diff --git a/test/e2e/docs/dgx-station-dispatch.md b/test/e2e/docs/dgx-station-dispatch.md index ad01a521c8b..6196ae48a23 100644 --- a/test/e2e/docs/dgx-station-dispatch.md +++ b/test/e2e/docs/dgx-station-dispatch.md @@ -23,10 +23,12 @@ NEMOCLAW_REPO_ROOT="$(pwd)" bash install.sh --express-install --yes-i-accept-thi `--express-install` selects the existing Station Express recipe without its selection prompt. Station qualification and explicit software acceptance remain required. It rejects conflicting provider, profile, deferred-onboarding, and interactive Station options. Ordinary non-interactive installation retains its existing selection behavior. -The smoke checks the ready sandbox, running managed vLLM with the Super VL serving alias, the independently selected managed-image revision, and an assistant response through `inference.local`. -Before inference, vLLM health requests must succeed on loopback and the private OpenShell gateway and refuse every other assigned IPv4/IPv6 host address. -The candidate's command artifacts retain inspected bindings, probe targets, exit codes and HTTP statuses. -Cleanup attempts to retire installation state left after success or failure. Cleanup uses a fresh signal through the E2E cleanup registry. +The smoke checks the ready sandbox, running managed vLLM with the Super VL serving alias, and the independently selected managed-image revision. +Before inference, vLLM health requests must succeed on loopback and refuse every other assigned IPv4/IPv6 host address. +The candidate's command artifacts retain inspected bindings, probe targets, exit codes, and HTTP statuses. +It checks an assistant response through the authenticated native local provider attached to that sandbox. +Cleanup attempts to retire installation state left after success or failure. +Cleanup uses a fresh signal through the E2E cleanup registry. Timing evidence records completed smoke phases. Total duration spans installer start through smoke cleanup. Readiness and assertion durations can be absent after failure. vLLM startup remains inside installation. The cache check compares paths, file sizes, and symlink targets. The smoke also checks that the vLLM image remains after uninstall. diff --git a/test/e2e/e2e-cloud-experimental/checks/04-deepagents-code-fresh-reonboard.sh b/test/e2e/e2e-cloud-experimental/checks/04-deepagents-code-fresh-reonboard.sh index aaa632cf429..64797744f07 100755 --- a/test/e2e/e2e-cloud-experimental/checks/04-deepagents-code-fresh-reonboard.sh +++ b/test/e2e/e2e-cloud-experimental/checks/04-deepagents-code-fresh-reonboard.sh @@ -78,10 +78,8 @@ NODE } dcode_identity() { - # Invoke dcode by absolute path: `openshell sandbox exec -- dcode ...` runs - # without a login shell, so /usr/local/bin is not on PATH and a bare `dcode` - # resolves to "command not found". The image installs the launcher at - # /usr/local/bin/dcode (see agents/langchain-deepagents-code/Dockerfile). + # Use the image-owned absolute launcher path so identity checks do not + # depend on the shell PATH. openshell sandbox exec --name "$SANDBOX_NAME" -- /usr/local/bin/dcode identity 2>&1 } @@ -263,9 +261,22 @@ for login_profile in "$PERSONAL_LOGIN_PROFILE" "$HOSTILE_LOGIN_FALLBACK"; do /bin/bash -lc '/usr/local/lib/nemoclaw/dcode-managed-exec /usr/bin/printf %s MANAGED_EXEC_OK' 2>&1 )" || fail "managed exec failed with personal startup files present: $managed_output" [ "$managed_output" = MANAGED_EXEC_OK ] || fail "managed exec output contains personal startup output: $managed_output" - privileged_exec "$resource_handle" /bin/sh -c \ - "/usr/bin/env HOME=/sandbox BASH_ENV='$HOSTILE_SHELL_ENV' ENV='$HOSTILE_SHELL_ENV' /usr/local/bin/nemoclaw-start /usr/bin/true && test ! -e '$HOSTILE_PROFILE_MARKER'" \ - || fail "managed exec or root entrypoint failed or read personal startup code" + # Native workloads drop all capabilities. Root cannot change UID here; + # the runtime starts the entrypoint as the sandbox user instead. + if root_output="$(privileged_exec "$resource_handle" /usr/bin/env \ + HOME=/sandbox BASH_ENV="$HOSTILE_SHELL_ENV" ENV="$HOSTILE_SHELL_ENV" \ + /usr/local/bin/nemoclaw-start /usr/bin/true 2>&1)"; then + fail "native workload unexpectedly allowed the root entrypoint to change UID" + fi + printf '%s\n' "$root_output" | grep -Fq 'setpriv: setresuid failed: Operation not permitted' \ + || fail "root entrypoint failed without the expected UID-transition denial" + # The runtime entrypoint has no login shell. Preserve that boundary here so + # a transport login cannot run the hostile profile before the entrypoint. + openshell sandbox exec --name "$SANDBOX_NAME" --no-login-shell -- \ + /usr/bin/env HOME=/sandbox BASH_ENV="$HOSTILE_SHELL_ENV" ENV="$HOSTILE_SHELL_ENV" \ + /usr/local/bin/nemoclaw-start /bin/sh -c \ + "test \"\$(id -un)\" = sandbox && test ! -e \"\$1\"" sh "$HOSTILE_PROFILE_MARKER" \ + || fail "sandbox-user entrypoint failed or read personal startup code" # shellcheck disable=SC2016 # Read the variable set by the sandbox's personal profile. ordinary_output="$(openshell sandbox exec --name "$SANDBOX_NAME" -- /usr/bin/env -u NEMOCLAW_E2E_PERSONAL_PROFILE /bin/bash -lc 'printf %s "$NEMOCLAW_E2E_PERSONAL_PROFILE"' 2>&1)" \ diff --git a/test/e2e/e2e-cloud-experimental/checks/05-deepagents-code-landlock-readonly.sh b/test/e2e/e2e-cloud-experimental/checks/05-deepagents-code-landlock-readonly.sh index df0d6317eaf..08e505762fc 100755 --- a/test/e2e/e2e-cloud-experimental/checks/05-deepagents-code-landlock-readonly.sh +++ b/test/e2e/e2e-cloud-experimental/checks/05-deepagents-code-landlock-readonly.sh @@ -31,31 +31,51 @@ sandbox_exec() { privileged_dpkg_sentinel() { local action="$1" local probe_path="$2" + local receipt="${3:-}" NEMOCLAW_E2E_DPKG_ACTION="$action" \ NEMOCLAW_E2E_DPKG_PROBE="$probe_path" \ + NEMOCLAW_E2E_DPKG_RECEIPT="$receipt" \ NEMOCLAW_E2E_SANDBOX_NAME="$SANDBOX_NAME" \ node <<'NODE' -const { executePrivilegedSandboxCommand } = require("./dist/lib/sandbox/privileged-exec.js"); +const { executePrivilegedSandboxCommand, resolvePrivilegedSandboxTarget } = require("./dist/lib/sandbox/privileged-exec.js"); const action = process.env.NEMOCLAW_E2E_DPKG_ACTION; const probe = process.env.NEMOCLAW_E2E_DPKG_PROBE; const sandbox = process.env.NEMOCLAW_E2E_SANDBOX_NAME; if (!probe || !sandbox || !["prepare", "cleanup"].includes(action)) process.exit(2); +const recorded = action === "cleanup" ? JSON.parse(process.env.NEMOCLAW_E2E_DPKG_RECEIPT) : null; +const resourceHandle = recorded?.resourceHandle ?? resolvePrivilegedSandboxTarget(sandbox).resourceHandle; +// Root owns this nonsecret file. Mode 0666 makes Unix write permission a +// positive control, so the sandbox denial must come from its outer policy. +const source = String.raw`import os, stat, sys +operation, target, recorded_identity = sys.argv[1:] +if operation == "prepare": + fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + try: + os.fchmod(fd, 0o666) + observed = os.fstat(fd) + assert observed.st_uid == 0 and observed.st_nlink == 1 + assert stat.S_ISREG(observed.st_mode) and stat.S_IMODE(observed.st_mode) == 0o666 + print(f"{observed.st_dev}:{observed.st_ino}") + finally: + os.close(fd) +else: + observed = os.lstat(target) + assert stat.S_ISREG(observed.st_mode) and observed.st_uid == 0 and observed.st_nlink == 1 + assert f"{observed.st_dev}:{observed.st_ino}" == recorded_identity + os.unlink(target) +`; +const result = executePrivilegedSandboxCommand(sandbox, + ["/opt/venv/bin/python3", "-I", "-c", source, action, probe, recorded?.identity ?? ""], + { sanitizeEnvironment: true, expectedResourceHandle: resourceHandle }, +); +if (result.status !== 0 || result.signal || result.error) process.exit(1); +if (action === "prepare") { + const identity = result.stdout.toString().trim(); + if (!/^[0-9]+:[0-9]+$/.test(identity)) process.exit(1); + process.stdout.write(JSON.stringify({ resourceHandle, identity })); +} -const command = - action === "prepare" - ? [ - "sh", - "-c", - 'set -eu; probe="$1"; test ! -e "$probe"; install -o sandbox -g sandbox -m 600 /dev/null "$probe"', - "sh", - probe, - ] - : ["rm", "-f", "--", probe]; -const result = executePrivilegedSandboxCommand(sandbox, command, { - sanitizeEnvironment: true, -}); -process.exit(result.status ?? 1); NODE } @@ -98,14 +118,14 @@ else fi DPKG_SENTINEL="/var/lib/dpkg/nemoclaw-e2e-write-probe-${PPID:-$$}" -if privileged_dpkg_sentinel prepare "$DPKG_SENTINEL"; then +if DPKG_RECEIPT="$(privileged_dpkg_sentinel prepare "$DPKG_SENTINEL")"; then OUT=$(sandbox_exec "set -e; dpkg-query -W dpkg >/dev/null; echo DPKG_QUERY_OK; control=/tmp/nemoclaw-e2e-dpkg-write-control; printf 'control\\n' >\"\$control\"; rm -f \"\$control\"; echo CONTROL_WRITE_OK; if printf 'denied\\n' >'$DPKG_SENTINEL' 2>/dev/null; then echo DPKG_WRITE_UNEXPECTEDLY_SUCCEEDED; exit 1; fi; echo DPKG_WRITE_DENIED" || true) if echo "$OUT" | grep -q "DPKG_QUERY_OK" && echo "$OUT" | grep -q "CONTROL_WRITE_OK" && echo "$OUT" | grep -q "DPKG_WRITE_DENIED"; then pass "package metadata is readable and the package database remains read-only" else fail_test "package database read-only boundary failed" fi - if ! privileged_dpkg_sentinel cleanup "$DPKG_SENTINEL"; then + if ! privileged_dpkg_sentinel cleanup "$DPKG_SENTINEL" "$DPKG_RECEIPT"; then fail_test "package database sentinel cleanup failed" fi else diff --git a/test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh b/test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh index 9a8a365b8f7..6b6df869234 100755 --- a/test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh +++ b/test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh @@ -163,8 +163,10 @@ DENIAL_MARKERS = ( 'tunnel connection failed', ) -if not os.environ.get('DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL'): - print('FETCH_ERROR:managed fetch proxy delegation is absent') +from deepagents_code._nemoclaw_managed import _managed_fetch_uses_openshell + +if not _managed_fetch_uses_openshell(): + print('FETCH_ERROR:managed image identity is absent') raise SystemExit(0) result = fetch_url(sys.argv[1], timeout=8) @@ -180,7 +182,7 @@ if ( else: details = str(result.get('error', result)) lowered = details.lower() - if any(marker in lowered for marker in DENIAL_MARKERS): + if lowered == 'fetch url error: url destination is blocked' or any(marker in lowered for marker in DENIAL_MARKERS): print(f'FETCH_BLOCKED:{details}') else: print(f'FETCH_ERROR:{details}') @@ -207,7 +209,7 @@ expect_fetch_reached() { local output output="$(fetch_url_probe "$url" || true)" if echo "$output" | grep -Eq 'FETCH_SUCCESS:2[0-9]{2}:[1-9][0-9]*'; then - pass "Deep Agents fetch_url can reach approved ${label} host through the managed proxy" + pass "Deep Agents fetch_url can reach approved ${label} host through the mediated network" else fail_test "Deep Agents fetch_url could not reach approved ${label} host: $output" fi @@ -312,7 +314,7 @@ fi expect_reached "arbitrary Python" "GitHub" "https://api.github.com/" expect_reached "arbitrary Python" "PyPI" "https://pypi.org/" expect_blocked "arbitrary Python" "Tavily" "https://api.tavily.com/" -# The public helper only reconstructs the trusted proxy environment; OpenShell +# The public helper clears ambient proxy variables; OpenShell # must still enforce Tavily denial on the final managed-Python executable. expect_blocked \ "direct managed-exec Python" \ @@ -326,8 +328,7 @@ expect_blocked "arbitrary Python" "unapproved hosts" "https://example.com/" # Exercise the actual Deep Agents fetch_url transport. Unlike urllib, upstream # fetch_url disables ambient proxies to pin direct DNS results; the managed -# image patch must instead force the root-owned OpenShell proxy without honoring -# NO_PROXY. The raw GitHub path covers the separate read-only policy endpoint. +# image patch uses OpenShell's mediated network without honoring ambient proxies. The raw GitHub path covers the separate read-only policy endpoint. expect_fetch_reached \ "raw GitHub" \ "https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/README.md" @@ -340,13 +341,12 @@ import os import subprocess url = 'https://raw.githubusercontent.com/NVIDIA/NemoClaw/main/README.md' -proxy = os.environ.get('HTTPS_PROXY') or os.environ['HTTP_PROXY'] ca = os.environ['SSL_CERT_FILE'] def request(method): command = [ '/usr/bin/curl', '--disable', '--silent', '--show-error', '--noproxy', '', - '--proxy', proxy, '--cacert', ca, '--connect-timeout', '10', + '--proxy', '', '--cacert', ca, '--connect-timeout', '10', '--max-time', '30', '--write-out', '\n%{http_code}', ] command += ['--head'] if method == 'HEAD' else ['--request', method] diff --git a/test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh b/test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh index 6cb5d34c52c..0982f85c75a 100755 --- a/test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh +++ b/test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh @@ -311,14 +311,10 @@ SHIM ) sandbox_login_proxy_contract() { - # Keep the remote login contract in one quoted shell expression so the - # fixture can dispatch it atomically. inference.local is intentionally absent from - # NO_PROXY: OpenShell does not need to provision inference.local DNS/hosts - # into the sandbox because its managed proxy owns this L7 route. Adding - # inference.local here would bypass that proxy and force a direct DNS lookup. + # Keep one quoted command so the fixture dispatches the login contract atomically. local contract_command # shellcheck disable=SC2016 - contract_command='set -euo pipefail; contract_fail() { printf "%s\n" "NEMOCLAW_DCODE_PROXY_ENV_FAIL:$1"; exit 1; }; proxy_file_metadata() { stat -c "%u:%a" "$1" 2>/dev/null || stat -f "%u:%Lp" "$1" 2>/dev/null; }; [ "${HOME:-}" = /sandbox ] || contract_fail home; runtime_uid="$(id -u)" || contract_fail runtime-user; sandbox_uid="$(id -u sandbox)" || contract_fail runtime-user; [ "$runtime_uid" != 0 ] && [ "$runtime_uid" = "$sandbox_uid" ] || contract_fail runtime-user; for file in /usr/local/share/nemoclaw/dcode-proxy-host /usr/local/share/nemoclaw/dcode-proxy-port; do [ -f "$file" ] && [ ! -L "$file" ] && [ "$(proxy_file_metadata "$file")" = "0:444" ] || contract_fail proxy-file-trust; done; trusted_proxy_host="$(cat /usr/local/share/nemoclaw/dcode-proxy-host)" || contract_fail proxy-file-read; trusted_proxy_port="$(cat /usr/local/share/nemoclaw/dcode-proxy-port)" || contract_fail proxy-file-read; proxy_env=/tmp/nemoclaw-proxy-env.sh; [ -f "$proxy_env" ] && [ ! -L "$proxy_env" ] && [ "$(proxy_file_metadata "$proxy_env")" = "${runtime_uid}:444" ] || contract_fail proxy-env-file-metadata; [ -z "${ALL_PROXY+x}" ] || contract_fail all-proxy; [ -z "${all_proxy+x}" ] || contract_fail lower-all-proxy; proxy_url="${HTTP_PROXY:-}"; case "$proxy_url" in http://*:*) ;; *) contract_fail proxy-shape ;; esac; case "$proxy_url" in *"@"*) contract_fail proxy-credentials ;; esac; expected_proxy_url="http://${trusted_proxy_host}:${trusted_proxy_port}"; [ "$proxy_url" = "$expected_proxy_url" ] || contract_fail proxy-source; [ "$proxy_url" = "${HTTPS_PROXY:-}" ] || contract_fail https-proxy; [ "$proxy_url" = "${http_proxy:-}" ] || contract_fail lower-http-proxy; [ "$proxy_url" = "${https_proxy:-}" ] || contract_fail lower-https-proxy; expected_no_proxy="localhost,127.0.0.1,::1,${trusted_proxy_host}"; [ "${NO_PROXY:-}" = "$expected_no_proxy" ] || contract_fail no-proxy; [ "${no_proxy:-}" = "$expected_no_proxy" ] || contract_fail lower-no-proxy; printf "%s\n" "NEMOCLAW_DCODE_PROXY_ENV_OK"' + contract_command='set -euo pipefail; contract_fail() { printf "%s\n" "NEMOCLAW_DCODE_PROXY_ENV_FAIL:$1"; exit 1; }; proxy_file_metadata() { stat -c "%u:%a" "$1" 2>/dev/null || stat -f "%u:%Lp" "$1" 2>/dev/null; }; [ "${HOME:-}" = /sandbox ] || contract_fail home; runtime_uid="$(id -u)" || contract_fail runtime-user; sandbox_uid="$(id -u sandbox)" || contract_fail runtime-user; [ "$runtime_uid" != 0 ] && [ "$runtime_uid" = "$sandbox_uid" ] || contract_fail runtime-user; proxy_env=/tmp/nemoclaw-proxy-env.sh; [ -f "$proxy_env" ] && [ ! -L "$proxy_env" ] && [ "$(proxy_file_metadata "$proxy_env")" = "${runtime_uid}:444" ] || contract_fail proxy-env-file-metadata; for name in HTTP_PROXY HTTPS_PROXY NO_PROXY http_proxy https_proxy no_proxy ALL_PROXY all_proxy OPENAI_PROXY grpc_proxy GRPC_PROXY NODE_USE_ENV_PROXY DEEPAGENTS_CODE_FETCH_URL_TRUSTED_PROXY_URL; do [ -z "${!name+x}" ] || contract_fail ambient-proxy; done; printf "%s\n" "NEMOCLAW_DCODE_PROXY_ENV_OK"' sandbox_login_exec "$contract_command" } @@ -711,24 +707,24 @@ main() { fi fi - # 3. The login shell loaded the exact normalized proxy contract from .profile. + # 3. The login shell cleared ambient proxy routes from .profile. proxy_contract_output="$(sandbox_login_proxy_contract || true)" if printf '%s\n' "$proxy_contract_output" | grep -Fxq "NEMOCLAW_DCODE_PROXY_ENV_OK"; then - pass "login shell loaded the normalized managed proxy environment" + pass "login shell loaded the managed environment without ambient proxies" else proxy_contract_reason="$(printf '%s\n' "$proxy_contract_output" | sed -n 's/^NEMOCLAW_DCODE_PROXY_ENV_FAIL:\([a-z-]*\)$/\1/p' | tail -n1)" - fail_test "login shell did not load the normalized managed proxy environment (${proxy_contract_reason:-unknown contract mismatch})" + fail_test "login shell did not load the managed environment without ambient proxies (${proxy_contract_reason:-unknown contract mismatch})" fi - # 4. The managed route is reachable through the normalized login-shell proxy. + # 4. The selected route is reachable from the managed login shell. # Native NVIDIA is proved through the attached-provider dcode PONG below. if [ "$route_contract" = "managed" ]; then - route_output="$(sandbox_login_exec "curl -sS -o /dev/null -w 'HTTP_CODE:%{http_code}' --proxy \"\${HTTPS_PROXY}\" --noproxy \"\${NO_PROXY}\" --max-time 30 https://inference.local/v1/models" || true)" + route_output="$(sandbox_login_exec "curl -sS -o /dev/null -w 'HTTP_CODE:%{http_code}' --proxy '' --noproxy '' --max-time 30 https://inference.local/v1/models" || true)" route_code="$(printf '%s' "$route_output" | sed -n 's/.*HTTP_CODE:\([0-9][0-9][0-9]\).*/\1/p' | tail -n1)" if [ "$route_code" = "200" ]; then - pass "login-shell proxy reached https://inference.local/v1/models" + pass "login-shell request reached https://inference.local/v1/models" else - fail_test "login-shell proxy did not receive HTTP 200 from https://inference.local/v1/models (HTTP ${route_code:-000})" + fail_test "login-shell request did not receive HTTP 200 from https://inference.local/v1/models (HTTP ${route_code:-000})" fi fi diff --git a/test/e2e/e2e-cloud-experimental/checks/11-deepagents-code-observability.sh b/test/e2e/e2e-cloud-experimental/checks/11-deepagents-code-observability.sh index 396f7e05a59..539e5d66c67 100755 --- a/test/e2e/e2e-cloud-experimental/checks/11-deepagents-code-observability.sh +++ b/test/e2e/e2e-cloud-experimental/checks/11-deepagents-code-observability.sh @@ -130,21 +130,47 @@ fi command -v ip >/dev/null 2>&1 || fail "host ip command is required" command -v curl >/dev/null 2>&1 || fail "host curl command is required" -bind_output="$(openshell sandbox exec --name "$SANDBOX_NAME" -- \ - sh -c "getent ahostsv4 ${COLLECTOR_HOST} | awk 'NR == 1 { print \"NEMOCLAW_OTLP_BIND_IP=\" \$1 }'" \ - 2>&1)" \ - || fail "could not resolve $COLLECTOR_HOST from the sandbox: $bind_output" -OTLP_BIND_IP="$( - printf '%s\n' "$bind_output" | tr -d '\r' | sed -n 's/^NEMOCLAW_OTLP_BIND_IP=//p' | tail -n 1 -)" -case "$OTLP_BIND_IP" in - 10.* | 192.168.* | 172.1[6-9].* | 172.2[0-9].* | 172.3[01].* | 169.254.2.2) ;; - *) fail "sandbox resolved $COLLECTOR_HOST to an unsupported host address '$OTLP_BIND_IP'" ;; -esac -if ! ip -o -4 address show \ - | awk '{ sub(/\/.*/, "", $4); print $4 }' \ - | grep -Fxq "$OTLP_BIND_IP"; then - fail "sandbox bridge address $OTLP_BIND_IP is not assigned to a host interface" +# Native DNS returns synthetic workload addresses. The selected runtime owns +# the supervisor's host route; it is the collector's bind address. +runtime_bind="$( + SANDBOX_NAME="$SANDBOX_NAME" node <<'NODE' +const { getSandbox } = require("./dist/lib/state/registry.js"); +const { currentRuntimeProviderBundles, requireRuntimeProviderBundleForSandbox } = require("./dist/lib/onboard/runtime-provider/selection.js"); +const { resolvePrivilegedSandboxTarget } = require("./dist/lib/sandbox/privileged-exec.js"); +const entry = getSandbox(process.env.SANDBOX_NAME); +if (!entry || entry.gatewayName !== process.env.OPENSHELL_GATEWAY) process.exit(1); +const provider = requireRuntimeProviderBundleForSandbox(entry, currentRuntimeProviderBundles()); +const target = resolvePrivilegedSandboxTarget(process.env.SANDBOX_NAME); +if (target.providerId !== provider.identity.id || !target.resourceHandle) process.exit(1); +const runtime = provider.gateway.observeHostRuntime({ environment: process.env, platform: process.platform }); +if (provider.identity.id === "docker" && process.platform === "linux" && + runtime.network.supervisorNetwork?.name === "host" && + runtime.network.supervisorNetwork.hostAddress === "127.0.0.1") { + process.stdout.write("127.0.0.1"); +} else if (provider.identity.id === "podman") { + process.stdout.write("podman"); +} else process.exit(1); +NODE +)" || fail "could not establish the selected runtime's collector route" +if [ "$runtime_bind" = 127.0.0.1 ]; then + OTLP_BIND_IP="$runtime_bind" +else + bind_output="$(openshell sandbox exec --name "$SANDBOX_NAME" -- \ + sh -c "getent ahostsv4 ${COLLECTOR_HOST} | awk 'NR == 1 { print \"NEMOCLAW_OTLP_BIND_IP=\" \$1 }'" \ + 2>&1)" \ + || fail "could not resolve $COLLECTOR_HOST from the sandbox: $bind_output" + OTLP_BIND_IP="$( + printf '%s\n' "$bind_output" | tr -d '\r' | sed -n 's/^NEMOCLAW_OTLP_BIND_IP=//p' | tail -n 1 + )" + case "$OTLP_BIND_IP" in + 10.* | 192.168.* | 172.1[6-9].* | 172.2[0-9].* | 172.3[01].* | 169.254.2.2) ;; + *) fail "sandbox resolved $COLLECTOR_HOST to an unsupported host address '$OTLP_BIND_IP'" ;; + esac + if ! ip -o -4 address show \ + | awk '{ sub(/\/.*/, "", $4); print $4 }' \ + | grep -Fxq "$OTLP_BIND_IP"; then + fail "sandbox bridge address $OTLP_BIND_IP is not assigned to a host interface" + fi fi "$TSX" "$CAPTURE_SERVER" "$CAPTURE_DIR" "$OTLP_BIND_IP" "$COLLECTOR_PORT" "$DECOY_PORT" \ @@ -175,12 +201,14 @@ request_count() { python_probe_source() { cat <<'PY' import sys +import json +import socket import urllib.error +import urllib.parse import urllib.request -# NemoClaw's exec wrapper sources the managed, credential-free OpenShell proxy -# route. Keep it intact so this probe exercises the same enforcement path -# as managed DCode and Relay instead of attempting an unsupported direct socket. +# Preserve the managed exec environment so this probe uses the same OpenShell +# enforcement path as managed DCode and Relay. method, url, body = sys.argv[1:] data = body.encode("utf-8") if method != "GET" else None request = urllib.request.Request( @@ -190,6 +218,13 @@ request = urllib.request.Request( headers={"content-type": "application/x-protobuf"}, ) try: + target = urllib.parse.urlsplit(url) + port = target.port or (443 if target.scheme == "https" else 80) + try: + addresses = sorted({item[4][0] for item in socket.getaddrinfo(target.hostname, port, type=socket.SOCK_STREAM)}) + except socket.gaierror as error: + raise urllib.error.URLError(error) from error + print("NEMOCLAW_OTLP_PROBE_DNS=" + json.dumps({"hostname": target.hostname, "port": port, "addresses": addresses})) with urllib.request.urlopen(request, timeout=10) as response: print(f"REACHED:{response.status}") except urllib.error.HTTPError as error: @@ -220,15 +255,18 @@ expect_blocked_without_capture() { local url="$3" local before output status after denial_state before="$(request_count)" + openshell logs "$SANDBOX_NAME" -n 500 --source sandbox --since 2m >"${CAPTURE_DIR}/dns-before.log" set +e output="$(sandbox_python_probe "$method" "$url" "NEMOCLAW_OTLP_DENIED_PROBE" 2>&1)" status=$? set -e sleep 1 + openshell logs "$SANDBOX_NAME" -n 500 --source sandbox --since 2m >"${CAPTURE_DIR}/dns-after.log" after="$(request_count)" [ "$status" -ne 0 ] || fail "$label unexpectedly returned success: $output" [ "$after" = "$before" ] || fail "$label reached the host capture server" - denial_state="$(printf '%s\n' "$output" | "$TSX" "$CONTRACT_HELPER" denial-state)" \ + denial_state="$(printf '%s\n' "$output" | "$TSX" "$CONTRACT_HELPER" denial-state "$status" \ + "${CAPTURE_DIR}/dns-before.log" "${CAPTURE_DIR}/dns-after.log" "$url")" \ || fail "$label denial classifier failed: $output" [ "$denial_state" = "policy-denied" ] \ || fail "$label failed without confirmed OpenShell policy-denial evidence: $output" @@ -276,18 +314,27 @@ openshell sandbox exec --name "$SANDBOX_NAME" -- test -x /usr/bin/curl >/dev/nul || fail "/usr/bin/curl is absent or not executable in the sandbox" before_binary="$(request_count)" set +e +# Native connect denials need curl's errno diagnostic; omit outgoing headers. +# shellcheck disable=SC2016 # Probe variables expand inside the sandbox shell. binary_output="$("$CLI" "$SANDBOX_NAME" exec -- \ - /usr/bin/curl --fail-with-body -sS --max-time 10 -X POST \ - -H 'content-type: application/x-protobuf' \ - --data-binary 'NEMOCLAW_OTLP_DENIED_BINARY_PROBE' \ - "http://${COLLECTOR_HOST}:${COLLECTOR_PORT}/v1/traces" 2>&1)" + sh -c ' + probe_stderr=$(mktemp) || exit 2 + trap '\''rm -f "$probe_stderr"'\'' EXIT + LC_ALL=C /usr/bin/curl --verbose --fail-with-body -sS --max-time 10 -X POST \ + --write-out "\nNEMOCLAW_OTLP_CURL_HTTP_CODE=%{http_code}\n" \ + -H "content-type: application/x-protobuf" \ + --data-binary "NEMOCLAW_OTLP_DENIED_BINARY_PROBE" "$1" 2>"$probe_stderr" + probe_status=$? + sed -n "/^\\* Immediate connect fail for /p; /^\\* connect to .* failed: /p; /^curl: /p" "$probe_stderr" >&2 + exit "$probe_status" + ' sh "$OTLP_TRACE_URL" 2>&1)" binary_status=$? set -e sleep 1 after_binary="$(request_count)" [ "$binary_status" -ne 0 ] || fail "unmanaged curl binary unexpectedly reached OTLP: $binary_output" [ "$after_binary" = "$before_binary" ] || fail "unmanaged curl binary reached the host collector" -binary_denial_state="$(printf '%s\n' "$binary_output" | "$TSX" "$CONTRACT_HELPER" denial-state)" \ +binary_denial_state="$(printf '%s\n' "$binary_output" | "$TSX" "$CONTRACT_HELPER" denial-state "$binary_status")" \ || fail "unmanaged curl denial classifier failed: $binary_output" [ "$binary_denial_state" = "policy-denied" ] \ || fail "unmanaged curl failed without confirmed OpenShell policy-denial evidence: $binary_output" diff --git a/test/e2e/fixtures/admin-approval-connect.sh b/test/e2e/fixtures/admin-approval-connect.sh index 10509eab712..4f77b6bbb4c 100644 --- a/test/e2e/fixtures/admin-approval-connect.sh +++ b/test/e2e/fixtures/admin-approval-connect.sh @@ -69,11 +69,17 @@ if sys.argv[2] == 'cron-run': reason=run.get('errorReason') reason=reason if isinstance(reason, str) and reason in reasons else 'unavailable' print(f'ADMIN_CRON_ERROR_REASON={reason}', file=sys.stderr) + duration=run.get('durationMs') + # Bound the terminal runtime only; exclude booleans and arbitrary metadata. + if type(duration) is int and 0 <= duration <= 86400000: + print(f'ADMIN_CRON_DURATION_MS={duration}', file=sys.stderr) diagnostics=run.get('diagnostics') entries=diagnostics.get('entries') if isinstance(diagnostics, dict) else None sources=('cron-preflight', 'cron-setup', 'model-preflight', 'agent-run', 'tool', 'exec', 'delivery') # Observe fixed error-text signatures, not provider or tool causality. messages={ + 'cron isolated agent run aborted': 'message-agent-aborted', + 'cron: job execution timed out': 'message-cron-deadline', # Pinned incomplete-turn texts: the reply budget ended without visible output. "Agent couldn't generate a response.": 'message-agent-incomplete-turn', "\u26a0\ufe0f Agent couldn't generate a response. Please try again.": 'message-agent-incomplete-turn', @@ -87,6 +93,7 @@ if sys.argv[2] == 'cron-run': http_signatures={str(code): f'error-text-http-{code}' for code in range(400, 600)} # Anchored subset of the pinned assistant-error-format HTTP grammar. http_patterns=( + r'\u26a0\ufe0f [A-Za-z0-9_./:-]+ request failed \(provider internal error, HTTP ([45][0-9]{2})\)\. This is usually temporary \u2014 try again shortly\.', r'(?:http\s*)?([45][0-9]{2})(?:(?:\s*:\s*|\s+)[\s\S]+)?', r'api\s*error\s*\(([45][0-9]{2})\)(?:\s*:\s*[\s\S]*)?', r'(?:status code|unexpected status|http status)\s*[:=]?\s*([45][0-9]{2})\b(?:\s*[:,]?\s*(?:message\s*:\s*)?[\s\S]*)?', @@ -118,6 +125,14 @@ if sys.argv[2] == 'cron-run': print('ADMIN_CRON_ERROR_SIGNATURES=' + (','.join(signatures) or 'unavailable'), file=sys.stderr) raise SystemExit(0) selection_checks=( + ('selection-state-busy', r'OpenClawPairingStateRetryableError: canonical pairing-state database is busy'), + ('selection-state-changed', r'OpenClawPairingStateRetryableError: canonical pairing-state .*changed'), + ('selection-state-wal-incomplete', r'OpenClawPairingStateRetryableError: canonical pairing-state WAL is missing its shared-memory sidecar'), + ('selection-state-unvalidated', r'OpenClawPairingStateRetryableError: SQLite reopened an unvalidated pairing-state file identity'), + ('selection-state-unsafe', r'OSError: unsafe canonical pairing-state'), + ('selection-state-missing', r'FileNotFoundError:'), + ('selection-state-permission', r'PermissionError:'), + ('selection-state-schema', r'ValueError: unsupported canonical device state schema'), ('selection-state-read', r'canonical pairing-state|canonical device state|canonical primary device identity|FileNotFoundError|PermissionError|sqlite3\.'), ('selection-local-identity', r'local CLI identity'), ('selection-pending-count', r'expected exactly one pending request, found [0-9]+'), diff --git a/test/e2e/fixtures/availability-env.ts b/test/e2e/fixtures/availability-env.ts index 60560cc940f..774245c77ac 100644 --- a/test/e2e/fixtures/availability-env.ts +++ b/test/e2e/fixtures/availability-env.ts @@ -16,6 +16,7 @@ export const AVAILABILITY_PROBE_EXTRA_ENV_KEYS = [ "E2E_MANAGED_IMAGE_REVISION", "E2E_MANAGED_IMAGE_COHORT_RECEIPT", "GITHUB_WORKSPACE", + "XDG_BIN_HOME", "XDG_CONFIG_HOME", "NEMOCLAW_E2E_EXPECTED_SHA", "NEMOCLAW_E2E_MANAGED_IMAGE_CATALOG", diff --git a/test/e2e/fixtures/clients/gateway.ts b/test/e2e/fixtures/clients/gateway.ts index e0960d6b088..b67a7d1e72a 100644 --- a/test/e2e/fixtures/clients/gateway.ts +++ b/test/e2e/fixtures/clients/gateway.ts @@ -163,7 +163,13 @@ export class GatewayClient { timeoutMs: 15_000, }, ); - if (pid.exitCode === 0 && pid.stdout.trim()) { + if ( + pid.exitCode === 0 && + !pid.timedOut && + pid.signal === null && + pid.stderr.trim() === "" && + /^[1-9][0-9]*$/.test(pid.stdout.trim()) + ) { return { kind: "pid", id: pid.stdout.trim() }; } @@ -182,7 +188,23 @@ export class GatewayClient { .filter(([, name]) => name === DEFAULT_GATEWAY_CONTAINER) .map(([id]) => id) .filter((id): id is string => Boolean(id)); - if (ids.length > 1) throw new Error("OpenShell gateway runtime identity is ambiguous."); + if ( + pid.exitCode !== 1 || + pid.timedOut || + pid.signal !== null || + pid.stdout.trim() !== "" || + pid.stderr.trim() !== "" || + container.exitCode !== 0 || + container.timedOut || + container.signal !== null || + container.stderr.trim() !== "" || + container.stdout + .split(/\r?\n/u) + .some((line) => line.trim() !== "" && !/^[a-f0-9]+\s+\S+$/u.test(line.trim())) || + ids.length > 1 + ) { + throw new Error("OpenShell gateway runtime observation failed or its identity is ambiguous."); + } const [id] = ids; return id ? { kind: "container", id } : null; } @@ -237,9 +259,10 @@ export class GatewayClient { }); const statusText = `${status.stdout}\n${status.stderr}`; if ( + (status.exitCode !== 0 && status.exitCode !== 1) || status.timedOut || status.signal !== null || - !/disconnected|no (?:active )?gateway|connection refused|does not exist|not found/iu.test( + !/disconnected|no (?:active )?gateway|connection refused|gateway[^\n]*(?:does not exist|not found)/iu.test( statusText, ) ) { @@ -340,7 +363,10 @@ export class GatewayClient { * The PID is accepted only while the process exists and its `/proc` start * identity still matches the second field recorded by the supervisor. */ - async resolveGatewayIdentity(instance: NemoClawInstance): Promise { + async resolveGatewayIdentity( + instance: Pick, + options: Pick = {}, + ): Promise { const script = "set -e; " + 'record="$(cat /tmp/nemoclaw-gateway.pid 2>/dev/null || true)"; ' + @@ -357,12 +383,15 @@ export class GatewayClient { 'printf "%s %s %s %s\\n" "$pid" "$expected_start" "$actual_start" "$state"'; const result = await this.sandbox.exec(instance.sandboxName, ["sh", "-c", script], { - artifactName: `gateway-resolve-pid-${instance.sandboxName}`, - env: probeEnv(), + artifactName: options.artifactName ?? `gateway-resolve-pid-${instance.sandboxName}`, + env: { ...probeEnv(), ...options.env }, }); const identity = result.stdout.trim().match(/^([0-9]+) ([0-9]+) ([0-9]+) ([A-Za-z])$/); if ( result.exitCode !== 0 || + result.timedOut || + result.signal !== null || + result.stderr.trim() !== "" || !identity || identity[2] !== identity[3] || /^(?:X|Z)$/.test(identity[4]) diff --git a/test/e2e/fixtures/clients/host.ts b/test/e2e/fixtures/clients/host.ts index 53a4581c766..01444c90988 100644 --- a/test/e2e/fixtures/clients/host.ts +++ b/test/e2e/fixtures/clients/host.ts @@ -31,6 +31,7 @@ export interface ForwardCleanupOptions extends ShellProbeRunOptions { export interface SandboxCleanupOptions extends ShellProbeRunOptions { preserveGatewayRegistration?: boolean; + cleanupGateway?: boolean; } export interface ForwardListenerEvidence { @@ -345,9 +346,14 @@ export class HostCliClient { sandboxName: string, options: SandboxCleanupOptions = {}, ): Promise { - const { preserveGatewayRegistration = false, ...probeOptions } = options; + const { + preserveGatewayRegistration = false, + cleanupGateway = false, + ...probeOptions + } = options; const args = [sandboxName, "destroy", "--yes"]; if (preserveGatewayRegistration) args.push("--no-cleanup-gateway"); + else if (cleanupGateway) args.push("--cleanup-gateway"); return await this.nemoclaw(args, { artifactName: `destroy-sandbox-${artifactLabel(sandboxName)}`, env: buildAvailabilityProbeEnv(), diff --git a/test/e2e/fixtures/cloudflared-command.ts b/test/e2e/fixtures/cloudflared-command.ts new file mode 100644 index 00000000000..b915a4dd7f3 --- /dev/null +++ b/test/e2e/fixtures/cloudflared-command.ts @@ -0,0 +1,38 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import path from "node:path"; + +/** A local command fixture for package, extraction, and executable checks. */ +export function cloudflaredCommandFixture(failure: string, bytes: Buffer) { + let packagePath = ""; + const command = async (name: string, args: string[]) => { + let exitCode = 0; + let stdout = ""; + if (name === "curl") { + packagePath = args[args.indexOf("--output") + 1]; + fs.writeFileSync(packagePath, failure === "digest" ? "substituted" : bytes); + } else if (name === "dpkg-deb") { + if (failure === "extract") exitCode = 1; + else { + const binary = path.join(args[2], "usr", "bin", "cloudflared"); + fs.mkdirSync(path.dirname(binary), { recursive: true }); + fs.writeFileSync(binary, "fixture binary", { mode: 0o700 }); + } + } else { + exitCode = failure === "execute" ? 126 : 0; + stdout = `cloudflared version ${failure === "version" ? "2025.1.1" : "2026.6.1"}`; + } + return { + command: [name, ...args], + exitCode, + stdout, + stderr: "", + signal: null, + timedOut: false, + artifacts: { stdout: "", stderr: "", result: "" }, + }; + }; + return { command, packagePath: () => packagePath }; +} diff --git a/test/e2e/fixtures/curl-policy-denial.ts b/test/e2e/fixtures/curl-policy-denial.ts new file mode 100644 index 00000000000..7996c3558e1 --- /dev/null +++ b/test/e2e/fixtures/curl-policy-denial.ts @@ -0,0 +1,130 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { resultText } from "./clients/command.ts"; +import type { ShellProbeResult } from "./shell-probe.ts"; + +const MCP_CURL_HTTP_CODE_MARKER = "NEMOCLAW_MCP_CURL_HTTP_CODE="; +export const INFERENCE_CURL_HTTP_CODE_MARKER = "NEMOCLAW_INFERENCE_CURL_HTTP_CODE="; + +/** Keep connect errors while withholding verbose outgoing headers from artifacts. */ +export const INFERENCE_CURL_DIAGNOSTIC_SCRIPT = `curl() ( + probe_stderr=$(mktemp) || exit 2 + trap 'rm -f "$probe_stderr"' EXIT + LC_ALL=C command curl --verbose --write-out '\n${INFERENCE_CURL_HTTP_CODE_MARKER}%{http_code}\n' "$@" 2>"$probe_stderr" + probe_status=$? + sed -n '/^\\* Immediate connect fail for /p; /^\\* connect to .* failed: /p; /^curl: /p' "$probe_stderr" >&2 + exit "$probe_status" +)`; + +/** + * Require a policy response or curl's explicit connect permission error. + * Exit 7 alone also represents refused connections and is not denial evidence. + */ +export function isExpectedMcpCurlPolicyDenial( + result: Pick, + httpCodeMarker = MCP_CURL_HTTP_CODE_MARKER, + connectDeniedHttpCode: "000" | "302" = "000", +): boolean { + if (result.timedOut) return false; + + const records = result.stdout.split(/\r?\n/u).filter((line) => line.startsWith(httpCodeMarker)); + if (records.length !== 1) return false; + const httpCode = records[0]?.slice(httpCodeMarker.length); + if (result.exitCode === 0) { + return httpCode === "403" && /policy[_ ]denied|not allowed by any policy/iu.test(result.stdout); + } + // A followed redirect retains its last response code when the next connect is denied. + if (result.exitCode === 7 && httpCode === connectDeniedHttpCode) { + // curl 8.14.1 cf-socket.c reports connect(2)'s errno here. Require every + // attempted address to fail with EACCES, not a mixed transport failure. + const failures = result.stderr + .split(/\r?\n/u) + .filter( + (line) => + line.startsWith("* Immediate connect fail for ") || + (line.startsWith("* connect to ") && line.includes(" failed: ")), + ); + return ( + failures.length > 0 && + failures.every((line) => + /^\* Immediate connect fail for [0-9a-f:.]+: Permission denied$/iu.test(line), + ) + ); + } + + return ( + result.exitCode === 56 && + /curl:\s*\(56\)\s*CONNECT tunnel failed,\s*response 403/i.test(result.stderr) + ); +} + +type DnsDenialProbe = Pick< + ShellProbeResult, + "exitCode" | "stdout" | "stderr" | "signal" | "timedOut" +>; + +export function isExpectedMcpDnsPolicyDenial( + result: DnsDenialProbe, + before: DnsDenialProbe, + after: DnsDenialProbe, + hostname: string, + httpCodeMarker = MCP_CURL_HTTP_CODE_MARKER, + connectDeniedHttpCode: "000" | "302" = "000", +): boolean { + const lines = result.stdout.split(/\r?\n/u); + const codes = lines.filter((line) => line.startsWith(httpCodeMarker)); + const locations = lines.filter((line) => /^location:/iu.test(line)); + const location = locations[0]?.slice("location:".length).trim() ?? ""; + const expectedResponse = + connectDeniedHttpCode === "000" + ? result.stdout.trim() === `${httpCodeMarker}000` + : codes.length === 1 && + codes[0] === `${httpCodeMarker}302` && + /^HTTP\/1\.[01] 302 /mu.test(result.stdout) && + locations.length === 1 && + URL.canParse(location) && + new URL(location).hostname === hostname; + if ( + result.timedOut || + result.signal !== null || + result.exitCode !== 6 || + !expectedResponse || + !result.stderr.split(/\r?\n/u).includes(`curl: (6) Could not resolve host: ${hostname}`) || + [before, after].some((logs) => logs.timedOut || logs.signal !== null || logs.exitCode !== 0) + ) + return false; + // OpenShell 0.1.3 filters resolved addresses before issuing a synthetic DNS + // answer. Curl exit 6 alone cannot distinguish that denial from broken DNS. + const events = ["policy_dns_no_valid_address", "policy_dns_ineligible"].map( + (reason) => `NET:REFUSE [MED] DENIED ${hostname} [reason:${reason}]`, + ); + const previous = new Set(resultText(before).split(/\r?\n/u)); + return resultText(after) + .split(/\r?\n/u) + .some((line) => events.some((event) => line.endsWith(event)) && !previous.has(line)); +} + +/** Confirm one request; only repeat audit reads while its DNS refusal reaches the gateway. */ +export async function waitForMcpCurlPolicyDenial( + result: DnsDenialProbe, + before: DnsDenialProbe, + readDnsAudit: (suffix: string) => Promise, + hostname: string, + httpCodeMarker = MCP_CURL_HTTP_CODE_MARKER, + connectDeniedHttpCode: "000" | "302" = "000", +): Promise { + let policyDenied = isExpectedMcpCurlPolicyDenial(result, httpCodeMarker, connectDeniedHttpCode); + for (let attempt = 1; !policyDenied && result.exitCode === 6 && attempt <= 5; attempt += 1) { + policyDenied = isExpectedMcpDnsPolicyDenial( + result, + before, + await readDnsAudit(`after-${attempt}`), + hostname, + httpCodeMarker, + connectDeniedHttpCode, + ); + if (!policyDenied && attempt < 5) await new Promise((resolve) => setTimeout(resolve, 250)); + } + return policyDenied; +} diff --git a/test/e2e/fixtures/dns-rebinding-server.ts b/test/e2e/fixtures/dns-rebinding-server.ts new file mode 100644 index 00000000000..69722673a4d --- /dev/null +++ b/test/e2e/fixtures/dns-rebinding-server.ts @@ -0,0 +1,206 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import dgram from "node:dgram"; +import net from "node:net"; + +const MAX_DNS_BYTES = 8192; + +/** Read one bounded DNS frame, including fragmented TCP deliveries. */ +export function readDnsTcpFrame(client: net.Socket): Promise { + return new Promise((accept, reject) => { + let bytes = Buffer.alloc(0); + const finish = (error: Error | null, payload?: Buffer) => { + client.off("data", receive); + client.off("error", failed); + client.off("end", ended); + client.off("close", ended); + if (error) reject(error); + else accept(payload!); + }; + const failed = (error: Error) => finish(error); + const ended = () => finish(new Error("DNS connection ended before a complete frame")); + const receive = (chunk: Buffer) => { + bytes = Buffer.concat([bytes, chunk]); + if (bytes.length < 2) return; + const length = bytes.readUInt16BE(0); + if (length < 12 || length > MAX_DNS_BYTES || bytes.length > length + 2) { + finish(new Error("Invalid DNS frame")); + } else if (bytes.length === length + 2) { + client.pause(); + finish(null, bytes.subarray(2)); + } + }; + client.on("data", receive); + client.once("error", failed); + client.once("end", ended); + client.once("close", ended); + }); +} + +/** Override one IPv4 fixture answer; preserve the question and return no AAAA answer. */ +export function reboundDnsAnswer(query: Buffer, hostname: string, address: string): Buffer | null { + if (query.length < 17 || query.length > MAX_DNS_BYTES || query.readUInt16BE(4) !== 1) return null; + const labels: string[] = []; + let offset = 12; + while (offset < query.length && query[offset] !== 0) { + const length = query[offset++]!; + if (length > 63 || offset + length >= query.length) return null; + const label = query.subarray(offset, offset + length).toString("utf8"); + if (!/^[a-z0-9-]+$/iu.test(label)) return null; + labels.push(label); + offset += length; + } + offset += 1; + if (offset + 4 > query.length || labels.join(".").toLowerCase() !== hostname) return null; + const type = query.readUInt16BE(offset); + if ((type !== 1 && type !== 28) || query.readUInt16BE(offset + 2) !== 1) return null; + if (net.isIP(address) !== 4) throw new Error("DNS rebinding requires an IPv4 fixture address"); + const reply = Buffer.from(query.subarray(0, offset + 4)); + reply.writeUInt16BE(0x8080 | (query.readUInt16BE(2) & 0x0100), 2); + reply.writeUInt16BE(type === 1 ? 1 : 0, 6); + reply.writeUInt32BE(0, 8); + // The answer points to the unchanged question name and has TTL zero. + return type === 28 + ? reply + : Buffer.concat([ + reply, + Buffer.from([192, 12, 0, 1, 0, 1, 0, 0, 0, 0, 0, 4, ...address.split(".").map(Number)]), + ]); +} + +export async function startRebindingDnsServer( + hostname: string, + upstream: string, + upstreamPort = 53, +) { + const family = net.isIP(upstream); + if ( + !family || + hostname.length > 253 || + !/^(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z0-9-]+$/u.test(hostname) + ) { + throw new Error("DNS fixture requires one literal upstream and a canonical hostname"); + } + const socketType = family === 4 ? "udp4" : "udp6"; + const loopback = family === 4 ? "127.0.0.1" : "::1"; + let address: string | null = null; + let answerCount = 0; + let closed = false; + const pending = new Set(); + const clients = new Set(); + const resolve = async (query: Buffer, tcpQuery = false): Promise => { + const answer = address ? reboundDnsAnswer(query, hostname, address) : null; + if (answer) { + if (answer.readUInt16BE(6) === 1) answerCount += 1; + return answer; + } + if (query.length < 12 || query.length > MAX_DNS_BYTES || pending.size >= 32) { + throw new Error("DNS fixture query limit exceeded"); + } + if (tcpQuery) { + return await new Promise((accept, reject) => { + const client = net.createConnection({ host: upstream, port: upstreamPort }); + clients.add(client); + client.on("close", () => clients.delete(client)); + client.once("error", reject); + client.setTimeout(3000, () => client.destroy(new Error("DNS upstream timed out"))); + const prefix = Buffer.alloc(2); + prefix.writeUInt16BE(query.length); + client.on("connect", () => client.write(Buffer.concat([prefix, query]))); + void readDnsTcpFrame(client).then( + (response) => { + if (response.readUInt16BE(0) === query.readUInt16BE(0)) accept(response); + else reject(new Error("Upstream DNS response identity changed")); + client.destroy(); + }, + (error: Error) => { + reject(error); + client.destroy(); + }, + ); + }); + } + return await new Promise((accept, reject) => { + const socket = dgram.createSocket(socketType); + pending.add(socket); + const timer = setTimeout(() => finish(new Error("DNS upstream timed out")), 3000); + const finish = (error: Error | null, response?: Buffer) => { + if (!pending.delete(socket)) return; + clearTimeout(timer); + socket.close(); + if (error) reject(error); + else accept(response!); + }; + socket.once("error", (error) => finish(error)); + socket.on("message", (response) => { + if ( + response.length >= 12 && + response.length <= MAX_DNS_BYTES && + response.readUInt16BE(0) === query.readUInt16BE(0) + ) { + finish(null, response); + } + }); + socket.connect(upstreamPort, upstream, () => socket.send(query)); + }); + }; + const udp = dgram.createSocket(socketType); + udp.on("message", (query, peer) => { + void resolve(query) + .then((reply) => udp.send(reply, peer.port, peer.address)) + .catch(() => {}); + }); + await new Promise((accept, reject) => { + udp.once("error", reject); + udp.bind(0, loopback, accept); + }); + const port = udp.address().port; + const tcp = net.createServer((client) => { + if (clients.size >= 32) { + client.destroy(); + return; + } + clients.add(client); + client.on("close", () => clients.delete(client)); + client.on("error", () => client.destroy()); + client.setTimeout(3000, () => client.destroy()); + void readDnsTcpFrame(client) + .then((query) => resolve(query, true)) + .then((reply) => { + const prefix = Buffer.alloc(2); + prefix.writeUInt16BE(reply.length); + client.end(Buffer.concat([prefix, reply])); + }) + .catch(() => client.destroy()); + }); + try { + await new Promise((accept, reject) => { + tcp.once("error", reject); + tcp.listen(port, loopback, accept); + }); + } catch (error) { + udp.close(); + throw error; + } + return { + port, + family, + answers: () => answerCount, + setAddress(value: string) { + if (net.isIP(value) !== 4) throw new Error("DNS rebinding requires an IPv4 fixture address"); + address = value; + answerCount = 0; + }, + async close() { + if (closed) return; + closed = true; + for (const client of clients) client.destroy(new Error("DNS fixture stopped")); + for (const socket of pending) socket.emit("error", new Error("DNS fixture stopped")); + udp.close(); + await new Promise((accept, reject) => + tcp.close((error) => (error ? reject(error) : accept())), + ); + }, + }; +} diff --git a/test/e2e/fixtures/external-gateway-health-config.ts b/test/e2e/fixtures/external-gateway-health-config.ts new file mode 100644 index 00000000000..7b1ba7af15a --- /dev/null +++ b/test/e2e/fixtures/external-gateway-health-config.ts @@ -0,0 +1,30 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import type { DockerDriverGatewayLocalTlsBundle } from "../../../src/lib/onboard/docker-driver-gateway-local-tls.ts"; + +/** Public health uses server TLS; sandboxes receive only the gateway CA, never user credentials. */ +export function externalGatewayHealthConfig( + address: string, + port: number, + tls: DockerDriverGatewayLocalTlsBundle, +): string { + return [ + "[openshell]", + "version = 2", + "", + "[openshell.gateway]", + `bind_address = ${JSON.stringify(`${address}:${String(port)}`)}`, + 'compute_driver = "docker"', + "disable_tls = false", + `guest_tls_ca = ${JSON.stringify(tls.caPath)}`, + "", + "[openshell.gateway.tls]", + `cert_path = ${JSON.stringify(tls.serverCertPath)}`, + `key_path = ${JSON.stringify(tls.serverKeyPath)}`, + "", + "[openshell.gateway.auth]", + "allow_unauthenticated_users = true", + "", + ].join("\n"); +} diff --git a/test/e2e/fixtures/gateway-provider-policy-binding.ts b/test/e2e/fixtures/gateway-provider-policy-binding.ts index cf51db4b0ef..0b0ca7e4923 100644 --- a/test/e2e/fixtures/gateway-provider-policy-binding.ts +++ b/test/e2e/fixtures/gateway-provider-policy-binding.ts @@ -53,14 +53,14 @@ export function bindFixtureProviderPolicyEndpoint( host: string, port: number, protocol: string, - restMethods?: readonly ("GET" | "POST")[], + allowMethods?: readonly ("GET" | "POST" | "WEBSOCKET_TEXT")[], ): void { const policy = readPolicy(policyFile); const { endpoint } = findFixturePolicyEndpoint(policy, host, port, protocol); endpoint.credential_binding = { provider: providerName }; - if (restMethods) { + if (allowMethods) { delete endpoint.access; - endpoint.rules = restMethods.map((method) => ({ allow: { method, path: "/**" } })); + endpoint.rules = allowMethods.map((method) => ({ allow: { method, path: "/**" } })); } fs.writeFileSync(policyFile, YAML.stringify(policy)); fs.chmodSync(policyFile, 0o600); diff --git a/test/e2e/fixtures/gateway-providers.ts b/test/e2e/fixtures/gateway-providers.ts index 901157aff13..864222eb4ad 100644 --- a/test/e2e/fixtures/gateway-providers.ts +++ b/test/e2e/fixtures/gateway-providers.ts @@ -92,7 +92,7 @@ async function setFixtureProviderPolicyBinding( }; readonly expectedBinaries?: readonly string[]; readonly removeBinding?: boolean; - readonly restMethods?: readonly ("GET" | "POST")[]; + readonly allowMethods?: readonly ("GET" | "POST" | "WEBSOCKET_TEXT")[]; }, ): Promise { const policy = await host.command( @@ -120,7 +120,7 @@ async function setFixtureProviderPolicyBinding( options.endpoint.host, options.endpoint.port, options.endpoint.protocol, - options.restMethods, + options.allowMethods, ); if (options.expectedBinaries) { assertFixtureProviderPolicyEndpointBinaries( @@ -162,6 +162,7 @@ export async function rebindFixtureProviderPolicyEndpoint( options: { readonly artifactName: string; readonly credentialEnv: string; + readonly allowMethods?: readonly ("GET" | "POST" | "WEBSOCKET_TEXT")[]; readonly endpoint: { readonly host: string; readonly port: number | string; @@ -199,6 +200,7 @@ export async function rebindFixtureProviderPolicyEndpoint( ...commandOptions, artifactName: `${options.artifactName}-policy-rebound`, endpoint: { ...options.endpoint, port: endpointPort }, + allowMethods: options.allowMethods, }); } @@ -236,7 +238,7 @@ export async function applyFixtureProviderPolicyEndpoint( }); const policyHost = "host.openshell.internal"; - const methods = + const methods: readonly ("GET" | "POST" | "WEBSOCKET_TEXT")[] = options.protocol === "rest" ? (options.restMethods ?? ["GET", "POST"]) : ["GET", "WEBSOCKET_TEXT"]; @@ -247,8 +249,6 @@ export async function applyFixtureProviderPolicyEndpoint( "--add-endpoint", `${policyHost}:${String(endpointPort)}:read-write:${options.protocol}:enforce:${options.rewrite},allowed-ip=10.0.0.0/8,allowed-ip=172.16.0.0/12,allowed-ip=192.168.0.0/16`, ]; - for (const method of methods) - args.push("--add-allow", `${policyHost}:${String(endpointPort)}:${method}:/**`); args.push(...allowedBinaries.flatMap((binary) => ["--binary", binary]), "--wait"); await runFixtureOpenShell(host, args, commandOptions); await setFixtureProviderPolicyBinding(host, sandboxName, { @@ -256,7 +256,7 @@ export async function applyFixtureProviderPolicyEndpoint( artifactName: `${options.artifactName}-credential-binding`, endpoint: { host: policyHost, port: endpointPort, protocol: options.protocol }, expectedBinaries: allowedBinaries, - restMethods: options.protocol === "rest" ? (options.restMethods ?? ["GET", "POST"]) : undefined, + allowMethods: methods, }); } @@ -266,19 +266,22 @@ export async function createExtraGatewayProvider( options: { artifactName: string; env: NodeJS.ProcessEnv; + profilePath?: string; + credentialEnv?: string; }, ): Promise { assertProviderName(providerName); - const credential = options.env[EXTRA_PROVIDER_CREDENTIAL_ENV]; + const credentialEnv = options.credentialEnv ?? EXTRA_PROVIDER_CREDENTIAL_ENV; + const profilePath = options.profilePath ?? EXTRA_PROVIDER_PROFILE; + if (!CREDENTIAL_ENV.test(credentialEnv)) throw new Error("Unsafe fixture credential environment"); + const credential = options.env[credentialEnv]; if (!credential) { - throw new Error(`Missing provider credential env value: ${EXTRA_PROVIDER_CREDENTIAL_ENV}`); + throw new Error(`Missing provider credential env value: ${credentialEnv}`); } - const contract = parseCheckedInProviderProfileContract( - fs.readFileSync(EXTRA_PROVIDER_PROFILE, "utf8"), - ); + const contract = parseCheckedInProviderProfileContract(fs.readFileSync(profilePath, "utf8")); if (!contract) throw new Error("The E2E extra-provider profile is invalid."); const profileEnv = { ...options.env }; - delete profileEnv[EXTRA_PROVIDER_CREDENTIAL_ENV]; + delete profileEnv[credentialEnv]; const run = (args: string[], artifactName: string, env: NodeJS.ProcessEnv) => host.command(host.openshellCommandPath, scopeGatewayOpenshellArgs(args, "nemoclaw"), { artifactName, @@ -287,7 +290,7 @@ export async function createExtraGatewayProvider( timeoutMs: 60_000, }); const imported = await run( - ["provider", "profile", "import", "--file", EXTRA_PROVIDER_PROFILE], + ["provider", "profile", "import", "--file", profilePath], `${options.artifactName}-profile-import`, profileEnv, ); @@ -314,7 +317,7 @@ export async function createExtraGatewayProvider( "--type", contract.profileId, "--credential", - EXTRA_PROVIDER_CREDENTIAL_ENV, + credentialEnv, ], options.artifactName, options.env, diff --git a/test/e2e/fixtures/gpu-install-failure.ts b/test/e2e/fixtures/gpu-install-failure.ts new file mode 100644 index 00000000000..6319fa86550 --- /dev/null +++ b/test/e2e/fixtures/gpu-install-failure.ts @@ -0,0 +1,168 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import path from "node:path"; +import type { SandboxClient } from "./clients/sandbox.ts"; +import { resultText } from "./clients/index.ts"; +import type { ArtifactSink } from "./artifacts.ts"; +import type { ShellProbeResult } from "./shell-probe.ts"; +import { stripAnsi } from "../live/json-envelope.ts"; +import { getNemoclawStateRoot } from "../../../src/lib/state/state-root.ts"; + +/** Preserve this install attempt's evidence before the fixture destroys its sandbox. */ +export async function captureGpuInstallFailure( + install: Pick, + sandbox: Pick, + artifacts: Pick, + environment: NodeJS.ProcessEnv, + sandboxName: string, +): Promise { + if (install.exitCode === 0) return; + const output = stripAnsi(resultText(install)); + const attempts = new Set( + [...output.matchAll(/ai\.nvidia\.nemoclaw\.create-attempt=([a-f0-9]{62})(?![a-f0-9])/gu)].map( + (match) => match[1], + ), + ); + if (attempts.size !== 1) return; + const attempt = [...attempts][0]; + try { + const root = path.join(getNemoclawStateRoot(environment.HOME), "onboard-failures"); + const directories = new Set( + [...output.matchAll(/^\s*Diagnostics saved: (.+)$/gmu)].map((match) => match[1]), + ); + const directory = directories.size === 1 ? [...directories][0] : undefined; + const started = Date.parse(install.startedAt ?? ""); + const finished = Date.parse(install.finishedAt ?? ""); + if (directory && path.dirname(directory) === root) { + const meta = fs.lstatSync(directory); + const rootMeta = fs.lstatSync(root); + const expectedSuffix = `-${sandboxName.slice(0, 80)}`; + const timestamp = path.basename(directory).slice(0, -expectedSuffix.length); + const emittedAt = Date.parse( + timestamp.replace(/T(\d{2})-(\d{2})-(\d{2})-(\d{3})Z$/u, "T$1:$2:$3.$4Z"), + ); + if ( + meta.isDirectory() && + !meta.isSymbolicLink() && + rootMeta.isDirectory() && + !rootMeta.isSymbolicLink() && + rootMeta.uid === process.getuid?.() && + (rootMeta.mode & 0o077) === 0 && + fs.realpathSync(root) === root && + meta.uid === process.getuid?.() && + (meta.mode & 0o077) === 0 && + meta.birthtimeMs >= started && + meta.birthtimeMs <= finished && + emittedAt >= started && + emittedAt <= finished && + path.basename(directory).endsWith(expectedSuffix) + ) { + const file = path.join(directory, "sandbox-container.log"); + const fd = fs.openSync( + file, + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | fs.constants.O_NONBLOCK, + ); + try { + const before = fs.fstatSync(fd); + if ( + before.isFile() && + before.nlink === 1 && + before.uid === meta.uid && + (before.mode & 0o077) === 0 && + before.size <= 8192 + ) { + // Read through the verified descriptor; accept output only after file and parent + // metadata remain stable. + const buffer = Buffer.alloc(8193); + const length = fs.readSync(fd, buffer, { + offset: 0, + length: buffer.length, + position: 0, + }); + const after = fs.fstatSync(fd); + const parents = [ + [root, rootMeta], + [directory, meta], + ] as const; + const stableParents = parents.every(([name, expected]) => { + const current = fs.lstatSync(name); + return ( + current.isDirectory() && + current.dev === expected.dev && + current.ino === expected.ino && + current.uid === expected.uid && + current.mode === expected.mode && + current.ctimeMs === expected.ctimeMs + ); + }); + if ( + stableParents && + length === before.size && + length <= 8192 && + before.size === after.size && + before.mtimeMs === after.mtimeMs && + before.ctimeMs === after.ctimeMs && + before.mode === after.mode && + before.nlink === after.nlink + ) { + await artifacts.writeText( + "gpu-install-failure-container.log", + buffer.subarray(0, length).toString("utf8"), + ); + } + } + } finally { + fs.closeSync(fd); + } + } + } + } catch { + // A missing or untrusted bundle must not replace the installer failure. + } + try { + const observed = await sandbox.openshell(["sandbox", "get", sandboxName, "--output", "json"], { + env: environment, + timeoutMs: 15_000, + captureLimitBytes: 64 * 1024, + persistArtifacts: false, + }); + if (observed.exitCode !== 0 || observed.timedOut || observed.signal !== null) return; + const status = JSON.parse(observed.stdout); + if ( + status.name !== sandboxName || + status.labels?.["ai.nvidia.nemoclaw.create-attempt"] !== attempt + ) + return; + const numeric = Object.fromEntries( + ["exit_code", "restart_count", "main_process_started_at_ms", "next_restart_at_ms"] + .filter((key) => Number.isSafeInteger(status[key])) + .map((key) => [key, status[key]]), + ); + const phase = [ + "Unspecified", + "Provisioning", + "Ready", + "Error", + "Deleting", + "Stopping", + "Stopped", + "Starting", + "Completed", + "Unknown", + ].includes(status.phase) + ? status.phase + : "unavailable"; + await artifacts.writeJson("gpu-install-failure-process.json", { + attempt, + phase, + mainProcessObserved: + typeof status.main_process_instance_id === "string" && + status.main_process_instance_id.length > 0, + ...numeric, + }); + } catch { + // Raw status, policy, environment, and error strings are never artifact output. + } +} diff --git a/test/e2e/fixtures/hermes-acp-live.ts b/test/e2e/fixtures/hermes-acp-live.ts index e4a00c1aaf5..bc174069452 100644 --- a/test/e2e/fixtures/hermes-acp-live.ts +++ b/test/e2e/fixtures/hermes-acp-live.ts @@ -222,7 +222,7 @@ async function terminateRemoteHermesAcp( return result.exitCode === 0; } -async function verifyNoRemoteHermesAcpProcess( +export async function verifyNoRemoteHermesAcpProcess( sandbox: SandboxClient, sandboxName: string, env: NodeJS.ProcessEnv, @@ -234,7 +234,13 @@ async function verifyNoRemoteHermesAcpProcess( env: hermesAcpLiveHostEnv(env), timeoutMs: 30_000, }); - if (result.exitCode === 1) return true; + if (result.exitCode === 1) + return ( + !result.timedOut && + result.signal === null && + result.stdout.trim() === "" && + result.stderr.trim() === "" + ); if (result.exitCode !== 0) return false; await sleep(1_000); } diff --git a/test/e2e/fixtures/hermes-switchyard-package.ts b/test/e2e/fixtures/hermes-switchyard-package.ts index 11d3ddbbecd..95b98ece712 100644 --- a/test/e2e/fixtures/hermes-switchyard-package.ts +++ b/test/e2e/fixtures/hermes-switchyard-package.ts @@ -260,8 +260,11 @@ with TemporaryDirectory() as directory, LocalUpstream('efficient') as efficient, os.environ[name] = value assert nemo_relay.plugin.report() is None capture_maps('after_shutdown') +# Record loader state without requiring immediate unmapping after shutdown. +print('Switchyard library mapped after shutdown:', str(library) in Path('/proc/self/maps').read_text(), + file=sys.stderr, flush=True) # The separate probe must not change the gateway's routing state. assert library.name not in Path(f'/proc/{pid}/maps').read_text() -print('Switchyard package, disabled gateway, adaptive route, native activation, and cleanup verified') +print('Switchyard package, disabled gateway, adaptive route, native activation, and cleanup verified', flush=True) PY `; diff --git a/test/e2e/fixtures/host-address.ts b/test/e2e/fixtures/host-address.ts index 19e488d905c..a23f4c67514 100644 --- a/test/e2e/fixtures/host-address.ts +++ b/test/e2e/fixtures/host-address.ts @@ -71,6 +71,14 @@ export async function discoverHostAddress( } return { source: "runtime-provider", address: runtimeProviderAddress, probe: null }; } + return discoverHostInterfaceAddress(host, artifactName); +} + +/** Host-network listeners need a real interface address, not a sandbox gateway alias. */ +export async function discoverHostInterfaceAddress( + host: HostCliClient, + artifactName: string, +): Promise { const probe = await host.command( "bash", [ diff --git a/test/e2e/fixtures/inference-adapter.ts b/test/e2e/fixtures/inference-adapter.ts index 1abef226c9c..20319af4d6e 100644 --- a/test/e2e/fixtures/inference-adapter.ts +++ b/test/e2e/fixtures/inference-adapter.ts @@ -2,6 +2,10 @@ // SPDX-License-Identifier: Apache-2.0 import { randomBytes } from "node:crypto"; +import { + nativeLocalCredentialReference, + NATIVE_LOCAL_AUTH_HEADER_SCRIPT, +} from "../../../src/lib/inference/native-local/agent-config.ts"; import { DEFAULT_CLOUD_MODEL } from "../../../src/lib/inference/config.ts"; import { nativeHostedAuthHeaderScript } from "../../../src/lib/inference/native-provider/contract.ts"; import { @@ -469,19 +473,22 @@ export function buildSandboxInferenceChatCommand( requestBody: string, logicalProvider: string, endpointUrl?: string, + maxTimeSeconds: 60 | 90 = 90, ): [string, ...string[]] { const nativeCustom = logicalProvider === "compatible-endpoint" && endpointUrl; if (logicalProvider === NVIDIA_HOSTED_LOGICAL_PROVIDER || nativeCustom) { const auth = nativeCustom - ? nativeHostedAuthHeaderScript({ - credentialEnv: "COMPATIBLE_API_KEY", - api: "openai-completions", - }) + ? nativeLocalCredentialReference(logicalProvider, endpointUrl, "openclaw") + ? NATIVE_LOCAL_AUTH_HEADER_SCRIPT + : nativeHostedAuthHeaderScript({ + credentialEnv: "COMPATIBLE_API_KEY", + api: "openai-completions", + }) : NATIVE_NVIDIA_AUTH_HEADER_SCRIPT; return [ "bash", "-c", - `${auth}; exec curl -fsS --max-time 90 "$1" -H 'Content-Type: application/json' -H "$AUTH_HEADER" --data-raw "$2"`, + `${auth}; exec curl -fsS --max-time ${maxTimeSeconds} "$1" -H 'Content-Type: application/json' -H "$AUTH_HEADER" --data-raw "$2"`, "full-e2e-native-inference", `${(nativeCustom ? endpointUrl : NVIDIA_HOSTED_NATIVE_ENDPOINT).replace(/\/+$/u, "")}/chat/completions`, requestBody, @@ -491,7 +498,7 @@ export function buildSandboxInferenceChatCommand( "curl", "-fsS", "--max-time", - "90", + String(maxTimeSeconds), "https://inference.local/v1/chat/completions", "-H", "Content-Type: application/json", diff --git a/test/e2e/fixtures/native-plugin-fixture.ts b/test/e2e/fixtures/native-plugin-fixture.ts new file mode 100644 index 00000000000..dc69bc4bbbf --- /dev/null +++ b/test/e2e/fixtures/native-plugin-fixture.ts @@ -0,0 +1,70 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { shellQuote } from "../../../src/lib/core/shell-quote.ts"; + +export type NativePluginVersion = "v1" | "v2"; + +export function nativeWeatherPluginWriteScript(version: NativePluginVersion): string { + const packageJson = JSON.stringify({ + name: "@nemoclaw/e2e-native-weather", + version: version === "v1" ? "1.0.0" : "2.0.0", + type: "module", + main: "index.js", + files: ["index.js", "openclaw.plugin.json"], + openclaw: { extensions: ["./index.js"] }, + peerDependencies: { openclaw: ">=2026.7.1" }, + }); + const manifest = JSON.stringify({ + id: "weather", + name: "NemoClaw Native Weather", + version: version === "v1" ? "1.0.0" : "2.0.0", + description: "Dependency-free native plugin lifecycle fixture", + activation: { onStartup: true }, + contracts: { tools: ["get_weather"] }, + configSchema: { + type: "object", + properties: {}, + additionalProperties: false, + }, + }); + const entrypoint = `const plugin = { + id: "weather", + name: "NemoClaw Native Weather", + version: ${JSON.stringify(version === "v1" ? "1.0.0" : "2.0.0")}, + description: "Dependency-free native plugin lifecycle fixture", + register(api) { + api.registerTool({ + name: "get_weather", + label: "Get Weather", + description: "Return deterministic weather data.", + parameters: { + type: "object", + properties: { location: { type: "string" } }, + required: ["location"], + additionalProperties: false, + }, + async execute(_toolCallId, params) { + const details = { + location: String(params.location), + condition: "clear", + fixtureVersion: ${JSON.stringify(version)}, + }; + return { content: [{ type: "text", text: JSON.stringify(details) }], details }; + }, + }); + }, +}; +export default plugin; +`; + return [ + "set -eu", + "source_dir=/sandbox/e2e-native-weather", + 'rm -rf -- "$source_dir"', + 'mkdir -p -- "$source_dir"', + `printf '%s' ${shellQuote(packageJson)} > "$source_dir/package.json"`, + `printf '%s' ${shellQuote(manifest)} > "$source_dir/openclaw.plugin.json"`, + `printf '%s' ${shellQuote(entrypoint)} > "$source_dir/index.js"`, + 'HOME=/sandbox openclaw plugins install --force --accept-capabilities "$source_dir"', + ].join("\n"); +} diff --git a/test/e2e/fixtures/openclaw-agent-output.ts b/test/e2e/fixtures/openclaw-agent-output.ts index 64a62c7402a..73353727cbe 100644 --- a/test/e2e/fixtures/openclaw-agent-output.ts +++ b/test/e2e/fixtures/openclaw-agent-output.ts @@ -137,7 +137,7 @@ export function nativeStateProcessIdentitiesAreValid( const gateways = processes.filter((row) => row[4] === "openclaw-gatewa"); const gateway = gateways[0]; const parent = processes.find((row) => row[2] === gateway?.[3]); - const supervisor = processes.find((row) => row[2] === "1"); + const init = processes.find((row) => row[2] === "1"); return ( result.timedOut === false && result.exitCode === 0 && @@ -151,8 +151,8 @@ export function nativeStateProcessIdentitiesAreValid( parent[3] === "1" && gateway?.[0] === parent[0] && gateway?.[1] === parent[1] && - supervisor?.[0] === "0" && - supervisor[1] === "0" && - supervisor[3] === "0" + init?.[0] === gateway?.[0] && + init?.[1] === gateway?.[1] && + init?.[3] === "0" ); } diff --git a/test/e2e/fixtures/openshell-v0116-qualification.ts b/test/e2e/fixtures/openshell-v0116-qualification.ts index aacf6a870c9..a6846b56173 100644 --- a/test/e2e/fixtures/openshell-v0116-qualification.ts +++ b/test/e2e/fixtures/openshell-v0116-qualification.ts @@ -9,6 +9,34 @@ export const OPENSHELL_V0116_QUALIFICATION = Object.freeze({ version: "0.0.116", }); +export const OPENSHELL_V012_QUALIFICATION = Object.freeze({ + sourceRevision: "6648bd0c290efbc41ba131ee9831ee45cd431f94", + supervisorImage: + "ghcr.io/nvidia/openshell/supervisor@sha256:d7b5264bb6bc56f4796e6fa3617b8e4a8d785be0b7293542efd8cc250b0fb67a", + version: "0.1.2", +}); + +export const OPENSHELL_V013_QUALIFICATION = Object.freeze({ + sourceRevision: "e1f3c82caa3ed3b65de22889ae7ef32a774878ef", + supervisorImage: + "ghcr.io/nvidia/openshell/supervisor@sha256:4ab2b970cdb02c7518ba4ccbbadb4861ae9a8d0b8770e5b160697f0453b0a8e4", + version: "0.1.3", +}); + +export const OPENSHELL_V015_QUALIFICATION = Object.freeze({ + sourceRevision: "bdb956f2cd65a635f1294fccf1d138d263bf650f", + supervisorImage: + "ghcr.io/nvidia/openshell/supervisor@sha256:fabcd01f656e9d23df4386f82e80b4f48b90f3452d7d5c58889007643a069161", + version: "0.1.5", +}); + +export const OPENSHELL_V017_QUALIFICATION = Object.freeze({ + sourceRevision: "5d25211b25a8fc887e8d5c606d57d6ab22f1d76c", + supervisorImage: + "ghcr.io/nvidia/openshell/supervisor@sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06", + version: "0.1.7", +}); + export function exactGatewayRelease(versionOutput: string): string | null { const tokens = versionOutput.match(/(?:^|\s)v?(\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.-]+)?)(?=\s|$)/gu); if (tokens?.length !== 1) return null; diff --git a/test/e2e/fixtures/owned-sandbox-cleanup.ts b/test/e2e/fixtures/owned-sandbox-cleanup.ts index 50149492c8b..6f4d77cd7d1 100644 --- a/test/e2e/fixtures/owned-sandbox-cleanup.ts +++ b/test/e2e/fixtures/owned-sandbox-cleanup.ts @@ -1,13 +1,17 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { MANAGED_IMAGE_AGENTS } from "../../../src/lib/onboard/managed-image/agents.ts"; +import { managedStartupStateRootOwnership } from "../../../src/lib/onboard/managed-startup/state-roots.ts"; import { resolveSandboxGatewayName } from "../../../src/lib/onboard/gateway-binding.ts"; import { getSandbox } from "../../../src/lib/state/registry.ts"; import { buildAvailabilityProbeEnv } from "./availability-env.ts"; import type { CleanupRegistry } from "./cleanup.ts"; import { cleanupAcquiredResource } from "./cleanup-resources.ts"; import type { HostCliClient } from "./clients/host.ts"; -import type { SandboxClient } from "./clients/sandbox.ts"; +import { type SandboxClient, validateSandboxName } from "./clients/sandbox.ts"; +import type { RuntimeProviderPrerequisite } from "./runtime-provider.ts"; +import type { ShellProbeRunOptions } from "./shell-probe.ts"; import { initializeGatewayForCleanup } from "./gateway-runtime-start.ts"; function buildOwnedSandboxCleanupEnv( @@ -28,6 +32,94 @@ function buildOwnedSandboxCleanupEnv( }; } +/** Avoid named CLI recovery when this job has no recorded sandbox to destroy. */ +export function cleanupRegisteredSandbox( + sandboxName: string, + cleanup: () => Promise, +): Promise { + return cleanupAcquiredResource(getSandbox(sandboxName) !== null, cleanup); +} + +/** Verify absence after onboarding released its gateway before creating a sandbox. */ +export async function cleanupOwnedSandboxAfterOnboard( + sandbox: Pick, + runtimeProvider: Pick, + sandboxName: string, + options: ShellProbeRunOptions, +): Promise { + validateSandboxName(sandboxName); + const gatewayName = resolveSandboxGatewayName({ + gatewayName: options.env?.OPENSHELL_GATEWAY?.trim() || "nemoclaw", + }); + const scopedOptions = { + ...options, + env: { ...options.env, OPENSHELL_GATEWAY: gatewayName }, + }; + if (await sandbox.hasGatewayForInitialCleanup(gatewayName, scopedOptions)) { + await sandbox.cleanupSandbox(sandboxName, scopedOptions); + return; + } + if (getSandbox(sandboxName) !== null) { + throw new Error( + `Cannot verify cleanup of registered sandbox '${sandboxName}' without its gateway.`, + ); + } + // Include stopped workloads and supervisors; state volumes can outlive a failed create. + const requests = [ + { + kind: "containers", + args: [ + "container", + "ps", + "--all", + "--no-trunc", + "--filter", + `label=openshell.ai/sandbox-name=${sandboxName}`, + "--format", + "{{.ID}}", + ], + }, + ...MANAGED_IMAGE_AGENTS.flatMap((agent) => + managedStartupStateRootOwnership({ agent, sandboxName }).map((root) => ({ + kind: root.resourceIdentity, + args: [ + "volume", + "ls", + "--filter", + `name=${root.resourceIdentity}`, + "--format", + "{{.Name}}", + ], + })), + ), + ]; + const inventories = await Promise.all( + requests.map(({ kind, args }) => + runtimeProvider.command(args, { + ...scopedOptions, + artifactName: `cleanup-${sandboxName}-${kind}`, + captureLimitBytes: 4096, + timeoutMs: Math.min(options.timeoutMs ?? 30_000, 30_000), + }), + ), + ); + if ( + inventories.some( + (inventory) => + inventory.exitCode !== 0 || + inventory.signal !== null || + inventory.timedOut || + inventory.stdout !== "" || + inventory.stderr !== "", + ) || + getSandbox(sandboxName) !== null + ) { + throw new Error( + `Cannot verify runtime absence for sandbox '${sandboxName}' after its gateway was removed.`, + ); + } +} + /** Prepare a sandbox name exclusively owned by this isolated qualification job. */ export async function prepareOwnedSandboxForOnboard( host: Pick, @@ -37,8 +129,8 @@ export async function prepareOwnedSandboxForOnboard( orphanGatewayName?: string, baseEnv: NodeJS.ProcessEnv = process.env, ): Promise { - const cleanupRegisteredSandbox = (artifactName: string) => - cleanupAcquiredResource(getSandbox(sandboxName) !== null, () => + const cleanupRegistered = (artifactName: string) => + cleanupRegisteredSandbox(sandboxName, () => host.cleanupSandbox(sandboxName, { artifactName, env: buildOwnedSandboxCleanupEnv(sandboxName, orphanGatewayName, baseEnv), @@ -46,7 +138,7 @@ export async function prepareOwnedSandboxForOnboard( }), ); cleanup.trackDisposable(`destroy sandbox ${sandboxName}`, () => - cleanupRegisteredSandbox("cleanup-destroy-sandbox"), + cleanupRegistered("cleanup-destroy-sandbox"), ); const cleanupOwnedSandbox = async (artifactName: string) => { const openshellCleanupEnv = buildOwnedSandboxCleanupEnv( @@ -74,5 +166,5 @@ export async function prepareOwnedSandboxForOnboard( cleanupOwnedSandbox("cleanup-delete-openshell-sandbox"), ); await cleanupOwnedSandbox("precleanup-delete-openshell-sandbox"); - await cleanupRegisteredSandbox("precleanup-destroy-sandbox"); + await cleanupRegistered("precleanup-destroy-sandbox"); } diff --git a/test/e2e/fixtures/phases/lifecycle-dcode-invalid-credential.ts b/test/e2e/fixtures/phases/lifecycle-dcode-invalid-credential.ts index 0903abaf895..8c54a332d15 100644 --- a/test/e2e/fixtures/phases/lifecycle-dcode-invalid-credential.ts +++ b/test/e2e/fixtures/phases/lifecycle-dcode-invalid-credential.ts @@ -4,9 +4,10 @@ import { randomUUID } from "node:crypto"; import { DCODE_MANAGED_EXEC_LAUNCHER } from "../../../../src/lib/actions/sandbox/connect-inference-route-probe.ts"; +import { parseCliOpenShellProviderMetadata } from "../../../../src/lib/adapters/openshell/provider-metadata-cli.ts"; import { nativeHostedAuthHeaderScript } from "../../../../src/lib/inference/native-provider/contract.ts"; import { hostedNativeProviderForAttachment } from "../../../../src/lib/inference/native-provider/hosted-attachment.ts"; -import { recordedNativeProviderAttachment } from "../../../../src/lib/inference/native-provider/recorded-selection.ts"; + import { buildAvailabilityProbeEnv } from "../availability-env.ts"; import { assertExitZero, outputContainsReadySandbox, resultText } from "../clients/command.ts"; import type { HostCliClient } from "../clients/host.ts"; @@ -44,8 +45,12 @@ const NATIVE_STATE_CASES = [ export interface DcodeInvalidCredentialRebuildOptions { gatewayName: string; providerName: string; - credentialEnv: string; + providerId: string; + profileId: string; + endpointUrl: string; + /** Fully resolved native completion endpoint, used only by the managed probe launcher. */ endpoint: string; + credentialEnv: string; model: string; validCredential: string; } @@ -80,10 +85,10 @@ export function dcodeInvalidCredentialRebuildOptionsFromRegistryEntry( if (!validCredential) { throw new Error("DCode invalid-credential lifecycle requires the original provider credential"); } - const logicalProvider = requiredString(entry, "provider"); - if (logicalProvider !== HOSTED_INFERENCE_PROVIDER_NAME) { + const providerName = requiredString(entry, "provider"); + if (providerName !== HOSTED_INFERENCE_PROVIDER_NAME) { throw new Error( - `DCode invalid-credential lifecycle requires provider '${HOSTED_INFERENCE_PROVIDER_NAME}', got '${logicalProvider}'`, + `DCode invalid-credential lifecycle requires provider '${HOSTED_INFERENCE_PROVIDER_NAME}', got '${providerName}'`, ); } const credentialEnv = @@ -95,29 +100,27 @@ export function dcodeInvalidCredentialRebuildOptionsFromRegistryEntry( `DCode invalid-credential lifecycle requires credential env '${HOSTED_INFERENCE_CREDENTIAL_ENV}'`, ); } - const attachment = recordedNativeProviderAttachment({ - provider: logicalProvider, - endpointUrl: typeof entry.endpointUrl === "string" ? entry.endpointUrl : undefined, - credentialEnv, - preferredInferenceApi: - typeof entry.preferredInferenceApi === "string" ? entry.preferredInferenceApi : undefined, - nativeHostedProviderAttachment: entry.nativeHostedProviderAttachment, - }); - const nativeProvider = attachment && hostedNativeProviderForAttachment(attachment); + const receipt = entry.nativeHostedProviderAttachment; + const definition = hostedNativeProviderForAttachment(receipt); if ( - !nativeProvider?.custom || - nativeProvider.api !== "openai-completions" || - nativeProvider.credentialEnv !== credentialEnv + !definition?.custom || + definition.logicalProvider !== providerName || + definition.api !== "openai-completions" || + definition.credentialEnv !== credentialEnv || + entry.nativeLocalProviderAttachment !== undefined ) { throw new Error( - "DCode invalid-credential lifecycle requires an attached native completion provider", + "DCode invalid-credential lifecycle requires its native hosted provider attachment", ); } return { gatewayName: requiredString(entry, "gatewayName"), - providerName: nativeProvider.providerName, + providerName: definition.providerName, + providerId: (receipt as { providerId: string }).providerId, + profileId: definition.profileId, + endpointUrl: definition.endpoint, + endpoint: `${definition.endpoint.replace(/\/+$/u, "")}/chat/completions`, credentialEnv, - endpoint: `${nativeProvider.endpoint.replace(/\/+$/u, "")}/chat/completions`, model: requiredString(entry, "model"), validCredential, }; @@ -129,8 +132,11 @@ export function isDcodeInvalidCredentialRebuildOptions( return ( "gatewayName" in options && "providerName" in options && - "credentialEnv" in options && + "providerId" in options && + "profileId" in options && + "endpointUrl" in options && "endpoint" in options && + "credentialEnv" in options && "model" in options && "validCredential" in options ); @@ -280,6 +286,28 @@ async function updateProviderCredential( phase: string, redactionValues: string[], ): Promise { + const queryOptions = { + env: gatewayEnv(options.gatewayName), + redactionValues, + timeoutMs: 30_000, + }; + const provider = await deps.host.command( + "openshell", + ["provider", "get", "-g", options.gatewayName, options.providerName], + { ...queryOptions, artifactName: `lifecycle-dcode-provider-owner-${phase}` }, + ); + const metadata = + provider.exitCode === 0 ? parseCliOpenShellProviderMetadata(provider.stdout) : null; + if ( + metadata?.name !== options.providerName || + metadata.type !== options.profileId || + metadata.revision?.id !== options.providerId || + metadata.configKeys.length !== 0 || + metadata.credentialKeys.length !== 1 || + metadata.credentialKeys[0] !== options.credentialEnv + ) { + throw new Error(`DCode provider ownership changed before credential ${phase}`); + } const result = await deps.host.command( "openshell", [ @@ -378,7 +406,7 @@ export async function simulateDcodeInvalidCredentialRebuild( const baseRedactions = [options.validCredential]; const names = await deps.host.command( "openshell", - ["sandbox", "list", "--names", "--limit", "2", "-g", options.gatewayName], + ["sandbox", "list", "--names", "--page-size", "2", "-g", options.gatewayName], { artifactName: "lifecycle-dcode-gateway-sandboxes", env: gatewayEnv(options.gatewayName), diff --git a/test/e2e/fixtures/phases/runtime.ts b/test/e2e/fixtures/phases/runtime.ts index bf9d875eab6..328c0fdb281 100644 --- a/test/e2e/fixtures/phases/runtime.ts +++ b/test/e2e/fixtures/phases/runtime.ts @@ -1,6 +1,9 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { NATIVE_LOCAL_AUTH_HEADER_SCRIPT } from "../../../../src/lib/inference/native-local/agent-config.ts"; +import { shellQuote } from "../../../../src/lib/core/shell-quote.ts"; +import { trustedSandboxShellScript } from "../clients/sandbox.ts"; import { buildAvailabilityProbeEnv } from "../availability-env.ts"; import { assertExitZero } from "../clients/command.ts"; import type { @@ -297,26 +300,38 @@ export class RuntimePhaseFixture { instance: Pick, options: InferenceRuntimeChatOptions & { readonly route?: InferenceRoute; + readonly modelRouter?: boolean; } = {}, ): Promise { - const endpoint = inferenceRouteUrl(options.route, CHAT_COMPLETIONS_PATH); + const endpoint = options.modelRouter + ? `http://host.openshell.internal:4000${CHAT_COMPLETIONS_PATH}` + : inferenceRouteUrl(options.route, CHAT_COMPLETIONS_PATH); const payload = openAiChatPayload(options); - const result = await this.sandbox.exec( - instance.sandboxName, - [ - "curl", - "-fsS", - "--max-time", - curlMaxTime(options), - "-H", - "Content-Type: application/json", - ...headerArgs(options.headers), - "--data-raw", - payload, - endpoint, - ], - shellOptions(options, "runtime-inference-local-chat-completion"), - ); + const command = [ + "curl", + "-fsS", + "--max-time", + curlMaxTime(options), + "-H", + "Content-Type: application/json", + ...headerArgs(options.headers), + "--data-raw", + payload, + endpoint, + ]; + const result = options.modelRouter + ? await this.sandbox.execShell( + instance.sandboxName, + trustedSandboxShellScript( + `${NATIVE_LOCAL_AUTH_HEADER_SCRIPT}; ${command.map(shellQuote).join(" ")} -H "$AUTH_HEADER"`, + ), + shellOptions(options, "runtime-native-model-router-chat-completion"), + ) + : await this.sandbox.exec( + instance.sandboxName, + command, + shellOptions(options, "runtime-inference-local-chat-completion"), + ); assertExitZero(result, "inference.local chat completion probe"); assertChatCompletionShape( parseJsonBody(result.stdout, "inference.local chat completion"), diff --git a/test/e2e/fixtures/portable-hermes-transaction-live.ts b/test/e2e/fixtures/portable-hermes-transaction-live.ts index 1bd1d3436ad..43133299117 100644 --- a/test/e2e/fixtures/portable-hermes-transaction-live.ts +++ b/test/e2e/fixtures/portable-hermes-transaction-live.ts @@ -11,6 +11,7 @@ import { assertHermesPortableOpenShellExecutableAuthority, captureHermesPortableOpenShellExecutableAuthority, } from "../../../src/lib/adapters/openshell/resolve-shared.ts"; +import { parseStrictOpenShellSandboxListJson } from "../../../src/lib/adapters/openshell/sandbox-identity.ts"; import { loadAgent } from "../../../src/lib/agent/defs.ts"; import { createHermesPortableBuildContextPlan, @@ -34,6 +35,26 @@ import { normalizeSandboxInferenceRouteSelection } from "../../../src/lib/state/ import { registryEntryGatewayPort } from "../../../src/lib/state/gateway-registry.ts"; import { streamSandboxCreate } from "../../../src/lib/sandbox/create-stream.ts"; +export function portableHermesTransactionSandboxIsPresent( + sandboxName: string, + result: { status: number | null; stdout?: string | Buffer | null; error?: unknown }, +): boolean { + if (result.error || result.status !== 0) { + throw new Error( + `OpenShell Hermes transaction cleanup observation failed (exit ${String(result.status)})`, + ); + } + const sandboxes = parseStrictOpenShellSandboxListJson(String(result.stdout ?? "")); + if (!sandboxes) { + throw new Error("OpenShell Hermes transaction cleanup observation was malformed"); + } + const matches = sandboxes.filter((sandbox) => sandbox.name === sandboxName); + if (matches.length > 1) { + throw new Error("OpenShell Hermes transaction cleanup found duplicate sandbox names"); + } + return matches.length === 1; +} + export async function withAmbientPodmanHost(host: string, action: () => Promise): Promise { const previous = process.env.CONTAINER_HOST; process.env.CONTAINER_HOST = host; @@ -185,6 +206,7 @@ export async function proveHermesPortableOnboardingTransaction(input: { "--policy", request.policyPath!, "--no-tty", + "--detach", "--", ...request.startupCommand, ]; @@ -196,7 +218,7 @@ export async function proveHermesPortableOnboardingTransaction(input: { initialPhase: "create", readyCheck: () => observeHermesPortableSandbox(sandboxName, gatewayName, capture).kind === "present", - readyCheckOutputPatterns: [/Setting up NemoClaw/u], + readyCheckOutputPatterns: [], waitForReadyTermination: true, }, ); @@ -271,37 +293,11 @@ export async function proveHermesPortableOnboardingTransaction(input: { primaryFailed = true; primaryFailure = error; } - const sandboxIsPresent = (): boolean => { - const result = capture(["sandbox", "list", "-g", gatewayName, "-o", "json"]); - if (result.error || result.status !== 0) { - throw new Error( - `OpenShell Hermes transaction cleanup observation failed (exit ${String(result.status)})`, - ); - } - let parsed: unknown; - try { - parsed = JSON.parse(String(result.stdout)); - } catch { - throw new Error("OpenShell Hermes transaction cleanup observation was malformed"); - } - if ( - !Array.isArray(parsed) || - !parsed.every( - (sandbox) => - sandbox !== null && - typeof sandbox === "object" && - !Array.isArray(sandbox) && - typeof (sandbox as { name?: unknown }).name === "string", - ) - ) { - throw new Error("OpenShell Hermes transaction cleanup observation was malformed"); - } - const matches = parsed.filter((sandbox) => sandbox.name === sandboxName); - if (matches.length > 1) { - throw new Error("OpenShell Hermes transaction cleanup found duplicate sandbox names"); - } - return matches.length === 1; - }; + const sandboxIsPresent = (): boolean => + portableHermesTransactionSandboxIsPresent( + sandboxName, + capture(["sandbox", "list", "-g", gatewayName, "-o", "json"]), + ); const cleanupFailures: unknown[] = []; try { if (sandboxIsPresent()) { diff --git a/test/e2e/fixtures/registered-inference.ts b/test/e2e/fixtures/registered-inference.ts new file mode 100644 index 00000000000..bed3b4d4fd7 --- /dev/null +++ b/test/e2e/fixtures/registered-inference.ts @@ -0,0 +1,328 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { INFERENCE_CURL_DIAGNOSTIC_SCRIPT } from "./curl-policy-denial.ts"; +import fs from "node:fs"; +import path from "node:path"; +import { randomUUID } from "node:crypto"; +import { + assertEndpointResolvesPublic, + type EndpointDnsLookupFn, +} from "../../../src/lib/security/trusted-private-endpoint.ts"; +import { inspectInferenceProviderProfile } from "../../../src/lib/adapters/openshell/provider-profile.ts"; +import { createExtraGatewayProvider } from "./gateway-providers.ts"; +import type { HostCliClient } from "./clients/host.ts"; +import { resultText } from "./clients/command.ts"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; +import { normalizeNativeLocalProviderAttachment } from "../../../src/lib/inference/native-local/contract.ts"; +import { recordedNativeProviderAttachment } from "../../../src/lib/inference/native-provider/recorded-selection.ts"; +import { buildSandboxInferenceChatCommand } from "./inference-adapter.ts"; +import { parseCliOpenShellProviderMetadata } from "../../../src/lib/adapters/openshell/provider-metadata-cli.ts"; +import type { SandboxClient } from "./clients/sandbox.ts"; +import { buildAvailabilityProbeEnv } from "./availability-env.ts"; + +/** Read the published selection again after a failed backup or successful recreation. */ +export function registeredCompatibleChatCommand( + sandboxName: string, + model: string, + payload: string, +): [string, ...string[]] { + const entry = getSandbox(sandboxName); + const local = normalizeNativeLocalProviderAttachment(entry?.nativeLocalProviderAttachment); + const hosted = entry && recordedNativeProviderAttachment(entry); + if ( + entry?.provider !== "compatible-endpoint" || + entry.model !== model || + (entry.nativeLocalProviderAttachment !== undefined && !local) || + Boolean(local && hosted) || + (local && + (local.sandboxName !== sandboxName || + local.gatewayName !== entry.gatewayName || + local.provider !== entry.provider || + local.agent !== (entry.agent === "deepagents" ? undefined : entry.agent) || + (local.api ?? "openai-completions") !== "openai-completions")) || + (hosted && + (hosted.custom?.api !== "openai-completions" || + hosted.custom.credentialEnv !== "COMPATIBLE_API_KEY")) || + (!local && !hosted) + ) + throw new Error("The registered compatible inference selection does not match this probe"); + return buildSandboxInferenceChatCommand( + payload, + entry.provider, + (local ?? hosted)!.endpointUrl, + 60, + ); +} + +/** Keep HTTP responses and native connect denials visible without outgoing headers. */ +export function registeredCompatibleResponseCommand( + sandboxName: string, + model: string, + payload: string, + followRedirect: boolean, +): [string, ...string[]] { + const command = registeredCompatibleChatCommand(sandboxName, model, payload); + command[2] = command[2].replace( + "exec curl -fsS", + `${INFERENCE_CURL_DIAGNOSTIC_SCRIPT}; curl -sS --include${followRedirect ? " --location --post302 --max-redirs 3" : ""}`, + ); + return command; +} + +/** Bind the sibling snapshot to the real provider revision and attachment. */ +export async function readRegisteredLocalRoute( + sandbox: SandboxClient, + sandboxName: string, + artifactName: string, +) { + const entry = getSandbox(sandboxName); + const receipt = normalizeNativeLocalProviderAttachment(entry?.nativeLocalProviderAttachment); + if ( + !receipt || + receipt.sandboxName !== sandboxName || + receipt.gatewayName !== entry?.gatewayName || + receipt.provider !== entry.provider || + receipt.agent !== (entry.agent === "deepagents" ? undefined : entry.agent) + ) + throw new Error("The sibling has no matching native local inference receipt"); + const options = { env: buildAvailabilityProbeEnv(), timeoutMs: 30_000 }; + const provider = await sandbox.openshell( + ["provider", "get", "-g", receipt.gatewayName, receipt.providerName], + { ...options, artifactName: `${artifactName}-provider` }, + ); + const attachment = await sandbox.openshell( + ["sandbox", "provider", "list", "-g", receipt.gatewayName, sandboxName], + { ...options, artifactName: `${artifactName}-attachment` }, + ); + const metadata = + provider.exitCode === 0 ? parseCliOpenShellProviderMetadata(provider.stdout) : null; + const ok = + metadata?.name === receipt.providerName && + metadata.type === receipt.profileId && + metadata.revision?.id === receipt.providerId && + metadata.configKeys.length === 0 && + metadata.credentialKeys.length === 1 && + metadata.credentialKeys[0] === receipt.credentialEnv && + attachment.exitCode === 0 && + attachment.stdout.split(/\s+/u).includes(receipt.providerName); + return { + exitCode: ok ? 0 : 1, + stdout: JSON.stringify({ provider: entry!.provider, model: entry!.model, receipt }), + stderr: ok ? "" : "Sibling native provider identity or attachment mismatch", + }; +} + +export const OPERATOR_SWITCH_API_KEY = "openclaw-operator-switch-synthetic-credential"; +export const OPERATOR_SWITCH_PROVIDER = "e2e-operator-inference"; +export const OPERATOR_SWITCH_CREDENTIAL_ENV = "NEMOCLAW_E2E_OPERATOR_TOKEN"; + +/** The fixture owns only the provider IDs it creates in its private test gateway. */ +export async function operatorInferenceSwitchFixture( + host: HostCliClient, + options: { + home: string; + env: NodeJS.ProcessEnv; + endpointUrl: string; + lookup?: EndpointDnsLookupFn; + credentialValue: string; + sandboxName: string; + model: string; + }, +) { + const endpoint = new URL(options.endpointUrl); + if (endpoint.protocol !== "https:" || endpoint.username || endpoint.password || endpoint.hash) + throw new Error("Operator fixture requires a credential-free HTTPS endpoint"); + const admitted = await assertEndpointResolvesPublic(options.endpointUrl, options.lookup); + if (!admitted.ok || !admitted.addresses?.length) + throw new Error("Operator fixture requires public endpoint IP pins"); + const profileId = `e2e-operator-${randomUUID()}`; + const profilePath = path.join(options.home, `${profileId}.json`); + const profile = { + id: profileId, + display_name: "E2E operator inference", + description: "Synthetic switch fixture", + category: "inference", + credentials: [ + { + name: "api_key", + env_vars: [OPERATOR_SWITCH_CREDENTIAL_ENV], + required: true, + auth_style: "bearer", + header_name: "authorization", + query_param: "", + }, + ], + endpoints: [ + { + host: endpoint.hostname, + port: Number(endpoint.port || 443), + protocol: "rest", + enforcement: "enforce", + allowed_ips: admitted.addresses, + rules: [ + { allow: { method: "GET", path: `${endpoint.pathname}/models` } }, + { allow: { method: "POST", path: `${endpoint.pathname}/chat/completions` } }, + ], + }, + ], + binaries: [ + "/usr/bin/python3", + "/usr/bin/python3.13", + "/usr/local/bin/node", + "/usr/bin/node", + "/usr/local/bin/curl", + "/usr/bin/curl", + ], + inference_capable: true, + }; + fs.writeFileSync(profilePath, JSON.stringify(profile), { flag: "wx", mode: 0o600 }); + const env = { ...options.env, [OPERATOR_SWITCH_CREDENTIAL_ENV]: options.credentialValue }; + let ownedId: string | undefined; + let originalDigest: string | undefined; + const run = (args: string[], artifactName: string) => + host.command(host.openshellCommandPath, [...args, "-g", "nemoclaw"], { + artifactName, + env: options.env, + timeoutMs: 60_000, + redactionValues: [options.credentialValue], + }); + const inspect = async () => { + const result = await run( + ["provider", "get", OPERATOR_SWITCH_PROVIDER], + "operator-provider-identity", + ); + const provider = + result.exitCode === 0 ? parseCliOpenShellProviderMetadata(result.stdout) : null; + if ( + provider?.name !== OPERATOR_SWITCH_PROVIDER || + provider.type !== profileId || + !provider.revision + ) + throw new Error("Operator fixture provider identity is unavailable"); + return provider; + }; + const create = async () => { + const absent = await run( + ["provider", "get", OPERATOR_SWITCH_PROVIDER], + "operator-provider-absence", + ); + if ( + absent.exitCode === 0 || + !/provider not found|requested entity was not found/iu.test(resultText(absent)) + ) + throw new Error("Operator fixture provider absence is unproven"); + await createExtraGatewayProvider(host, OPERATOR_SWITCH_PROVIDER, { + artifactName: "operator-provider-create", + env, + profilePath, + credentialEnv: OPERATOR_SWITCH_CREDENTIAL_ENV, + }); + ownedId = (await inspect()).revision!.id; + const exported = await run( + ["provider", "profile", "export", profileId, "--output", "json"], + "operator-original-profile", + ); + if (exported.exitCode !== 0) throw new Error("Operator profile export failed"); + originalDigest ??= inspectInferenceProviderProfile(exported.stdout)?.digest; + if (!originalDigest) throw new Error("Operator profile has no inference projection"); + return ownedId; + }; + const remove = async (detach: boolean) => { + if (!ownedId || (await inspect()).revision!.id !== ownedId) + throw new Error("Refusing to delete an unowned operator provider identity"); + if (detach) { + const result = await run( + ["sandbox", "provider", "detach", options.sandboxName, OPERATOR_SWITCH_PROVIDER], + "operator-provider-detach", + ); + if (result.exitCode !== 0) throw new Error("Could not detach the fixture-owned provider"); + } + const result = await run( + ["provider", "delete", OPERATOR_SWITCH_PROVIDER], + "operator-provider-delete", + ); + if (result.exitCode !== 0) throw new Error("Could not delete the fixture-owned provider"); + ownedId = undefined; + }; + return { + ...options, + create, + async replace() { + await remove(true); + return create(); + }, + async cleanup() { + if (!ownedId) return; + const attached = await run( + ["sandbox", "provider", "list", options.sandboxName], + "operator-cleanup-attachments", + ); + if ( + attached.exitCode !== 0 && + !/sandbox.*not found|requested entity was not found/iu.test(resultText(attached)) + ) + throw new Error("Operator cleanup sandbox identity is unavailable"); + await remove( + attached.exitCode === 0 && attached.stdout.split(/\s+/u).includes(OPERATOR_SWITCH_PROVIDER), + ); + }, + async evidence() { + const registry = JSON.parse( + fs.readFileSync(path.join(options.home, ".nemoclaw", "sandboxes.json"), "utf8"), + ); + const selected = registry.sandboxes?.[options.sandboxName]; + const provider = await inspect(); + const exported = await run( + ["provider", "profile", "export", profileId, "--output", "json"], + "operator-profile-readback", + ); + const attachments = await run( + ["sandbox", "provider", "list", options.sandboxName], + "operator-attachment-readback", + ); + return { + selected, + provider, + profileDigest: + exported.exitCode === 0 + ? inspectInferenceProviderProfile(exported.stdout)?.digest + : undefined, + attachmentExitCode: attachments.exitCode, + attached: attachments.stdout.split(/\s+/u).includes(OPERATOR_SWITCH_PROVIDER), + }; + }, + expected(providerId: string) { + const profileDigest = originalDigest; + return { + selected: { + provider: OPERATOR_SWITCH_PROVIDER, + model: options.model, + registeredInferenceProviderAttachment: { + schemaVersion: 1, + gatewayName: "nemoclaw", + sandboxName: options.sandboxName, + providerName: OPERATOR_SWITCH_PROVIDER, + providerId, + profileId, + profileDigest, + endpointUrl: options.endpointUrl, + inferenceApi: "openai-completions", + credentialEnv: OPERATOR_SWITCH_CREDENTIAL_ENV, + attachedBySelection: true, + }, + }, + provider: { + name: OPERATOR_SWITCH_PROVIDER, + type: profileId, + revision: { id: providerId }, + credentialKeys: [OPERATOR_SWITCH_CREDENTIAL_ENV], + configKeys: [], + }, + profileDigest, + attachmentExitCode: 0, + attached: true, + }; + }, + }; +} diff --git a/test/e2e/fixtures/routed-private-relay.ts b/test/e2e/fixtures/routed-private-relay.ts index 3343cbdbbf1..c62b924034f 100644 --- a/test/e2e/fixtures/routed-private-relay.ts +++ b/test/e2e/fixtures/routed-private-relay.ts @@ -11,6 +11,7 @@ import path from "node:path"; import { isOperatorTrustablePrivateIp } from "../../../src/lib/security/trusted-private-endpoint.ts"; import { assertExitZero } from "./clients/command.ts"; import type { HostCliClient } from "./clients/host.ts"; +import { discoverHostInterfaceAddress } from "./host-address.ts"; import { RuntimeProviderPrerequisite } from "./runtime-provider.ts"; const RELAY_PORT = 8443; @@ -95,9 +96,7 @@ export async function startRoutedPrivateRelay(options: { upstreamHost: string; upstreamPort: number; }): Promise { - const runtime = new RuntimeProviderPrerequisite(options.host, (reason) => { - throw new Error(reason); - }); + const runtime = new RuntimeProviderPrerequisite(options.host); const sandboxHandle = await runtime.resolveSandboxResourceHandle(options.sandboxName, { artifactName: "routed-private-relay-sandbox-resource", timeoutMs: 30_000, @@ -110,6 +109,22 @@ export async function startRoutedPrivateRelay(options: { const networkNames = Object.keys(JSON.parse(networks.stdout) as Record); assert.equal(networkNames.length, 1, "sandbox must have one exact runtime network"); const networkName = networkNames[0] as string; + if (networkName === "none") { + // OpenShell 0.1.3 isolates the workload network and runs its supervisor on + // the host network. The caller's HTTPS fixture already listens on 0.0.0.0; + // retain its authentication and diagnostics without creating another relay. + const { address } = await discoverHostInterfaceAddress( + options.host, + "routed-private-host-address", + ); + assert.equal( + isOperatorTrustablePrivateIp(address), + true, + "host listener must use an operator-trustable private address", + ); + // The caller owns the existing listener and its cleanup. + return Object.freeze({ address, port: options.upstreamPort, close: async () => {} }); + } const relayName = `nemoclaw-private-relay-${process.pid}-${randomBytes(4).toString("hex")}`; const snapshot = async (phase: string): Promise => { const directory = await mkdtemp(path.join(os.tmpdir(), "nemoclaw-relay-snapshot-")).catch( diff --git a/test/e2e/fixtures/runtime-provider.ts b/test/e2e/fixtures/runtime-provider.ts index a771b5c6352..6d759eb5b86 100644 --- a/test/e2e/fixtures/runtime-provider.ts +++ b/test/e2e/fixtures/runtime-provider.ts @@ -12,6 +12,7 @@ export type RuntimeProviderSkip = (reason: string) => never; export type E2eRuntimeProviderId = "docker" | "podman"; const SANITIZED_PRIVILEGED_ENVIRONMENT = [ + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "BASH_ENV=", "ENV=", "GCONV_PATH=", @@ -138,7 +139,7 @@ export class RuntimeProviderPrerequisite { "--filter", `label=${SANDBOX_NAME_LABEL}=${sandboxName}`, "--format", - "{{.ID}}", + '{{.ID}}\t{{.Label "openshell.ai/isolation-role"}}', ], options, ); @@ -154,8 +155,19 @@ export class RuntimeProviderPrerequisite { } const handles = result.stdout .split(/\r?\n/u) - .map((line) => line.trim().toLowerCase()) - .filter(Boolean); + .map((line) => line.trim()) + .filter(Boolean) + .flatMap((line) => { + const [id = "", role = "", ...unexpected] = line.split("\t"); + if ( + !CONTAINER_ID.test(id) || + unexpected.length > 0 || + !["", "sandbox", "supervisor"].includes(role) + ) { + throw new Error(`${this.displayName} returned malformed sandbox resource metadata.`); + } + return role === "supervisor" ? [] : [id]; + }); if (handles.length !== 1 || !CONTAINER_ID.test(handles[0] ?? "")) { throw new Error( `${this.displayName} sandbox '${sandboxName}' resolved ${String(handles.length)} runtime resources; expected exactly one.`, diff --git a/test/e2e/fixtures/security-posture-containers.ts b/test/e2e/fixtures/security-posture-containers.ts new file mode 100644 index 00000000000..fe9dac1bc1f --- /dev/null +++ b/test/e2e/fixtures/security-posture-containers.ts @@ -0,0 +1,262 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import type { + RuntimeProviderCommandCapture, + RuntimeProviderPrivilegedSandboxTarget, +} from "../../../src/lib/onboard/runtime-provider/contract.ts"; +import { CURRENT_RUNTIME_PROVIDER_BUNDLES } from "../../../src/lib/onboard/runtime-provider/current.ts"; +import { requireRuntimeProviderBundleForSandbox } from "../../../src/lib/onboard/runtime-provider/selection.ts"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; +import { OPENSHELL_V017_QUALIFICATION } from "./openshell-v0116-qualification.ts"; + +type Capture = (args: readonly string[]) => RuntimeProviderCommandCapture; +type Owner = { uid: number; gid: number }; +type ContainerObservation = { + id: string; + image: string; + startedAt: string; + pid: number; + user: string; + entrypoint: string[]; + sandboxId: string; + namespace: string; +}; + +export interface SecurityContainerPair { + evidence: "container-launch-configuration-and-supervisor-health"; + workload: ContainerObservation; + supervisor: ContainerObservation; +} + +const FULL_ID = /^[a-f0-9]{64}$/u; + +function requireObservation(condition: unknown): asserts condition { + if (!condition) throw new Error("OpenShell container security observation failed"); +} + +function contentId(value: unknown): string { + requireObservation(typeof value === "string"); + const id = value.replace(/^sha256:/u, ""); + requireObservation(FULL_ID.test(id)); + return id; +} + +function captureJson(capture: Capture, args: readonly string[]): Record { + const result = capture(args); + // Do not echo engine output: failures can contain unrelated container metadata. + requireObservation(result.status === 0 && !result.error && result.stdout.length < 16_384); + try { + const parsed: unknown = JSON.parse(result.stdout); + requireObservation(parsed !== null && typeof parsed === "object" && !Array.isArray(parsed)); + return parsed as Record; + } catch { + throw new Error("OpenShell container security observation returned invalid metadata"); + } +} + +function inspectFormat(provider: string, role: "sandbox" | "supervisor"): string { + // Project only the required metadata. Never collect Env, health logs, auth mounts, + // or supervisor arguments (which can contain proxy configuration). + const fields: Record = { + id: ".Id", + image: ".Image", + user: ".Config.User", + entrypoint: ".Config.Entrypoint", + running: ".State.Running", + paused: ".State.Paused", + pid: ".State.Pid", + startedAt: ".State.StartedAt", + privileged: ".HostConfig.Privileged", + capAdd: ".HostConfig.CapAdd", + capDrop: ".HostConfig.CapDrop", + securityOpt: ".HostConfig.SecurityOpt", + network: ".HostConfig.NetworkMode", + readonlyRootfs: ".HostConfig.ReadonlyRootfs", + name: 'index .Config.Labels "openshell.ai/sandbox-name"', + sandboxId: 'index .Config.Labels "openshell.ai/sandbox-id"', + namespace: 'index .Config.Labels "openshell.ai/sandbox-namespace"', + role: 'index .Config.Labels "openshell.ai/isolation-role"', + managed: + provider === "docker" + ? 'index .Config.Labels "openshell.ai/managed-by"' + : 'index .Config.Labels "openshell.managed"', + ...(role === "sandbox" ? { command: ".Config.Cmd" } : {}), + ...(provider === "podman" + ? { effectiveCaps: ".EffectiveCaps", boundingCaps: ".BoundingCaps" } + : {}), + }; + return `{${Object.entries(fields) + .map(([key, expression]) => `"${key}":{{json (${expression})}}`) + .join(",")}}`; +} + +function inspectContainer( + capture: Capture, + provider: string, + id: string, + sandboxName: string, + role: "sandbox" | "supervisor", + owner: Owner, +): ContainerObservation { + requireObservation(FULL_ID.test(id)); + const record = captureJson(capture, [ + "container", + "inspect", + "--format", + inspectFormat(provider, role), + id, + ]); + const expectedUser = + role === "supervisor" && provider === "docker" ? "65534:65534" : `${owner.uid}:${owner.gid}`; + const executable = + role === "supervisor" + ? "/openshell-supervisor" + : provider === "docker" + ? "/.openshell/runtime/openshell-sandbox" + : "/opt/openshell/bin/openshell-sandbox"; + requireObservation( + record.id === id && + record.running === true && + record.paused === false && + Number.isSafeInteger(record.pid) && + Number(record.pid) > 0 && + typeof record.startedAt === "string" && + Number.isFinite(Date.parse(record.startedAt)) && + record.user === expectedUser && + JSON.stringify(record.entrypoint) === JSON.stringify([executable]) && + record.name === sandboxName && + record.role === role && + record.managed === (provider === "docker" ? "openshell" : "true") && + typeof record.sandboxId === "string" && + /^[a-zA-Z0-9_-]{1,128}$/u.test(record.sandboxId) && + typeof record.namespace === "string" && + record.namespace.length <= 128 && + record.privileged === false && + (record.capAdd === null || JSON.stringify(record.capAdd) === "[]") && + Array.isArray(record.capDrop) && + record.capDrop.length > 0 && + (provider === "docker" + ? record.capDrop.includes("ALL") + : record.effectiveCaps === null && record.boundingCaps === null) && + Array.isArray(record.securityOpt) && + record.securityOpt.some( + (value) => value === "no-new-privileges" || value === "no-new-privileges:true", + ) && + record.network === (role === "sandbox" ? "none" : "host") && + (role !== "supervisor" || provider !== "docker" || record.readonlyRootfs === true) && + (role !== "sandbox" || + JSON.stringify(record.command) === + JSON.stringify(["--bootstrap", "/.openshell/channel/sandbox/bootstrap.json"])), + ); + return { + id, + image: contentId(record.image), + startedAt: record.startedAt as string, + pid: record.pid as number, + user: expectedUser, + entrypoint: [executable], + sandboxId: record.sandboxId as string, + namespace: record.namespace as string, + }; +} + +/** Observe engine-owned launch configuration, not the bytes behind /proc/1/exe. */ +export function observeSecurityContainerPair( + capture: Capture, + target: RuntimeProviderPrivilegedSandboxTarget, + sandboxName: string, + owner: Owner, +): SecurityContainerPair { + const provider = target.providerId; + requireObservation(provider === "docker" || provider === "podman"); + const workload = inspectContainer( + capture, + provider, + target.resourceHandle, + sandboxName, + "sandbox", + owner, + ); + const discovered = capture([ + "ps", + "--all", + "--no-trunc", + "--filter", + `label=openshell.ai/sandbox-id=${workload.sandboxId}`, + "--filter", + "label=openshell.ai/isolation-role=supervisor", + "--format", + "{{.ID}}", + ]); + requireObservation( + discovered.status === 0 && !discovered.error && discovered.stdout.length < 1024, + ); + const supervisorId = discovered.stdout.trim(); + requireObservation(FULL_ID.test(supervisorId) && supervisorId !== workload.id); + const supervisor = inspectContainer( + capture, + provider, + supervisorId, + sandboxName, + "supervisor", + owner, + ); + requireObservation( + supervisor.sandboxId === workload.sandboxId && supervisor.namespace === workload.namespace, + ); + const image = captureJson(capture, [ + "image", + "inspect", + "--format", + '{"id":{{json .Id}}}', + OPENSHELL_V017_QUALIFICATION.supervisorImage, + ]); + requireObservation(supervisor.image === contentId(image.id)); + const socket = + provider === "docker" ? "/run/openshell/health.sock" : "/run/openshell/supervisor-health.sock"; + const health = capture([ + "exec", + supervisor.id, + "/openshell-supervisor", + "health", + "--socket", + socket, + ]); + requireObservation(health.status === 0 && !health.error); + // Detect restart or replacement while the native health command ran. + requireObservation( + JSON.stringify(supervisor) === + JSON.stringify( + inspectContainer(capture, provider, supervisor.id, sandboxName, "supervisor", owner), + ), + ); + requireObservation( + JSON.stringify(workload) === + JSON.stringify( + inspectContainer(capture, provider, workload.id, sandboxName, "sandbox", owner), + ), + ); + return { evidence: "container-launch-configuration-and-supervisor-health", workload, supervisor }; +} + +export function inspectSecurityContainerPair( + sandboxName: string, + target: RuntimeProviderPrivilegedSandboxTarget, + owner: Owner, +): SecurityContainerPair { + const sandbox = getSandbox(sandboxName); + requireObservation(sandbox !== null); + const provider = requireRuntimeProviderBundleForSandbox( + sandbox, + CURRENT_RUNTIME_PROVIDER_BUNDLES, + ); + const engine = provider.containerEngine; + requireObservation(provider.identity.id === target.providerId && engine.supported); + return observeSecurityContainerPair( + (args) => engine.capture("sandbox-lifecycle", args, 30_000), + target, + sandboxName, + owner, + ); +} diff --git a/test/e2e/fixtures/security-posture.ts b/test/e2e/fixtures/security-posture.ts index cb715ff620f..185271416f0 100644 --- a/test/e2e/fixtures/security-posture.ts +++ b/test/e2e/fixtures/security-posture.ts @@ -6,6 +6,7 @@ import type { RuntimeProviderPrivilegedSandboxTarget, } from "../../../src/lib/onboard/runtime-provider/contract.ts"; import { liveE2eManagedImageCatalog } from "../../../src/lib/onboard/workload/preparation.ts"; +import { managedImageRuntimeIdentity } from "../../../src/lib/onboard/managed-image/agents.ts"; import { executePrivilegedSandboxCommand, resolvePrivilegedSandboxTarget, @@ -14,6 +15,10 @@ import { buildAvailabilityProbeEnv } from "./availability-env.ts"; import type { HostCliClient } from "./clients/host.ts"; import { type SandboxClient, trustedSandboxShellScript } from "./clients/sandbox.ts"; import type { ShellProbeResult } from "./shell-probe.ts"; +import { + inspectSecurityContainerPair, + type SecurityContainerPair, +} from "./security-posture-containers.ts"; export type SecurityPostureAgent = "hermes" | "openclaw"; @@ -31,7 +36,8 @@ export interface ProcessSecurityStatus { export interface ProcessSecurityIdentity { argv: string[]; - executable: string; + // Null records PermissionError from /proc//exe, not an inferred path. + executable: string | null; pid: number; ppid: number; state: string; @@ -45,11 +51,13 @@ export interface SplitProcessSecurityReport { observedProcEntries: number; sandboxGid: number; sandboxUid: number; - supervisor: ProcessSecurityIdentity; - version: 2; + workloadBoundary: ProcessSecurityIdentity; + version: 3; } export interface SecurityPostureSummary { + processScope: "workload-container"; + containerPair: SecurityContainerPair; capabilitySurfaces: { connect: CapabilitySurfaceReport; entrypoint: CapabilitySurfaceReport; @@ -65,7 +73,7 @@ export interface SecurityPostureSummary { }; splitProcess: { childSupervisor: ProcessSecurityIdentity; - supervisor: ProcessSecurityIdentity; + workloadBoundary: ProcessSecurityIdentity; }; startupLogClean: true; } @@ -88,16 +96,16 @@ export interface SecurityPostureExpectations { export interface SecurityPostureDependencies { environment?: NodeJS.ProcessEnv; + inspectContainerPair?: typeof inspectSecurityContainerPair; executePrivilegedCommand?: typeof executePrivilegedSandboxCommand; resolvePrivilegedTarget?: typeof resolvePrivilegedSandboxTarget; } -const OPENSHELL_SUPERVISOR_EXECUTABLE = "/opt/openshell/bin/openshell-sandbox"; -const OPENSHELL_SUPERVISOR_ARGV = [ - OPENSHELL_SUPERVISOR_EXECUTABLE, - "--workdir", - "/sandbox", -] as const; +const WORKLOAD_BOUNDARY_EXECUTABLES = Object.freeze({ + docker: "/.openshell/runtime/openshell-sandbox", + podman: "/opt/openshell/bin/openshell-sandbox", +}); +const WORKLOAD_BOOTSTRAP_ARGV = ["--bootstrap", "/.openshell/channel/sandbox/bootstrap.json"]; const SYSTEM_BASH_EXECUTABLES = ["/bin/bash", "/usr/bin/bash"] as const; const NEMOCLAW_START_SUPERVISOR_PATHS = [ "nemoclaw-start", @@ -109,29 +117,15 @@ const MAX_PROC_ENTRIES = 32_768; const MAX_CENSUS_STABILITY_ATTEMPTS = 4; const MAX_CENSUS_DIAGNOSTIC_IDENTITIES = 16; const CAPABILITY_SURFACE_MARKER = "NEMOCLAW_SECURITY_CAPABILITY_SURFACE"; -const EXPECTED_OPENSHELL_VERSION = "0.0.116"; +const EXPECTED_OPENSHELL_VERSION = "0.1.7"; const REVISION_PATTERN = /^[0-9a-f]{40}$/u; -// The pinned OpenShell supervisor has the Docker default capabilities plus -// NET_ADMIN, SYS_ADMIN, SYS_PTRACE, and SYSLOG. Freeze the -// resulting Linux capability mask so additions and removals both require an -// explicit security review. -export const OPENSHELL_SUPERVISOR_CAPABILITY_MASK = "00000004a82c35fb"; -export const PODMAN_OPENSHELL_SUPERVISOR_CAPABILITY_MASK = "00000004002811cd"; - -const OPENSHELL_SUPERVISOR_CAPABILITY_MASKS = Object.freeze({ - docker: OPENSHELL_SUPERVISOR_CAPABILITY_MASK, - podman: PODMAN_OPENSHELL_SUPERVISOR_CAPABILITY_MASK, -}); - -function supervisorCapabilityMask(providerId: string): string { - const mask = - OPENSHELL_SUPERVISOR_CAPABILITY_MASKS[ - providerId as keyof typeof OPENSHELL_SUPERVISOR_CAPABILITY_MASKS - ]; - if (!mask) { - throw new Error(`security-posture has no reviewed capability mask for '${providerId}'`); +function workloadBoundaryArgv(providerId: string): string[] { + const executable = + WORKLOAD_BOUNDARY_EXECUTABLES[providerId as keyof typeof WORKLOAD_BOUNDARY_EXECUTABLES]; + if (!Object.hasOwn(WORKLOAD_BOUNDARY_EXECUTABLES, providerId)) { + throw new Error(`security-posture has no reviewed workload boundary for '${providerId}'`); } - return mask; + return [executable, ...WORKLOAD_BOOTSTRAP_ARGV]; } export const SPLIT_PROCESS_SECURITY_PROBE = String.raw`import grp @@ -144,12 +138,11 @@ PROC_ROOT = Path("/proc") MAX_PROC_ENTRIES = ${MAX_PROC_ENTRIES} MAX_CENSUS_STABILITY_ATTEMPTS = ${MAX_CENSUS_STABILITY_ATTEMPTS} MAX_CENSUS_DIAGNOSTIC_IDENTITIES = ${MAX_CENSUS_DIAGNOSTIC_IDENTITIES} -OPENSHELL_SUPERVISOR_ARGV = tuple(item.encode("utf-8") for item in ${JSON.stringify(OPENSHELL_SUPERVISOR_ARGV)}) -OPENSHELL_SUPERVISOR_EXECUTABLE = ${JSON.stringify(OPENSHELL_SUPERVISOR_EXECUTABLE)} NEMOCLAW_START_SUPERVISOR = tuple(item.encode("utf-8") for item in ${JSON.stringify(NEMOCLAW_START_SUPERVISOR_PATHS)}) BASH = tuple(item.encode("utf-8") for item in ${JSON.stringify(BASH_ARGV0)}) SYSTEM_BASH_EXECUTABLES = set(${JSON.stringify(SYSTEM_BASH_EXECUTABLES)}) LIVE_PROCESS_STATES = set(${JSON.stringify(LIVE_PROCESS_STATES)}) +WORKLOAD_BOUNDARY_COMMANDS = ${JSON.stringify(Object.values(WORKLOAD_BOUNDARY_EXECUTABLES).map((executable) => [executable, ...WORKLOAD_BOOTSTRAP_ARGV]))} def argv_for(path): raw = (path / "cmdline").read_bytes() @@ -196,6 +189,16 @@ def selected_status(raw): "noNewPrivs": (values.get("NoNewPrivs") or [""])[0], } +def executable_for(path, pid): + try: + return os.readlink(path / "exe") + except PermissionError: + # OpenShell 0.1.3 makes PID 1 nondumpable before reading its bootstrap. + # Do not invent an executable identity or hide denied child inspection. + if int(pid) != 1: + raise + return None + def stable_process(pid): path = PROC_ROOT / str(pid) before = os.stat(path, follow_symlinks=False) @@ -204,13 +207,13 @@ def stable_process(pid): ) first_status = selected_status((path / "status").read_text(encoding="utf-8")) first_argv = argv_for(path) - first_executable = os.readlink(path / "exe") + first_executable = executable_for(path, pid) second_state, second_ppid, second_start_time = stat_identity( (path / "stat").read_text(encoding="utf-8") ) second_status = selected_status((path / "status").read_text(encoding="utf-8")) second_argv = argv_for(path) - second_executable = os.readlink(path / "exe") + second_executable = executable_for(path, pid) after = os.stat(path, follow_symlinks=False) if first_state not in LIVE_PROCESS_STATES or second_state not in LIVE_PROCESS_STATES: raise RuntimeError(f"process {pid} is not live") @@ -375,31 +378,30 @@ sandbox_uid = sandbox_user.pw_uid sandbox_gid = sandbox_group.gr_gid if sandbox_user.pw_gid != sandbox_gid: raise RuntimeError("sandbox user and group identities disagree") -supervisor_before = stable_process(1) +if os.getresuid() != (sandbox_uid,) * 3 or os.getresgid() != (sandbox_gid,) * 3: + raise RuntimeError("process observer must run as the sandbox owner") +boundary_before = stable_process(1) +if boundary_before["argv"] not in WORKLOAD_BOUNDARY_COMMANDS: + raise RuntimeError("unexpected OpenShell workload boundary command") ( observed_proc_entries, child_supervisors, observed_child_supervisors, ) = acquire_stable_child_supervisor_census() -supervisor_after = stable_process(1) -if stable_security_identity(supervisor_before) != stable_security_identity(supervisor_after): - raise RuntimeError("OpenShell supervisor changed during inspection") -if ( - tuple(supervisor_before["argv"]) != tuple(item.decode("ascii") for item in OPENSHELL_SUPERVISOR_ARGV) - or supervisor_before["executable"] != OPENSHELL_SUPERVISOR_EXECUTABLE -): - raise RuntimeError("unexpected OpenShell supervisor command") +boundary_after = stable_process(1) +if stable_security_identity(boundary_before) != stable_security_identity(boundary_after): + raise RuntimeError("OpenShell workload boundary changed during inspection") if any( item["executable"] not in SYSTEM_BASH_EXECUTABLES for item in observed_child_supervisors ): raise RuntimeError("unexpected nemoclaw-start child supervisor executable") print(json.dumps({ - "version": 2, + "version": 3, "observedProcEntries": observed_proc_entries, "sandboxUid": sandbox_uid, "sandboxGid": sandbox_gid, - "supervisor": supervisor_before, + "workloadBoundary": boundary_before, "childSupervisors": child_supervisors, "observedChildSupervisors": observed_child_supervisors, }, sort_keys=True))`; @@ -498,7 +500,10 @@ function processIdentity(value: unknown, label: string): ProcessSecurityIdentity const process = requiredRecord(value, label); const identity = { argv: requiredStringArray(process.argv, `${label}.argv`), - executable: requiredString(process.executable, `${label}.executable`), + executable: + process.executable === null + ? null + : requiredString(process.executable, `${label}.executable`), pid: requiredInteger(process.pid, `${label}.pid`, 1), ppid: requiredInteger(process.ppid, `${label}.ppid`, 0), state: requiredString(process.state, `${label}.state`), @@ -636,46 +641,41 @@ function canonicalNemoclawStartSupervisorArgv(argv: string[]): boolean { ); } -function validateSupervisor( +function validateWorkloadBoundary( process: ProcessSecurityIdentity, + sandboxUid: number, sandboxGid: number, - expectedCapabilityMask: string, + providerId: string, ): void { if (process.pid !== 1 || process.ppid !== 0) { throw new Error( - `OpenShell supervisor expected pid=1 ppid=0, got ${process.pid}/${process.ppid}`, + `OpenShell workload boundary expected pid=1 ppid=0, got ${process.pid}/${process.ppid}`, ); } + const argv = workloadBoundaryArgv(providerId); if ( - process.executable !== OPENSHELL_SUPERVISOR_EXECUTABLE || - process.argv.length !== OPENSHELL_SUPERVISOR_ARGV.length || - process.argv.some((argument, index) => argument !== OPENSHELL_SUPERVISOR_ARGV[index]) + process.argv.length !== argv.length || + process.argv.some((argument, index) => argument !== argv[index]) ) { - throw new Error("PID 1 does not have the expected OpenShell supervisor command"); + throw new Error("PID 1 does not have the expected OpenShell workload boundary command"); } - requireExactIds(process.status.uid, 0, "OpenShell supervisor Uid"); - requireExactIds(process.status.gid, 0, "OpenShell supervisor Gid"); - requireExactSupplementaryGroups(process.status.groups, [0], "OpenShell supervisor Groups", [ - [0, sandboxGid], - ]); - for (const field of ["capInh", "capPrm", "capEff", "capBnd", "capAmb"] as const) { - requireCapabilityHex(process.status[field], `OpenShell supervisor ${field}`); - } - if (process.status.capInh !== "0000000000000000") { - throw new Error(`OpenShell supervisor CapInh drifted to ${process.status.capInh}`); - } - for (const field of ["capPrm", "capEff", "capBnd"] as const) { - if (process.status[field] !== expectedCapabilityMask) { - throw new Error( - `OpenShell supervisor ${field} expected ${expectedCapabilityMask}, got ${process.status[field]}`, - ); - } - } - if (process.status.capAmb !== "0000000000000000") { - throw new Error(`OpenShell supervisor CapAmb drifted to ${process.status.capAmb}`); + if (process.executable !== null) { + throw new Error( + "OpenShell workload boundary must deny executable-link access to the sandbox owner", + ); } + requireExactIds(process.status.uid, sandboxUid, "OpenShell workload boundary Uid"); + requireExactIds(process.status.gid, sandboxGid, "OpenShell workload boundary Gid"); + requireExactSupplementaryGroups( + process.status.groups, + [sandboxGid], + "OpenShell workload boundary Groups", + ); + requireZeroCapabilities(process.status, "OpenShell workload boundary"); if (process.status.noNewPrivs !== "1") { - throw new Error(`OpenShell supervisor expected NoNewPrivs=1, got ${process.status.noNewPrivs}`); + throw new Error( + `OpenShell workload boundary expected NoNewPrivs=1, got ${process.status.noNewPrivs}`, + ); } } @@ -685,12 +685,17 @@ function validateNemoclawStartProcess( sandboxGid: number, ): void { if (process.pid === 1) { - throw new Error("nemoclaw-start process must not replace the OpenShell supervisor at PID 1"); + throw new Error( + "nemoclaw-start process must not replace the OpenShell workload boundary at PID 1", + ); } if (!canonicalNemoclawStartSupervisorArgv(process.argv)) { throw new Error("nemoclaw-start process does not have the expected argv"); } - if (!(SYSTEM_BASH_EXECUTABLES as readonly string[]).includes(process.executable)) { + if ( + process.executable === null || + !(SYSTEM_BASH_EXECUTABLES as readonly string[]).includes(process.executable) + ) { throw new Error( `nemoclaw-start process expected the system Bash executable, got ${process.executable}`, ); @@ -772,11 +777,10 @@ function processIdentityArray(value: unknown, label: string): ProcessSecurityIde export function validateSplitProcessSecurityReport( value: unknown, - expectedCapabilityMask = OPENSHELL_SUPERVISOR_CAPABILITY_MASK, + providerId = "docker", ): SplitProcessSecurityReport { - requireCapabilityHex(expectedCapabilityMask, "reviewed OpenShell supervisor capability mask"); const report = requiredRecord(value, "split-process security report"); - if (report.version !== 2) throw new Error("split-process security report version must be 2"); + if (report.version !== 3) throw new Error("split-process security report version must be 3"); const observedProcEntries = requiredInteger( report.observedProcEntries, "split-process security report observedProcEntries", @@ -787,7 +791,7 @@ export function validateSplitProcessSecurityReport( } const sandboxUid = requiredInteger(report.sandboxUid, "sandbox uid", 1); const sandboxGid = requiredInteger(report.sandboxGid, "sandbox gid", 1); - const supervisor = processIdentity(report.supervisor, "supervisor"); + const workloadBoundary = processIdentity(report.workloadBoundary, "workloadBoundary"); const childSupervisors = processIdentityArray( report.childSupervisors, "split-process security report childSupervisors", @@ -801,7 +805,7 @@ export function validateSplitProcessSecurityReport( "split-process security report retained more child supervisors than observed processes", ); } - validateSupervisor(supervisor, sandboxGid, expectedCapabilityMask); + validateWorkloadBoundary(workloadBoundary, sandboxUid, sandboxGid, providerId); for (const process of childSupervisors) { validateNemoclawStartProcess(process, sandboxUid, sandboxGid); } @@ -829,14 +833,14 @@ export function validateSplitProcessSecurityReport( observedProcEntries, sandboxGid, sandboxUid, - supervisor, - version: 2, + workloadBoundary, + version: 3, }; } export function parseSplitProcessSecurityReport( output: string, - expectedCapabilityMask = OPENSHELL_SUPERVISOR_CAPABILITY_MASK, + providerId = "docker", ): SplitProcessSecurityReport { let parsed: unknown; try { @@ -844,7 +848,7 @@ export function parseSplitProcessSecurityReport( } catch (error) { throw new Error("split-process security probe emitted invalid JSON", { cause: error }); } - return validateSplitProcessSecurityReport(parsed, expectedCapabilityMask); + return validateSplitProcessSecurityReport(parsed, providerId); } export function securityPostureEnabled(): boolean { @@ -910,11 +914,16 @@ export async function assertSecurityPosture( const splitProcessProbeCommand = ["/usr/bin/python3", "-I", "-c", SPLIT_PROCESS_SECURITY_PROBE]; const initialTarget: RuntimeProviderPrivilegedSandboxTarget = resolvePrivilegedTarget(sandboxName); + workloadBoundaryArgv(initialTarget.providerId); + const identity = managedImageRuntimeIdentity(agent); + const inspectContainerPair = dependencies.inspectContainerPair ?? inspectSecurityContainerPair; + const containerPair = inspectContainerPair(sandboxName, initialTarget, identity); const splitProcessProbe: RuntimeProviderPrivilegedSandboxCommandResult = executePrivilegedCommand( sandboxName, splitProcessProbeCommand, { expectedResourceHandle: initialTarget.resourceHandle, + sandboxUser: `${identity.uid}:${identity.gid}`, sanitizeEnvironment: true, timeout: 30_000, }, @@ -940,8 +949,17 @@ export async function assertSecurityPosture( } const splitProcess = parseSplitProcessSecurityReport( splitProcessProbe.stdout.toString("utf8"), - supervisorCapabilityMask(initialTarget.providerId), + initialTarget.providerId, ); + if (splitProcess.sandboxUid !== identity.uid || splitProcess.sandboxGid !== identity.gid) { + throw new Error("security-posture observed a different managed-image sandbox identity"); + } + if ( + JSON.stringify(containerPair) !== + JSON.stringify(inspectContainerPair(sandboxName, finalTarget, identity)) + ) { + throw new Error("OpenShell container pair changed during process inspection"); + } const execCapabilities = await sandbox.execShell( sandboxName, @@ -1079,9 +1097,10 @@ grep -q 'cannot modify config inside the sandbox' /tmp/nemoclaw-security-guard-p : await sandbox.execShell( sandboxName, trustedSandboxShellScript(String.raw` -. /tmp/nemoclaw-proxy-env.sh -openclaw config validate -test "$(openclaw config get gateway.mode)" = local +. /tmp/nemoclaw-proxy-env.sh && +openclaw config validate && +gateway_mode="$(openclaw config get gateway.mode)" && +test "$gateway_mode" = local `), { artifactName: "security-posture-native-config", @@ -1117,6 +1136,8 @@ tail -n 20 "$log" requireSuccess("sandbox startup log security posture", startLog); return { + processScope: "workload-container", + containerPair, capabilitySurfaces: { connect: connectCapabilitySurface, entrypoint: entrypointCapabilitySurface, @@ -1132,7 +1153,7 @@ tail -n 20 "$log" }, splitProcess: { childSupervisor, - supervisor: splitProcess.supervisor, + workloadBoundary: splitProcess.workloadBoundary, }, startupLogClean: true, }; diff --git a/test/e2e/lib/fake-openai-compatible-api.mts b/test/e2e/lib/fake-openai-compatible-api.mts index f1f717b5139..1f771ee12ec 100755 --- a/test/e2e/lib/fake-openai-compatible-api.mts +++ b/test/e2e/lib/fake-openai-compatible-api.mts @@ -89,19 +89,19 @@ function sendJson(res: ServerResponse, status: number, payload: unknown): void { res.end(body); } -function sendChatSse(res: ServerResponse, content: string): void { +function sendChatSse(res: ServerResponse, content: string, responseModel: string): void { const chunk = JSON.stringify({ id: "chatcmpl-fake-openai-compatible", object: "chat.completion.chunk", created: 0, - model, + model: responseModel, choices: [{ index: 0, delta: { role: "assistant", content }, finish_reason: null }], }); const doneChunk = JSON.stringify({ id: "chatcmpl-fake-openai-compatible", object: "chat.completion.chunk", created: 0, - model, + model: responseModel, choices: [{ index: 0, delta: {}, finish_reason: "stop" }], }); const body = `data: ${chunk}\n\ndata: ${doneChunk}\n\ndata: [DONE]\n\n`; @@ -337,6 +337,10 @@ const server = createServer(async (req, res) => { sendJson(res, 401, { error: { message: "missing bearer credential" } }); return; } + const responseModel = + typeof payload.model === "string" && additionalModels.includes(payload.model) + ? payload.model + : model; const content = requestedPromptReply(payload) ?? chatContent; const toolCall = requestedCanaryToolCall(payload, raw); if (toolCall && payload.stream) { @@ -344,7 +348,7 @@ const server = createServer(async (req, res) => { id: "chatcmpl-fake-openai-compatible", object: "chat.completion.chunk", created: 0, - model, + model: responseModel, choices: [ { index: 0, delta: { role: "assistant", tool_calls: [toolCall] }, finish_reason: null }, ], @@ -353,7 +357,7 @@ const server = createServer(async (req, res) => { id: "chatcmpl-fake-openai-compatible", object: "chat.completion.chunk", created: 0, - model, + model: responseModel, choices: [{ index: 0, delta: {}, finish_reason: "tool_calls" }], }); const body = `data: ${chunk}\n\ndata: ${done}\n\ndata: [DONE]\n\n`; @@ -365,14 +369,14 @@ const server = createServer(async (req, res) => { return; } if (payload.stream) { - sendChatSse(res, content); + sendChatSse(res, content, responseModel); return; } sendJson(res, 200, { id: "chatcmpl-fake-openai-compatible", object: "chat.completion", created: 0, - model, + model: responseModel, choices: [ { index: 0, diff --git a/test/e2e/lib/issue-4462-admin-request-selector.py b/test/e2e/lib/issue-4462-admin-request-selector.py index 5572cca2820..4df486f3190 100644 --- a/test/e2e/lib/issue-4462-admin-request-selector.py +++ b/test/e2e/lib/issue-4462-admin-request-selector.py @@ -8,6 +8,7 @@ import os import re import sys +import time from pathlib import Path data = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")) @@ -142,7 +143,16 @@ def identity_public_key(value): spec = importlib.util.spec_from_file_location("nemoclaw_admin_pairing_state", helper_path) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) -records, _metadata = module.read_openclaw_pairing_state(str(state_root), timeout=1) +retryable_state_error = getattr(module, "OpenClawPairingStateRetryableError", ()) +for attempt in range(3): + try: + records, _metadata = module.read_openclaw_pairing_state(str(state_root), timeout=1) + break + except retryable_state_error: + if attempt == 2: + raise + # Discard the raced snapshot; the next read repeats all file-identity checks. + time.sleep(0.05) identity = records.get("identity") if not isinstance(identity, dict): raise SystemExit("local CLI identity must be an object") diff --git a/test/e2e/live/agent-turn-latency-helpers.ts b/test/e2e/live/agent-turn-latency-helpers.ts index 6d7804da03f..ab9e57c06bc 100644 --- a/test/e2e/live/agent-turn-latency-helpers.ts +++ b/test/e2e/live/agent-turn-latency-helpers.ts @@ -5,9 +5,19 @@ import { openClawAgentResponseRecord, parseOpenClawJsonDocuments, } from "../../../src/lib/openclaw/agent-json-provenance.ts"; +import { parseCliOpenShellProviderMetadata } from "../../../src/lib/adapters/openshell/provider-metadata-cli.ts"; +import { getSandboxInferenceConfig } from "../../../src/lib/inference/config.ts"; +import { hermesApiMode } from "../../../src/lib/hermes-managed-route.ts"; +import { hostedNativeProviderForAttachment } from "../../../src/lib/inference/native-provider/hosted-attachment.ts"; +import { recordedNativeProviderAttachment } from "../../../src/lib/inference/native-provider/recorded-selection.ts"; +import { + NVIDIA_HOSTED_CREDENTIAL_ENV, + NVIDIA_HOSTED_NATIVE_ENDPOINT, +} from "../../../src/lib/inference/native-nvidia/contract.ts"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; import { execTimeout } from "../../helpers/timeouts.ts"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; -import { resultText, shellQuote } from "../fixtures/clients/command.ts"; +import { resultText } from "../fixtures/clients/command.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import { type SandboxClient, validateSandboxName } from "../fixtures/clients/sandbox.ts"; import type { E2EInferenceAdapter } from "../fixtures/inference-adapter.ts"; @@ -406,23 +416,98 @@ export async function cleanupTurnSandbox( } } -// Hosted custom endpoints attach a native provider to the sandbox, so the -// gateway-global route stays unconfigured. Sandbox status reports the recorded -// provider and model and exits nonzero when its live inference probe fails. +/** Read a live native route, or return null when its sandbox binding cannot be verified. */ export async function route( - host: HostCliClient, + sandbox: SandboxClient, sandboxName: string, agent: "openclaw" | "hermes", inference: AgentTurnInference, artifactName: string, progress?: Pick, -): Promise { - return await host.nemoclaw([sandboxName, "status"], { - artifactName, - env: env(sandboxName, agent, inference), +): Promise<{ provider: string; model: string } | null> { + const entry = getSandbox(sandboxName); + const receipt = entry && recordedNativeProviderAttachment(entry); + const definition = hostedNativeProviderForAttachment(receipt); + if ( + !entry || + !receipt || + (entry.agent ?? "openclaw") !== agent || + (!definition && entry.provider !== "nvidia-prod") + ) { + return null; + } + const gateway = entry.gatewayName || env(sandboxName, agent, inference).OPENSHELL_GATEWAY!; + const selected = getSandboxInferenceConfig( + inference.model, + entry.provider, + receipt.custom?.api, + true, + receipt.endpointUrl, + ); + const options = { + env: { ...env(sandboxName, agent, inference), OPENSHELL_GATEWAY: gateway }, onOutput: progress?.onOutput, - timeoutMs: 120_000, - }); + redactionValues: inference.redactionValues(), + timeoutMs: 30_000, + captureLimitBytes: 16 * 1024, + }; + const provider = await sandbox.openshell( + ["provider", "get", "-g", gateway, receipt.providerName], + { + ...options, + artifactName: `${artifactName}-provider`, + }, + ); + const attached = await sandbox.openshell( + ["sandbox", "provider", "list", "-g", gateway, sandboxName], + { + ...options, + artifactName: `${artifactName}-attachment`, + }, + ); + const config = await sandbox.exec( + sandboxName, + agent === "openclaw" + ? [ + "node", + "-e", + 'const c=JSON.parse(require("node:fs").readFileSync("/sandbox/.openclaw/openclaw.json","utf8")); const p=c.models.providers[process.argv[1]]; process.stdout.write(JSON.stringify({model:c.agents.defaults.model.primary,endpoint:p.baseUrl,api:p.api}));', + selected.providerKey, + ] + : [ + "/opt/hermes/.venv/bin/python3", + "-c", + 'import json,yaml; c=yaml.safe_load(open("/sandbox/.hermes/config.yaml")); m=c["model"]; print(json.dumps({"model":m["default"],"endpoint":m["base_url"],"api":m.get("api_mode"),"provider":m["provider"]}))', + ], + { ...options, artifactName: `${artifactName}-agent-selection` }, + ); + const metadata = + provider.exitCode === 0 ? parseCliOpenShellProviderMetadata(provider.stdout) : null; + const actual = config.exitCode === 0 ? JSON.parse(config.stdout) : null; + const prefix = `${selected.providerKey}/`; + if ( + !metadata || + metadata.name !== receipt.providerName || + metadata.type !== receipt.profileId || + metadata.revision?.id !== receipt.providerId || + metadata.configKeys.length !== 0 || + metadata.credentialKeys.length !== 1 || + metadata.credentialKeys[0] !== (definition?.credentialEnv ?? NVIDIA_HOSTED_CREDENTIAL_ENV) || + attached.exitCode !== 0 || + !attached.stdout.split(/\s+/u).includes(receipt.providerName) || + !actual || + typeof actual.model !== "string" || + actual.endpoint !== (definition?.endpoint ?? NVIDIA_HOSTED_NATIVE_ENDPOINT) || + actual.api !== + (agent === "openclaw" ? selected.inferenceApi : hermesApiMode(selected.inferenceApi)) || + (agent === "openclaw" ? !actual.model.startsWith(prefix) : actual.provider !== "custom") + ) { + return null; + } + return { + provider: definition?.logicalProvider ?? "nvidia-prod", + model: agent === "openclaw" ? actual.model.slice(prefix.length) : actual.model, + }; } export async function openclawTurn( diff --git a/test/e2e/live/agent-turn-latency.test.ts b/test/e2e/live/agent-turn-latency.test.ts index b3666cc8ae6..03897fb3669 100644 --- a/test/e2e/live/agent-turn-latency.test.ts +++ b/test/e2e/live/agent-turn-latency.test.ts @@ -135,16 +135,19 @@ runAgentTurnLatencyTest( expect(openclawInstall.exitCode, resultText(openclawInstall)).toBe(0); progress.phase("validate OpenClaw inference route"); const openclawRoute = await route( - host, + sandbox, OPENCLAW_SANDBOX, "openclaw", inference, "openclaw-route", progress, ); - expect(openclawRoute.exitCode, resultText(openclawRoute)).toBe(0); - for (const expected of [inference.expectedRouteProvider, inference.model]) { - expect(resultText(openclawRoute)).toContain(expected); + expect(openclawRoute).not.toBeNull(); + for (const [field, expected] of [ + ["provider", inference.expectedRouteProvider], + ["model", inference.model], + ] as const) { + expect(openclawRoute?.[field]).toBe(expected); } progress.phase("run OpenClaw hosted inference turn"); const firstTurn = await openclawTurn(host, inference, progress, { @@ -202,16 +205,19 @@ runAgentTurnLatencyTest( expect(hermesInstall.exitCode, resultText(hermesInstall)).toBe(0); progress.phase("validate Hermes inference route"); const hermesRoute = await route( - host, + sandbox, HERMES_SANDBOX, "hermes", inference, "hermes-route", progress, ); - expect(hermesRoute.exitCode, resultText(hermesRoute)).toBe(0); - for (const expected of [inference.expectedRouteProvider, inference.model]) { - expect(resultText(hermesRoute)).toContain(expected); + expect(hermesRoute).not.toBeNull(); + for (const [field, expected] of [ + ["provider", inference.expectedRouteProvider], + ["model", inference.model], + ] as const) { + expect(hermesRoute?.[field]).toBe(expected); } const payload = JSON.stringify({ diff --git a/test/e2e/live/bedrock-runtime-compatible-anthropic-raw-command.ts b/test/e2e/live/bedrock-runtime-compatible-anthropic-raw-command.ts index 157525dd1c9..265ab28507b 100644 --- a/test/e2e/live/bedrock-runtime-compatible-anthropic-raw-command.ts +++ b/test/e2e/live/bedrock-runtime-compatible-anthropic-raw-command.ts @@ -84,22 +84,16 @@ export async function runRawCommand( const redactionValues = [...(options.redactionValues ?? [])]; const progressName = progressCommandName(options.artifactName); emitProgressEvent(options.progress, `command ${progressName} started`); - let child: ReturnType; - try { - child = spawnObservedChild(command, args, { - activityLabel: `command: ${progressName}`, - progress: options.progress, - spawn: { - cwd: options.cwd ?? REPO_ROOT, - detached: true, - env: resolveLiveE2eWorkloadSourceEnv({ ...(options.env ?? {}) }), - stdio: [options.stdin === undefined ? "ignore" : "pipe", "pipe", "pipe"], - }, - }); - } catch (error) { - emitProgressEvent(options.progress, `command ${progressName} failed to start`); - throw error; - } + const child = spawnObservedChild(command, args, { + activityLabel: `command: ${progressName}`, + progress: options.progress, + spawn: { + cwd: options.cwd ?? REPO_ROOT, + detached: true, + env: resolveLiveE2eWorkloadSourceEnv({ ...(options.env ?? {}) }), + stdio: [options.stdin === undefined ? "ignore" : "pipe", "pipe", "pipe"], + }, + }); const fullCommand = [command, ...args]; const stdoutCapture = boundedOutputCapture(); const stderrCapture = boundedOutputCapture(); diff --git a/test/e2e/live/brave-search-helpers.ts b/test/e2e/live/brave-search-helpers.ts index 4e9322badb7..e64fafe2a2b 100644 --- a/test/e2e/live/brave-search-helpers.ts +++ b/test/e2e/live/brave-search-helpers.ts @@ -7,7 +7,7 @@ const BRAVE_REFERENCE_PATTERN = String.raw`openshell:resolve:env:(?:(?:v[0-9]{1, // Node. Cross-tree gateway environ reads are blocked by Yama; the original // #7425 regression observes this agent child and a fresh login shell. export const BRAVE_AGENT_BOUNDARY = String.raw` -import os, pathlib, re, signal, subprocess, sys, time +import pathlib, re, subprocess, sys, time def inspect(process): try: @@ -37,14 +37,16 @@ else: break time.sleep(0.05) finally: + # OpenShell denies process-group signals. Reap our direct child here; + # the test's administrator sandbox deletion owns descendant cleanup. try: - os.killpg(child.pid, signal.SIGTERM) + child.terminate() except ProcessLookupError: pass try: child.wait(timeout=5) except subprocess.TimeoutExpired: - os.killpg(child.pid, signal.SIGKILL) + child.kill() child.wait(timeout=5) sys.exit(status) `; diff --git a/test/e2e/live/channels-add-remove.test.ts b/test/e2e/live/channels-add-remove.test.ts index c1f596be5f7..4c747bd8f77 100644 --- a/test/e2e/live/channels-add-remove.test.ts +++ b/test/e2e/live/channels-add-remove.test.ts @@ -17,6 +17,7 @@ import { import { expect, test } from "../fixtures/e2e-test.ts"; import { startFakeDockerApi } from "../fixtures/fake-docker-api.ts"; import { startFakeOpenAiCompatibleServer } from "../fixtures/fake-openai-compatible.ts"; +import { discoverHostInterfaceAddress } from "../fixtures/host-address.ts"; import { applyFixtureProviderPolicyEndpoint, clearFixtureProviderPolicyEndpoint, @@ -119,6 +120,7 @@ async function onboardWithLocalBaseline(host: HostCliClient, endpointUrl: string NEMOCLAW_AGENT: "openclaw", NEMOCLAW_COMPAT_MODEL: BASELINE_MODEL, NEMOCLAW_ENDPOINT_URL: endpointUrl, + NEMOCLAW_TRUSTED_PRIVATE_INFERENCE_HOSTS: new URL(endpointUrl).hostname, NEMOCLAW_MODEL: BASELINE_MODEL, NEMOCLAW_PREFERRED_API: "openai-completions", NEMOCLAW_PROVIDER: "custom", @@ -393,14 +395,18 @@ test( `channels-add-remove live test is destructive and only accepts sandbox names with prefix ${TEST_SANDBOX_PREFIX}; got ${SANDBOX_NAME}`, ); } - // The OpenShell router reaches this fixture from its own network namespace, - // so runner loopback is not a valid advertised provider endpoint. + // Native custom providers need a routable endpoint outside the legacy bridge route. + const inferenceHost = await discoverHostInterfaceAddress( + host, + "channels-add-remove-inference-host-address", + ); + const inferenceEnv = { NEMOCLAW_TRUSTED_PRIVATE_INFERENCE_HOSTS: inferenceHost.address }; const baseline = await startFakeOpenAiCompatibleServer({ apiKey: BASELINE_API_KEY, host: "0.0.0.0", model: BASELINE_MODEL, progress, - publicHost: "host.openshell.internal", + publicHost: inferenceHost.address, requireAuth: true, }); cleanup.add("close channels add/remove baseline fixture", async () => { @@ -528,7 +534,7 @@ test( const baselineRequestCountBeforeCredentialReuseRebuild = baseline.requests().length; const rebuildAdd = await host.nemoclaw([SANDBOX_NAME, "rebuild", "--yes"], { artifactName: "phase-3-rebuild-after-add-without-host-compatible-key", - env: channelEnv(), + env: channelEnv(inferenceEnv), redactionValues: secretsToRedact, timeoutMs: REBUILD_TIMEOUT_MS, }); @@ -629,7 +635,7 @@ test( progress.phase("remove Telegram and rebuild sandbox"); const remove = await host.nemoclaw([SANDBOX_NAME, "channels", "remove", "telegram"], { artifactName: "phase-5-channels-remove-telegram", - env: channelEnv({ COMPATIBLE_API_KEY: apiKey }), + env: channelEnv({ ...inferenceEnv, COMPATIBLE_API_KEY: apiKey }), redactionValues: secretsToRedact, timeoutMs: COMMAND_TIMEOUT_MS, }); @@ -639,7 +645,7 @@ test( const rebuildRemove = await host.nemoclaw([SANDBOX_NAME, "rebuild", "--yes"], { artifactName: "phase-5-rebuild-after-remove-with-stale-telegram-env", - env: channelEnv({ COMPATIBLE_API_KEY: apiKey }), + env: channelEnv({ ...inferenceEnv, COMPATIBLE_API_KEY: apiKey }), redactionValues: secretsToRedact, timeoutMs: REBUILD_TIMEOUT_MS, }); @@ -656,7 +662,6 @@ test( attempts: 12, delayMs: 5_000, }); - expectHostTelegramPlan("removed", "after remove rebuild"); progress.phase("validate Telegram removal"); const removedTelegram = await readOpenClawTelegramState( diff --git a/test/e2e/live/cloudflared-prerequisite.ts b/test/e2e/live/cloudflared-prerequisite.ts index bc94226139b..8f40437e067 100644 --- a/test/e2e/live/cloudflared-prerequisite.ts +++ b/test/e2e/live/cloudflared-prerequisite.ts @@ -97,21 +97,6 @@ export async function resolveVerifiedCloudflaredBinary( if (actualSha256 !== pin.debSha256) { throw new Error(`cloudflared package SHA256 mismatch: expected ${pin.debSha256}`); } - const packageName = ( - await commandOutput(host, "dpkg-deb", ["-f", deb, "Package"], "package-name") - ).trim(); - const version = ( - await commandOutput(host, "dpkg-deb", ["-f", deb, "Version"], "package-version") - ).trim(); - const architecture = ( - await commandOutput(host, "dpkg-deb", ["-f", deb, "Architecture"], "package-architecture") - ).trim(); - if (packageName !== "cloudflared" || version !== pin.version || architecture !== "amd64") { - throw new Error( - `unexpected cloudflared package metadata: package=${packageName} version=${version} architecture=${architecture}`, - ); - } - const extracted = path.join(root, "extracted"); await commandOutput(host, "dpkg-deb", ["-x", deb, extracted], "extract"); const binary = path.join(extracted, "usr", "bin", "cloudflared"); diff --git a/test/e2e/live/common-egress-agent-helpers.ts b/test/e2e/live/common-egress-agent-helpers.ts index 737a5aa3d24..feed8739e90 100644 --- a/test/e2e/live/common-egress-agent-helpers.ts +++ b/test/e2e/live/common-egress-agent-helpers.ts @@ -27,6 +27,17 @@ import { isTransientProviderValidationFailure } from "./network-policy-transient export const COMMON_EGRESS_TEST_TIMEOUT_MS = 40 * 60_000; +export const WEATHER_RESPONSE_VALIDATION_PYTHON = [ + "import json,sys,math", + "d=json.load(open(sys.argv[1]))", + 'c=d["current"]', + 't=c["temperature_2m"]', + "assert type(t) in (int,float) and math.isfinite(t)", + 'assert abs(d["latitude"]-37.7749)<0.1 and abs(d["longitude"]+122.4194)<0.1', + 'assert isinstance(c["time"],str) and len(c["time"])>=16', + 'assert d["current_units"]["temperature_2m"]=="°C"', +].join("; "); + interface ChatCompletionLike { choices?: Array<{ message?: { @@ -47,7 +58,18 @@ export interface OpenClawToolRecord { target?: OpenClawToolTarget; } +type WebFetchErrorCategory = + | "http" + | "invalid-url" + | "invalid-max-chars" + | "disabled" + | "extraction" + | "fetch-failed" + | "other" + | "unknown"; + export interface OpenClawWebFetchResultEvidence { + errorCategory?: WebFetchErrorCategory; httpStatus?: number; toolError?: boolean; expectedContentMatches: boolean; @@ -123,7 +145,12 @@ export interface PersonalPublicFetchToolEvidenceArtifact { total: number; }; webFetchResultsWithinMaxChars: number; - webFetchDiagnostics: { httpStatus: number | null; toolError: boolean; paired: boolean }[]; + webFetchDiagnostics: { + httpStatus: number | null; + toolError: boolean; + paired: boolean; + errorCategory: WebFetchErrorCategory | null; + }[]; } export interface OpenClawToolEvidenceReductionFailureArtifact { @@ -499,6 +526,47 @@ export function reduceOpenClawToolEvidence( } return null; }; + const diagnosticStatusFrom = (payload: Record | null): number | undefined => { + const status = payload?.status; + // OpenClaw's tool adapter serializes non-2xx fetch errors in this fixed prefix. + // Read only the status, never the response detail following the colon. + const errorStatus = + payload?.status === "error" && + payload.tool === "web_fetch" && + typeof payload.error === "string" + ? /^Web fetch failed \(([1-5][0-9]{2})\):/u.exec(payload.error.slice(0, 32))?.[1] + : undefined; + return typeof status === "number" ? status : errorStatus ? Number(errorStatus) : undefined; + }; + const errorCategoryFrom = ( + payload: Record | null, + status: number | undefined, + ): WebFetchErrorCategory => { + if (typeof status === "number" && Number.isInteger(status) && status >= 400 && status <= 599) { + return "http"; + } + // The pinned tool adapter serializes Error.message in this envelope. + // Project fixed categories only; arbitrary messages and nested causes stay in the sandbox. + if ( + payload?.status !== "error" || + payload.tool !== "web_fetch" || + typeof payload.error !== "string" + ) { + return "unknown"; + } + switch (payload.error) { + case "Invalid URL: must be http or https": + return "invalid-url"; + case "maxChars must be a positive integer": + return "invalid-max-chars"; + case "web_fetch is disabled.": + return "disabled"; + case "fetch failed": + return "fetch-failed"; + default: + return payload.error.startsWith("Web fetch extraction failed: ") ? "extraction" : "other"; + } + }; const expectedUrl = normalizedUrl(expectedFetch?.url); const expectedContent = typeof expectedFetch?.content === "string" && @@ -508,17 +576,20 @@ export function reduceOpenClawToolEvidence( : null; const trajectoryDocuments = parseJsonLines(trajectoryJsonLines, "trajectory"); const projectedMessages: unknown[] = []; + let projectedRunId: string | null = null; for (const document of trajectoryDocuments) { const root = asRecord(document); if (root?.type !== "model.completed") continue; const data = asRecord(root.data); if (!Array.isArray(data?.messagesSnapshot)) continue; + projectedRunId = normalizedId(root.runId); projectedMessages.splice(0, projectedMessages.length, ...data.messagesSnapshot); } const toolCalls: OpenClawToolRecord[] = []; const sessionDocuments = parseJsonLines(sessionJsonLines, "session"); const allowedWrapperIdParts = new Set(); + const allowedWrapperIds = new Set(); const controlTargetViolationIds = new Set(); const sessionCallIds = new Set(); const sessionToolNames = new Set(); @@ -558,11 +629,77 @@ export function reduceOpenClawToolEvidence( controlTargetViolationIds.add(id); } if (name === "tool_call" && allowedControlTarget) { + allowedWrapperIds.add(id); allowedWrapperIdParts.add(sanitizeToolCallIdPart(id)); } } } + const readProjectedWrapperIds = (): Set => { + const ids = new Set(); + for (const document of projectedMessages) { + const root = asRecord(document); + const message = asRecord(root?.message ?? document); + if (message?.role !== "assistant" || !Array.isArray(message.content)) continue; + for (const value of message.content) { + const block = asRecord(value); + const id = normalizedId(block?.id ?? block?.toolCallId ?? block?.tool_call_id); + if ( + block?.type === "toolCall" && + normalizedName(block.name) === "tool_call" && + id && + isAllowedWebFetchControlTarget(block.arguments ?? block.input ?? block.args) + ) { + ids.add(id); + } + } + } + return ids; + }; + const projectedWrapperIds = readProjectedWrapperIds(); + const nestedMessages: unknown[] = []; const projectedTargetCallIds = new Set(); + for (const document of sessionDocuments) { + const root = asRecord(document); + const message = asRecord(root?.message ?? document); + if (message?.customType !== "openclaw.nested-tool.v1") continue; + const details = asRecord(message.details); + const result = asRecord(details?.result); + const parentId = normalizedId(details?.parentToolCallId); + const id = normalizedId(details?.toolCallId); + const name = normalizedName(details?.toolName); + const prefix = + parentId && name ? `tool_search_code:${sanitizeToolCallIdPart(parentId)}:${name}:` : ""; + if ( + message.role !== "custom" || + message.display !== true || + message.excludeFromContext !== true || + !details || + !result || + !Array.isArray(result.content) || + typeof details.isError !== "boolean" || + !projectedRunId || + normalizedId(details.runId) !== projectedRunId || + !parentId || + !allowedWrapperIds.has(parentId) || + !projectedWrapperIds.has(parentId) || + !id || + !name || + !id.startsWith(prefix) || + !/^[1-9][0-9]*$/u.test(id.slice(prefix.length)) + ) { + addError("nested tool evidence has invalid correlation or result"); + continue; + } + if (nestedMessages.length >= MAX_RECORDS * 2) { + addError("nested tool record limit exceeded"); + break; + } + nestedMessages.push( + { role: "assistant", content: [{ type: "toolCall", id, name, arguments: details.input }] }, + { ...result, role: "toolResult", toolCallId: id, toolName: name, isError: details.isError }, + ); + if (name === "web_fetch") projectedTargetCallIds.add(id); + } for (const document of projectedMessages) { const root = asRecord(document); const message = asRecord(root?.message ?? document); @@ -581,14 +718,13 @@ export function reduceOpenClawToolEvidence( if (associated) projectedTargetCallIds.add(id); } } - // Progressive Tool Search persists the model-visible control calls in the - // session, then projects each invoked target tool into the final trajectory - // snapshot. Prefer the complete projection whenever present so result - // content remains available before session persistence truncates it. Only a - // paired web_fetch target whose generated ID binds to an allowed native - // wrapper grants the control-tool exemption; direct-disclosure sessions - // remain web_fetch-only. - const proofMessages = projectedMessages.length > 0 ? projectedMessages : sessionDocuments; + // OpenClaw 2026.9.2 stores nested results outside model snapshots. Bind those + // records to the current run and an allowed wrapper before reducing them. + // A paired target result is still required for the control-tool exemption. + const proofMessages = [ + ...(projectedMessages.length > 0 ? projectedMessages : sessionDocuments), + ...nestedMessages, + ]; const callsById = new Map< string, { @@ -668,16 +804,8 @@ export function reduceOpenClawToolEvidence( const boundedPayloadText = typeof payload?.text === "string" ? payload.text.slice(0, 20_000) : ""; const status = payload?.status; - // OpenClaw's tool adapter serializes non-2xx fetch errors in this fixed prefix. - // Read only the status, never the response detail following the colon. - const errorStatus = - payload?.status === "error" && - payload.tool === "web_fetch" && - typeof payload.error === "string" - ? /^Web fetch failed \(([1-5][0-9]{2})\):/u.exec(payload.error.slice(0, 32))?.[1] - : undefined; - const diagnosticStatus = - typeof status === "number" ? status : errorStatus ? Number(errorStatus) : undefined; + const diagnosticStatus = diagnosticStatusFrom(payload); + const toolError = message.isError === true || payload?.status === "error"; const paired = call?.name === "web_fetch" && resultName === "web_fetch"; if (paired && callId && projectedTargetCallIds.has(callId)) { pairedProjectedTargetCallIds.add(callId); @@ -695,7 +823,8 @@ export function reduceOpenClawToolEvidence( diagnosticStatus <= 599 ? { httpStatus: diagnosticStatus } : {}), - toolError: message.isError === true || payload?.status === "error", + toolError, + ...(toolError ? { errorCategory: errorCategoryFrom(payload, diagnosticStatus) } : {}), expectedContentMatches: expectedContent !== null && boundedPayloadText.includes(expectedContent), expectedUrlMatches: @@ -973,6 +1102,20 @@ export function projectPersonalPublicFetchToolEvidenceArtifact( : null, toolError: result.toolError === true, paired: result.paired === true, + errorCategory: + result.toolError !== true + ? null + : [ + "http", + "invalid-url", + "invalid-max-chars", + "disabled", + "extraction", + "fetch-failed", + "other", + ].includes(result.errorCategory ?? "") + ? (result.errorCategory as WebFetchErrorCategory) + : "unknown", })), }; } diff --git a/test/e2e/live/common-egress-agent.test.ts b/test/e2e/live/common-egress-agent.test.ts index c258d0908cb..2af4ee2e1f0 100644 --- a/test/e2e/live/common-egress-agent.test.ts +++ b/test/e2e/live/common-egress-agent.test.ts @@ -37,6 +37,7 @@ import { classifyHermesAgentAssertion, classifyPreContractProviderValidationSkip, COMMON_EGRESS_TEST_TIMEOUT_MS, + WEATHER_RESPONSE_VALIDATION_PYTHON, parseChatContent, runHermesAgentAssertionRetry, type OpenClawPublicFetchExpectation, @@ -597,7 +598,7 @@ const hermesTest = process.env.NEMOCLAW_COMMON_EGRESS_SKIP_HERMES === "1" ? test describe.sequential("common-egress agent live targets", () => { openClawTest( - "C1 OpenClaw balanced excludes weather until explicitly added, then permits a verified wttr.in curl", + "C1 OpenClaw balanced excludes weather until explicitly added, then permits a verified Open-Meteo curl", { timeout: testTimeout(COMMON_EGRESS_TEST_TIMEOUT_MS), meta: { @@ -622,7 +623,7 @@ describe.sequential("common-egress agent live targets", () => { "OpenClaw balanced onboarding applies exactly five expected presets without weather", "explicit policy-add weather applies the weather common-egress endpoints", "balanced scope does not include the broader restcountries public-reference endpoint", - "a real OpenClaw agent turn validates one wttr.in response and leaves its body as proof", + "a real OpenClaw agent turn validates one Open-Meteo weather response and leaves its body as proof", "without a search provider or retrieved content, a fresh OpenClaw turn discloses unavailable search and returns no installation-documentation result", ], }); @@ -659,7 +660,7 @@ describe.sequential("common-egress agent live targets", () => { sandbox, OPENCLAW_BALANCED_SANDBOX, "c1-weather-before-add", - "wttr.in", + "api.open-meteo.com", ); progress.phase("add weather egress policy"); @@ -685,7 +686,7 @@ describe.sequential("common-egress agent live targets", () => { "c1-balanced-scope", "restcountries.com", ); - const weatherProofPath = `/tmp/nemoclaw-weather-proof-${Date.now()}-${process.pid}.txt`; + const weatherProofPath = `/tmp/nemoclaw-weather-proof-${Date.now()}-${process.pid}.json`; const clearWeatherProof = await sandbox.execShell( OPENCLAW_BALANCED_SANDBOX, trustedSandboxShellScript(`rm -f ${shellQuote(weatherProofPath)}`), @@ -699,34 +700,37 @@ describe.sequential("common-egress agent live targets", () => { progress.phase("run verified weather agent turn"); // The agent must leave the fetched body behind. The host-side assertion // independently validates it, so merely echoing the reply token cannot pass. + const validateWeatherResponse = `python3 -c ${shellQuote(WEATHER_RESPONSE_VALIDATION_PYTHON)}`; const weatherProofCommand = [ "set -eu", `proof=${shellQuote(weatherProofPath)}`, - "if test -s \"$proof\"; then printf 'WEATHER_AGENT_OK\\n'; exit 0; fi", + `if test -s "$proof"; then ${validateWeatherResponse} "$proof"; printf 'WEATHER_AGENT_OK\\n'; exit 0; fi`, "tmp=$(mktemp)", "trap 'rm -f \"$tmp\"' EXIT", - "curl -fsS --max-time 30 --output \"$tmp\" 'https://wttr.in/:help'", + "curl -fsS --max-time 30 --output \"$tmp\" 'https://api.open-meteo.com/v1/forecast?latitude=37.7749&longitude=-122.4194¤t=temperature_2m'", 'test -s "$tmp"', - "grep -Fq 'Usage:' \"$tmp\"", - "grep -Fq 'Special URLs:' \"$tmp\"", + `${validateWeatherResponse} "$tmp"`, 'mv "$tmp" "$proof"', "trap - EXIT", "printf 'WEATHER_AGENT_OK\\n'", ].join("; "); await runOpenClawAgentAssertion(host, sandbox, artifacts, { apiKey, + collectToolEvidenceOnFailure: true, expected: "WEATHER_AGENT_OK", + finalReplyContract: "exact-token", label: "c1-agent-weather", sandboxName: OPENCLAW_BALANCED_SANDBOX, - prompt: `Run exactly this shell command to verify the weather host curl path: + prompt: `Call the exec tool with exactly this shell command to verify the weather host curl path: ${weatherProofCommand} +Use a structured tool call. Do not print the command or tool-call markup as your answer. Do not use web_fetch, web_search, or any other weather provider. After it returns, reply with only WEATHER_AGENT_OK. Do not fetch any other URL.`, }); const weatherProof = await sandbox.execShell( OPENCLAW_BALANCED_SANDBOX, trustedSandboxShellScript( - `test -s ${shellQuote(weatherProofPath)} && grep -Fq 'Usage:' ${shellQuote(weatherProofPath)} && grep -Fq 'Special URLs:' ${shellQuote(weatherProofPath)} && sha256sum ${shellQuote(weatherProofPath)}`, + `${validateWeatherResponse} ${shellQuote(weatherProofPath)} && sha256sum ${shellQuote(weatherProofPath)}`, ), { artifactName: "c1-weather-agent-proof", diff --git a/test/e2e/live/cron-preflight-inference-local.test.ts b/test/e2e/live/cron-preflight-inference-local.test.ts index 02c18d1960e..5e44038ef77 100644 --- a/test/e2e/live/cron-preflight-inference-local.test.ts +++ b/test/e2e/live/cron-preflight-inference-local.test.ts @@ -16,7 +16,7 @@ const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? "e2e-cron-preflight"; const MODEL = process.env.NEMOCLAW_CRON_PREFLIGHT_MODEL ?? DEFAULT_HOSTED_INFERENCE_MODEL; test( - "native OpenClaw scheduled work reaches inference.local", + "native OpenClaw scheduled work reaches its attached inference provider", { timeout: testTimeout(30 * 60_000), meta: { @@ -47,10 +47,10 @@ test( await artifacts.target.declare({ id: "cron-preflight-inference-local", - boundary: "native OpenClaw cron add/run through managed inference.local", + boundary: "native OpenClaw cron add/run through its attached inference provider", contracts: [ "native OpenClaw creates scheduled work without NemoClaw lifecycle authorization", - "native cron run passes its provider preflight and reaches inference.local", + "native cron run passes its provider preflight and reaches the selected endpoint", "the native command reports a completed or queued run without endpoint-unreachable errors", ], }); @@ -227,7 +227,7 @@ test( status: "passed", nativeCronId: cronId, nativeRunAccepted: true, - inferenceRoute: "https://inference.local/v1", + inferenceRoute: hosted.endpointUrl, }); }, ); diff --git a/test/e2e/live/deepagents-observability-contract.ts b/test/e2e/live/deepagents-observability-contract.ts index 65415b23857..d963d0c3f8e 100644 --- a/test/e2e/live/deepagents-observability-contract.ts +++ b/test/e2e/live/deepagents-observability-contract.ts @@ -2,10 +2,12 @@ // SPDX-License-Identifier: Apache-2.0 import fs from "node:fs"; +import { isIP } from "node:net"; import path from "node:path"; import { pathToFileURL } from "node:url"; import * as policyDenialNamespace from "../../../src/lib/actions/sandbox/exec-policy-hint-detection.ts"; +import * as curlDenialNamespace from "../fixtures/curl-policy-denial.ts"; import type { DecodedOtlpSpan } from "./otlp-trace-decoder.ts"; import * as decoderNamespace from "./otlp-trace-decoder.ts"; import * as policyStateNamespace from "./policy-list-state.ts"; @@ -15,6 +17,7 @@ function moduleExports(namespace: T & { default?: T }): T { } const { isPolicyDenialLine } = moduleExports(policyDenialNamespace); +const { isExpectedMcpCurlPolicyDenial } = moduleExports(curlDenialNamespace); const { decodeExportTraceServiceRequest, otlpValueContains } = moduleExports(decoderNamespace); const { parsePolicyPresetState } = moduleExports(policyStateNamespace); @@ -145,8 +148,104 @@ export function assertDeepAgentsTraceContract( return { requestCount: bodies.length, spanCount: spans.length }; } -export function hasConfirmedOpenShellPolicyDenial(output: string): boolean { - if (output.split(/\r?\n/u).some(isPolicyDenialLine) || CONFIRMED_EXEC_HINT.test(output)) { +type DnsDenialAudit = { before: string; after: string; hostname: string; port?: number }; + +function hasFreshPythonTcpDenial(output: string, audit?: DnsDenialAudit): boolean { + if ( + !audit?.port || + !/^FAILED:URLError:$/mu.test(output) + ) { + return false; + } + const records = output + .split(/\r?\n/u) + .filter((line) => line.startsWith("NEMOCLAW_OTLP_PROBE_DNS=")); + if (records.length !== 1 || records[0].length > 4096) return false; + let probe: { hostname?: unknown; port?: unknown; addresses?: unknown }; + try { + probe = JSON.parse(records[0].slice("NEMOCLAW_OTLP_PROBE_DNS=".length)); + } catch { + return false; + } + if ( + !probe || + probe.hostname !== audit.hostname || + probe.port !== audit.port || + !Array.isArray(probe.addresses) + ) + return false; + const previous = new Set(audit.before.split(/\r?\n/u)); + if ( + audit.after + .split(/\r?\n/u) + .some( + (line) => + !previous.has(line) && + line.endsWith( + `NET:OPEN [MED] DENIED /opt/venv/bin/python3(0) -> ${audit.hostname}:${audit.port} [reason:transparent_tcp_policy_denied]`, + ), + ) + ) { + return true; + } + // OpenShell 0.1.3 owns 198.18.0.0/15 synthetic DNS mappings. A denied + // mapping must match this probe's DNS answer and requested port. + return probe.addresses.some( + (address) => + typeof address === "string" && + isIP(address) === 4 && + /^198\.(?:18|19)\./u.test(address) && + audit.after + .split(/\r?\n/u) + .some( + (line) => + !previous.has(line) && + line.endsWith( + `NET:OPEN [MED] DENIED /opt/venv/bin/python3(0) -> ${address}:${audit.port} [reason:transparent_tcp_mapping_denied]`, + ), + ), + ); +} + +function hasFreshPythonDnsDenial(output: string, audit?: DnsDenialAudit): boolean { + const previous = new Set(audit?.before.split(/\r?\n/u)); + return ( + audit !== undefined && + /^FAILED:URLError:$/mu.test( + output, + ) && + audit.after + .split(/\r?\n/u) + .some( + (line) => + !previous.has(line) && + ["policy_dns_ineligible", "policy_dns_no_valid_address"].some((reason) => + line.endsWith(`NET:REFUSE [MED] DENIED ${audit.hostname} [reason:${reason}]`), + ), + ) + ); +} + +export function hasConfirmedOpenShellPolicyDenial( + output: string, + curlExitCode?: number, + dnsAudit?: DnsDenialAudit, +): boolean { + if ( + output.split(/\r?\n/u).some(isPolicyDenialLine) || + CONFIRMED_EXEC_HINT.test(output) || + (curlExitCode === 7 && + (hasFreshPythonDnsDenial(output, dnsAudit) || hasFreshPythonTcpDenial(output, dnsAudit))) + ) { + return true; + } + if ( + curlExitCode === 7 && + isExpectedMcpCurlPolicyDenial( + { exitCode: curlExitCode, stdout: output, stderr: output, timedOut: false }, + "NEMOCLAW_OTLP_CURL_HTTP_CODE=", + ) + ) { return true; } // curl writes the response body to stdout and its error to stderr. Once the @@ -256,15 +355,26 @@ export function validateCaptureDirectory( } async function main(): Promise { - const [command, argument] = process.argv.slice(2); + const [command, argument, beforePath, afterPath, probeUrl] = process.argv.slice(2); const input = command === "validate-captures" ? "" : fs.readFileSync(0, "utf8"); if (command === "policy-state") { process.stdout.write(`${observabilityPresetState(input)}\n`); return; } if (command === "denial-state") { + const dnsAudit = + beforePath && afterPath && probeUrl + ? { + before: fs.readFileSync(beforePath, "utf8"), + after: fs.readFileSync(afterPath, "utf8"), + hostname: new URL(probeUrl).hostname, + port: Number( + new URL(probeUrl).port || (new URL(probeUrl).protocol === "https:" ? 443 : 80), + ), + } + : undefined; process.stdout.write( - `${hasConfirmedOpenShellPolicyDenial(input) ? "policy-denied" : "other-failure"}\n`, + `${hasConfirmedOpenShellPolicyDenial(input, Number(argument), dnsAudit) ? "policy-denied" : "other-failure"}\n`, ); return; } diff --git a/test/e2e/live/deepagents-otlp-capture-server.ts b/test/e2e/live/deepagents-otlp-capture-server.ts index 97396161b50..173ced9ff8d 100644 --- a/test/e2e/live/deepagents-otlp-capture-server.ts +++ b/test/e2e/live/deepagents-otlp-capture-server.ts @@ -331,6 +331,7 @@ async function main(): Promise { ); } const started = await startOtlpCaptureServers({ + allowLoopback: bindIp === "127.0.0.1", bindIp, captureDir, collectorPort, diff --git a/test/e2e/live/deferred-onboarding.test.ts b/test/e2e/live/deferred-onboarding.test.ts index 7695f62519a..23f8f6c3e03 100644 --- a/test/e2e/live/deferred-onboarding.test.ts +++ b/test/e2e/live/deferred-onboarding.test.ts @@ -13,6 +13,7 @@ import { resultText } from "../fixtures/clients/command.ts"; import { validateSandboxName } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { assertStockManagedImageReceipt } from "../fixtures/managed-image-receipt.ts"; +import { cleanupOwnedSandboxAfterOnboard } from "../fixtures/owned-sandbox-cleanup.ts"; import { REPO_ROOT } from "../fixtures/paths.ts"; import { captureSandboxFailureDiagnostics } from "../fixtures/sandbox-failure-diagnostics.ts"; @@ -71,7 +72,7 @@ test( ); cleanup.trackDisposable(`delete OpenShell sandbox ${SANDBOX_NAME}`, () => cleanupWhenOpenShellAvailable(host, commandOptions, () => - sandbox.cleanupSandbox(SANDBOX_NAME, commandOptions), + cleanupOwnedSandboxAfterOnboard(sandbox, runtimeProvider, SANDBOX_NAME, commandOptions), ), ); cleanup.trackDisposable(`destroy deferred sandbox ${SANDBOX_NAME}`, () => diff --git a/test/e2e/live/dgx-express.test.ts b/test/e2e/live/dgx-express.test.ts index e372c23ef80..12e51c6490c 100644 --- a/test/e2e/live/dgx-express.test.ts +++ b/test/e2e/live/dgx-express.test.ts @@ -5,11 +5,16 @@ import assert from "node:assert/strict"; +import { normalizeNativeLocalProviderAttachment } from "../../../src/lib/inference/native-local/contract.ts"; +import { + NATIVE_LOCAL_AUTH_HEADER_SCRIPT, + NATIVE_LOCAL_CREDENTIAL_ENV, +} from "../../../src/lib/inference/native-local/agent-config.ts"; +import { shellQuote } from "../../../src/lib/core/shell-quote.ts"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; import { loadServingCatalog } from "../../../src/lib/inference/serving/catalog-loader.ts"; import { materializeHostLocalVllmSelection } from "../../../src/lib/inference/serving/host-local-vllm-selection.ts"; -import { resolveManagedVllmBridgeHost } from "../../../src/lib/inference/serving/vllm-host-local-network.ts"; import { buildVllmServeCommand } from "../../../src/lib/inference/vllm-models.ts"; -import { buildLocalManagedVllmDockerEnv } from "../../../src/lib/inference/vllm-docker-env.ts"; import { LOCAL_MODEL_PROFILE_ENABLED_ENV, LOCAL_MODEL_PROFILE_RUNTIME_ENV, @@ -30,7 +35,7 @@ import { } from "../support/spark-express-vllm-safety.ts"; import { cleanupSandbox, - expectOpenAiChatThroughSandbox, + openAiContent, requireLivePrerequisites, } from "./inference-routing-helpers.ts"; @@ -67,6 +72,7 @@ interface VllmContainerInspection { readonly IpcMode: string; readonly NetworkMode: string; readonly ShmSize: number; + readonly PortBindings: Record>; }; readonly Mounts: Array<{ Destination: string; RW: boolean; Type: string }>; } @@ -385,6 +391,8 @@ test( expect(inspection.HostConfig.NetworkMode).toBe(plan.recipe.spec.runtime.networkMode); expect(inspection.HostConfig.IpcMode).toBe(plan.recipe.spec.runtime.ipcMode); expect(inspection.HostConfig.ShmSize).toBe(plan.recipe.spec.runtime.sharedMemoryBytes); + const portBindings = inspection.HostConfig.PortBindings["8000/tcp"]; + expect(portBindings).toEqual([{ HostIp: "127.0.0.1", HostPort: "8000" }]); expect(inspection.HostConfig.DeviceRequests).toEqual( expect.arrayContaining([expect.objectContaining({ Count: -1, Capabilities: [["gpu"]] })]), ); @@ -399,9 +407,7 @@ test( ); const publicationEnv = e2eEnv(); - const probes = vllmPublicationProbeUrls( - resolveManagedVllmBridgeHost(undefined, buildLocalManagedVllmDockerEnv({}, publicationEnv)), - ); + const probes = vllmPublicationProbeUrls(); await artifacts.writeJson("spark-express-vllm-publication-probes", probes); const allowed = await host.command( "curl", @@ -416,7 +422,7 @@ test( }, ); expect(allowed.exitCode, resultText(allowed)).toBe(0); - expect(allowed.stdout.trim().split("\n")).toEqual(["0 200", "0 200"]); + expect(allowed.stdout.trim().split("\n")).toEqual(["0 200"]); const refused = await host.command( "curl", [ @@ -433,26 +439,56 @@ test( expect(refused.stdout.trim().split("\n")).toEqual(probes.denied.map(() => "7 000")); progress.phase("prove sandbox inference and unrelated egress denial"); - await expectOpenAiChatThroughSandbox( - sandbox, - SANDBOX_NAME, - plan.recipe.spec.model.servedName, - [], - "spark-express-inference-local-chat", + const route = normalizeNativeLocalProviderAttachment( + getSandbox(SANDBOX_NAME)?.nativeLocalProviderAttachment, ); - const denied = await sandbox.execShell( + expect( + route?.sandboxName === SANDBOX_NAME && + route.gatewayName === "nemoclaw" && + route.agent === "openclaw" && + route.provider === "vllm-local" && + route.authMode === "authenticated" && + route.credentialEnv === NATIVE_LOCAL_CREDENTIAL_ENV, + "The managed vLLM probe requires its authenticated native attachment.", + ).toBe(true); + const payload = JSON.stringify({ + model: plan.recipe.spec.model.servedName, + messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], + max_tokens: 1024, + }); + const chat = await sandbox.execShell( SANDBOX_NAME, trustedSandboxShellScript( - "status=0; code=$(curl -sS -o /dev/null -w '%{http_connect}' --max-time 20 https://example.com/) || status=$?; printf '%s %s' \"$status\" \"$code\"", + `${NATIVE_LOCAL_AUTH_HEADER_SCRIPT}; exec curl -sS --max-time 60 ${shellQuote(`${route!.endpointUrl}/chat/completions`)} -H "$AUTH_HEADER" -H 'Content-Type: application/json' --data-raw ${shellQuote(payload)}`, ), + { artifactName: "spark-express-native-vllm-chat", env: e2eEnv(), timeoutMs: 90_000 }, + ); + expect(chat.exitCode, resultText(chat)).toBe(0); + expect( + openAiContent(JSON.parse(chat.stdout)), + "The native vLLM response has no chat content.", + ).not.toBe(""); + const denied = await sandbox.exec( + SANDBOX_NAME, + [ + "curl", + "-sS", + "-o", + "/dev/null", + "-w", + "%{http_connect}", + "--max-time", + "20", + "https://example.com/", + ], { artifactName: "spark-express-unrelated-egress-denied", env: e2eEnv(), timeoutMs: 30_000, }, ); - expect(denied.exitCode, resultText(denied)).toBe(0); - expect(denied.stdout.trim()).toBe("56 403"); + expect(denied.exitCode, resultText(denied)).toBe(56); + expect(denied.stdout.trim()).toBe("403"); }, ); @@ -493,9 +529,7 @@ test( cleanup: () => progress.phase("clean up the job runtime"), }, verifyPublication: async (env) => { - const probes = vllmPublicationProbeUrls( - resolveManagedVllmBridgeHost(undefined, buildLocalManagedVllmDockerEnv({}, env)), - ); + const probes = vllmPublicationProbeUrls(); await artifacts.writeJson("station-express-vllm-publication-probes", probes); const allowed = await host.command( "curl", @@ -510,7 +544,7 @@ test( }, ); expect(allowed.exitCode, resultText(allowed)).toBe(0); - expect(allowed.stdout.trim().split("\n")).toEqual(["0 200", "0 200"]); + expect(allowed.stdout.trim().split("\n")).toEqual(["0 200"]); const refused = await host.command( "curl", [ diff --git a/test/e2e/live/dns-rebinding-hosts-fixture.ts b/test/e2e/live/dns-rebinding-hosts-fixture.ts index 14412f3afa3..618ad8060fc 100644 --- a/test/e2e/live/dns-rebinding-hosts-fixture.ts +++ b/test/e2e/live/dns-rebinding-hosts-fixture.ts @@ -2,10 +2,14 @@ // SPDX-License-Identifier: Apache-2.0 import os from "node:os"; +import fs from "node:fs"; +import { randomUUID } from "node:crypto"; +import { startRebindingDnsServer } from "../fixtures/dns-rebinding-server.ts"; import path from "node:path"; import { shellQuote } from "../../../src/lib/core/shell-quote"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; +import { assertExitZero } from "../fixtures/clients/command.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import { RuntimeProviderPrerequisite } from "../fixtures/runtime-provider.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -13,18 +17,178 @@ import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; export interface DnsRebindingHostsFixture { hostname: string; hostBackupPath: string; - sandboxBackupPath: string; + backupReady: boolean; + removeBackup: () => void; + containerId: string; + pid: number; + cgroup: string; + upstream: string; + comment: string; + server: Awaited>; + restored: boolean; } -function assertHostFixtureProbeSucceeded(result: ShellProbeResult, label: string): void { - if (result.exitCode === 0) return; - throw new Error(`${label}\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`); +async function resolveResolverHosts(host: HostCliClient, sandboxName: string) { + const runtime = new RuntimeProviderPrerequisite(host); + const inspect = async (id: string) => { + if (!/^[a-f0-9]{64}$/u.test(id)) throw new Error("DNS fixture requires one full container ID"); + const result = await runtime.command( + [ + "container", + "inspect", + "--format", + '{"id":{{json .Id}},"name":{{json (index .Config.Labels "openshell.ai/sandbox-name")}},"role":{{json (index .Config.Labels "openshell.ai/isolation-role")}},"sandboxId":{{json (index .Config.Labels "openshell.ai/sandbox-id")}},"namespace":{{json (index .Config.Labels "openshell.ai/sandbox-namespace")}},"running":{{json .State.Running}},"pid":{{json .State.Pid}},"resolvPath":{{json .ResolvConfPath}},"networkMode":{{json .HostConfig.NetworkMode}}}', + id, + ], + { artifactName: "dns-rebinding-resolver-identity", timeoutMs: 30_000 }, + ); + assertExitZero(result, "inspect DNS resolver container"); + const record = JSON.parse(result.stdout) as Record; + if (record.id !== id || record.name !== sandboxName || record.running !== true) { + throw new Error("DNS resolver container identity or running state changed"); + } + return record; + }; + let id = await runtime.resolveSandboxResourceHandle(sandboxName, { + artifactName: "dns-rebinding-workload-resource", + timeoutMs: 30_000, + }); + let record = await inspect(id); + const { sandboxId, namespace } = record; + if ( + record.role !== "sandbox" || + typeof sandboxId !== "string" || + !/^[a-zA-Z0-9_-]{1,128}$/u.test(sandboxId) || + typeof namespace !== "string" || + !/^[a-zA-Z0-9_-]{1,128}$/u.test(namespace) + ) { + throw new Error("DNS fixture requires the workload's sandbox identity"); + } + const supervisor = await runtime.command( + [ + "container", + "ps", + "--no-trunc", + "--filter", + `label=openshell.ai/sandbox-id=${sandboxId}`, + "--filter", + `label=openshell.ai/sandbox-namespace=${namespace}`, + "--filter", + "label=openshell.ai/isolation-role=supervisor", + "--format", + "{{.ID}}", + ], + { artifactName: "dns-rebinding-supervisor-resource", timeoutMs: 30_000 }, + ); + assertExitZero(supervisor, "discover DNS supervisor"); + id = supervisor.stdout.trim(); + record = await inspect(id); + if ( + record.role !== "supervisor" || + record.sandboxId !== sandboxId || + record.namespace !== namespace || + record.networkMode !== "host" + ) { + throw new Error("DNS fixture requires the selected workload's host-network supervisor"); + } + const { pid, resolvPath } = record; + if ( + !Number.isSafeInteger(pid) || + Number(pid) <= 1 || + typeof resolvPath !== "string" || + !path.isAbsolute(resolvPath) || + path.normalize(resolvPath) !== resolvPath || + !resolvPath.includes(`/${id}/`) || + path.basename(resolvPath) !== "resolv.conf" + ) { + throw new Error("DNS fixture requires the resolver process and engine-owned resolver file"); + } + const probe = await host.command( + "sudo", + [ + "-n", + "python3", + "-I", + "-c", + ` +import ipaddress, json, pathlib, sys +pid, container, resolver = sys.argv[1:] +start = pathlib.Path('/proc/' + pid + '/stat').read_text().rsplit(')', 1)[1].split()[19] +rows = pathlib.Path('/proc/' + pid + '/cgroup').read_text().splitlines() +assert len(rows) == 1 and rows[0].startswith('0::/') +cgroup = rows[0][4:] +parts = cgroup.split('/') +assert cgroup and all(part not in ('', '.', '..') for part in parts) +assert any(part == container or part.endswith('-' + container + '.scope') for part in parts) +assert not pathlib.Path(resolver).is_symlink() +upstream = None +for line in pathlib.Path(resolver).read_text().splitlines(): + fields = line.split('#', 1)[0].split() + if len(fields) < 2 or fields[0] != 'nameserver' or '%' in fields[1]: continue + try: upstream = str(ipaddress.ip_address(fields[1])); break + except ValueError: pass +assert upstream is not None +assert pathlib.Path('/proc/' + pid + '/stat').read_text().rsplit(')', 1)[1].split()[19] == start +print(json.dumps({'cgroup': cgroup, 'upstream': upstream})) +`, + String(pid), + id, + resolvPath, + ], + { artifactName: "dns-rebinding-resolver-cgroup", timeoutMs: 30_000 }, + ); + assertExitZero(probe, "bind DNS fixture to the supervisor process cgroup"); + const { cgroup, upstream } = JSON.parse(probe.stdout) as { cgroup: string; upstream: string }; + const current = await inspect(id); + if (current.pid !== pid || current.resolvPath !== resolvPath) + throw new Error("DNS supervisor changed during setup"); + return { containerId: id, pid: Number(pid), cgroup, upstream }; } -function runtimeInvocation(host: HostCliClient) { - return new RuntimeProviderPrerequisite(host, (reason) => { - throw new Error(reason); - }).hostInvocation([]); +function dnsRule(fixture: DnsRebindingHostsFixture, protocol: string): string[] { + return [ + "-p", + protocol, + "-d", + fixture.upstream, + "--dport", + "53", + "-m", + "cgroup", + "--path", + fixture.cgroup, + "-m", + "comment", + "--comment", + fixture.comment, + "-j", + "REDIRECT", + "--to-ports", + String(fixture.server.port), + ]; +} + +async function firewall( + host: HostCliClient, + fixture: DnsRebindingHostsFixture, + action: string, + protocol: string, +) { + return host.command( + "sudo", + [ + "-n", + fixture.server.family === 4 ? "iptables" : "ip6tables", + "-w", + "5", + "-t", + "nat", + action, + "OUTPUT", + ...dnsRule(fixture, protocol), + ], + { artifactName: `dns-rebinding-rule-${action}-${protocol}`, timeoutMs: 30_000 }, + ); } export async function setupDnsRebindingHostsFixture( @@ -32,48 +196,51 @@ export async function setupDnsRebindingHostsFixture( sandboxName: string, hostname: string, ): Promise { - const tempDir = process.env.RUNNER_TEMP ?? os.tmpdir(); - const suffix = `${process.pid}-${sandboxName}`; - const fixture = { - hostname, - hostBackupPath: path.join(tempDir, `nemoclaw-rebind-hosts-host-${suffix}`), - sandboxBackupPath: path.join(tempDir, `nemoclaw-rebind-hosts-sandbox-${suffix}`), - }; - const runtime = runtimeInvocation(host); - const result = await host.command( - "bash", - [ - "-lc", - [ - "set -euo pipefail", - 'runtime_command=("$@")', - `sandbox_name=${shellQuote(sandboxName)}`, - `hostname=${shellQuote(hostname)}`, - `host_backup=${shellQuote(fixture.hostBackupPath)}`, - `sandbox_backup=${shellQuote(fixture.sandboxBackupPath)}`, - 'container_id="$("${runtime_command[@]}" container ps --filter "label=openshell.ai/sandbox-name=${sandbox_name}" --format \'{{.ID}}\' | head -n 1)"', - '[ -n "$container_id" ] || { echo "OpenShell sandbox container not found" >&2; exit 1; }', - "sudo -n true", - 'rm -f "$host_backup" "$sandbox_backup"', - 'sudo -n cat /etc/hosts > "$host_backup"', - '"${runtime_command[@]}" container exec "$container_id" cat /etc/hosts > "$sandbox_backup"', - 'if grep -Fq "$hostname" "$host_backup" || grep -Fq "$hostname" "$sandbox_backup"; then rm -f "$host_backup" "$sandbox_backup"; echo "DNS rebinding fixture hostname already exists in /etc/hosts" >&2; exit 1; fi', - ].join("\n"), - "dns-rebinding-backup-hosts", - runtime.command, - ...runtime.args, - ], - { - artifactName: "dns-rebinding-backup-hosts", - env: buildAvailabilityProbeEnv(), - timeoutMs: 60_000, - }, + const identity = await resolveResolverHosts(host, sandboxName); + const comment = `nemoclaw-dns-${randomUUID()}`; + const directory = fs.mkdtempSync( + path.join(process.env.RUNNER_TEMP ?? os.tmpdir(), "nemoclaw-dns-"), ); - assertHostFixtureProbeSucceeded( - result, - "back up host and sandbox hosts files for DNS rebinding proof", - ); - return fixture; + let fixture: DnsRebindingHostsFixture | undefined; + try { + fixture = { + hostname, + hostBackupPath: path.join(directory, "hosts"), + backupReady: false, + ...identity, + server: await startRebindingDnsServer(hostname, identity.upstream), + comment, + restored: false, + removeBackup: () => fs.rmSync(directory, { recursive: true, force: true }), + }; + const backup = await host.command( + "bash", + [ + "-lc", + [ + "set -euo pipefail", + "umask 077", + "set -C", + `sudo -n cat /etc/hosts > ${shellQuote(fixture.hostBackupPath)}`, + `if grep -Fq ${shellQuote(hostname)} ${shellQuote(fixture.hostBackupPath)}; then echo 'DNS fixture hostname already exists' >&2; exit 1; fi`, + ].join("\n"), + ], + { artifactName: "dns-rebinding-backup-hosts", timeoutMs: 30_000 }, + ); + assertExitZero(backup, "back up host DNS mapping"); + fixture.backupReady = true; + for (const protocol of ["udp", "tcp"]) { + assertExitZero( + await firewall(host, fixture, "-A", protocol), + "redirect selected supervisor DNS", + ); + } + return fixture; + } catch (error) { + if (fixture) await restoreDnsRebindingHostsFixture(host, sandboxName, fixture); + else fs.rmdirSync(directory); + throw error; + } } export async function remapDnsRebindingHostname( @@ -83,6 +250,13 @@ export async function remapDnsRebindingHostname( address: string, artifactName: string, ): Promise { + const current = await resolveResolverHosts(host, sandboxName); + if ( + fixture.restored || + Object.entries(current).some(([key, value]) => fixture[key as keyof typeof current] !== value) + ) { + throw new Error("DNS supervisor identity changed before remapping"); + } const resolverCheck = [ 'const dns = require("node:dns");', "const [hostname, expected] = process.argv.slice(1);", @@ -93,32 +267,22 @@ export async function remapDnsRebindingHostname( " process.exit(addresses.length === 1 && addresses[0] === expected ? 0 : 1);", "});", ].join(" "); - const runtime = runtimeInvocation(host); + fixture.server.setAddress(address); const result = await host.command( "bash", [ "-lc", [ "set -euo pipefail", - 'runtime_command=("$@")', - `sandbox_name=${shellQuote(sandboxName)}`, `hostname=${shellQuote(fixture.hostname)}`, `expected_ip=${shellQuote(address)}`, `host_backup=${shellQuote(fixture.hostBackupPath)}`, - `sandbox_backup=${shellQuote(fixture.sandboxBackupPath)}`, - '[ -s "$host_backup" ] && [ -s "$sandbox_backup" ] || { echo "DNS rebinding hosts backups are missing" >&2; exit 1; }', - 'container_id="$("${runtime_command[@]}" container ps --filter "label=openshell.ai/sandbox-name=${sandbox_name}" --format \'{{.ID}}\' | head -n 1)"', - '[ -n "$container_id" ] || { echo "OpenShell sandbox container not found" >&2; exit 1; }', + '[ -s "$host_backup" ] || { echo "DNS rebinding host backup is missing" >&2; exit 1; }', 'sudo -n tee /etc/hosts < "$host_backup" >/dev/null', 'printf "\\n%s %s\\n" "$expected_ip" "$hostname" | sudo -n tee -a /etc/hosts >/dev/null', - '"${runtime_command[@]}" container exec --user 0 --interactive "$container_id" sh -c \'cat > /etc/hosts\' < "$sandbox_backup"', - 'printf "\\n%s %s\\n" "$expected_ip" "$hostname" | "${runtime_command[@]}" container exec --user 0 --interactive "$container_id" tee -a /etc/hosts >/dev/null', `node -e ${shellQuote(resolverCheck)} "$hostname" "$expected_ip"`, - '"${runtime_command[@]}" container exec "$container_id" grep -F "$expected_ip $hostname" /etc/hosts >/dev/null', ].join("\n"), "dns-rebinding-remap-hosts", - runtime.command, - ...runtime.args, ], { artifactName, @@ -126,67 +290,86 @@ export async function remapDnsRebindingHostname( timeoutMs: 60_000, }, ); - assertHostFixtureProbeSucceeded(result, `map DNS rebinding fixture hostname to ${address}`); + assertExitZero(result, `map DNS rebinding fixture hostname to ${address}`); } export async function restoreDnsRebindingHostsFixture( host: HostCliClient, - sandboxName: string, + _sandboxName: string, fixture: DnsRebindingHostsFixture, ): Promise { - const runtime = runtimeInvocation(host); - const result = await host.command( - "bash", - [ - "-lc", + if (fixture.restored) return; + const failures: string[] = []; + const observe = async (result: Promise, label: string) => { + try { + return await result; + } catch { + failures.push(label); + return null; + } + }; + for (const protocol of ["udp", "tcp"]) { + const present = await observe( + firewall(host, fixture, "-C", protocol), + `inspect ${protocol} DNS rule`, + ); + if (present?.exitCode === 0) { + const removed = await observe( + firewall(host, fixture, "-D", protocol), + `remove ${protocol} DNS rule`, + ); + if (removed?.exitCode !== 0) failures.push(`remove ${protocol} DNS rule`); + } + } + const remaining = await observe( + host.command( + "sudo", [ - // Cleanup must report the exact failed operation. An implicit `errexit` - // here can turn a transient file/container race into an unexplained - // exit 1 with empty stdout/stderr, which defeats the cleanup artifact. - "set -uo pipefail", - 'runtime_command=("$@")', - `sandbox_name=${shellQuote(sandboxName)}`, - `host_backup=${shellQuote(fixture.hostBackupPath)}`, - `sandbox_backup=${shellQuote(fixture.sandboxBackupPath)}`, - 'if [ ! -f "$host_backup" ] && [ ! -f "$sandbox_backup" ]; then echo "DNS rebinding hosts backups already absent"; exit 0; fi', - "host_restore_failed=0", - 'if [ -f "$host_backup" ]; then', - ' if ! sudo -n tee /etc/hosts < "$host_backup" >/dev/null; then', - ' echo "failed to restore host /etc/hosts" >&2; host_restore_failed=1', - ' elif ! cmp -s "$host_backup" /etc/hosts; then', - ' echo "host /etc/hosts differs after restoration" >&2; host_restore_failed=1', - " else", - ' echo "restored host /etc/hosts"', - " fi", - "else", - ' echo "host /etc/hosts backup is missing while sandbox backup remains" >&2; host_restore_failed=1', - "fi", - 'if [ -f "$sandbox_backup" ]; then', - " sandbox_restored=0", - " for attempt in 1 2 3; do", - ' container_id="$("${runtime_command[@]}" container ps --filter "label=openshell.ai/sandbox-name=${sandbox_name}" --format \'{{.ID}}\' 2>/dev/null | head -n 1 || true)"', - ' if [ -n "$container_id" ] && "${runtime_command[@]}" container exec --user 0 --interactive "$container_id" sh -c \'cat > /etc/hosts\' < "$sandbox_backup"; then sandbox_restored=1; break; fi', - ' [ "$attempt" -eq 3 ] || sleep 1', - " done", - ' if [ "$sandbox_restored" -eq 1 ]; then echo "restored sandbox /etc/hosts"; else echo "::warning::could not restore ephemeral sandbox /etc/hosts; cleanup will destroy the sandbox" >&2; fi', - "fi", - 'if [ "$host_restore_failed" -ne 0 ]; then exit 1; fi', - 'if ! rm -f "$host_backup" "$sandbox_backup"; then echo "failed to remove DNS rebinding hosts backups" >&2; exit 1; fi', - 'echo "removed DNS rebinding hosts backups"', - "exit 0", - ].join("\n"), - "dns-rebinding-restore-hosts", - runtime.command, - ...runtime.args, - ], - { - artifactName: "dns-rebinding-restore-hosts", - env: buildAvailabilityProbeEnv(), - timeoutMs: 60_000, - }, - ); - assertHostFixtureProbeSucceeded( - result, - "restore host and sandbox hosts files after DNS rebinding proof", + "-n", + fixture.server.family === 4 ? "iptables" : "ip6tables", + "-w", + "5", + "-t", + "nat", + "-S", + "OUTPUT", + ], + { artifactName: "dns-rebinding-rules-absent", timeoutMs: 30_000 }, + ), + "prove owned DNS rules absent", ); + if (remaining?.exitCode !== 0 || remaining.stdout.includes(fixture.comment)) + failures.push("prove owned DNS rules absent"); + const result = fixture.backupReady + ? await observe( + host.command( + "bash", + [ + "-lc", + [ + "set -euo pipefail", + `backup=${shellQuote(fixture.hostBackupPath)}`, + '[ -s "$backup" ] || { echo "DNS host backup is missing" >&2; exit 1; }', + ' sudo -n tee /etc/hosts < "$backup" >/dev/null', + ' cmp -s "$backup" /etc/hosts', + ' rm -- "$backup"', + ].join("\n"), + ], + { artifactName: "dns-rebinding-restore-hosts", timeoutMs: 30_000 }, + ), + "restore host DNS mapping", + ) + : { exitCode: 0 }; + if (result?.exitCode !== 0) failures.push("restore host DNS mapping"); + try { + await fixture.server.close(); + } catch { + failures.push("close DNS forwarder"); + } + if (result?.exitCode === 0) { + fixture.backupReady = false; + fixture.removeBackup(); + } + fixture.restored = failures.length === 0; + if (failures.length) throw new Error(`DNS fixture cleanup failed: ${failures.join(", ")}`); } diff --git a/test/e2e/live/double-onboard.test.ts b/test/e2e/live/double-onboard.test.ts index 24d3f3c536e..1dbc85b9454 100644 --- a/test/e2e/live/double-onboard.test.ts +++ b/test/e2e/live/double-onboard.test.ts @@ -19,6 +19,11 @@ import { CLI_ENTRYPOINT } from "../fixtures/paths.ts"; import type { TestProgress } from "../fixtures/progress.ts"; import type { RuntimeProviderPrerequisite } from "../fixtures/runtime-provider.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; +import { + INFERENCE_CURL_DIAGNOSTIC_SCRIPT, + INFERENCE_CURL_HTTP_CODE_MARKER, + isExpectedMcpCurlPolicyDenial, +} from "../fixtures/curl-policy-denial.ts"; const REGISTRY_FILE = path.join(os.homedir(), ".nemoclaw", "sandboxes.json"); const SANDBOX_A = process.env.NEMOCLAW_DOUBLE_ONBOARD_SANDBOX_A ?? "e2e-double-a"; @@ -244,7 +249,7 @@ async function cleanupDoubleOnboardResources( async function gatewayRuntimeId(gateway: GatewayClient): Promise { const runtime = await gateway.resolveHostRuntime(); - return runtime?.kind === "container" ? `${runtime.kind}:${runtime.id}` : (runtime?.kind ?? ""); + return runtime ? `${runtime.kind}:${runtime.id}` : ""; } async function prerequisiteOrSkip( @@ -358,6 +363,7 @@ test( ); const fake = await startFakeOpenAiCompatibleServer({ + additionalModels: ["changed-model"], host: "0.0.0.0", port: Number(process.env.NEMOCLAW_FAKE_PORT ?? 0), progress, @@ -576,7 +582,11 @@ test( const denied = siblingInference ? await sandbox.exec( SANDBOX_A, - ["curl", "-sS", "--max-time", "15", "http://host.openshell.internal:11434/v1/models"], + [ + "sh", + "-c", + `${INFERENCE_CURL_DIAGNOSTIC_SCRIPT}; curl -sS --max-time 15 http://host.openshell.internal:11434/v1/models`, + ], { artifactName: "phase-4-sibling-native-endpoint-denial", env: commandEnv(), @@ -584,7 +594,11 @@ test( }, ) : null; - denied && expect(JSON.parse(denied.stdout)).toMatchObject({ error: "policy_denied" }); + denied && + expect( + isExpectedMcpCurlPolicyDenial(denied, INFERENCE_CURL_HTTP_CODE_MARKER), + resultText(denied), + ).toBe(true); siblingInference && progress.phase("verify sibling native endpoint accepts authenticated requests"); @@ -595,7 +609,7 @@ test( [ "sh", "-lc", - `${NATIVE_LOCAL_AUTH_HEADER_SCRIPT}; curl -fsS --max-time 15 http://host.openshell.internal:11434/v1/chat/completions -H "$AUTH_HEADER" -H 'Content-Type: application/json' --data '{"model":"test-model","messages":[{"role":"user","content":"ping"}]}'`, + `${NATIVE_LOCAL_AUTH_HEADER_SCRIPT}; ${INFERENCE_CURL_DIAGNOSTIC_SCRIPT}; curl -fsS --max-time 15 http://host.openshell.internal:11434/v1/chat/completions -H "$AUTH_HEADER" -H 'Content-Type: application/json' --data '{"model":"test-model","messages":[{"role":"user","content":"ping"}]}'`, ], { artifactName: "phase-4-sibling-native-endpoint-authenticated-post", diff --git a/test/e2e/live/external-gateway-health-helpers.ts b/test/e2e/live/external-gateway-health-helpers.ts index 6f301276a4b..2f9f84e1402 100644 --- a/test/e2e/live/external-gateway-health-helpers.ts +++ b/test/e2e/live/external-gateway-health-helpers.ts @@ -11,13 +11,14 @@ import YAML from "yaml"; import { getDockerDriverGatewayLocalTlsBundle } from "../../../dist/lib/onboard/docker-driver-gateway-local-tls"; import type { ArtifactSink } from "../fixtures/artifacts.ts"; import type { CleanupRegistry } from "../fixtures/cleanup.ts"; +import { externalGatewayHealthConfig } from "../fixtures/external-gateway-health-config.ts"; import { externalGatewayHealthProcessStopped, stopExternalGatewayHealthGateway, } from "../fixtures/external-gateway-health-process.ts"; import { exactGatewayRelease, - OPENSHELL_V0116_QUALIFICATION, + OPENSHELL_V017_QUALIFICATION, } from "../fixtures/openshell-v0116-qualification.ts"; import { spawnObservedChild } from "../fixtures/observed-child-process.ts"; import type { TestProgress } from "../fixtures/progress.ts"; @@ -197,7 +198,7 @@ export async function startPreparedExternalTlsGateway({ skip, }: ScenarioFixtures): Promise { const gatewayBin = resolveGatewayBin(); - if (!gatewayBin) skip("openshell-gateway 0.0.116 is required"); + if (!gatewayBin) skip("openshell-gateway 0.1.7 is required"); progress.phase("confirm the exact OpenShell gateway and SDK prerequisites"); const version = await shellProbe.run( @@ -214,7 +215,7 @@ export async function startPreparedExternalTlsGateway({ }, ); requireProbeSuccess(version, "OpenShell gateway version check"); - if (exactGatewayRelease(version.stdout) !== OPENSHELL_V0116_QUALIFICATION.version) { + if (exactGatewayRelease(version.stdout) !== OPENSHELL_V017_QUALIFICATION.version) { throw new Error("The OpenShell gateway release does not match the required release."); } @@ -241,28 +242,7 @@ export async function startPreparedExternalTlsGateway({ ); requireProbeSuccess(certificate, "OpenShell gateway certificate generation"); const configPath = path.join(stateDir, "gateway.toml"); - fs.writeFileSync( - configPath, - [ - "[openshell]", - "version = 1", - "", - "[openshell.gateway]", - `bind_address = "${address}:${String(port)}"`, - "compute_drivers = []", - "disable_tls = false", - "", - "[openshell.gateway.tls]", - `cert_path = ${JSON.stringify(tls.serverCertPath)}`, - `key_path = ${JSON.stringify(tls.serverKeyPath)}`, - "require_client_auth = false", - "", - "[openshell.gateway.auth]", - "allow_unauthenticated_users = true", - "", - ].join("\n"), - { mode: 0o600 }, - ); + fs.writeFileSync(configPath, externalGatewayHealthConfig(address, port, tls), { mode: 0o600 }); progress.phase("launch a TLS gateway without client-certificate authentication"); let gatewayOutput = ""; @@ -298,12 +278,12 @@ export async function startPreparedExternalTlsGateway({ path.join(blueprintRoot, "blueprint.yaml"), YAML.stringify({ version: "1.0.0", - min_openshell_version: OPENSHELL_V0116_QUALIFICATION.version, - max_openshell_version: OPENSHELL_V0116_QUALIFICATION.version, + min_openshell_version: OPENSHELL_V017_QUALIFICATION.version, + max_openshell_version: OPENSHELL_V017_QUALIFICATION.version, openshell_target: { endpoint: `https://${address}:${String(port)}`, workspace: "default", - expected_release: OPENSHELL_V0116_QUALIFICATION.version, + expected_release: OPENSHELL_V017_QUALIFICATION.version, lifecycle: "external", trust: { ca_file: tls.caPath }, authentication: { credential_file: authenticationPath }, @@ -316,7 +296,7 @@ export async function startPreparedExternalTlsGateway({ return { authenticationPath, blueprintRoot, - expectedRelease: OPENSHELL_V0116_QUALIFICATION.version, + expectedRelease: OPENSHELL_V017_QUALIFICATION.version, privateStateRoot: stateDir, }; } diff --git a/test/e2e/live/full-e2e-inference-probe.ts b/test/e2e/live/full-e2e-inference-probe.ts index 55374cdec8b..08e550401dc 100644 --- a/test/e2e/live/full-e2e-inference-probe.ts +++ b/test/e2e/live/full-e2e-inference-probe.ts @@ -1,6 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + normalizeNativeLocalProviderAttachment, + NATIVE_LOCAL_CREDENTIAL_ENV, +} from "../../../src/lib/inference/native-local/contract.ts"; import { parseCliOpenShellProviderMetadata } from "../../../src/lib/adapters/openshell/provider-metadata-cli.ts"; import { hostedNativeProviderForAttachment } from "../../../src/lib/inference/native-provider/hosted-attachment.ts"; import { getSandboxInferenceConfig } from "../../../src/lib/inference/config.ts"; @@ -340,20 +344,33 @@ export async function readFullE2eInferenceRoute(input: { entry: SandboxEntry | null; logicalProvider: string; model: string; + timeoutMs?: number; env: NodeJS.ProcessEnv; redactionValues: string[]; }): Promise<{ ok: boolean; detail: string }> { - const receipt = input.entry?.nativeHostedProviderAttachment; - if (receipt?.custom) { - const definition = hostedNativeProviderForAttachment(receipt); + const gateway = input.entry?.gatewayName || input.env.OPENSHELL_GATEWAY || "nemoclaw"; + const local = normalizeNativeLocalProviderAttachment(input.entry?.nativeLocalProviderAttachment); + if ( + input.entry?.nativeLocalProviderAttachment && + (!local || + local.sandboxName !== input.sandboxName || + local.gatewayName !== gateway || + local.agent !== "openclaw" || + local.provider !== input.logicalProvider || + local.credentialEnv !== NATIVE_LOCAL_CREDENTIAL_ENV) + ) + return { ok: false, detail: "Invalid local native ownership" }; + const hosted = input.entry?.nativeHostedProviderAttachment; + const receipt = local ?? (hosted?.custom ? hosted : undefined); + if (receipt) { + const definition = local ?? (hosted && hostedNativeProviderForAttachment(hosted)); if (!definition) return { ok: false, detail: "Invalid custom native ownership" }; const options = { env: input.env, redactionValues: input.redactionValues, - timeoutMs: 60_000, + timeoutMs: input.timeoutMs ?? 60_000, captureLimitBytes: 16 * 1024, }; - const gateway = input.entry?.gatewayName || input.env.OPENSHELL_GATEWAY || "nemoclaw"; const provider = await input.sandbox.openshell( ["provider", "get", "-g", gateway, receipt.providerName], { ...options, artifactName: "phase-3-native-custom-provider-metadata" }, @@ -367,9 +384,10 @@ export async function readFullE2eInferenceRoute(input: { const route = getSandboxInferenceConfig( input.model, input.logicalProvider, - receipt.custom.api, + local ? (local.api ?? "openai-completions") : hosted?.custom?.api, true, receipt.endpointUrl, + "openclaw", ); // Inspect the selected agent's real configuration inside the sandbox. // Keep credential references out of retained command output. @@ -415,7 +433,7 @@ export async function readFullE2eInferenceRoute(input: { agent.stdout === "selected agent route verified"; return { ok, - detail: `Custom native provider ownership, attachment and selected agent route: ${ok ? "verified" : "mismatch"}`, + detail: `${local ? "Local" : "Custom"} native provider ownership, attachment and selected agent route: ${ok ? "verified" : "mismatch"}`, }; } const native = await readPublicNvidiaSwitchAttachmentEvidence({ @@ -431,7 +449,7 @@ export async function readFullE2eInferenceRoute(input: { const route = await input.sandbox.openshell(["inference", "get"], { artifactName: "phase-3-openshell-inference-get", env: input.env, - timeoutMs: 60_000, + timeoutMs: input.timeoutMs ?? 60_000, }); return { ok: route.exitCode === 0 && route.stdout.includes(input.model), detail: route.stdout }; } diff --git a/test/e2e/live/full-e2e-native-model.ts b/test/e2e/live/full-e2e-native-model.ts index 73f939cc2a9..19d63cf455f 100644 --- a/test/e2e/live/full-e2e-native-model.ts +++ b/test/e2e/live/full-e2e-native-model.ts @@ -2,6 +2,10 @@ // SPDX-License-Identifier: Apache-2.0 import { randomUUID } from "node:crypto"; +import { + nativeLocalCredentialReference, + NATIVE_LOCAL_CREDENTIAL_ENV, +} from "../../../src/lib/inference/native-local/agent-config.ts"; import { NVIDIA_HOSTED_LOGICAL_PROVIDER, NVIDIA_HOSTED_NATIVE_ENDPOINT, @@ -49,7 +53,9 @@ export function buildNativeModelRestartFixture( ? NVIDIA_HOSTED_NATIVE_ENDPOINT : "https://inference.local/v1", apiKey: nativeCustom - ? "${COMPATIBLE_API_KEY}" + ? nativeLocalCredentialReference(logicalProvider, endpointUrl, "openclaw") + ? "${" + NATIVE_LOCAL_CREDENTIAL_ENV + "}" + : "${COMPATIBLE_API_KEY}" : nativeNvidia ? "${NVIDIA_INFERENCE_API_KEY}" : "unused", diff --git a/test/e2e/live/full-e2e.test.ts b/test/e2e/live/full-e2e.test.ts index f5afcd7554c..a08384cdeaf 100644 --- a/test/e2e/live/full-e2e.test.ts +++ b/test/e2e/live/full-e2e.test.ts @@ -571,6 +571,13 @@ sha256sum /sandbox/.bashrc /sandbox/.profile > /tmp/nemoclaw-e2e-profiles.sha256 [SANDBOX_NAME, "gateway", "restart"], "phase-4-restart-after-native-model-restore", ); + await captureNativePluginFailureReadiness( + input.sandbox, + restartAfterNativeModelRestore, + SANDBOX_NAME, + "phase-4-restart-after-native-model-restore", + env(), + ); }, ); const nativeReply = parseOpenClawGatewayModelRun(nativeTurn.stdout); @@ -1139,11 +1146,14 @@ test( }, ); - const nativeEndpointUrl = getSandbox(SANDBOX_NAME)?.nativeHostedProviderAttachment?.endpointUrl; + const selectedEntry = getSandbox(SANDBOX_NAME); + const nativeEndpointUrl = + selectedEntry?.nativeLocalProviderAttachment?.endpointUrl ?? + selectedEntry?.nativeHostedProviderAttachment?.endpointUrl; const inferenceRoute = await readFullE2eInferenceRoute({ sandbox, sandboxName: SANDBOX_NAME, - entry: getSandbox(SANDBOX_NAME), + entry: selectedEntry, logicalProvider: hosted.providerName, model: hosted.model, env: env(), diff --git a/test/e2e/live/gpu-double-onboard.test.ts b/test/e2e/live/gpu-double-onboard.test.ts index 3c094bbc81d..8c307274470 100644 --- a/test/e2e/live/gpu-double-onboard.test.ts +++ b/test/e2e/live/gpu-double-onboard.test.ts @@ -17,6 +17,7 @@ import { resultText } from "../fixtures/clients/command.ts"; import { type HostCliClient } from "../fixtures/clients/host.ts"; import { type SandboxClient, validateSandboxName } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; +import type { LifecyclePhaseFixture } from "../fixtures/phases/lifecycle.ts"; import { nativeLocalChatCommand } from "../fixtures/native-local-chat.ts"; import { CLI_ENTRYPOINT, REPO_ROOT } from "../fixtures/paths.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -61,7 +62,11 @@ async function nemoclaw( }); } -async function preCleanup(host: HostCliClient, sandbox: SandboxClient): Promise { +async function preCleanup( + host: HostCliClient, + sandbox: SandboxClient, + lifecycle: LifecyclePhaseFixture, +): Promise { await nemoclaw(host, [SANDBOX_NAME, "destroy", "--yes"], "pre-cleanup-nemoclaw-destroy").catch( () => undefined, ); @@ -72,13 +77,17 @@ async function preCleanup(host: HostCliClient, sandbox: SandboxClient): Promise< timeoutMs: 60_000, }) .catch(() => undefined); - await sandbox - .openshell(["gateway", "destroy", "-g", "nemoclaw"], { - artifactName: "pre-cleanup-openshell-gateway-destroy", - env: env(), - timeoutMs: 60_000, - }) - .catch(() => undefined); + await lifecycle.stopGatewayRuntime({ env: env(), userServiceMode: "permanent" }); + await cleanupWhenOpenShellAvailable( + host, + { artifactName: "pre-cleanup-probe-openshell-gateway", env: env(), timeoutMs: 30_000 }, + () => + host.cleanupGatewayRegistration("nemoclaw", { + artifactName: "pre-cleanup-openshell-gateway", + env: env(), + timeoutMs: 60_000, + }), + ); await cleanupOllama(host, "pre-cleanup-ollama-processes").catch(() => undefined); } @@ -164,6 +173,7 @@ test( artifacts, cleanup: cleanupRegistry, host, + lifecycle, progress, runtimeProvider, sandbox, @@ -201,26 +211,21 @@ test( const stop = await cleanupOllama(host, "cleanup-ollama-processes"); expect(stop.exitCode, resultText(stop)).toBe(0); }); - const gatewayCleanupOptions = { - artifactName: "cleanup-openshell-gateway-destroy", - env: env(), - timeoutMs: 60_000, - }; - cleanupRegistry.trackGateway( - { - cleanupGatewayRegistration: (name: string) => - cleanupWhenOpenShellAvailable( - host, - { - artifactName: "cleanup-probe-openshell-gateway", - env: gatewayCleanupOptions.env, - timeoutMs: 30_000, - }, - () => host.cleanupGatewayRegistration(name, gatewayCleanupOptions), - ), + cleanupRegistry.trackDisposable( + "stop GPU gateway runtime before removing its state", + async () => { + await lifecycle.stopGatewayRuntime({ env: env(), userServiceMode: "permanent" }); + await cleanupWhenOpenShellAvailable( + host, + { artifactName: "cleanup-probe-openshell-gateway", env: env(), timeoutMs: 30_000 }, + () => + host.cleanupGatewayRegistration("nemoclaw", { + artifactName: "cleanup-openshell-gateway", + env: env(), + timeoutMs: 60_000, + }), + ); }, - "nemoclaw", - gatewayCleanupOptions, ); const openshellSandboxCleanupOptions = { artifactName: "cleanup-openshell-sandbox-delete", @@ -260,7 +265,7 @@ test( SANDBOX_NAME, nemoclawSandboxCleanupOptions, ); - await preCleanup(host, sandbox); + await preCleanup(host, sandbox, lifecycle); progress.phase("install Ollama runtime"); const installOllama = await host.command( diff --git a/test/e2e/live/gpu-e2e-helpers.ts b/test/e2e/live/gpu-e2e-helpers.ts index 562a262b435..d9439e25e6f 100644 --- a/test/e2e/live/gpu-e2e-helpers.ts +++ b/test/e2e/live/gpu-e2e-helpers.ts @@ -17,6 +17,17 @@ import { pollUntil } from "../fixtures/polling.ts"; import type { TestProgress } from "../fixtures/progress.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; import { stripAnsi } from "./json-envelope.ts"; +import { parseCliOpenShellProviderMetadata } from "../../../src/lib/adapters/openshell/provider-metadata-cli.ts"; +import { + normalizeNativeLocalProviderAttachment, + type NativeLocalProviderAttachment, +} from "../../../src/lib/inference/native-local/contract.ts"; +import { + NATIVE_LOCAL_AUTH_HEADER_SCRIPT, + NATIVE_LOCAL_CREDENTIAL_ENV, +} from "../../../src/lib/inference/native-local/agent-config.ts"; +import type { SandboxEntry } from "../../../src/lib/state/registry.ts"; +import { shellQuote } from "../fixtures/clients/command.ts"; import { cleanupOllama, env, SANDBOX_NAME, PROXY_PORT } from "./gpu-e2e-ollama.ts"; export { @@ -64,6 +75,63 @@ export function waitForAttachedOllama( } export const CLI = CLI_ENTRYPOINT; +export async function readGpuOllamaRoute( + sandbox: SandboxClient, + entry: SandboxEntry | null, + environment: NodeJS.ProcessEnv, +): Promise { + const receipt = normalizeNativeLocalProviderAttachment(entry?.nativeLocalProviderAttachment); + const gateway = environment.OPENSHELL_GATEWAY ?? "nemoclaw"; + if ( + !receipt || + entry?.name !== SANDBOX_NAME || + (entry.gatewayName ?? gateway) !== gateway || + (entry.agent ?? "openclaw") !== "openclaw" || + entry.provider !== "ollama-local" || + receipt.provider !== "ollama-local" || + receipt.agent !== "openclaw" || + receipt.sandboxName !== SANDBOX_NAME || + receipt.gatewayName !== gateway || + receipt.authMode !== "authenticated" || + receipt.credentialEnv !== NATIVE_LOCAL_CREDENTIAL_ENV + ) + return null; + const options = { env: environment, timeoutMs: 30_000, captureLimitBytes: 16 * 1024 }; + const provider = await sandbox.openshell( + ["provider", "get", "-g", gateway, receipt.providerName], + { ...options, artifactName: "gpu-ollama-native-provider" }, + ); + const attached = await sandbox.openshell( + ["sandbox", "provider", "list", "-g", gateway, SANDBOX_NAME], + { ...options, artifactName: "gpu-ollama-native-attachment" }, + ); + const metadata = + provider.exitCode === 0 ? parseCliOpenShellProviderMetadata(provider.stdout) : null; + return metadata?.name === receipt.providerName && + metadata.type === receipt.profileId && + metadata.revision?.id === receipt.providerId && + metadata.configKeys.length === 0 && + metadata.credentialKeys.length === 1 && + metadata.credentialKeys[0] === NATIVE_LOCAL_CREDENTIAL_ENV && + attached.exitCode === 0 && + attached.stdout.split(/\s+/u).includes(receipt.providerName) + ? receipt + : null; +} + +export function gpuOllamaChatCommand(route: NativeLocalProviderAttachment, model: string): string { + const payload = JSON.stringify({ + model, + messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], + // Preserve the routed-inference check without consuming a reasoning-token budget. + reasoning_effort: "none", + seed: 0, + temperature: 0, + max_tokens: 32, + }); + return `${NATIVE_LOCAL_AUTH_HEADER_SCRIPT}; curl -sS --max-time 120 ${shellQuote(`${route.endpointUrl}/chat/completions`)} -H 'Content-Type: application/json' -H "$AUTH_HEADER" --data ${shellQuote(payload)}`; +} + function isShellProbeResult(value: unknown): value is ShellProbeResult { return ( typeof value === "object" && @@ -344,19 +412,3 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $token" "http:/ }, ); } - -export async function detectOllamaModel(host: HostCliClient): Promise { - return ( - process.env.NEMOCLAW_MODEL || - ( - await host.command( - "bash", - [ - "-lc", - 'curl -sf http://127.0.0.1:11434/api/tags | python3 -c \'import json,sys; m=json.load(sys.stdin).get("models",[]); print(m[0]["name"] if m else "")\'', - ], - { artifactName: "detect-ollama-model", env: env(), timeoutMs: 30_000 }, - ) - ).stdout.trim() - ); -} diff --git a/test/e2e/live/gpu-e2e.test.ts b/test/e2e/live/gpu-e2e.test.ts index e81a15feedd..e8811a1ffa6 100644 --- a/test/e2e/live/gpu-e2e.test.ts +++ b/test/e2e/live/gpu-e2e.test.ts @@ -4,10 +4,10 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; +import { captureGpuInstallFailure } from "../fixtures/gpu-install-failure.ts"; import { EXPORTED_VLLM_PROFILE_ID } from "../../../src/lib/config/model.ts"; import type { V1Alpha1OllamaProxyService } from "../../../src/lib/config/v1alpha1-export.ts"; import { cleanupLocalModelRuntimes } from "../../../src/lib/inference/local-model-profile/cleanup.ts"; -import { normalizeNativeLocalProviderAttachment } from "../../../src/lib/inference/native-local/contract.ts"; import { HOST_LOCAL_VLLM_CONTAINER_NAME } from "../../../src/lib/inference/serving/vllm-host-local-lifecycle.ts"; import { loadManagedVllmApiKey } from "../../../src/lib/inference/vllm-api-key.ts"; import { getSandbox } from "../../../src/lib/state/registry.ts"; @@ -17,7 +17,6 @@ import { resultText } from "../fixtures/clients/index.ts"; import { trustedSandboxShellScript } from "../fixtures/clients/sandbox.ts"; import { parseConfigExport } from "../fixtures/phases/config-export-validation.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; -import { nativeLocalChatCommand } from "../fixtures/native-local-chat.ts"; import { assertAgentExecutionSucceeded, assertGpuInstallProofs, @@ -27,7 +26,6 @@ import { cleanupGpu, cleanupOllama, createGpuPrivateHome, - detectOllamaModel, ensureOllama, env, hasExactReadyPhase, @@ -36,6 +34,8 @@ import { proxyStatus, REPO_ROOT, readTokenFileChecked, + readGpuOllamaRoute, + gpuOllamaChatCommand, restartProxy, SANDBOX_NAME, startAttachedOllama, @@ -168,6 +168,8 @@ test( "--all", "--filter", `label=openshell.ai/sandbox-name=${SANDBOX_NAME}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", "--format", "{{.Names}}\t{{.State}}\t{{.Status}}", ], @@ -194,12 +196,10 @@ test( expect(retainedSandboxContainer.State).toBe("running"); expect(retainedSandboxContainer.Status).toMatch(/\(healthy\)/i); - // OpenClaw local inference uses a sandbox-owned provider behind the auth proxy, not the - // shared gateway route (#13293). - const nativeEndpoint = normalizeNativeLocalProviderAttachment( - getSandbox(SANDBOX_NAME)?.nativeLocalProviderAttachment, - )?.endpointUrl; - expect(nativeEndpoint).toBe(`http://host.openshell.internal:${PROXY_PORT}/v1`); + const route = await readGpuOllamaRoute(sandbox, getSandbox(SANDBOX_NAME), env()); + expect(route, "native Ollama provider identity and attachment must match").toMatchObject({ + endpointUrl: `http://host.openshell.internal:${PROXY_PORT}/v1`, + }); progress.phase("validate Ollama proxy credential boundary"); const tokenRecord = readTokenFileChecked(ollamaProxyTokenFile()); @@ -242,24 +242,10 @@ test( ).toBe(""); progress.phase("run sandbox native Ollama chat"); - const model = await detectOllamaModel(host); + const model = env().NEMOCLAW_MODEL!; const chat = await sandbox.execShell( SANDBOX_NAME, - trustedSandboxShellScript( - nativeLocalChatCommand( - nativeEndpoint ?? "http://missing-native-local-attachment.invalid/v1", - JSON.stringify({ - model, - messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], - // Keep this assertion about routed inference, not the model's reasoning-token budget. - reasoning_effort: "none", - seed: 0, - temperature: 0, - max_tokens: 32, - }), - "-sS", - ), - ), + trustedSandboxShellScript(gpuOllamaChatCommand(route!, model)), { artifactName: "sandbox-native-ollama-chat", env: env(), timeoutMs: 150_000 }, ); expect(chatContent(chat.stdout)).toMatch(/pong/i); @@ -404,6 +390,7 @@ test( timeoutMs: execTimeout(60 * 60_000), }, ); + await captureGpuInstallFailure(install, sandbox, artifacts, hermesResponseEnv(), SANDBOX_NAME); expect(install.exitCode, resultText(install)).toBe(0); progress.phase("run Hermes initial, resumed, and continued replies"); diff --git a/test/e2e/live/hermes-discord.test.ts b/test/e2e/live/hermes-discord.test.ts index 4619952711c..2482f5c5409 100644 --- a/test/e2e/live/hermes-discord.test.ts +++ b/test/e2e/live/hermes-discord.test.ts @@ -1,6 +1,12 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + NETWORK_POLICY_ERROR_REPORT, + NETWORK_POLICY_RESULT_PREFIX, + isNetworkPolicyDenial, +} from "../support/network-policy-probe.ts"; + import fs from "node:fs"; import { setTimeout as sleep } from "node:timers/promises"; @@ -303,7 +309,7 @@ const request = http.request({ response.setEncoding("utf8"); response.on("data", (chunk) => { body += chunk; }); response.on("end", () => { - console.log("response " + response.statusCode + " " + body.slice(0, 200)); + console.log(${JSON.stringify(NETWORK_POLICY_RESULT_PREFIX)} + JSON.stringify({ status: response.statusCode, response: body.slice(0, 200) })); }); }); request.on("upgrade", () => { @@ -312,7 +318,7 @@ request.on("upgrade", () => { }); request.setTimeout(20000, () => request.destroy(new Error("timeout"))); request.on("error", (error) => { - console.log("error " + error.message); + ${NETWORK_POLICY_ERROR_REPORT} }); request.end(); NODE`, @@ -347,12 +353,12 @@ const request = http.request({ response.setEncoding("utf8"); response.on("data", (chunk) => { body += chunk; }); response.on("end", () => { - console.log("response " + response.statusCode + " " + body.slice(0, 200)); + console.log(${JSON.stringify(NETWORK_POLICY_RESULT_PREFIX)} + JSON.stringify({ status: response.statusCode, response: body.slice(0, 200) })); }); }); request.setTimeout(20000, () => request.destroy(new Error("timeout"))); request.on("error", (error) => { - console.log("error " + error.message); + ${NETWORK_POLICY_ERROR_REPORT} }); request.end(); NODE`, @@ -538,9 +544,11 @@ test( fakeGateway.port, redactionValues, ); - expect(resultText(deniedNodeGateway)).toMatch( - /response 403|policy[_ ]denied|not allowed by any policy/i, - ); + expect( + isNetworkPolicyDenial(deniedNodeGateway) && + !readDiscordRestRequests(fakeGateway.captureFile).some((row) => row.event === "upgrade"), + resultText(deniedNodeGateway), + ).toBe(true); const nativeGateway = await runHermesPythonDiscordGatewayProof( sandbox, @@ -612,9 +620,11 @@ test( fakeRest.port, redactionValues, ); - expect(resultText(deniedNodeRest)).toMatch( - /(?=[\s\S]*TOKEN_PLACEHOLDER true)(?=[\s\S]*(?:response 403|policy[_ ]denied|not allowed by any policy))/iu, - ); + expect( + isNetworkPolicyDenial(deniedNodeRest) && + /^TOKEN_PLACEHOLDER true$/mu.test(deniedNodeRest.stdout), + resultText(deniedNodeRest), + ).toBe(true); expect( readDiscordRestRequests(fakeRest.captureFile), "denied Node REST request changed the fake Discord capture", diff --git a/test/e2e/live/hermes-e2e.test.ts b/test/e2e/live/hermes-e2e.test.ts index da87d1def7b..a06ab5a2e22 100644 --- a/test/e2e/live/hermes-e2e.test.ts +++ b/test/e2e/live/hermes-e2e.test.ts @@ -1,17 +1,24 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + buildNetworkPolicyRequest, + isNetworkPolicyDenial, +} from "../support/network-policy-probe.ts"; + import fs from "node:fs"; import os from "node:os"; import path from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; import type { SandboxEntry } from "../../../src/lib/state/registry/types.ts"; +import { normalizeNativeLocalProviderAttachment } from "../../../src/lib/inference/native-local/contract.ts"; import { recordedNativeProviderAttachment } from "../../../src/lib/inference/native-provider/recorded-selection.ts"; import { buildSandboxInferenceChatCommand } from "../fixtures/inference-adapter.ts"; import { HERMES_E2E_TEST_TIMEOUT_MS } from "../../../tools/e2e/hermes-timeout-contract.mts"; import { execTimeout, testTimeout } from "../../helpers/timeouts.ts"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import { assertStockManagedImageReceipt } from "../fixtures/managed-image-receipt.ts"; +import { cleanupWhenOpenShellAvailable } from "../fixtures/cleanup-resources.ts"; import { resultText, shellQuote } from "../fixtures/clients/command.ts"; import { trustedSandboxShellScript, validateSandboxName } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; @@ -271,12 +278,16 @@ test( timeoutMs: 60_000, }), ); - await preCleanBestEffort(() => - sandbox.openshell(["gateway", "destroy", "-g", "nemoclaw"], { - artifactName: `${label}-openshell-gateway-destroy`, - env: commandEnv(), - timeoutMs: 60_000, - }), + await lifecycle.stopGatewayRuntime({ env: commandEnv(), userServiceMode: "permanent" }); + await cleanupWhenOpenShellAvailable( + host, + { artifactName: `${label}-probe-openshell-gateway`, env: commandEnv(), timeoutMs: 30_000 }, + () => + host.cleanupGatewayRegistration("nemoclaw", { + artifactName: `${label}-openshell-gateway`, + env: commandEnv(), + timeoutMs: 60_000, + }), ); }; @@ -435,17 +446,15 @@ test( const deniedEgress = await sandbox.exec( SANDBOX_NAME, - ["curl", "-fsS", "--connect-timeout", "5", "--max-time", "15", "https://example.com/"], + buildNetworkPolicyRequest("https://example.com/"), { artifactName: "phase-3-unintended-egress-denied", env: commandEnv(), timeoutMs: 30_000, }, ); - expect(deniedEgress.exitCode, resultText(deniedEgress)).not.toBe(0); - expect(resultText(deniedEgress)).toMatch( - /CONNECT tunnel failed, response 403|The requested URL returned error: 403|policy[_ ]denied|not allowed by any policy/i, - ); + expect(deniedEgress.exitCode, resultText(deniedEgress)).toBe(0); + expect(isNetworkPolicyDenial(deniedEgress), resultText(deniedEgress)).toBe(true); // Continue Phase 3 with Hermes health and sandbox state. let health: ShellProbeResult | undefined; @@ -882,7 +891,9 @@ test( expectPong(`${inference.mode} direct chat`, directChat); const chatSelection = registryEntry(SANDBOX_NAME) as unknown as SandboxEntry; - const chatAttachment = recordedNativeProviderAttachment(chatSelection); + const chatAttachment = + normalizeNativeLocalProviderAttachment(chatSelection.nativeLocalProviderAttachment) ?? + recordedNativeProviderAttachment(chatSelection); const sandboxChat = await sandbox.exec( SANDBOX_NAME, buildSandboxInferenceChatCommand( diff --git a/test/e2e/live/hermes-gpu-startup-fallback.ts b/test/e2e/live/hermes-gpu-startup-fallback.ts index 1a8b9cbcebd..8308754399b 100644 --- a/test/e2e/live/hermes-gpu-startup-fallback.ts +++ b/test/e2e/live/hermes-gpu-startup-fallback.ts @@ -62,10 +62,11 @@ export function resolveHermesGpuStartupScenario( export function extractHermesGpuDiagnosticsDirectory(output: string): string { return ( - output.match(/Pre-rollback diagnostics saved:\s*(\S+)/u)?.[1] ?? - output.match(/Docker GPU diagnostics saved:\s*(\S+)/u)?.[1] ?? - output.match(/Native GPU diagnostics saved:\s*(\S+)/u)?.[1] ?? - "" + [ + ...output.matchAll( + /(?:^|\n)\s*(?:(?:Pre-rollback|Docker GPU|Native GPU) diagnostics|Diagnostics) saved:\s*(\S+)/gu, + ), + ].at(-1)?.[1] ?? "" ); } diff --git a/test/e2e/live/hermes-gpu-startup-integrity.ts b/test/e2e/live/hermes-gpu-startup-integrity.ts index 3bb7ddcb490..543574ad99c 100644 --- a/test/e2e/live/hermes-gpu-startup-integrity.ts +++ b/test/e2e/live/hermes-gpu-startup-integrity.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 export interface HermesManagedStartupIntegrityPaths { + pythonPath: string; configPath: string; envPath: string; strictHashPath: string; @@ -12,6 +13,7 @@ export interface HermesManagedStartupIntegrityPaths { } const DEFAULT_PATHS: HermesManagedStartupIntegrityPaths = { + pythonPath: "/opt/hermes/.venv/bin/python3", configPath: "/sandbox/.hermes/config.yaml", envPath: "/sandbox/.hermes/.env", strictHashPath: "/etc/nemoclaw/hermes.config-hash", @@ -34,13 +36,14 @@ export function buildHermesManagedStartupIntegrityScript( ): string { const paths = { ...DEFAULT_PATHS, ...overrides }; return `set -eu -/usr/bin/python3 -I - <<'PY' +${pythonLiteral(paths.pythonPath)} -I - <<'PY' import hashlib import os from pathlib import Path import re import secrets import stat +import yaml config_path = Path(${pythonLiteral(paths.configPath)}) env_path = Path(${pythonLiteral(paths.envPath)}) @@ -137,7 +140,54 @@ compat_digests = parse_hash(compat_hash_bytes, "Hermes compatibility hash") strict_config_digest, strict_env_digest = strict_digests compat_config_digest, compat_env_digest = compat_digests if not secrets.compare_digest(strict_config_digest, digest(config_bytes)): - fail("Hermes config differs from the strict startup base") + original, metadata = read_regular( + config_path.parent / ".native-local-startup-config", "Hermes startup config witness", 4 * 1024 * 1024 + ) + if ( + stat.S_IMODE(metadata.st_mode) != 0o600 + or metadata.st_uid != _config_metadata.st_uid + or metadata.st_gid != _config_metadata.st_gid + or not secrets.compare_digest(strict_config_digest, digest(original)) + ): + fail("Hermes startup config witness does not match the strict anchor") + # Authenticate the original bytes before interpreting any proposed credential changes. + base = yaml.safe_load(original) + current = yaml.safe_load(config_bytes) + try: + provider = base["_nemoclaw_upstream"]["provider"] + if provider not in ("ollama-local", "vllm-local", "llama-cpp-local", "compatible-endpoint"): + fail("Hermes startup config has no managed native provider") + reference = current["model"]["api_key"] + if not isinstance(reference, str) or re.fullmatch( + r"openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_NEMOCLAW_LOCAL_INFERENCE_TOKEN", reference + ) is None: + fail("Hermes startup config has no provider-bound credential") + selected = [entry for entry in base["custom_providers"] if entry.get("name") == provider] + if len(selected) != 1: + fail("Hermes startup config has ambiguous managed providers") + model, managed, custom = base["model"], base["providers"][provider], selected[0] + prior = model["api_key"] + if ( + base["_nemoclaw_upstream"]["provider_key"] != provider + or model["provider"] != "custom" + or managed["api_key"] != prior or custom["api_key"] != prior + or managed["api"] != model["base_url"] or custom["base_url"] != model["base_url"] + or not isinstance(prior, str) + or (prior != "\${NEMOCLAW_LOCAL_INFERENCE_TOKEN}" and re.fullmatch( + r"openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_NEMOCLAW_LOCAL_INFERENCE_TOKEN", prior + ) is None) + ): + fail("Hermes startup credential slots are inconsistent") + base["model"] = {**model, "api_key": reference} + base["providers"] = {**base["providers"], provider: {**managed, "api_key": reference}} + base["custom_providers"] = [ + {**entry, "api_key": reference} if entry is custom else entry + for entry in base["custom_providers"] + ] + if yaml.safe_dump(base, sort_keys=False).encode("utf-8") != config_bytes: + fail("Hermes config differs beyond the managed startup credentials") + except (KeyError, TypeError, AttributeError): + fail("Hermes startup config has invalid managed credential slots") if not secrets.compare_digest(strict_env_digest, digest(base_env_bytes)): fail("Hermes environment differs from the strict startup base beyond the generated API key") if not secrets.compare_digest(compat_config_digest, digest(config_bytes)): diff --git a/test/e2e/live/hermes-gpu-startup-proof.ts b/test/e2e/live/hermes-gpu-startup-proof.ts index 85774ce3a6e..ad21bdea80c 100644 --- a/test/e2e/live/hermes-gpu-startup-proof.ts +++ b/test/e2e/live/hermes-gpu-startup-proof.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { parseOpenShellSandboxId } from "../../../src/lib/adapters/openshell/sandbox-identity.ts"; +import { NATIVE_LOCAL_AUTH_HEADER_SCRIPT } from "../../../src/lib/inference/native-local/agent-config.ts"; import { type ManagedWorkloadAuthority, readManagedWorkloadAuthority, @@ -15,6 +17,7 @@ import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import { type HostCliClient, resultText, + shellQuote, type SandboxClient, trustedSandboxShellScript, } from "../fixtures/clients/index.ts"; @@ -137,6 +140,66 @@ export function assertHermesContainerImageAuthority( ).toBe(true); } +/** The fixture owns this host-alias endpoint; OpenShell supplies its scoped credential. */ +export function buildHermesGpuInferenceCommand(endpointUrl: string, model: string): string { + const payload = JSON.stringify({ + model, + messages: [{ role: "user", content: "reply with OK" }], + max_tokens: 8, + }); + return `${NATIVE_LOCAL_AUTH_HEADER_SCRIPT}; curl -fsS --max-time 60 ${shellQuote(`${endpointUrl}/chat/completions`)} -H 'Content-Type: application/json' -H "$AUTH_HEADER" --data ${shellQuote(payload)}`; +} + +interface HermesRuntimeInspection { + Id: string; + Image?: unknown; + ImageName?: unknown; + Config: { + Cmd?: unknown; + Entrypoint?: unknown; + Env: string[]; + Image?: unknown; + Labels: Record; + }; +} + +/** Main-process metadata belongs to the supervisor; the managed image belongs to its workload. */ +export function hermesGpuRuntimePair( + inspections: readonly HermesRuntimeInspection[], + sandboxName: string, + sandboxId: string | null, +): { workload: HermesRuntimeInspection; supervisor: HermesRuntimeInspection } | null { + const workload = inspections.find( + (row) => row.Config?.Labels?.["openshell.ai/isolation-role"] === "sandbox", + ); + const supervisor = inspections.find( + (row) => row.Config?.Labels?.["openshell.ai/isolation-role"] === "supervisor", + ); + const labels = workload?.Config?.Labels ?? {}; + if ( + inspections.length !== 2 || + !sandboxId || + !workload || + !supervisor || + workload.Id === supervisor.Id || + !labels["openshell.ai/sandbox-namespace"] || + !labels["openshell.ai/sandbox-workspace"] || + supervisor.Config.Labels["openshell.ai/sandbox-workspace"] !== undefined || + inspections.some( + (row) => + !/^[a-f0-9]{64}$/u.test(row.Id) || + !Array.isArray(row.Config?.Env) || + row.Config.Labels["openshell.ai/managed-by"] !== "openshell" || + row.Config.Labels["openshell.ai/sandbox-name"] !== sandboxName || + row.Config.Labels["openshell.ai/sandbox-id"] !== sandboxId || + row.Config.Labels["openshell.ai/sandbox-namespace"] !== + labels["openshell.ai/sandbox-namespace"], + ) + ) + return null; + return { workload, supervisor }; +} + export async function assertHermesGpuStartupProof({ env, gpuRoute, @@ -176,7 +239,7 @@ export async function assertHermesGpuStartupProof({ ); expect(pid1Topology.exitCode, resultText(pid1Topology)).toBe(0); expect(JSON.parse(pid1Topology.stdout)).toEqual({ - argv0: "/opt/openshell/bin/openshell-sandbox", + argv0: "/.openshell/runtime/openshell-sandbox", has_nemoclaw_start: false, }); @@ -184,10 +247,11 @@ export async function assertHermesGpuStartupProof({ [ "container", "ps", + "--no-trunc", "--filter", `label=openshell.ai/sandbox-name=${sandboxName}`, "--format", - "{{.ID}} {{.Names}}", + "{{.ID}}", ], { artifactName: "phase-4-gpu-startup-running-containers", @@ -200,12 +264,7 @@ export async function assertHermesGpuStartupProof({ .split(/\r?\n/) .map((line) => line.trim()) .filter(Boolean); - expect( - containerRows, - `expected one running container, got ${runningContainers.stdout}`, - ).toHaveLength(1); - const [containerId = ""] = containerRows[0].split(/\s+/, 1); - expect(containerId).not.toBe(""); + expect(containerRows.join("\n")).toMatch(/^[a-f0-9]{64}\n[a-f0-9]{64}$/u); const registryEntry = loadSandboxRegistry().sandboxes[sandboxName]; if (!registryEntry) { @@ -278,32 +337,34 @@ export async function assertHermesGpuStartupProof({ expect(startupConfig.exitCode, resultText(startupConfig)).toBe(0); expect(startupConfig.stdout.trim()).toBe("OK"); + // Inspect remains local to this proof: supervisor argv can carry opaque environment values. + // Neither stdout, stderr nor result metadata is persisted by this probe. const runtimeCommandBoundary = await runtimeProvider.command( - ["container", "inspect", containerId], + ["container", "inspect", ...containerRows], { artifactName: "phase-4-gpu-startup-runtime-command-boundary", env: buildAvailabilityProbeEnv(), + persistArtifacts: false, timeoutMs: 30_000, }, ); - expect(runtimeCommandBoundary.exitCode, resultText(runtimeCommandBoundary)).toBe(0); - const runtimeInspection = ( - JSON.parse(runtimeCommandBoundary.stdout) as Array<{ - Image?: unknown; - ImageName?: unknown; - Config?: { - Cmd?: unknown; - Entrypoint?: unknown; - Env?: string[]; - Image?: unknown; - }; - }> - )[0]; - const runtimeConfig = runtimeInspection?.Config ?? {}; + expect(runtimeCommandBoundary.exitCode, "runtime pair inspection must succeed").toBe(0); + const pair = hermesGpuRuntimePair( + JSON.parse(runtimeCommandBoundary.stdout), + sandboxName, + parseOpenShellSandboxId(resultText(openshellState)), + ); + expect( + pair, + "expected the exact workload and supervisor for the selected sandbox", + ).not.toBeNull(); + const { workload: runtimeInspection, supervisor } = pair!; + const containerId = runtimeInspection.Id; + const runtimeConfig = runtimeInspection.Config; const runtimeEnvironment = Object.fromEntries( - (runtimeConfig.Env ?? []) - .filter((entry) => entry.includes("=")) - .map((entry) => entry.split(/=(.*)/su).slice(0, 2) as [string, string]), + supervisor.Config.Env.filter((entry) => entry.includes("=")).map( + (entry) => entry.split(/=(.*)/su).slice(0, 2) as [string, string], + ), ); const intendedTokens = hermesRuntimeIntendedCommand(runtimeEnvironment); const commandBoundary = { @@ -320,7 +381,11 @@ export async function assertHermesGpuStartupProof({ ["nemoclaw-start", "/usr/local/bin/nemoclaw-start"].includes(intendedTokens.at(-1) ?? ""), }; expect(commandBoundary.has_openshell_sandbox_command).toBe(true); - expect(runtimeEnvironment.NEMOCLAW_MANAGED_BOOTSTRAP_IDENTITY).toBeUndefined(); + expect( + [...(runtimeConfig.Env ?? []), ...supervisor.Config.Env].filter((entry) => + entry.startsWith("NEMOCLAW_MANAGED_BOOTSTRAP_IDENTITY="), + ).length, + ).toBe(0); assertHermesContainerImageAuthority( commandBoundary.image, managedImageReference, @@ -347,6 +412,8 @@ export async function assertHermesGpuStartupProof({ "--all", "--filter", `label=openshell.ai/sandbox-name=${sandboxName}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", "--format", "{{.Names}}", ], diff --git a/test/e2e/live/hermes-gpu-startup.test.ts b/test/e2e/live/hermes-gpu-startup.test.ts index 99c6a93e5a7..093821b643e 100644 --- a/test/e2e/live/hermes-gpu-startup.test.ts +++ b/test/e2e/live/hermes-gpu-startup.test.ts @@ -4,6 +4,7 @@ import fs from "node:fs"; import path from "node:path"; +import { getNemoclawStateRoot } from "../../../src/lib/state/state-root.ts"; import { resolveSandboxLaunchForwardPorts } from "../../../src/lib/actions/sandbox/process-recovery"; import { createCliOpenShellForwardAdapter } from "../../../src/lib/adapters/openshell/forward-cli"; import { execTimeout, testTimeout } from "../../helpers/timeouts.ts"; @@ -31,6 +32,7 @@ import { } from "./hermes-gpu-startup-fallback.ts"; import { assertHermesGpuStartupProof, + buildHermesGpuInferenceCommand, HERMES_GPU_FALLBACK_DISCLOSURE_FRAGMENTS, } from "./hermes-gpu-startup-proof.ts"; @@ -51,6 +53,8 @@ const { route: GPU_ROUTE, scenario: GPU_STARTUP_SCENARIO } = resolveHermesGpuSta process.env.E2E_HERMES_GPU_STARTUP_SCENARIO, process.env.NEMOCLAW_DOCKER_GPU_PATCH === "1", ); +import { captureGpuInstallFailure } from "../fixtures/gpu-install-failure.ts"; + const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? (GPU_STARTUP_SCENARIO === "fallback" @@ -271,27 +275,35 @@ async function captureFailedGpuContainer( const script = String.raw`set -eu sandbox_filter="$1" diagnostics_dir="$2" -shift 2 +diagnostics_root="$3" +shift 3 runtime_command=("$@") -if [ -n "$diagnostics_dir" ] && [ -d "$diagnostics_dir" ]; then - printf '%s\n' "== pre-rollback diagnostics $diagnostics_dir ==" - for name in summary.txt patched-container-state.json docker-inspect.json docker-network-summary.txt docker-top.txt docker-logs.txt openshell-sandbox-get.txt openshell-sandbox-list.txt openshell-logs.txt; do - file="$diagnostics_dir/$name" - if [ -f "$file" ]; then - printf '%s\n' "available: $name" - printf '%s\n' "== redacted $name ==" - tail -c 8000 "$file" || true - printf '\n' - fi - done -else - printf '%s\n' "pre-rollback diagnostics directory unavailable: $diagnostics_dir" -fi +python3 -I - "$diagnostics_dir" "$diagnostics_root" <<'PYDIAGNOSTICS' +import os, pathlib, stat, sys +directory = pathlib.Path(sys.argv[1]) +root = pathlib.Path(sys.argv[2]).resolve() +names = "summary.txt patched-container-state.json docker-inspect.json docker-network-summary.txt docker-top.txt docker-logs.txt openshell-sandbox-get.txt openshell-sandbox-list.txt openshell-logs.txt openshell-gateway-relevant.log openshell-gateway-tail.log sandbox-container.log".split() +try: + info = directory.lstat() + if directory.parent.resolve() != root or stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode) or info.st_uid != os.getuid() or info.st_mode & 0o077: + raise ValueError("untrusted diagnostics directory") + for name in names: + file = directory / name + try: + fd = os.open(file, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) + except OSError: + continue + with os.fdopen(fd, "rb") as stream: + meta = os.fstat(stream.fileno()) + if not stat.S_ISREG(meta.st_mode) or meta.st_uid != os.getuid() or meta.st_nlink != 1 or meta.st_mode & 0o077: + continue + stream.seek(max(0, meta.st_size - 8000)) + print("== redacted " + name + " ==") + print(stream.read(8000).decode("utf-8", "replace")) +except (OSError, ValueError): + print("pre-rollback diagnostics directory unavailable or untrusted") +PYDIAGNOSTICS ids="$("${"${runtime_command[@]}"}" container ps --all --quiet --no-trunc --filter "$sandbox_filter")" -if [ -z "$ids" ]; then - printf '%s\n' "no runtime container found for $sandbox_filter" - exit 0 -fi for id in $ids; do if [[ ! "$id" =~ ^[a-f0-9]{64}$ ]]; then printf '%s\n' "ignored non-container identity" @@ -299,6 +311,41 @@ for id in $ids; do fi printf '%s\n' "== exact container $id state ==" "${"${runtime_command[@]}"}" container inspect --format '{{.State.Status}} {{.State.ExitCode}} {{.State.OOMKilled}} {{.RestartCount}}' "$id" 2>&1 || true + role="$("${"${runtime_command[@]}"}" container inspect --format '{{index .Config.Labels "openshell.ai/isolation-role"}}' "$id" 2>/dev/null)" || continue + [ "$role" = sandbox ] || continue + # Capture numeric guard inputs after startup fails; argv and environment can contain credentials. + "${"${runtime_command[@]}"}" exec -i "$id" /usr/bin/python3 -I - <<'PYIDENTITY' || true +import itertools, json, os, re, stat +fields = {"Uid", "Gid", "Groups", "PPid", "NSpid", "NoNewPrivs", "CapInh", "CapPrm", "CapEff", "CapBnd", "CapAmb"} +try: + metadata = os.stat("/proc/1/status", follow_symlinks=False) + with open("/proc/1/status", "r", encoding="ascii") as stream: + status = stream.read(8193) + if len(status) > 8192: + raise ValueError("oversized status") + observed = {} + for line in status.splitlines(): + key, separator, value = line.partition(":") + if separator and key in fields and re.fullmatch(r"[0-9a-fA-F \t]*", value): + observed[key] = value.split() + print(json.dumps({"pid1_status_owner": [metadata.st_uid, metadata.st_gid], "pid1_status": observed})) +except (OSError, UnicodeError, ValueError): + print("PID 1 identity diagnostics unavailable") +try: + with os.scandir("/dev/dri") as entries: + recognized = filter(lambda entry: re.fullmatch(r"(?:card|renderD)[0-9]+", entry.name), + itertools.islice(entries, 256)) + device_paths = {entry.name: entry.path for entry in recognized} + paths = {"/dev": "/dev", "/dev/dri": "/dev/dri", **device_paths} + devices = {} + for name, path in paths.items(): + info = os.stat(path, follow_symlinks=False) + devices[name] = {"uid": info.st_uid, "gid": info.st_gid, "mode": info.st_mode, + "device": [os.major(info.st_rdev), os.minor(info.st_rdev)] if stat.S_ISCHR(info.st_mode) else None} + print(json.dumps({"drm_devices": devices})) +except (OSError, ValueError): + print("DRM device diagnostics unavailable") +PYIDENTITY done`; await captureDiagnosticsBestEffort(() => host.command( @@ -309,6 +356,7 @@ done`; "hermes-gpu-failure-diagnostics", sandboxFilter, preRollbackDiagnosticsDir, + path.join(getNemoclawStateRoot(), "onboard-failures"), runtimeInvocation.command, ...runtimeInvocation.args, ], @@ -470,6 +518,7 @@ test( redactionValues: [FAKE_API_KEY], timeoutMs: execTimeout(60 * 60_000), }); + await captureGpuInstallFailure(install, sandbox, artifacts, env, SANDBOX_NAME); const gpuDiagnosticsDir = extractHermesGpuDiagnosticsDirectory(resultText(install)); await (install.exitCode !== 0 ? captureFailedGpuContainer(host, runtimeProvider, gpuDiagnosticsDir) @@ -552,15 +601,7 @@ test( progress.phase("exercise authenticated GPU inference route"); const inference = await sandbox.execShell( SANDBOX_NAME, - trustedSandboxShellScript( - `curl -fsS --max-time 60 https://inference.local/v1/chat/completions -H 'Content-Type: application/json' --data '${JSON.stringify( - { - model: FAKE_MODEL, - messages: [{ role: "user", content: "reply with OK" }], - max_tokens: 8, - }, - )}'`, - ), + trustedSandboxShellScript(buildHermesGpuInferenceCommand(fake.baseUrl, FAKE_MODEL)), { artifactName: "phase-5-authenticated-inference-post", env: commandEnv(), diff --git a/test/e2e/live/hermes-inference-switch-helpers.ts b/test/e2e/live/hermes-inference-switch-helpers.ts index 9c4ca940ecb..96276932fb0 100644 --- a/test/e2e/live/hermes-inference-switch-helpers.ts +++ b/test/e2e/live/hermes-inference-switch-helpers.ts @@ -11,6 +11,10 @@ import { DEFAULT_CLOUD_MODEL, resolveAgentInferenceApi, } from "../../../src/lib/inference/config.ts"; +import { requireHostedProviderAttachment } from "../../../src/lib/inference/native-provider/hosted-attachment.ts"; +import { nativeHostedAuthHeaderScript } from "../../../src/lib/inference/native-provider/contract.ts"; +import { parseCliOpenShellProviderMetadata } from "../../../src/lib/adapters/openshell/provider-metadata-cli.ts"; +import { shellQuote } from "../../../src/lib/core/shell-quote.ts"; import { NVIDIA_HOSTED_NATIVE_ENDPOINT } from "../../../src/lib/inference/native-nvidia/index.ts"; import { NATIVE_NVIDIA_AUTH_HEADER_SCRIPT } from "../../../src/lib/inference/native-nvidia/contract.ts"; import { execTimeout } from "../../helpers/timeouts.ts"; @@ -26,11 +30,10 @@ import { } from "../fixtures/clients/sandbox.ts"; import { type CompatibleAnthropicSwitchBinding, - compatibleAnthropicMockEndpointUrl, compatibleAnthropicSwitchBinding, compatibleAnthropicSwitchEnv, - requireCompatibleAnthropicProviderAbsent, } from "../fixtures/compatible-anthropic-switch.ts"; +import { discoverHostAddress } from "../fixtures/host-address.ts"; import { expect } from "../fixtures/e2e-test.ts"; import type { FakeOpenAiCompatibleServer } from "../fixtures/fake-openai-compatible.ts"; import { @@ -78,8 +81,9 @@ export const SWITCH_API = process.env.NEMOCLAW_SWITCH_INFERENCE_API ?? "openai-c export const RUNTIME_SWITCH_API = resolveAgentInferenceApi("hermes", SWITCH_PROVIDER, SWITCH_API) ?? SWITCH_API; const SWITCH_MOCK_PORT = Number.parseInt(process.env.NEMOCLAW_SWITCH_MOCK_PORT ?? "0", 10); +const SWITCH_NATIVE_CUSTOM = process.env.NEMOCLAW_SWITCH_NATIVE_CUSTOM === "1"; const INSTALL_ATTEMPTS = process.env.CI === "true" || process.env.GITHUB_ACTIONS === "true" ? 3 : 1; -export const PROXY_RESOLUTION_PROVIDER = "hermes-proxy-resolution-e2e"; +export const PROXY_RESOLUTION_PROVIDER = "compatible-endpoint"; export const PROXY_RESOLUTION_MODEL = "nvidia/nemotron-proxy-resolution-e2e"; export const PROXY_FORBIDDEN_MARKERS = [ "openshell:resolve:env:", @@ -218,26 +222,22 @@ export function env(apiKey?: string, extra: NodeJS.ProcessEnv = {}): NodeJS.Proc return { ...out, ...extra }; } -export async function expectOpenAiProvider( - host: HostCliClient, - providerName: string, - credentialEnv: string, -): Promise { +export async function expectOpenAiProvider(host: HostCliClient, value: unknown): Promise { + const receipt = requireHostedProviderAttachment(value, "compatible-anthropic-endpoint"); const provider = await host.command( "openshell", - ["provider", "get", "-g", "nemoclaw", providerName], + ["provider", "get", "-g", "nemoclaw", receipt?.providerName ?? ""], { - artifactName: `${providerName}-openai-provider-metadata`, + artifactName: "compatible-anthropic-native-provider-metadata", env: env(), timeoutMs: 30_000, }, ); - const output = resultText(provider); - expect(provider.exitCode, output).toBe(0); - const plain = stripAnsi(output); - expect(plain).toMatch(/^\s*Type:\s*openai\s*$/imu); - expect(plain).toContain(credentialEnv); - expect(plain).toContain("OPENAI_BASE_URL"); + expect(provider.exitCode, resultText(provider)).toBe(0); + const metadata = parseCliOpenShellProviderMetadata(provider.stdout); + expect(metadata?.type).toBe(receipt?.profileId); + expect(metadata?.credentialKeys).toEqual(["COMPATIBLE_ANTHROPIC_API_KEY"]); + expect(metadata?.configKeys).toEqual([]); } export async function prepareProxyResolutionRoute({ @@ -259,46 +259,29 @@ export async function prepareProxyResolutionRoute({ const model = mockBaseline ? PROXY_RESOLUTION_MODEL : SWITCH_MODEL; const requestOffset = mockBaseline?.requests().length ?? 0; - const registered = await host.command( - "openshell", - [ - "provider", - "create", - "-g", - "nemoclaw", - "--name", - PROXY_RESOLUTION_PROVIDER, - "--type", - "openai", - "--credential", - "NVIDIA_INFERENCE_API_KEY", - "--config", - `OPENAI_BASE_URL=${endpoint}`, - ], - { - artifactName: "register-proxy-resolution-provider", - env: env(undefined, { NVIDIA_INFERENCE_API_KEY: apiKey }), - redactionValues, - timeoutMs: 120_000, - }, - ); - expect(registered.exitCode, resultText(registered)).toBe(0); - const setRoute = await host.command( "node", [ CLI, "inference", "set", + "--sandbox", + SANDBOX_NAME, "--provider", PROXY_RESOLUTION_PROVIDER, "--model", model, + "--endpoint-url", + endpoint, + "--credential-env", + "COMPATIBLE_API_KEY", + "--inference-api", + "openai-completions", "--no-verify", ], { artifactName: "set-proxy-resolution-route", - env: env(), + env: env(undefined, { COMPATIBLE_API_KEY: apiKey }), redactionValues, timeoutMs: 180_000, }, @@ -563,7 +546,11 @@ function openAiSseResponse(res: http.ServerResponse, chunks: unknown[]): void { ); } -async function startMockAnthropicProvider(): Promise { +async function startMockAnthropicProvider( + host: HostCliClient, +): Promise { + const trustedPrivateHost = (await discoverHostAddress(host, "native-anthropic-endpoint-address")) + .address; const server = http.createServer((req, res) => { const url = new URL(req.url ?? "/", "http://mock.local"); if (req.method === "GET" && url.pathname === "/health") @@ -686,7 +673,7 @@ async function startMockAnthropicProvider(): Promise closeServer(server), }; } @@ -697,15 +684,11 @@ export async function prepareCompatibleAnthropicSwitchBinding( ): Promise { if (SWITCH_PROVIDER !== "compatible-anthropic-endpoint" || SWITCH_API !== "anthropic-messages") return null; - const mock = mockAnthropicSwitchEnabled() ? await startMockAnthropicProvider() : undefined; + const mock = mockAnthropicSwitchEnabled() ? await startMockAnthropicProvider(host) : undefined; mock && cleanup.add("close compatible Anthropic switch mock", () => mock.close()); const binding = compatibleAnthropicSwitchBinding( process.env.NEMOCLAW_SWITCH_ENDPOINT_URL ?? mock?.endpointUrl ?? "", ); - await requireCompatibleAnthropicProviderAbsent(host, { - artifactName: "compatible-anthropic-provider-absent-before-switch", - env: env(), - }); return { ...binding, endpointUrl: openAiSurfaceEndpointUrl(binding.endpointUrl) }; } @@ -775,6 +758,13 @@ export async function runHermesInferenceSetWithRetry( artifactName: `${options.artifactNamePrefix ?? "hermes-inference-set"}-${attempt}`, env: env(undefined, { ...compatibleAnthropicSwitchEnv(options.compatibleBinding ?? null), + ...(SWITCH_NATIVE_CUSTOM && options.compatibleBinding + ? { + NEMOCLAW_TRUSTED_PRIVATE_INFERENCE_HOSTS: new URL( + options.compatibleBinding.endpointUrl, + ).hostname, + } + : {}), ...(options.publicNvidiaApiKey ? { NVIDIA_INFERENCE_API_KEY: options.publicNvidiaApiKey } : {}), @@ -816,13 +806,6 @@ export function parseHermesGatewayPid( return result.exitCode === 0 && /^[1-9][0-9]*$/u.test(pid) ? pid : undefined; } -export function expectedBaseUrl(): string { - if (SWITCH_PROVIDER === PUBLIC_NVIDIA_SWITCH_PROVIDER) return NVIDIA_HOSTED_NATIVE_ENDPOINT; - return RUNTIME_SWITCH_API === "anthropic-messages" - ? "https://inference.local" - : "https://inference.local/v1"; -} - export function inferenceLocalMaxTokens(api: string = RUNTIME_SWITCH_API): number { return api === "anthropic-messages" ? 32 : 100; } @@ -834,18 +817,12 @@ export function expectedApiMode(): string | undefined { ]).get(RUNTIME_SWITCH_API); } -// This live lane runs on ubuntu-latest and intentionally uses GNU grep's -// POSIX ERE character classes; support tests pin the accepted scalar shapes. -export const API_KEY_SHAPE_PATTERN = `^[[:space:]]*api_key:[[:space:]]*("sk-[^"[:space:]]+"|'sk-[^'[:space:]]+'|sk-[^"'[:space:]]+)[[:space:]]*$`; -const NVIDIA_REFERENCE = - "openshell:resolve:env:(v[0-9]{1,20}|s[a-f0-9]{64})_NVIDIA_INFERENCE_API_KEY"; -export const NVIDIA_API_KEY_SHAPE_PATTERN = `^[[:space:]]*api_key:[[:space:]]*("${NVIDIA_REFERENCE}"|'${NVIDIA_REFERENCE}'|${NVIDIA_REFERENCE})[[:space:]]*$`; - export function apiKeyShapeCommand(provider: string = SWITCH_PROVIDER): string[] { - const pattern = + const reference = provider === PUBLIC_NVIDIA_SWITCH_PROVIDER - ? NVIDIA_API_KEY_SHAPE_PATTERN - : API_KEY_SHAPE_PATTERN; + ? "openshell:resolve:env:(v[0-9]{1,20}|s[a-f0-9]{64})_NVIDIA_INFERENCE_API_KEY" + : "\\$\\{COMPATIBLE_ANTHROPIC_API_KEY\\}"; + const pattern = `^[[:space:]]*api_key:[[:space:]]*("${reference}"|'${reference}'|${reference})[[:space:]]*$`; return ["grep", "-Eq", pattern, "/sandbox/.hermes/config.yaml"]; } @@ -869,11 +846,16 @@ export async function hashCheck( ); } -export async function strictHashPerms(sandbox: SandboxClient): Promise { +export async function strictHashState( + sandbox: SandboxClient, + artifact: string, +): Promise { return await sandbox.execShell( SANDBOX_NAME, - trustedSandboxShellScript("stat -c '%u %a' /etc/nemoclaw/hermes.config-hash"), - { artifactName: "hermes-strict-hash-perms", env: env(), timeoutMs: 30_000 }, + trustedSandboxShellScript( + "set -eu; sha256sum /etc/nemoclaw/hermes.config-hash; stat -c '%u %a' /etc/nemoclaw/hermes.config-hash", + ), + { artifactName: `hermes-strict-hash-${artifact}`, env: env(), timeoutMs: 30_000 }, ); } @@ -902,16 +884,20 @@ function quotePayload(payload: string): string { return payload.replace(/'/gu, `'\\''`); } -export function sandboxInferenceCommand(payload: string): string { - if (SWITCH_PROVIDER === PUBLIC_NVIDIA_SWITCH_PROVIDER) { - // The upstream URL is the managed route for an attached OpenShell - // provider. OpenShell authorizes this profile-scoped request and replaces - // the placeholder without exposing the provider credential to the sandbox. - return `${NATIVE_NVIDIA_AUTH_HEADER_SCRIPT}; curl -sS --fail-with-body --write-out '%{stderr}\\n__NEMOCLAW_HTTP_STATUS__=%{http_code}\\n' --max-time 90 ${NVIDIA_HOSTED_NATIVE_ENDPOINT}/chat/completions -H 'Content-Type: application/json' -H "$AUTH_HEADER" -d '${quotePayload(payload)}'`; - } - return RUNTIME_SWITCH_API === "anthropic-messages" - ? `curl -sS --fail-with-body --write-out '%{stderr}\\n__NEMOCLAW_HTTP_STATUS__=%{http_code}\\n' --max-time 90 https://inference.local/v1/messages -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01' -d '${quotePayload(payload)}'` - : `curl -sS --fail-with-body --write-out '%{stderr}\\n__NEMOCLAW_HTTP_STATUS__=%{http_code}\\n' --max-time 90 https://inference.local/v1/chat/completions -H 'Content-Type: application/json' -d '${quotePayload(payload)}'`; +export function sandboxInferenceCommand( + payload: string, + endpointUrl: string, + provider = SWITCH_PROVIDER, +): string { + const headers = + provider === PUBLIC_NVIDIA_SWITCH_PROVIDER + ? NATIVE_NVIDIA_AUTH_HEADER_SCRIPT + : nativeHostedAuthHeaderScript({ + credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", + api: RUNTIME_SWITCH_API, + }); + const target = `${openAiSurfaceEndpointUrl(endpointUrl)}/chat/completions`; + return `${headers}; curl -sS --fail-with-body --write-out '%{stderr}\\n__NEMOCLAW_HTTP_STATUS__=%{http_code}\\n' --max-time 90 ${shellQuote(target)} -H 'Content-Type: application/json' -H "$AUTH_HEADER" -d '${quotePayload(payload)}'`; } export function hermesApiCommand(payload: string): string { diff --git a/test/e2e/live/hermes-inference-switch.test.ts b/test/e2e/live/hermes-inference-switch.test.ts index 562d4f4d586..37795d1ccdb 100644 --- a/test/e2e/live/hermes-inference-switch.test.ts +++ b/test/e2e/live/hermes-inference-switch.test.ts @@ -1,8 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { NVIDIA_HOSTED_NATIVE_ENDPOINT } from "../../../src/lib/inference/native-nvidia/index.ts"; import { testTimeout } from "../../helpers/timeouts.ts"; -import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import { resultText } from "../fixtures/clients/index.ts"; import { trustedSandboxShellScript } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; @@ -44,7 +44,7 @@ import { SWITCH_API, SWITCH_MODEL, SWITCH_PROVIDER, - strictHashPerms, + strictHashState, useLocalHermesInferenceBaseline, } from "./hermes-inference-switch-helpers.ts"; import { @@ -187,6 +187,7 @@ test( provider: mockBaseline ? "compatible-endpoint" : PUBLIC_NVIDIA_SWITCH_PROVIDER, model: hostedInstallModel(env(apiKey, installEnv)), }); + const strictHashBefore = await strictHashState(sandbox, "before-switch"); progress.phase("prove native local Hermes startup and model switch"); mockBaseline && (await expectNativeLocalHermesTurn({ @@ -261,8 +262,7 @@ test( switchBinding && (await expectOpenAiProvider( host, - "compatible-anthropic-endpoint", - "COMPATIBLE_ANTHROPIC_API_KEY", + registryState().registry.sandboxes?.[SANDBOX_NAME]?.nativeHostedProviderAttachment, )); progress.phase("validate switched route and mutable config"); @@ -282,7 +282,7 @@ test( }, ); expect(health.exitCode, resultText(health)).toBe(0); - expect(resultText(health)).toMatch(/ok/i); + expect(JSON.parse(health.stdout).status).toBe("ok"); const route = await host.command("node", [CLI, "inference", "get"], { artifactName: "nemoclaw-inference-route", @@ -330,13 +330,16 @@ test( model: SWITCH_MODEL, }); - const strictHash = await hashCheck(sandbox, "/etc/nemoclaw/hermes.config-hash", "strict"); - expect(strictHash.exitCode, resultText(strictHash)).toBe(0); + // Native config writes update the mutable hash. The root-owned image anchor stays unchanged. + const strictHashAfter = await strictHashState(sandbox, "after-switch"); + expect(strictHashBefore.exitCode, resultText(strictHashBefore)).toBe(0); + expect(strictHashAfter.exitCode, resultText(strictHashAfter)).toBe(0); + expect(strictHashAfter.stdout).toBe(strictHashBefore.stdout); const compatHash = await hashCheck(sandbox, "/sandbox/.hermes/.config-hash", "compat"); expect(compatHash.exitCode, resultText(compatHash)).toBe(0); - const strictPerms = await strictHashPerms(sandbox); - expect(strictPerms.stdout.trim()).toMatch(/^0\s+[0-7]+$/u); - expect(Number.parseInt(strictPerms.stdout.trim().split(/\s+/u)[1], 8) & 0o222).toBe(0); + const strictPerms = strictHashAfter.stdout.trim().split("\n")[1] ?? ""; + expect(strictPerms).toMatch(/^0\s+[0-7]+$/u); + expect(Number.parseInt(strictPerms.split(/\s+/u)[1], 8) & 0o222).toBe(0); maybeAssertEnvHashStable( envHashBefore, @@ -390,10 +393,7 @@ test( canonicalEndpoint(durableEndpointUrl), ); expect(state.registry.sandboxes?.[SANDBOX_NAME]?.credentialEnv).toBe(durableCredentialEnv); - expect(state.registry.sandboxes?.[SANDBOX_NAME]?.preferredInferenceApi).toBe( - publicSwitch ? null : RUNTIME_SWITCH_API, - ); - expect(state.registry.sandboxes?.[SANDBOX_NAME]?.nimContainer).toBeNull(); + // API metadata persistence is covered by inference-set-compatible-provider tests. progress.phase("exercise sandbox inference and Hermes API"); const inferenceLocalPayload = JSON.stringify({ model: SWITCH_MODEL, @@ -408,7 +408,12 @@ test( run: (attempt) => sandbox.execShell( SANDBOX_NAME, - trustedSandboxShellScript(sandboxInferenceCommand(inferenceLocalPayload)), + trustedSandboxShellScript( + sandboxInferenceCommand( + inferenceLocalPayload, + switchEndpointUrl ?? NVIDIA_HOSTED_NATIVE_ENDPOINT, + ), + ), { artifactName: `hermes-sandbox-inference-chat-after-switch-${attempt}`, env: env(), @@ -493,12 +498,9 @@ test( }, ); expect(reapplied.exitCode, resultText(reapplied)).toBe(0); - const strictHashAfterCli = await hashCheck( - sandbox, - "/etc/nemoclaw/hermes.config-hash", - "strict-after-cli", - ); + const strictHashAfterCli = await strictHashState(sandbox, "after-cli"); expect(strictHashAfterCli.exitCode, resultText(strictHashAfterCli)).toBe(0); + expect(strictHashAfterCli.stdout).toBe(strictHashBefore.stdout); const compatHashAfterCli = await hashCheck( sandbox, "/sandbox/.hermes/.config-hash", @@ -513,7 +515,12 @@ test( run: (attempt) => sandbox.execShell( SANDBOX_NAME, - trustedSandboxShellScript(sandboxInferenceCommand(inferenceLocalPayload)), + trustedSandboxShellScript( + sandboxInferenceCommand( + inferenceLocalPayload, + switchEndpointUrl ?? NVIDIA_HOSTED_NATIVE_ENDPOINT, + ), + ), { artifactName: `hermes-sandbox-inference-after-cli-${attempt}`, env: env(), @@ -537,12 +544,16 @@ test( mockBaseline, redactionValues, }); - const persistedProxyRoute = await sandbox.openshell(["inference", "get", "-g", "nemoclaw"], { - artifactName: "proxy-resolution-route-after-set", - env: env(), - redactionValues, - timeoutMs: 30_000, - }); + const persistedProxyRoute = await host.command( + "node", + [CLI, SANDBOX_NAME, "inference", "get"], + { + artifactName: "proxy-resolution-route-after-set", + env: env(), + redactionValues, + timeoutMs: 30_000, + }, + ); expect(persistedProxyRoute.exitCode, resultText(persistedProxyRoute)).toBe(0); expect(parseInferenceRoute(persistedProxyRoute.stdout)).toEqual({ provider: PROXY_RESOLUTION_PROVIDER, diff --git a/test/e2e/live/hermes-sandbox-secret-boundary.test.ts b/test/e2e/live/hermes-sandbox-secret-boundary.test.ts index 8a90245a9b3..a8ff052ecf0 100644 --- a/test/e2e/live/hermes-sandbox-secret-boundary.test.ts +++ b/test/e2e/live/hermes-sandbox-secret-boundary.test.ts @@ -28,7 +28,6 @@ from pathlib import Path secret_key_re = re.compile(r"(^|_)(TOKEN|KEY|SECRET|PASSWORD|CREDENTIAL|API)(_|$)") slack_alias_re = re.compile(r"^(xoxb|xapp)-OPENSHELL-RESOLVE-ENV-[A-Z0-9_]+$") allowed_nonsecret_keys = {"API_SERVER_HOST", "API_SERVER_PORT"} -allowed_raw_secret_keys = set() allowed_literals = {"", "[STRIPPED_BY_MIGRATION]"} required_remote_toolsets = { "web", @@ -70,8 +69,6 @@ def env_violations(path: Path) -> list[str]: continue if key in allowed_nonsecret_keys: continue - if key in allowed_raw_secret_keys: - continue if not secret_key_re.search(key): continue value = unquote(value) @@ -139,9 +136,6 @@ if violations: toolsets = parse_platform_toolsets(config_path.read_text(encoding="utf-8")) api_server_toolsets = set(toolsets.get("api_server", [])) -if not api_server_toolsets: - print("platform_toolsets.api_server missing", file=sys.stderr) - sys.exit(1) missing = sorted(required_remote_toolsets - api_server_toolsets) if missing: print(f"platform_toolsets.api_server missing expected Hermes toolsets: {missing}", file=sys.stderr) diff --git a/test/e2e/live/https-pin-compatible-server.ts b/test/e2e/live/https-pin-compatible-server.ts index 97a5d783d4a..01780253c4b 100644 --- a/test/e2e/live/https-pin-compatible-server.ts +++ b/test/e2e/live/https-pin-compatible-server.ts @@ -24,18 +24,11 @@ export interface FakeHttpsCompatibleRequest { } export interface FakeHttpsCompatibleServer extends StartedHttpServer { + readonly certificate: Buffer; requests(): readonly FakeHttpsCompatibleRequest[]; setChatRedirect(location: string | null): void; } -function requireTcpPort(server: https.Server): number { - const address = server.address(); - if (!address || typeof address === "string") { - throw new Error("fake HTTPS compatible endpoint did not bind to a TCP port"); - } - return address.port; -} - function generateEphemeralTlsMaterial(): { dir: string; cert: Buffer; key: Buffer } { const dir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-https-pin-tls-")); const keyPath = path.join(dir, "server.key"); @@ -59,6 +52,8 @@ function generateEphemeralTlsMaterial(): { dir: string; cert: Buffer; key: Buffe "1", "-subj", "/CN=nemoclaw-https-pin-e2e", + "-addext", + "subjectAltName=IP:127.0.0.1", "-keyout", keyPath, "-out", @@ -140,9 +135,10 @@ export async function startFakeHttpsCompatibleServer(options: { jsonResponse(res, 404, { error: { message: "not found" } }); }); - await listenOnRandomPort(server); + const port = await listenOnRandomPort(server); return { - port: requireTcpPort(server), + port, + certificate: tls.cert, requests: () => requests, setChatRedirect: (location) => { chatRedirectLocation = location; diff --git a/test/e2e/live/inference-routing-helpers.ts b/test/e2e/live/inference-routing-helpers.ts index b38be9987c2..0c544425296 100644 --- a/test/e2e/live/inference-routing-helpers.ts +++ b/test/e2e/live/inference-routing-helpers.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { parseOpenClawGatewayModelRun } from "./openclaw-inference-switch-helpers.ts"; +import { registeredCompatibleChatCommand } from "../fixtures/registered-inference.ts"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -314,17 +316,7 @@ function expectOnboardFailure(result: RawRunResult, label: string): void { expect(result.exitCode, `${label} unexpectedly succeeded\n${redacted}`).not.toBe(0); } -function parseJsonBody(body: string, label: string): unknown { - try { - return JSON.parse(body); - } catch (error) { - throw new Error( - `${label} response was not JSON: ${error instanceof Error ? error.message : String(error)}`, - ); - } -} - -function openAiContent(json: unknown): string { +export function openAiContent(json: unknown): string { if (!json || typeof json !== "object") return ""; const choices = (json as { choices?: unknown }).choices; if (!Array.isArray(choices)) return ""; @@ -341,33 +333,13 @@ function openAiContent(json: unknown): string { return ""; } -function anthropicContent(json: unknown): string { - if (!json || typeof json !== "object") return ""; - const content = (json as { content?: unknown }).content; - if (Array.isArray(content)) { - return content - .map((part) => { - if ( - part && - typeof part === "object" && - typeof (part as { text?: unknown }).text === "string" - ) { - return (part as { text: string }).text; - } - return ""; - }) - .join("") - .trim(); - } - return openAiContent(json); -} - async function expectOpenAiChatThroughSandbox( sandbox: SandboxClient, sandboxName: string, model: string, redactionValues: readonly string[], artifactName: string, + consumer: "curl" | "openclaw-gateway" = "curl", ): Promise { const payload = JSON.stringify({ model, @@ -376,17 +348,18 @@ async function expectOpenAiChatThroughSandbox( }); const response = await sandbox.exec( sandboxName, - [ - "curl", - "-sS", - "--max-time", - "60", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", - payload, - ], + consumer === "openclaw-gateway" + ? [ + "openclaw", + "infer", + "model", + "run", + "--gateway", + "--json", + "--prompt", + "Reply with exactly one word: PONG", + ] + : registeredCompatibleChatCommand(sandboxName, model, payload), { artifactName, env: buildAvailabilityProbeEnv(), @@ -395,45 +368,16 @@ async function expectOpenAiChatThroughSandbox( }, ); expect(response.exitCode, resultText(response)).toBe(0); - const content = openAiContent(parseJsonBody(response.stdout, artifactName)); - expect(content, `no chat content in response: ${response.stdout.slice(0, 500)}`).not.toBe(""); -} - -async function expectAnthropicMessageThroughSandbox( - sandbox: SandboxClient, - sandboxName: string, - model: string, - redactionValues: readonly string[], -): Promise { - const payload = JSON.stringify({ - model, - messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], - max_tokens: 50, - }); - const response = await sandbox.exec( - sandboxName, - [ - "curl", - "-sS", - "--max-time", - "60", - "https://inference.local/v1/messages", - "-H", - "Content-Type: application/json", - "--data-raw", - payload, - ], - { - artifactName: "anthropic-inference-local-message", - env: buildAvailabilityProbeEnv(), - redactionValues: [...redactionValues], - timeoutMs: 90_000, - }, - ); - expect(response.exitCode, resultText(response)).toBe(0); - const content = anthropicContent(parseJsonBody(response.stdout, "anthropic inference.local")); - expect(content, `no Anthropic content in response: ${response.stdout.slice(0, 500)}`).not.toBe( - "", + const gatewayRun = + consumer === "openclaw-gateway" ? parseOpenClawGatewayModelRun(response.stdout) : null; + const content = + consumer === "openclaw-gateway" + ? gatewayRun?.provider === "inference" && gatewayRun.model === model + ? gatewayRun.text + : "" + : openAiContent(JSON.parse(response.stdout)); + expect(content, `unexpected chat content in response: ${response.stdout.slice(0, 500)}`).toBe( + "PONG", ); } @@ -447,21 +391,6 @@ export function requireProviderSmokeSelected(provider: "openrouter", skip: SkipF } } -export async function verifyProcessListCredentialIsolation( - artifacts: ArtifactSink, - processList: RawRunResult, - apiKey: string, -): Promise { - if (processList.exitCode === 0 && processList.stdout.trim()) { - expect(processList.stdout.includes(apiKey), redactedResultText(processList)).toBe(false); - } else { - await artifacts.writeJson("tc-inf-05-process-list-skipped.json", { - reason: "ps not available in hardened sandbox", - exitCode: processList.exitCode, - }); - } -} - export async function verifyCredentialPlaceholder( artifacts: ArtifactSink, placeholderValue: string, @@ -480,7 +409,6 @@ export async function verifyCredentialPlaceholder( export { CREDENTIAL_CLASSIFICATION_PATTERN, cleanupSandbox, - expectAnthropicMessageThroughSandbox, expectNoActiveSandbox, expectOnboardFailure, expectOnboardSuccess, diff --git a/test/e2e/live/inference-routing.test.ts b/test/e2e/live/inference-routing.test.ts index ad39ed2e5e2..6b91b122431 100644 --- a/test/e2e/live/inference-routing.test.ts +++ b/test/e2e/live/inference-routing.test.ts @@ -2,6 +2,11 @@ // SPDX-License-Identifier: Apache-2.0 import { approveOpenClawAdminScope } from "./openclaw-admin-scope.ts"; +import { registeredCompatibleResponseCommand } from "../fixtures/registered-inference.ts"; +import { + INFERENCE_CURL_HTTP_CODE_MARKER, + waitForMcpCurlPolicyDenial, +} from "../fixtures/curl-policy-denial.ts"; import fs from "node:fs"; import { getSandbox } from "../../../src/lib/state/registry.ts"; import { hostedCredentialScanCommand } from "./inference-routing-credential-scan.ts"; @@ -324,45 +329,6 @@ async function runRuntimeIdentityE2EScenario( env: openshellEnv, timeoutMs: 30_000, }); - const settingsBefore = await sandbox.openshell(["settings", "get", "--global", "--json"], { - artifactName: `${artifactPrefix}-provider-policy-setting-before`, - env: openshellEnv, - timeoutMs: 30_000, - }); - expect(settingsBefore.exitCode, resultText(settingsBefore)).toBe(0); - const settingsDocument = JSON.parse(settingsBefore.stdout) as { - settings?: Record; - }; - const priorProvidersV2Setting = settingsDocument.settings?.providers_v2_enabled; - const restoreSettingArgs = new Map([ - ["", ["settings", "delete", "--global", "--key", "providers_v2_enabled", "--yes"]], - [ - "false", - ["settings", "set", "--global", "--key", "providers_v2_enabled", "--value", "false", "--yes"], - ], - [ - "true", - ["settings", "set", "--global", "--key", "providers_v2_enabled", "--value", "true", "--yes"], - ], - ]).get(priorProvidersV2Setting ?? ""); - expect(restoreSettingArgs).toBeDefined(); - cleanup.add("restore OpenShell provider-derived policy setting", async () => { - const restored = await sandbox.openshell(restoreSettingArgs!, { - artifactName: `${artifactPrefix}-provider-policy-setting-restore`, - env: openshellEnv, - timeoutMs: 30_000, - }); - expect(restored.exitCode, resultText(restored)).toBe(0); - }); - const enableProviderPolicy = await sandbox.openshell( - ["settings", "set", "--global", "--key", "providers_v2_enabled", "--value", "true", "--yes"], - { - artifactName: `${artifactPrefix}-provider-policy-setting-enable`, - env: openshellEnv, - timeoutMs: 30_000, - }, - ); - expect(enableProviderPolicy.exitCode, resultText(enableProviderPolicy)).toBe(0); progress.phase("start the public OAuth issuer and protected resource"); const oauth = await startRuntimeIdentityOAuthServer({ clientId, @@ -523,7 +489,7 @@ async function runRuntimeIdentityE2EScenario( tsxPath, "--input-type=module", "--eval", - `const { main } = await import(${JSON.stringify(runnerPath)}); await main(["apply"], { runtimeIdentityProfilePolicy: ${JSON.stringify(runtimeIdentityProfilePolicy)} });`, + `const { main } = await import(${JSON.stringify(runnerPath)}); const { default: hostRunner } = await import(${JSON.stringify(path.join(REPO_ROOT, "src/lib/blueprint-runner.ts"))}); const { observeBlueprintNativeInference } = hostRunner; await main(["apply"], { runtimeIdentityProfilePolicy: ${JSON.stringify(runtimeIdentityProfilePolicy)}, nativeInferenceObserver: observeBlueprintNativeInference });`, ], { artifactName: `${artifactPrefix}-runtime-identity-apply`, @@ -723,7 +689,7 @@ test ); test( - "TC-INF-09 local compatible endpoint routes through inference.local (#5744)", + "TC-INF-09 local compatible endpoint uses its attached provider (#5744)", { timeout: ONBOARD_SINGLE_FINAL_HANDOFF_TEST_TIMEOUT_MS, meta: { @@ -731,7 +697,7 @@ test( "confirm compatible-endpoint prerequisites", "start the local compatible endpoint", "onboard to the compatible endpoint", - "request sandbox chat through inference.local", + "request OpenClaw chat through its attached provider", ], }, }, @@ -771,7 +737,8 @@ test( id: "inference-routing-compatible-endpoint", contract: [ "a custom OpenAI-compatible endpoint onboards", - "sandbox inference.local routes chat to compatible endpoint", + "OpenClaw gateway uses the selected native model and returns chat content", + "the compatible endpoint receives an authenticated chat POST", ], endpointUrl: fake.baseUrl, model, @@ -793,14 +760,23 @@ test( ONBOARD_FINAL_HANDOFF_COMMAND_TIMEOUT_MS, ); expectOnboardSuccess(onboard, "TC-INF-09 compatible-endpoint onboard"); - progress.phase("request sandbox chat through inference.local"); + progress.phase("request OpenClaw chat through its attached provider"); + await approveOpenClawAdminScope( + host, + sandbox, + sandboxName, + sandboxAccessEnv(), + [apiKey], + false, + ); const sandboxRequestOffset = fake.requests().length; await expectOpenAiChatThroughSandbox( sandbox, sandboxName, model, [apiKey], - "compatible-endpoint-inference-local-chat", + "compatible-endpoint-openclaw-chat", + "openclaw-gateway", ); expect( fake @@ -810,14 +786,15 @@ test( (request) => request.auth === "ok" && request.method === "POST" && - request.path === "/v1/chat/completions", + request.path === "/v1/chat/completions" && + request.model === model, ), ).toBe(true); }, ); test( - "TC-INF-09 Docker bearerless vLLM validation (#14083)", + "TC-INF-12 Docker bearerless vLLM validation (#14083)", { timeout: 5 * 60_000, meta: { @@ -833,7 +810,7 @@ test( await artifacts.target.declare({ id: "inference-routing-bearerless-vllm-validation", contract: [ - "production bearerless provider validation reaches private bindings across distinct Docker gateways", + "production bearerless provider validation reaches private bindings through host networking on loopback", ], }); progress.phase("prepare owned bearerless vLLM cleanup"); @@ -844,7 +821,7 @@ test( expect(id === null || /^[a-f0-9]{64}$/u.test(id), "invalid cleanup container ID").toBe(true); const removed = id ? await host.command("docker", ["--context", "default", "rm", "--force", id], { - artifactName: "tc-inf-09-bearerless-vllm-cleanup", + artifactName: "tc-inf-12-bearerless-vllm-cleanup", env: buildAvailabilityProbeEnv(), timeoutMs: 30_000, }) @@ -864,7 +841,6 @@ const assert = require('node:assert/strict'); const { execFileSync, spawnSync } = require('node:child_process'); const fs = require('node:fs'); const local = require('./dist/lib/inference/local.js'); -const { resolveManagedVllmBridgeHost } = require('./dist/lib/inference/serving/vllm-host-local-network.js'); const docker = (...args) => execFileSync('docker', ['--context', 'default', ...args], { encoding: 'utf8', timeout: 120_000, stdio: ['ignore', 'pipe', 'pipe'], }); @@ -874,32 +850,30 @@ const existing = spawnSync('docker', ['--context', 'default', 'container', 'insp }); assert.notEqual(existing.status, 0, 'refusing to replace an existing managed container'); assert.match(existing.stderr, /No such (?:object|container)/i); -const bridge = resolveManagedVllmBridgeHost(); const defaultNetwork = JSON.parse(docker('network', 'inspect', 'bridge')); -const openshellNetwork = JSON.parse(docker('network', 'inspect', 'openshell-docker')); -assert.notEqual(bridge, defaultNetwork[0].IPAM.Config[0].Gateway, 'the gateways must differ'); +assert.notEqual('127.0.0.1', defaultNetwork[0].IPAM.Config[0].Gateway, 'the probe must not use Docker gateway routing'); const cidfile = process.argv[1]; docker('create', '--name', 'nemoclaw-vllm', '--cidfile', cidfile, '--label', 'com.nvidia.nemoclaw.managed-vllm=true', - '--publish', '127.0.0.1:18080:8000', '--publish', bridge + ':18080:8000', + '--publish', '127.0.0.1:18080:8000', 'node:24.22.1-trixie-slim@sha256:43b9ba1142b8dcf33fa8f922ca1be85714adc444e0586c7cf7726daaeabc1bea', - 'node', '-e', "require('node:http').createServer((req, res) => { res.setHeader('content-type', 'application/json'); require('node:fs').appendFileSync('/tmp/requests', [req.method, req.url, req.headers.host, req.socket.remoteAddress].join(' ') + '\\n'); res.end(JSON.stringify({data:[{id:'bearerless-bridge-proof'}]})); }).listen(8000, '0.0.0.0');"); + 'node', '-e', "require('node:http').createServer((req, res) => { res.setHeader('content-type', 'application/json'); require('node:fs').appendFileSync('/tmp/requests', [req.method, req.url, req.headers.host, req.socket.remoteAddress].join(' ') + '\\n'); res.end(JSON.stringify({data:[{id:'bearerless-loopback-proof'}]})); }).listen(8000, '0.0.0.0');"); const id = fs.readFileSync(cidfile, 'utf8').trim(); docker('start', id); const inspection = JSON.parse(docker('inspect', id)); ${BEARERLESS_VLLM_READINESS_SCRIPT} const command = local.getLocalProviderContainerReachabilityCheck('vllm-local'); -assert(command?.includes('host.openshell.internal:' + bridge), 'the production probe must select the published bridge'); +assert(command?.includes('host.openshell.internal:127.0.0.1') && command[command.indexOf('--network') + 1] === 'host', 'the production probe must use host networking and loopback'); const result = local.validateLocalProvider('vllm-local'); const requests = docker('exec', id, 'cat', '/tmp/requests'); -console.log(JSON.stringify({defaultNetwork, openshellNetwork, inspection, command, result, requests})); +console.log(JSON.stringify({defaultNetwork, inspection, command, result, requests})); assert.deepEqual(result, {ok: true}); assert.match(requests, /^GET \/v1\/models host\.openshell\.internal:18080 \S+$/m); `, cidfile, ], { - artifactName: "tc-inf-09-bearerless-vllm-container-validation", + artifactName: "tc-inf-12-bearerless-vllm-container-validation", cwd: REPO_ROOT, env: { ...buildAvailabilityProbeEnv(), @@ -915,7 +889,7 @@ assert.match(requests, /^GET \/v1\/models host\.openshell\.internal:18080 \S+$/m ); test( - "TC-INF-11 DNS-backed HTTPS custom endpoint routes through the local pinning adapter (#6141)", + "TC-INF-11 native HTTPS inference denies rebound DNS and private redirects (#6141)", { timeout: ONBOARD_SINGLE_FINAL_HANDOFF_TEST_TIMEOUT_MS, meta: { @@ -925,8 +899,8 @@ test( "start the public HTTPS compatible endpoint", "onboard with the placeholder endpoint", "switch to the DNS-backed HTTPS endpoint", - "verify pinned route isolation and DNS rebinding", - "verify private redirect rejection", + "verify authenticated inference and private redirect rejection", + "verify DNS rebinding rejection", ], }, }, @@ -953,11 +927,8 @@ test( await fake.close(); } }); - // A genuinely public, DNS-resolvable, publicly-trusted-certificate origin - // is required: the adapter's SSRF preflight rejects loopback/private - // addresses, and only a real TLS trust chain exercises its SNI-pinned - // certificate validation. This reuses the same trycloudflare.com quick - // tunnel mechanism as the MCP-bridge DNS-rebinding coverage. + // Use a public trusted TLS origin so native DNS admission and certificate + // verification exercise the same boundary as a hosted provider. const cloudflaredBin = await resolveVerifiedCloudflaredBinary(cleanup, host); const tunnel = await startPublicMcpHttpsTunnel({ cloudflaredBin, @@ -974,27 +945,16 @@ test( id: "https-pin-runtime-adapter-dns-backed-endpoint", issue: 6141, contract: [ - "inference set routes a DNS-backed HTTPS endpoint through the local pinning adapter", - "OpenShell's own policy view never references the real upstream hostname", - "a real chat completion round-trips through the pinned TLS connection to the public endpoint", - "a DNS rebind of the upstream hostname after inference set does not redirect adapter traffic", - "an upstream redirect to a private target is rejected without relaying Location or reaching the target", + "inference set attaches the selected HTTPS provider with exact public destination addresses", + "an authenticated chat completion reaches the selected HTTPS endpoint", + "a DNS rebind is denied before upstream traffic or credentials leave the sandbox", + "a client following the upstream redirect is denied before the private target receives any request", ], endpointUrl, model, }); - // Onboarding's own SSRF preflight (assertEndpointResolvesPublic) only - // rejects private/internal addresses; it does not fail closed on - // DNS-backed HTTPS the way the HTTPS Pin Runtime adapter's call site does, - // and onboarding never wires that adapter itself (only - // inference-set-route-containment.ts's normalizeCustomEndpointUrl does, on - // the `inference set --endpoint-url` path). Onboard with a disposable - // plain-HTTP placeholder endpoint first -- the same shape TC-INF-09 already - // onboards successfully with -- then switch to the DNS-backed HTTPS - // endpoint through `inference set --endpoint-url`, the actual #6141 call - // site this test exercises. - // Advertise localhost so onboarding exercises its host-bridge rewrite, but - // listen beyond host loopback so the resulting sandbox route can reach it. + // Start with the existing local route, then exercise its native hosted + // transition. The old provider must no longer authorize the private target. const placeholder = await startFakeOpenAiCompatibleServer({ apiKey, chatContent: "placeholder", @@ -1059,33 +1019,21 @@ test( env: { ...buildAvailabilityProbeEnv(), COMPATIBLE_API_KEY: apiKey }, progress, redactionValues: [apiKey], - timeoutMs: 60_000, + // Match the OpenClaw switch lane: selection also restarts and pairs the gateway. + timeoutMs: 120_000, }, ); expect( inferenceSet.exitCode, `TC-INF-11 inference set https-pin endpoint failed\n${redactedResultText(inferenceSet)}`, ).toBe(0); - progress.phase("verify pinned route isolation and DNS rebinding"); - // OpenShell's own network-policy view is a second, independent witness: - // it must never learn the real upstream hostname either, only the local - // adapter's host.openshell.internal boundary that everything else here - // already resolves through. - const policy = await sandbox.openshell(["policy", "get", "--full", sandboxName], { - artifactName: "tc-inf-11-policy-get-https-pin", - env: buildAvailabilityProbeEnv(), - timeoutMs: 30_000, - }); - const policyText = resultText(policy).replace(/\u001b\[[0-9;]*m/g, ""); - expect(policy.exitCode, policyText).toBe(0); - expect(policyText).not.toContain(endpointHostname); + progress.phase("verify authenticated inference and private redirect rejection"); + const hostsFixture = await setupDnsRebindingHostsFixture(host, sandboxName, endpointHostname); + cleanup.add(`restore https-pin DNS rebinding hosts fixture for ${sandboxName}`, () => + restoreDnsRebindingHostsFixture(host, sandboxName, hostsFixture), + ); const sandboxRequestOffset = fake.requests().length; - // OpenShell 0.0.85 refreshes the sandbox-side inference bundle every five - // seconds. Because this switch intentionally keeps the same provider/model - // identity while replacing only its endpoint binding, an immediate request - // can still use the placeholder route cached before `inference set`. Poll - // through two refresh intervals, but accept success only after the real - // pinned upstream records the authenticated request. + // Retain bounded propagation polling and require a new authenticated request. let routeProbeAttempt = 0; await expect .poll( @@ -1096,7 +1044,7 @@ test( sandboxName, model, [apiKey], - `https-pin-endpoint-inference-local-chat-${routeProbeAttempt}`, + `https-pin-endpoint-native-chat-${routeProbeAttempt}`, ); return fake .requests() @@ -1111,72 +1059,30 @@ test( { interval: 5_000, timeout: 11_000 }, ) .toBe(true); - // The assertions above only prove the *initial* `inference set` reached - // the real target. They do not prove the adapter is resistant to a DNS - // record changing after the route is already pinned -- the exact - // SSRF/DNS-rebinding vulnerability the pinning mechanism exists to close. - // Rebind the tunnel hostname to a reserved, unreachable documentation - // address (RFC 5737 TEST-NET-1) now that the route is registered: if the - // adapter re-resolved DNS per request instead of using the addresses it - // already pinned, this chat call would fail to connect instead of - // succeeding. - const hostsFixture = await setupDnsRebindingHostsFixture(host, sandboxName, endpointHostname); - cleanup.add(`restore https-pin DNS rebinding hosts fixture for ${sandboxName}`, () => - restoreDnsRebindingHostsFixture(host, sandboxName, hostsFixture), - ); - await remapDnsRebindingHostname( - host, - sandboxName, - hostsFixture, - "192.0.2.1", - "tc-inf-11-dns-rebind-after-inference-set", - ); - const rebindRequestOffset = fake.requests().length; - await expectOpenAiChatThroughSandbox( - sandbox, - sandboxName, - model, - [apiKey], - "https-pin-endpoint-dns-rebinding-chat", - ); - expect( - fake - .requests() - .slice(rebindRequestOffset) - .some( - (request) => - request.auth === "ok" && - request.method === "POST" && - request.path === "/v1/chat/completions", - ), - ).toBe(true); - await restoreDnsRebindingHostsFixture(host, sandboxName, hostsFixture); - progress.phase("verify private redirect rejection"); + const readDnsAudit = (suffix: string) => + sandbox.openshell( + ["logs", sandboxName, "-n", "500", "--source", "sandbox", "--since", "2m"], + { + artifactName: `tc-inf-11-dns-rebinding-audit-${suffix}`, + env: buildAvailabilityProbeEnv(), + redactionValues: [apiKey], + timeoutMs: 30_000, + }, + ); const privateTargetRequestOffset = placeholder.requests().length; - const redirectTarget = new URL("chat/completions", `${placeholder.baseUrl}/`).toString(); + const privateTarget = new URL("chat/completions", `${placeholder.baseUrl}/`); + privateTarget.hostname = "host.openshell.internal"; + const redirectTarget = privateTarget.toString(); fake.setChatRedirect(redirectTarget); const redirectPayload = JSON.stringify({ model, messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], max_tokens: 50, }); + const redirectAuditBefore = await readDnsAudit("before-private-redirect"); const redirect = await sandbox.exec( sandboxName, - [ - "curl", - "-sS", - "--include", - "--location", - "--max-redirs", - "3", - "--max-time", - "60", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", - redirectPayload, - ], + registeredCompatibleResponseCommand(sandboxName, model, redirectPayload, true), { artifactName: "tc-inf-11-private-redirect-rejection", env: buildAvailabilityProbeEnv(), @@ -1185,11 +1091,61 @@ test( }, ); const redirectText = resultText(redirect); - expect(redirect.exitCode, redirectText).toBe(0); - expect(redirectText).toMatch(/HTTP\/1\.[01] 502/u); - expect(redirectText).toContain("redirect_blocked"); - expect(redirectText.toLowerCase()).not.toContain("location:"); + expect( + await waitForMcpCurlPolicyDenial( + redirect, + redirectAuditBefore, + readDnsAudit, + privateTarget.hostname, + INFERENCE_CURL_HTTP_CODE_MARKER, + "302", + ), + redirectText, + ).toBe(true); + expect(redirectText).toMatch(/HTTP\/1\.[01] 302/u); + expect(redirectText.toLowerCase()).toContain(`location: ${redirectTarget}`.toLowerCase()); expect(placeholder.requests()).toHaveLength(privateTargetRequestOffset); + fake.setChatRedirect(null); + progress.phase("verify DNS rebinding rejection"); + // OpenShell re-resolves DNS and rejects answers outside allowed_ips. + // Unlike the old pinning adapter, this route must fail after the rebind. + await remapDnsRebindingHostname( + host, + sandboxName, + hostsFixture, + "192.0.2.1", + "tc-inf-11-dns-rebind-after-inference-set", + ); + const rebindRequestOffset = fake.requests().length; + const dnsAuditBefore = await readDnsAudit("before"); + const rebind = await sandbox.exec( + sandboxName, + registeredCompatibleResponseCommand( + sandboxName, + model, + JSON.stringify({ + model, + messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], + max_tokens: 50, + }), + false, + ), + { + artifactName: "tc-inf-11-dns-rebinding-denial", + env: buildAvailabilityProbeEnv(), + redactionValues: [apiKey], + timeoutMs: 90_000, + }, + ); + const policyDenied = await waitForMcpCurlPolicyDenial( + rebind, + dnsAuditBefore, + readDnsAudit, + endpointHostname, + INFERENCE_CURL_HTTP_CODE_MARKER, + ); + expect(policyDenied && hostsFixture.server.answers() > 0, resultText(rebind)).toBe(true); + expect(fake.requests()).toHaveLength(rebindRequestOffset); }, ); diff --git a/test/e2e/live/launch-agent-turn.ts b/test/e2e/live/launch-agent-turn.ts index 2775edbbea6..9e238ec9acb 100644 --- a/test/e2e/live/launch-agent-turn.ts +++ b/test/e2e/live/launch-agent-turn.ts @@ -1103,6 +1103,30 @@ function removeBaseline() { finish(0); } +async function waitForPtyMonitorExit() { + const deadline = performance.now() + 5_000; + while (true) { + let stats; + try { + stats = fs.lstatSync(ptyMonitorSocketPath); + } catch (error) { + if (error && error.code === "ENOENT") finish(0); + finish(2, "pty_socket_unavailable"); + } + if ( + !stats.isSocket() || + stats.uid !== process.getuid() || + !exactMode(stats, 0o600) || + stats.nlink !== 1 + ) { + finish(2, "pty_socket_invalid"); + } + const remaining = deadline - performance.now(); + if (remaining <= 0) finish(2, "pty_monitor_socket_still_present"); + await new Promise((resolve) => setTimeout(resolve, Math.min(50, remaining))); + } +} + function removePtyMonitorRoot() { let before; try { @@ -1388,6 +1412,9 @@ try { else if (mode === "input-mode") qualifyTuiInputMode(); else if (mode === "qualify") qualifyTurns(); else if (mode === "cleanup-baseline") removeBaseline(); + else if (mode === "cleanup-pty-wait") { + waitForPtyMonitorExit().catch(() => finish(2, "pty_monitor_cleanup_failed")); + } else if (mode === "cleanup-pty") removePtyMonitorRoot(); else finish(2, "mode_invalid"); } catch { @@ -1433,10 +1460,7 @@ remove_pty_monitor() { } wait_for_pty_monitor_exit() { - for _ in {1..100}; do - [[ ! -S "$pty_monitor_root/pty-input-mode.sock" ]] && return - sleep 0.05 - done + session_evidence cleanup-pty-wait } cleanup() { @@ -1461,7 +1485,11 @@ cleanup() { tail -c 2048 "$evidence_error" >&2 || true cleanup_status=1 fi - wait_for_pty_monitor_exit + if ! (wait_for_pty_monitor_exit) >/dev/null 2>"$evidence_error"; then + echo "launch PTY monitor wait failed" >&2 + tail -c 2048 "$evidence_error" >&2 || true + cleanup_status=1 + fi if ! (remove_pty_monitor) >/dev/null 2>"$evidence_error"; then echo "launch PTY monitor cleanup failed" >&2 tail -c 2048 "$evidence_error" >&2 || true diff --git a/test/e2e/live/launchable-smoke.test.ts b/test/e2e/live/launchable-smoke.test.ts index e29bda4547c..cdd7ae2a4fe 100644 --- a/test/e2e/live/launchable-smoke.test.ts +++ b/test/e2e/live/launchable-smoke.test.ts @@ -2,6 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 import { randomUUID } from "node:crypto"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; +import { buildSandboxInferenceChatCommand } from "../fixtures/inference-adapter.ts"; +import { readFullE2eInferenceRoute } from "./full-e2e-inference-probe.ts"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -14,10 +17,8 @@ import { validateSandboxName } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { DEFAULT_HOSTED_INFERENCE_MODEL, - HOSTED_INFERENCE_PROVIDER_NAME, requireHostedInferenceConfig, } from "../fixtures/hosted-inference.ts"; -import { buildSandboxInferenceChatCommand } from "../fixtures/inference-adapter.ts"; import { parseOpenClawAgentText } from "../fixtures/openclaw-agent-output.ts"; import { REPO_ROOT } from "../fixtures/paths.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -28,14 +29,13 @@ import { bootstrapInstallSudoArgs } from "../fixtures/bootstrap-install-command. // This is intentionally a single live test instead of a new fixture // family: the contract is the real Ubuntu bootstrap path, so the test invokes // scripts/brev-launchable-ci-cpu.sh via sudo, then proves the bootstrap-built -// CLI can onboard, reach its native hosted provider, and run an OpenClaw agent turn +// CLI can onboard, verify native inference, and run an OpenClaw agent turn // through Vitest. const BOOTSTRAP_SCRIPT = path.join(REPO_ROOT, "scripts", "brev-launchable-ci-cpu.sh"); const BOOTSTRAP_SENTINEL = "/var/run/nemoclaw-launchable-ready"; const MODEL = process.env.NEMOCLAW_MODEL ?? process.env.NEMOCLAW_COMPAT_MODEL ?? DEFAULT_HOSTED_INFERENCE_MODEL; -const EXPECTED_ROUTE_PROVIDER = HOSTED_INFERENCE_PROVIDER_NAME; const DEFAULT_SANDBOX_NAME = `e2e-boot-${randomUUID().slice(0, 8)}`; const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? DEFAULT_SANDBOX_NAME; const TEST_TIMEOUT_MS = testTimeout(30 * 60_000); @@ -121,9 +121,9 @@ async function cleanupBootstrapState(host: HostCliClient, cloneDir: string): Pro await runBash( host, [ - `if command -v nemoclaw >/dev/null 2>&1; then nemoclaw ${JSON.stringify(SANDBOX_NAME)} destroy --yes 2>/dev/null || true; fi`, - `if command -v openshell >/dev/null 2>&1; then openshell sandbox delete ${JSON.stringify(SANDBOX_NAME)} 2>/dev/null || true; fi`, - "if command -v openshell >/dev/null 2>&1; then openshell gateway destroy -g nemoclaw 2>/dev/null || true; fi", + `nemoclaw ${JSON.stringify(SANDBOX_NAME)} destroy --yes 2>/dev/null || true`, + `openshell sandbox delete ${JSON.stringify(SANDBOX_NAME)} 2>/dev/null || true`, + "openshell gateway destroy -g nemoclaw 2>/dev/null || true", `sudo rm -rf ${JSON.stringify(cloneDir)} 2>/dev/null || rm -rf ${JSON.stringify(cloneDir)} || true`, ].join("\n"), { @@ -140,7 +140,8 @@ async function sleep(ms: number): Promise { async function expectPongFromSandboxInference( sandboxExec: (command: string[], artifactName: string) => Promise, - endpointUrl: string, + logicalProvider: string, + endpointUrl?: string, ): Promise { const payload = JSON.stringify({ model: MODEL, @@ -152,10 +153,10 @@ async function expectPongFromSandboxInference( let lastResult: ShellProbeResult | undefined; for (let attempt = 1; attempt <= 3; attempt += 1) { lastResult = await sandboxExec( - buildSandboxInferenceChatCommand(payload, EXPECTED_ROUTE_PROVIDER, endpointUrl), + buildSandboxInferenceChatCommand(payload, logicalProvider, endpointUrl, 60), `phase-6-sandbox-inference-attempt-${attempt}`, ); - if (lastResult.stdout.trim()) { + if (lastResult.exitCode === 0 && !lastResult.timedOut && lastResult.stdout.trim()) { try { lastContent = parseChatContent(lastResult.stdout); } catch { @@ -167,7 +168,7 @@ async function expectPongFromSandboxInference( } throw new Error( - `sandbox native hosted inference expected PONG after 3 attempts; last content='${lastContent}'; ` + + `sandbox inference expected PONG after 3 attempts; last content='${lastContent}'; ` + `stdout='${lastResult?.stdout.slice(0, 300) ?? ""}'; stderr='${lastResult?.stderr.slice(0, 300) ?? ""}'`, ); } @@ -211,8 +212,6 @@ test( const hosted = requireHostedInferenceConfig(secrets); const apiKey = hosted.apiKey; - expect(fs.existsSync(BOOTSTRAP_SCRIPT), `${BOOTSTRAP_SCRIPT} missing`).toBe(true); - const sudo = await host.command("sudo", ["-n", "true"], { artifactName: "prereq-passwordless-sudo", env: runEnv(), @@ -294,34 +293,7 @@ test( "the dev integration target must install a dev-channel OpenShell build", ).toBe(true); - const nodeVersion = await host.command( - "node", - [ - "-p", - "JSON.stringify({version: process.version, major: Number(process.versions.node.split('.')[0])})", - ], - { artifactName: "phase-3-node-version", env: pathEnv, timeoutMs: 30_000 }, - ); - expectExitZero(nodeVersion, "node version probe"); - const node = JSON.parse(nodeVersion.stdout) as { - version: string; - major: number; - }; - await artifacts.writeJson("node-version.json", node); - expect( - node.major, - `Node.js too old after bootstrap install: ${node.version}`, - ).toBeGreaterThanOrEqual(20); - - const runtimeAfterInstall = await runtimeProvider.command(["info"], { - artifactName: "phase-3-runtime-info-after-install", - env: pathEnv, - timeoutMs: 30_000, - }); - expectExitZero(runtimeAfterInstall, `${runtimeProvider.displayName} running after install`); expect(fs.existsSync(BOOTSTRAP_SENTINEL), `${BOOTSTRAP_SENTINEL} missing`).toBe(true); - expect(fs.existsSync(path.join(cloneDir, ".git")), `${cloneDir}/.git missing`).toBe(true); - expect(fs.existsSync(path.join(cloneDir, "dist")), `${cloneDir}/dist missing`).toBe(true); expect( fs.existsSync(path.join(cloneDir, "nemoclaw", "dist")), `${cloneDir}/nemoclaw/dist missing`, @@ -382,7 +354,18 @@ test( }); expectExitZero(status, `nemoclaw ${SANDBOX_NAME} status`); - expect(status.stdout).toMatch(new RegExp(`Provider:\\s+${EXPECTED_ROUTE_PROVIDER}`, "i")); + const entry = getSandbox(SANDBOX_NAME); + const inferenceConfig = await readFullE2eInferenceRoute({ + sandbox, + sandboxName: SANDBOX_NAME, + entry, + logicalProvider: hosted.providerName, + model: MODEL, + env: pathEnv, + redactionValues: [apiKey], + timeoutMs: 30_000, + }); + expect(inferenceConfig.ok, inferenceConfig.detail).toBe(true); const gatewayContainer = await runtimeProvider.command( ["container", "ps", "--format", "{{.Names}}"], @@ -436,7 +419,12 @@ test( env: pathEnv, timeoutMs: INFERENCE_TIMEOUT_MS, }); - await expectPongFromSandboxInference(sandboxExec, hosted.endpointUrl); + await expectPongFromSandboxInference( + sandboxExec, + hosted.providerName, + entry?.nativeLocalProviderAttachment?.endpointUrl ?? + entry?.nativeHostedProviderAttachment?.endpointUrl, + ); const sessionId = `e2e-bootstrap-${Date.now()}-${randomUUID()}`; const agent = await sandboxExec( diff --git a/test/e2e/live/managed-image-activation-e2e-helpers.ts b/test/e2e/live/managed-image-activation-e2e-helpers.ts index dfc3cffca41..67574044c25 100644 --- a/test/e2e/live/managed-image-activation-e2e-helpers.ts +++ b/test/e2e/live/managed-image-activation-e2e-helpers.ts @@ -2,6 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 import fs from "node:fs"; +import { randomUUID } from "node:crypto"; +import { discoverHostInterfaceAddress } from "../fixtures/host-address.ts"; import os from "node:os"; import path from "node:path"; import { setTimeout as sleep } from "node:timers/promises"; @@ -46,11 +48,141 @@ import { } from "../fixtures/fake-openai-compatible.ts"; import { captureIssue4462FailureDiagnostics } from "../fixtures/issue-4462-diagnostics.ts"; import { initializeGatewayForCleanup } from "../fixtures/gateway-runtime-start.ts"; +import { captureSandboxFailureDiagnostics } from "../fixtures/sandbox-failure-diagnostics.ts"; import type { LifecyclePhaseFixture } from "../fixtures/phases/lifecycle.ts"; import { pollUntil } from "../fixtures/polling.ts"; import type { TestProgress } from "../fixtures/progress.ts"; +import { createDockerRuntimeProviderBundle } from "../../../src/lib/onboard/runtime-provider/docker.ts"; +import { createCurrentPodmanRuntimeProviderBundle } from "../../../src/lib/onboard/runtime-provider/podman.ts"; +import { managedStartupStateRoots } from "../../../src/lib/onboard/managed-startup/state-roots.ts"; +import { preflightManagedStateVolumes } from "../../../src/lib/onboard/managed-workload/managed-state-volumes.ts"; + +const STATE_PROBE_IMAGE = + "node:24.18.1-trixie-slim@sha256:ac39e4b5fcb2b1b34b20364fd58b2e898f3bb80731ee6f62a7536f9df3d6aadc"; const API_KEY = "nemoclaw-managed-activation-e2e-key"; + +/** Disposable legacy state for the attachment-refusal check. */ +export function createLegacyManagedVolumeFixture(cleanup: CleanupRegistry) { + const sandboxName = `mv-${randomUUID().replaceAll("-", "").slice(0, 16)}`; + const runtimeProvider = + process.env.NEMOCLAW_GATEWAY_RUNTIME === "podman" + ? createCurrentPodmanRuntimeProviderBundle() + : createDockerRuntimeProviderBundle(); + // Both selected built-in factories expose the container engine; no provider registry input is used. + const engine = runtimeProvider.containerEngine as Extract< + typeof runtimeProvider.containerEngine, + { supported: true } + >; + const volumes = new Set(); + const helpers = new Set(); + let unresolvedHelperCreation = false; + const root = managedStartupStateRoots({ + agent: "openclaw", + sandboxName, + agentIdentity: { uid: 1000, gid: 1000 }, + })[0]!; + const run = (args: readonly string[], timeout?: number) => { + const previouslyUnresolved = unresolvedHelperCreation; + if (args[0] === "create") unresolvedHelperCreation = true; + if (args[0] === "volume" && args[1] === "create") volumes.add(args.at(-1)!); + const result = engine.capture("workload-cleanup", args, Math.min(timeout ?? 30_000, 120_000)); + if (args[0] === "create") { + const id = String(result.stdout ?? "").trim(); + if (result.status === 0 && /^[a-f0-9]{64}$/u.test(id)) { + helpers.add(id); + unresolvedHelperCreation = previouslyUnresolved; + } + } + if (result.status === 0 && args[0] === "rm") helpers.delete(args.at(-1)!); + return result; + }; + const command = (args: readonly string[]) => { + const result = run(args, 120_000); + if (result.status !== 0 || result.error) + throw new Error( + `Volume qualification command ${args[0]} failed: ${String(result.stderr ?? result.error ?? "")}`, + ); + return String(result.stdout ?? "").trim(); + }; + cleanup.add("remove only the volume qualification fixtures", () => { + for (const helper of helpers) command(["rm", "--force", helper]); + if (unresolvedHelperCreation) + throw new Error("Fixture helper creation was ambiguous; cleanup is not established"); + for (const volume of volumes) command(["volume", "rm", volume]); + }); + const marker = randomUUID(); + const probe = (volume: string, program: string, readonly = true) => { + const id = command([ + "create", + "--name", + `nemoclaw-volume-probe-${randomUUID()}`, + "--pull", + "never", + "--network", + "none", + "--read-only", + ...(runtimeProvider.identity.id === "podman" + ? ["--volume", `${volume}:/state:${readonly ? "ro" : "rw"},nocopy`] + : [ + "--mount", + `type=volume,src=${volume},dst=/state,volume-nocopy${readonly ? ",readonly" : ""}`, + ]), + "--entrypoint", + "/usr/local/bin/node", + STATE_PROBE_IMAGE, + "-e", + program, + ]); + try { + return command(["start", "--attach", id]); + } finally { + command(["rm", "--force", id]); + } + }; + const seed = () => { + command(["pull", "--quiet", STATE_PROBE_IMAGE]); + command([ + "volume", + "create", + ...Object.entries(root.ownershipLabels).flatMap(([key, value]) => [ + "--label", + `${key}=${value}`, + ]), + root.resourceIdentity, + ]); + probe( + root.resourceIdentity, + `const fs=require("node:fs"); fs.writeFileSync("/state/marker", ${JSON.stringify(marker)}); fs.chownSync("/state/marker",1000,1000); fs.chmodSync("/state/marker",0o640); fs.symlinkSync("marker","/state/link");`, + false, + ); + }; + const readback = `const fs=require("node:fs"); const s=fs.lstatSync("/state/marker"); process.stdout.write(JSON.stringify([fs.readFileSync("/state/marker","utf8"),s.uid,s.gid,s.mode,fs.readlinkSync("/state/link")]));`; + return { sandboxName, root, marker, seed, probe, readback, runtimeProvider }; +} + +/** Real engine/provider evidence; deterministic identity and failure matrices stay in unit tests. */ +export async function qualifyManagedVolumePreservation( + { artifacts, cleanup }: Pick, + phases: { refusal: () => void; evidence: () => void }, +): Promise { + const { root, seed, probe, readback, runtimeProvider } = + createLegacyManagedVolumeFixture(cleanup); + seed(); + const before = probe(root.resourceIdentity, readback); + phases.refusal(); + expect(() => preflightManagedStateVolumes({ roots: [root] }, { runtimeProvider })).toThrow( + "lacks OpenShell attachment approval", + ); + expect(probe(root.resourceIdentity, readback)).toBe(before); + phases.evidence(); + await artifacts.writeJson("legacy-managed-volume-preservation.json", { + provider: runtimeProvider.identity.id, + source: root.resourceIdentity, + attachmentRejected: true, + contentAndMetadataPreserved: true, + }); +} const MODEL = "nemoclaw-managed-activation-model"; const GATEWAY = "nemoclaw"; const AGENT_TIMEOUT_MS = 3 * 60_000; @@ -112,6 +244,14 @@ type RuntimeFixtures = { readonly progress: TestProgress; readonly sandbox: SandboxClient; }; +type ManagedActivationPhases = { + readonly agents: Record< + ShippedManagedImageAgent, + { onboard: () => void; publicLifecycle: () => void; cleanup: () => void } + >; + readonly hermesSecretBoundary: () => void; + readonly externalImages: () => void; +}; export function managedActivationOnboardArgs( catalogPath: string, @@ -178,27 +318,13 @@ function exactCatalog( ): ReadonlyMap { const document = JSON.parse(fs.readFileSync(catalogPath, "utf8")) as ManagedImageContractCatalog; const platform = managedImagePlatformForNodeArchitecture(process.arch); - expect( - platform, - `managed activation E2E does not support host architecture ${process.arch}`, - ).not.toBeNull(); const contracts = new Map(); - let revision: string | null = null; - let cohort: string | null = null; for (const agent of SHIPPED_MANAGED_IMAGE_AGENTS) { + // The parser rejects a platform mismatch, including null on an unsupported host. const contract = parseManagedImageContractV1(document[agent], agent, platform!); - revision ??= contract.source.revision; - cohort ??= contract.source.cohort; - if (contract.source.revision !== revision || contract.source.cohort !== cohort) { - throw new Error("managed activation catalog is not one exact all-agent publication cohort"); - } contracts.set(agent, contract); } - if ( - Object.keys(document).sort().join("\n") !== [...SHIPPED_MANAGED_IMAGE_AGENTS].sort().join("\n") - ) { - throw new Error("managed activation catalog must contain exactly the shipped agents"); - } + // Catalog shape and cohort rejection belong to the production loader's source tests. return contracts; } @@ -214,6 +340,7 @@ function commandEnv( NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1", NEMOCLAW_COMPAT_MODEL: MODEL, NEMOCLAW_ENDPOINT_URL: endpointUrl, + NEMOCLAW_TRUSTED_PRIVATE_INFERENCE_HOSTS: new URL(endpointUrl).hostname, NEMOCLAW_IGNORE_RUNTIME_RESOURCES: "1", NEMOCLAW_GATEWAY_RUNTIME: gatewayRuntime, NEMOCLAW_MANAGED_ACTIVATION_CATALOG: catalogPath, @@ -338,7 +465,7 @@ async function inspectRuntimeImageId( }; } -async function inspectRuntimeSandboxContainerId( +export async function inspectRuntimeSandboxContainerId( host: HostCliClient, containerEngine: ContainerEngine, sandboxName: string, @@ -347,7 +474,15 @@ async function inspectRuntimeSandboxContainerId( ): Promise>> { return await host.command( containerEngine, - ["ps", "-aq", "--filter", `label=openshell.ai/sandbox-name=${sandboxName}`], + [ + "ps", + "-aq", + "--no-trunc", + "--filter", + `label=openshell.ai/sandbox-name=${sandboxName}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", + ], { artifactName, env, timeoutMs: 30_000 }, ); } @@ -360,7 +495,6 @@ function expectManagedReceipt(sandboxName: string, contract: ManagedImageContrac release: contract.source.release, sourceRevision: contract.source.revision, sourceCohort: contract.source.cohort, - shared: true, }); } @@ -667,24 +801,64 @@ export async function waitForManagedActivationSandboxAbsence( }); } -async function verifyExactCleanup( +const RUNTIME_SANDBOX_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$/u; +export async function inspectActivationSandboxId( + host: HostCliClient, + sandboxName: string, + env: NodeJS.ProcessEnv, + observation: "before-cleanup" | "before-rebuild" | "after-rebuild", +): Promise { + const engine = env.NEMOCLAW_GATEWAY_RUNTIME === "podman" ? "podman" : "docker"; + const result = await host.command( + engine, + [ + "ps", + "-a", + "--filter", + `label=openshell.ai/sandbox-name=${sandboxName}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", + "--format", + '{{.Label "openshell.ai/sandbox-id"}}', + ], + { + artifactName: `activation-runtime-sandbox-id-${sandboxName}-${observation}`, + env, + timeoutMs: 30_000, + }, + ); + const id = result.stdout.trim(); + expect( + result.exitCode === 0 && !result.timedOut && RUNTIME_SANDBOX_ID_PATTERN.test(id), + `Runtime sandbox identity must be unambiguous before cleanup: ${resultText(result)}`, + ).toBe(true); + return id; +} + +export async function verifyExactCleanup( host: HostCliClient, sandbox: SandboxClient, sandboxName: string, env: NodeJS.ProcessEnv, + runtimeSandboxIds: readonly string[], ): Promise { + expect( + runtimeSandboxIds.length > 0 && + runtimeSandboxIds.every((id) => RUNTIME_SANDBOX_ID_PATTERN.test(id)), + "Cleanup requires the observed runtime sandbox identities", + ).toBe(true); const containerEngine = env.NEMOCLAW_GATEWAY_RUNTIME === "podman" ? "podman" : "docker"; await pollUntil({ artifactPrefix: `post-destroy-absence-${sandboxName}`, deadlineMs: 60_000, delayMs: 1_000, probe: async (_attempt, artifactName) => { - const openshellList = await sandbox.list({ + const openshellList = sandbox.list({ artifactName: `${artifactName}-openshell-list`, env, timeoutMs: 30_000, }); - const containers = await host.command( + const containers = host.command( containerEngine, ["ps", "-aq", "--filter", `label=openshell.ai/sandbox-name=${sandboxName}`], { @@ -693,64 +867,49 @@ async function verifyExactCleanup( timeoutMs: 30_000, }, ); - return { containers, openshellList }; + const volumes = []; + for (const id of new Set(runtimeSandboxIds)) { + volumes.push( + host.command( + containerEngine, + ["volume", "ls", "-q", "--filter", `label=openshell.ai/sandbox-id=${id}`], + { + artifactName: `${artifactName}-${containerEngine}-volumes-${id}`, + env, + timeoutMs: 30_000, + }, + ), + ); + } + const [observedList, observedContainers, observedVolumes] = await Promise.all([ + openshellList, + containers, + Promise.all(volumes), + ]); + return { + containers: observedContainers, + openshellList: observedList, + volumes: observedVolumes, + }; }, - terminal: ({ containers, openshellList }) => { + terminal: ({ containers, openshellList, volumes }) => { if (openshellList.exitCode !== 0) { return `list OpenShell sandboxes after managed activation destroy failed: ${resultText(openshellList)}`; } if (containers.exitCode !== 0) { return `inspect ${containerEngine} inventory after managed activation destroy failed: ${resultText(containers)}`; } + const failedVolumes = volumes.find((result) => result.exitCode !== 0 || result.timedOut); + if (failedVolumes) { + return `inspect ${containerEngine} runtime volumes after managed activation destroy failed: ${resultText(failedVolumes)}`; + } return undefined; }, - accept: ({ containers, openshellList }) => - !outputContainsSandbox(openshellList, sandboxName) && containers.stdout.trim() === "", + accept: ({ containers, openshellList, volumes }) => + !outputContainsSandbox(openshellList, sandboxName) && + containers.stdout.trim() === "" && + volumes.every((result) => result.stdout.trim() === ""), }); - // pollUntil returns only after both successful inventory commands prove - // absence. Its terminal and deadline failures already reject this cleanup. -} - -function enterOnboardPhase(progress: TestProgress, agent: ShippedManagedImageAgent): void { - switch (agent) { - case "openclaw": - progress.phase("onboard and exercise OpenClaw"); - return; - case "hermes": - progress.phase("onboard and exercise Hermes"); - return; - case "langchain-deepagents-code": - progress.phase("onboard and exercise Deep Agents Code"); - return; - } -} - -function enterPublicLifecyclePhase(progress: TestProgress, agent: ShippedManagedImageAgent): void { - switch (agent) { - case "openclaw": - progress.phase("stop and start OpenClaw through public NemoClaw lifecycle"); - return; - case "hermes": - progress.phase("stop and start Hermes through public NemoClaw lifecycle"); - return; - case "langchain-deepagents-code": - progress.phase("stop and start Deep Agents Code through public NemoClaw lifecycle"); - return; - } -} - -function enterCleanupPhase(progress: TestProgress, agent: ShippedManagedImageAgent): void { - switch (agent) { - case "openclaw": - progress.phase("destroy and verify OpenClaw cleanup"); - return; - case "hermes": - progress.phase("destroy and verify Hermes cleanup"); - return; - case "langchain-deepagents-code": - progress.phase("destroy and verify Deep Agents Code cleanup"); - return; - } } export async function collectOnboardFailureRuntimeDiagnostics( @@ -858,12 +1017,21 @@ export async function collectOnboardFailureRuntimeDiagnostics( }, ); if (copy.exitCode !== 0) return; - const stat = fs.lstatSync(copiedLog); - if (!stat.isFile() || stat.isSymbolicLink() || stat.size > 2 * 1024 * 1024) return; - await artifacts.writeText( - `managed-activation-onboard-failure-${agent}-container-${index + 1}-nemoclaw-start.log`, - fs.readFileSync(copiedLog, "utf8"), + // Check and read the same file; sandbox output must not redirect the read through a symlink. + const fd = fs.openSync( + copiedLog, + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | fs.constants.O_NONBLOCK, ); + try { + const stat = fs.fstatSync(fd); + if (!stat.isFile() || stat.size > 2 * 1024 * 1024) return; + await artifacts.writeText( + `managed-activation-onboard-failure-${agent}-container-${index + 1}-nemoclaw-start.log`, + fs.readFileSync(fd, "utf8"), + ); + } finally { + fs.closeSync(fd); + } } finally { fs.rmSync(copyRoot, { recursive: true, force: true }); } @@ -881,9 +1049,10 @@ export async function qualifyAgent( endpointUrl: string, agent: ShippedManagedImageAgent, contract: ManagedImageContractV1, + phases: ManagedActivationPhases, observeRequests?: () => readonly FakeOpenAiCompatibleRequest[], ): Promise { - const { artifacts, cleanup, host, lifecycle, progress, sandbox } = fixtures; + const { artifacts, cleanup, host, lifecycle, sandbox } = fixtures; const sandboxName = SANDBOX_NAMES[agent]; const env = commandEnv(guard, catalogPath, endpointUrl); cleanup.trackDisposable(`delete OpenShell sandbox ${sandboxName}`, () => @@ -892,7 +1061,7 @@ export async function qualifyAgent( cleanup.trackSandbox(host, sandboxName, { env, timeoutMs: 3 * 60_000 }); await preclean(host, lifecycle, sandbox, sandboxName, env); - enterOnboardPhase(progress, agent); + phases.agents[agent].onboard(); const onboard = await host.nemoclaw( managedActivationOnboardArgs(catalogPath, agent, sandboxName), { @@ -913,7 +1082,7 @@ export async function qualifyAgent( await runManagedActivationAgentTurn(sandbox, agent, sandboxName, "before", env, observeRequests); if (agent === "openclaw") await runOpenClawSubagentTurn(sandbox, sandboxName, env); if (agent === "hermes") { - progress.phase("prove Hermes secret-boundary refusal before native restart"); + phases.hermesSecretBoundary(); await proveHermesRestartSecretBoundary(host, sandbox, sandboxName, env); } const marker = `managed-activation-${agent}-${Date.now()}`; @@ -935,12 +1104,12 @@ export async function qualifyAgent( }, ); - enterPublicLifecyclePhase(progress, agent); + phases.agents[agent].publicLifecycle(); const stop = await host.nemoclaw([sandboxName, "stop"], { artifactName: `${agent}-public-stop`, env, redactionValues: [API_KEY], - timeoutMs: 120_000, + timeoutMs: 180_000, }); if (stop.exitCode !== 0) { await collectOnboardFailureRuntimeDiagnostics(artifacts, host, agent, sandboxName, env); @@ -987,16 +1156,22 @@ export async function qualifyAgent( ).toBe(true); await runManagedActivationAgentTurn(sandbox, agent, sandboxName, "after", env, observeRequests); - enterCleanupPhase(progress, agent); + phases.agents[agent].cleanup(); + const runtimeSandboxId = await inspectActivationSandboxId( + host, + sandboxName, + env, + "before-cleanup", + ); await sandbox.cleanupSandbox(sandboxName, { artifactName: `managed-activation-openshell-delete-${agent}`, env, timeoutMs: 120_000, }); - await verifyExactCleanup(host, sandbox, sandboxName, env); + await verifyExactCleanup(host, sandbox, sandboxName, env, [runtimeSandboxId]); } -async function qualifyExternalImage( +export async function qualifyExternalImage( fixtures: RuntimeFixtures, guard: ContainerBuildGuard, containerEngine: ContainerEngine, @@ -1103,12 +1278,24 @@ async function qualifyExternalImage( } } + const previousRuntimeSandboxId = await inspectActivationSandboxId( + host, + sandboxName, + env, + "before-rebuild", + ); const rebuild = await host.nemoclaw([sandboxName, "rebuild", "--yes"], { artifactName: `external-image-${agent}-rebuild`, env, redactionValues: [API_KEY], timeoutMs: ONBOARD_TIMEOUT_MS, }); + await captureSandboxFailureDiagnostics(host, rebuild, { + sandboxName, + artifactPrefix: `external-image-${agent}-rebuild-failure`, + redactionValues: [API_KEY], + captureGatewayLog: true, + }); if (rebuild.exitCode === 0) { await lifecycle.waitForSandboxReadyAfterGatewayRestart(sandboxName, { artifactNamePrefix: `external-image-${agent}-ready-after-rebuild`, @@ -1126,13 +1313,22 @@ async function qualifyExternalImage( registryEntry = registryDocument().sandboxes?.[sandboxName]; receipt = registryEntry?.workload; + const runtimeSandboxId = await inspectActivationSandboxId( + host, + sandboxName, + env, + "after-rebuild", + ); const destroy = await host.destroySandbox(sandboxName, { artifactName: `external-image-destroy-${agent}`, env, redactionValues: [API_KEY], timeoutMs: 15 * 60_000, }); - await verifyExactCleanup(host, sandbox, sandboxName, env); + await verifyExactCleanup(host, sandbox, sandboxName, env, [ + previousRuntimeSandboxId, + runtimeSandboxId, + ]); const afterInspection = await inspectRuntimeImageId( host, containerEngine, @@ -1169,9 +1365,11 @@ async function qualifyExternalImage( }; } -export async function qualifyManagedImageActivation(fixtures: RuntimeFixtures): Promise { +export async function qualifyManagedImageActivation( + fixtures: RuntimeFixtures, + phases: ManagedActivationPhases, +): Promise { const { artifacts, cleanup, host, progress } = fixtures; - progress.phase("validate exact candidate catalog and host runtime"); const catalogPath = requiredCatalogPath(); const contracts = exactCatalog(catalogPath); const containerEngine: ContainerEngine = @@ -1182,12 +1380,15 @@ export async function qualifyManagedImageActivation(fixtures: RuntimeFixtures): guard.dispose, ); cleanup.trackGateway(host, GATEWAY, { env: guard.env, timeoutMs: 60_000 }); - const runtimeInfo = await host.command(containerEngine, ["info"], { + await host.command(containerEngine, ["info"], { artifactName: `managed-activation-${containerEngine}-info`, env: guard.env, timeoutMs: 30_000, }); - expect(runtimeInfo.exitCode, resultText(runtimeInfo)).toBe(0); + const inferenceHost = await discoverHostInterfaceAddress( + host, + "managed-activation-inference-host-address", + ); const inference = await startFakeOpenAiCompatibleServer({ apiKey: API_KEY, chatContent: "PONG", @@ -1195,7 +1396,7 @@ export async function qualifyManagedImageActivation(fixtures: RuntimeFixtures): model: MODEL, port: 8000, progress, - publicHost: "host.openshell.internal", + publicHost: inferenceHost.address, requireAuth: true, requireAuthModels: true, requestCanaryMarker: "NEMOCLAW_MANAGED_SUBAGENT", @@ -1221,13 +1422,12 @@ export async function qualifyManagedImageActivation(fixtures: RuntimeFixtures): inference.baseUrl, agent, contracts.get(agent)!, + phases, agent === "hermes" ? inference.requests : undefined, ); } - progress.phase( - "prove buildless external-image onboarding, drift rejection, rebuild, and retention", - ); + phases.externalImages(); const externalImages = []; for (const agent of externalImageActivationAgents(containerEngine)) { externalImages.push( diff --git a/test/e2e/live/managed-image-activation-e2e.test.ts b/test/e2e/live/managed-image-activation-e2e.test.ts index d7189b54547..e920c21702a 100644 --- a/test/e2e/live/managed-image-activation-e2e.test.ts +++ b/test/e2e/live/managed-image-activation-e2e.test.ts @@ -2,10 +2,38 @@ // SPDX-License-Identifier: Apache-2.0 import { test } from "../fixtures/e2e-test.ts"; -import { qualifyManagedImageActivation } from "./managed-image-activation-e2e-helpers.ts"; +import { + qualifyManagedImageActivation, + qualifyManagedVolumePreservation, +} from "./managed-image-activation-e2e-helpers.ts"; const TIMEOUT_MS = 75 * 60_000; +test( + "legacy managed state remains unchanged when attachment approval is missing (#12603)", + { + timeout: 10 * 60_000, + meta: { + e2ePhases: [ + "create owned legacy state on the selected container engine", + "reject legacy state without changing its content or metadata", + "record legacy state preservation evidence", + ], + }, + }, + async ({ artifacts, cleanup, progress }) => { + progress.phase("create owned legacy state on the selected container engine"); + await qualifyManagedVolumePreservation( + { artifacts, cleanup }, + { + refusal: () => + progress.phase("reject legacy state without changing its content or metadata"), + evidence: () => progress.phase("record legacy state preservation evidence"), + }, + ); + }, +); + test( "candidate CLI activates managed images without builds and the selected runtime adopts public agent digests (#7744, #11932, #12241)", { @@ -36,6 +64,37 @@ test( syntheticBoundary: "Only the OpenAI-compatible inference response is synthetic; runtime construction and agent execution are real.", }); - await qualifyManagedImageActivation({ artifacts, cleanup, host, lifecycle, progress, sandbox }); + progress.phase("validate exact candidate catalog and host runtime"); + await qualifyManagedImageActivation( + { artifacts, cleanup, host, lifecycle, progress, sandbox }, + { + agents: { + openclaw: { + onboard: () => progress.phase("onboard and exercise OpenClaw"), + publicLifecycle: () => + progress.phase("stop and start OpenClaw through public NemoClaw lifecycle"), + cleanup: () => progress.phase("destroy and verify OpenClaw cleanup"), + }, + hermes: { + onboard: () => progress.phase("onboard and exercise Hermes"), + publicLifecycle: () => + progress.phase("stop and start Hermes through public NemoClaw lifecycle"), + cleanup: () => progress.phase("destroy and verify Hermes cleanup"), + }, + "langchain-deepagents-code": { + onboard: () => progress.phase("onboard and exercise Deep Agents Code"), + publicLifecycle: () => + progress.phase("stop and start Deep Agents Code through public NemoClaw lifecycle"), + cleanup: () => progress.phase("destroy and verify Deep Agents Code cleanup"), + }, + }, + hermesSecretBoundary: () => + progress.phase("prove Hermes secret-boundary refusal before native restart"), + externalImages: () => + progress.phase( + "prove buildless external-image onboarding, drift rejection, rebuild, and retention", + ), + }, + ); }, ); diff --git a/test/e2e/live/managed-image-protected-runtime-budget.ts b/test/e2e/live/managed-image-protected-runtime-budget.ts deleted file mode 100644 index be829ad7a53..00000000000 --- a/test/e2e/live/managed-image-protected-runtime-budget.ts +++ /dev/null @@ -1,66 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -import { HostCliClient } from "../fixtures/clients/host.ts"; - -const MINUTE = 60_000; -export const PROTECTED_RUNTIME_JOB_MINUTES = 90; -export const PROTECTED_RUNTIME_CLEANUP_MS = 5 * MINUTE; -export const PROTECTED_RUNTIME_EVIDENCE_MS = 8 * MINUTE; -export const PROTECTED_RUNTIME_RUNNER_HEADROOM_MS = 2 * MINUTE; -export const PROTECTED_RUNTIME_ROLLBACK_MS = 15 * MINUTE; -const BODY_MS = - PROTECTED_RUNTIME_JOB_MINUTES * MINUTE - - PROTECTED_RUNTIME_CLEANUP_MS - - PROTECTED_RUNTIME_EVIDENCE_MS - - PROTECTED_RUNTIME_RUNNER_HEADROOM_MS; - -export function protectedRuntimeDeadlines(startedAt: string | undefined, now = Date.now()) { - const start = Number(startedAt); - if (!Number.isSafeInteger(start) || start <= 0 || start > now) { - throw new Error("protected runtime job start must be a past epoch millisecond timestamp"); - } - if (now >= start + BODY_MS - PROTECTED_RUNTIME_ROLLBACK_MS) { - throw new Error( - "protected runtime setup exhausted the provider budget; qualification was not started", - ); - } - return { - providers: start + BODY_MS - PROTECTED_RUNTIME_ROLLBACK_MS, - rollback: start + BODY_MS, - testTimeoutMs: Math.max(1, start + BODY_MS - now), - }; -} - -// Preserve each command's failure guard while sharing the suite's remaining -// wall-clock budget. The underlying ShellProbe owns process-group termination -// and artifact capture; cleanup uses the original host with its own signal. -export function protectedRuntimeDeadlineHost( - host: Pick, - deadline: number, - now = Date.now, -): HostCliClient { - return new HostCliClient({ - async run(command, options = {}) { - const remaining = deadline - now(); - // Leave the existing supervisor's termination grace inside the deadline. - const killGraceMs = options.killGraceMs ?? 5_000; - if (remaining <= killGraceMs) { - throw new Error( - "protected runtime suite deadline exhausted; reserved cleanup and evidence remain", - ); - } - const result = await host.command(command.command, [...command.args], { - ...options, - killGraceMs, - timeoutMs: Math.min(options.timeoutMs ?? 60_000, remaining - killGraceMs), - }); - if (result.timedOut || now() >= deadline) { - throw new Error( - `protected runtime command exceeded its budget; see ${result.artifacts.result}`, - ); - } - return result; - }, - }); -} diff --git a/test/e2e/live/managed-image-protected-runtime-helpers.ts b/test/e2e/live/managed-image-protected-runtime-helpers.ts index c4aa05dce49..c44fb958efe 100644 --- a/test/e2e/live/managed-image-protected-runtime-helpers.ts +++ b/test/e2e/live/managed-image-protected-runtime-helpers.ts @@ -21,7 +21,7 @@ import { startOllamaAuthProxy, } from "../../../src/lib/inference/ollama/proxy.ts"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; -import type { HostCliClient } from "../fixtures/clients/host.ts"; +import { HostCliClient } from "../fixtures/clients/host.ts"; import { resultText } from "../fixtures/clients/index.ts"; import type { E2ETargetFixtures } from "../fixtures/e2e-test.ts"; import { expect } from "../fixtures/e2e-test.ts"; @@ -37,10 +37,67 @@ import { REPO_ROOT, } from "./gpu-e2e-helpers.ts"; -import { - protectedRuntimeDeadlineHost, - protectedRuntimeDeadlines, -} from "./managed-image-protected-runtime-budget.ts"; +const MINUTE = 60_000; +export const PROTECTED_RUNTIME_JOB_MINUTES = 90; +export const PROTECTED_RUNTIME_CLEANUP_MS = 5 * MINUTE; +export const PROTECTED_RUNTIME_EVIDENCE_MS = 8 * MINUTE; +export const PROTECTED_RUNTIME_RUNNER_HEADROOM_MS = 2 * MINUTE; +export const PROTECTED_RUNTIME_ROLLBACK_MS = 15 * MINUTE; +const BODY_MS = + PROTECTED_RUNTIME_JOB_MINUTES * MINUTE - + PROTECTED_RUNTIME_CLEANUP_MS - + PROTECTED_RUNTIME_EVIDENCE_MS - + PROTECTED_RUNTIME_RUNNER_HEADROOM_MS; + +export function protectedRuntimeDeadlines(startedAt: string | undefined, now = Date.now()) { + const start = Number(startedAt); + if (!Number.isSafeInteger(start) || start <= 0 || start > now) { + throw new Error("protected runtime job start must be a past epoch millisecond timestamp"); + } + if (now >= start + BODY_MS - PROTECTED_RUNTIME_ROLLBACK_MS) { + throw new Error( + "protected runtime setup exhausted the provider budget; qualification was not started", + ); + } + return { + providers: start + BODY_MS - PROTECTED_RUNTIME_ROLLBACK_MS, + rollback: start + BODY_MS, + testTimeoutMs: Math.max(1, start + BODY_MS - now), + }; +} + +// Preserve each command's failure guard while sharing the suite's remaining +// wall-clock budget. The underlying ShellProbe owns process-group termination +// and artifact capture; cleanup uses the original host with its own signal. +export function protectedRuntimeDeadlineHost( + host: Pick, + deadline: number, + now = Date.now, +): HostCliClient { + return new HostCliClient({ + async run(command, options = {}) { + const remaining = deadline - now(); + // Leave the existing supervisor's termination grace inside the deadline. + const killGraceMs = options.killGraceMs ?? 5_000; + if (remaining <= killGraceMs) { + throw new Error( + "protected runtime suite deadline exhausted; reserved cleanup and evidence remain", + ); + } + const result = await host.command(command.command, [...command.args], { + ...options, + killGraceMs, + timeoutMs: Math.min(options.timeoutMs ?? 60_000, remaining - killGraceMs), + }); + if (result.timedOut || now() >= deadline) { + throw new Error( + `protected runtime command exceeded its budget; see ${result.artifacts.result}`, + ); + } + return result; + }, + }); +} const OLLAMA_MODEL = "qwen3.5:9b"; const VLLM_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"; @@ -92,26 +149,14 @@ export function protectedProviderContainerName( if (!PROTECTED_MANAGED_IMAGE_COHORT_PATTERN.test(cohort)) { throw new Error("NEMOCLAW_PROTECTED_MANAGED_IMAGE_COHORT has an invalid protected cohort"); } - const name = `nemoclaw-mi-${kind}-${cohort}`; - if ( - name.length > PROTECTED_PROVIDER_CONTAINER_MAX_LENGTH || - !/^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$/u.test(name) - ) { - throw new Error("protected provider container name is outside the Docker name contract"); - } - return name; + // The fixed provider kinds and bounded cohort yield at most 58 Docker-safe characters. + return `nemoclaw-mi-${kind}-${cohort}`; } -export function protectedProviderReportedContainerId(name: string, stdout: string): string { - if ( - name.length > PROTECTED_PROVIDER_CONTAINER_MAX_LENGTH || - !/^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$/u.test(name) - ) { - throw new Error("protected provider container name is outside the Docker name contract"); - } +export function protectedProviderReportedContainerId(stdout: string): string { const candidate = stdout.trim(); if (!/^[a-f0-9]{64}$/u.test(candidate)) { - throw new Error(`provider start did not report one full container ID for ${name}`); + throw new Error("provider start did not report one full container ID"); } return candidate; } @@ -397,19 +442,7 @@ export function protectedOllamaReadinessCommand(logPath: string): ProtectedRunti return { command: "bash", captureLimitBytes: PROTECTED_READINESS_CAPTURE_LIMIT_BYTES, - args: [ - "-c", - `( -${startScript} -) -status=$? -if [ "$status" -eq 0 ]; then - printf 'managed-image-ollama-ready\n' -else - printf 'managed-image-ollama-not-ready status=%s\n' "$status" >&2 -fi -exit "$status"`, - ], + args: ["-c", startScript], }; } @@ -621,7 +654,7 @@ async function startProtectedVllm( }, ); expect(start.exitCode, resultText(start)).toBe(0); - state.reportedContainerId = protectedProviderReportedContainerId(state.name, start.stdout); + state.reportedContainerId = protectedProviderReportedContainerId(start.stdout); state.authority = await inspectProtectedProviderContainer( host, state, @@ -754,8 +787,6 @@ async function startProtectedNim( state: ProtectedProviderContainerState, apiKey: string, ): Promise { - const image = getImageForModel(NIM_CATALOG_MODEL); - if (!image) throw new Error("protected NIM catalog model has no image"); // NIM's production login and platform-aware pull are synchronous. Run them // in the supervised child so a slow registry cannot block the suite deadline. const prepare = await host.command( @@ -781,6 +812,8 @@ async function startProtectedNim( }, ); expect(prepare.exitCode, resultText(prepare)).toBe(0); + // pullNimImage in the successful child validated this same pinned catalog entry. + const image = getImageForModel(NIM_CATALOG_MODEL)!; const start = await host.command( "docker", [ @@ -818,7 +851,7 @@ async function startProtectedNim( }, ); expect(start.exitCode, resultText(start)).toBe(0); - state.reportedContainerId = protectedProviderReportedContainerId(state.name, start.stdout); + state.reportedContainerId = protectedProviderReportedContainerId(start.stdout); state.authority = await inspectProtectedProviderContainer( host, state, @@ -955,13 +988,9 @@ export function protectedProviderFinalInventoryCommand( async function proveOwnedRuntimeInventoryClean( host: HostCliClient, cohort: string, - providerStates: readonly ProtectedProviderContainerState[], + vllmName: string, + nimName: string, ): Promise { - const vllmName = providerStates.find(({ kind }) => kind === "vllm")?.name; - const nimName = providerStates.find(({ kind }) => kind === "nim")?.name; - if (!vllmName || !nimName) { - throw new Error("protected provider inventory requires vLLM and NIM container names"); - } const command = protectedProviderFinalInventoryCommand(cohort, vllmName, nimName); const result = await host.command(command.command, command.args, { artifactName: "final-managed-image-owned-runtime-inventory", @@ -1005,100 +1034,88 @@ export async function qualifyProtectedManagedImageRuntime( expect(result.exitCode, resultText(result)).toBe(0); }); - let activePhase = "validate protected host runtime"; - try { - const docker = await host.command("docker", ["info"], { - artifactName: "docker-info", - env: buildAvailabilityProbeEnv(), - timeoutMs: 30_000, - }); - expect(docker.exitCode, resultText(docker)).toBe(0); - const nvidia = await host.command("nvidia-smi", [], { - artifactName: "nvidia-smi", - env: buildAvailabilityProbeEnv(), - timeoutMs: 30_000, - }); - assertNvidiaAvailable(nvidia, (message) => { - throw new Error(message ?? "protected GPU runner is unavailable"); - }); - await assertProtectedProviderContainerAbsent(host, vllmState, "preflight-vllm-container"); - await assertProtectedProviderContainerAbsent(host, nimState, "preflight-nim-container"); + const docker = await host.command("docker", ["info"], { + artifactName: "docker-info", + env: buildAvailabilityProbeEnv(), + timeoutMs: 30_000, + }); + expect(docker.exitCode, resultText(docker)).toBe(0); + const nvidia = await host.command("nvidia-smi", [], { + artifactName: "nvidia-smi", + env: buildAvailabilityProbeEnv(), + timeoutMs: 30_000, + }); + assertNvidiaAvailable(nvidia, (message) => { + throw new Error(message ?? "protected GPU runner is unavailable"); + }); + await assertProtectedProviderContainerAbsent(host, vllmState, "preflight-vllm-container"); + await assertProtectedProviderContainerAbsent(host, nimState, "preflight-nim-container"); - activePhase = "qualify all managed agents with GPU-backed Ollama"; - progress.phase("qualify all managed agents with GPU-backed Ollama"); - const proxyToken = await startProtectedOllama(host); - await qualifyEveryAgent(host, contracts, "ollama", OLLAMA_MODEL, { - NEMOCLAW_OLLAMA_PROXY_TOKEN: proxyToken, - }); - await proveOllamaGpuPlacement(host); - killStaleProxy(); - const ollamaCleanup = await cleanupOllama(host, "stop-ollama-before-vllm"); - expect(ollamaCleanup.exitCode, resultText(ollamaCleanup)).toBe(0); + progress.phase("qualify all managed agents with GPU-backed Ollama"); + const proxyToken = await startProtectedOllama(host); + await qualifyEveryAgent(host, contracts, "ollama", OLLAMA_MODEL, { + NEMOCLAW_OLLAMA_PROXY_TOKEN: proxyToken, + }); + await proveOllamaGpuPlacement(host); + killStaleProxy(); + const ollamaCleanup = await cleanupOllama(host, "stop-ollama-before-vllm"); + expect(ollamaCleanup.exitCode, resultText(ollamaCleanup)).toBe(0); - activePhase = "qualify all managed agents with GPU-backed vLLM"; - progress.phase("qualify all managed agents with GPU-backed vLLM"); - const vllmImagePulledByRun = await startProtectedVllm(host, vllmState); - activeRecorder = await startProtectedInferenceRecorder({ - bindHost: "0.0.0.0", - port: PROTECTED_PROVIDER_ROUTE_PORT, - backendPort: PROTECTED_PROVIDER_BACKEND_PORT, - model: VLLM_MODEL, - }); - await qualifyEveryAgent(host, contracts, "vllm", VLLM_MODEL, { - NEMOCLAW_VLLM_LOCAL_TOKEN: activeRecorder.token, - NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_PORT: String(activeRecorder.port), - NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_TOKEN: activeRecorder.token, + progress.phase("qualify all managed agents with GPU-backed vLLM"); + const vllmImagePulledByRun = await startProtectedVllm(host, vllmState); + activeRecorder = await startProtectedInferenceRecorder({ + bindHost: "0.0.0.0", + port: PROTECTED_PROVIDER_ROUTE_PORT, + backendPort: PROTECTED_PROVIDER_BACKEND_PORT, + model: VLLM_MODEL, + }); + await qualifyEveryAgent(host, contracts, "vllm", VLLM_MODEL, { + NEMOCLAW_VLLM_LOCAL_TOKEN: activeRecorder.token, + NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_PORT: String(activeRecorder.port), + NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_TOKEN: activeRecorder.token, + }); + await activeRecorder.close(); + activeRecorder = null; + await removeProtectedProviderContainer(host, vllmState, "stop-vllm-before-nim"); + if (vllmImagePulledByRun) { + // Free the vLLM image this run pulled before NIM downloads its model. This + // frees space when the image store and NIM cache share a filesystem (#13135). + // The artifact records the result; NIM readiness gates. + await host.command("docker", ["image", "rm", VLLM_IMAGE], { + artifactName: "remove-run-pulled-vllm-image", + env: buildAvailabilityProbeEnv(), + timeoutMs: 120_000, }); - await activeRecorder.close(); - activeRecorder = null; - await removeProtectedProviderContainer(host, vllmState, "stop-vllm-before-nim"); - if (vllmImagePulledByRun) { - // Free the vLLM image this run pulled before NIM downloads its model. This - // frees space when the image store and NIM cache share a filesystem (#13135). - // The artifact records the result; NIM readiness gates. - await host.command("docker", ["image", "rm", VLLM_IMAGE], { - artifactName: "remove-run-pulled-vllm-image", - env: buildAvailabilityProbeEnv(), - timeoutMs: 120_000, - }); - } + } - activePhase = "qualify all managed agents with GPU-backed NVIDIA NIM"; - progress.phase("qualify all managed agents with GPU-backed NVIDIA NIM"); - const nimModel = await startProtectedNim(host, nimState, ngcApiKey); - activeRecorder = await startProtectedInferenceRecorder({ - bindHost: "0.0.0.0", - port: PROTECTED_PROVIDER_ROUTE_PORT, - backendPort: PROTECTED_PROVIDER_BACKEND_PORT, - model: nimModel, - }); - await qualifyEveryAgent(host, contracts, "nim", nimModel, { - NEMOCLAW_VLLM_LOCAL_TOKEN: activeRecorder.token, - NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_PORT: String(activeRecorder.port), - NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_TOKEN: activeRecorder.token, - }); - await activeRecorder.close(); - activeRecorder = null; - await removeProtectedProviderContainer(host, nimState, "stop-nim-before-rollback"); + progress.phase("qualify all managed agents with GPU-backed NVIDIA NIM"); + const nimModel = await startProtectedNim(host, nimState, ngcApiKey); + activeRecorder = await startProtectedInferenceRecorder({ + bindHost: "0.0.0.0", + port: PROTECTED_PROVIDER_ROUTE_PORT, + backendPort: PROTECTED_PROVIDER_BACKEND_PORT, + model: nimModel, + }); + await qualifyEveryAgent(host, contracts, "nim", nimModel, { + NEMOCLAW_VLLM_LOCAL_TOKEN: activeRecorder.token, + NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_PORT: String(activeRecorder.port), + NEMOCLAW_PROTECTED_INFERENCE_RECEIPT_TOKEN: activeRecorder.token, + }); + await activeRecorder.close(); + activeRecorder = null; + await removeProtectedProviderContainer(host, nimState, "stop-nim-before-rollback"); - activePhase = "prove all-agent managed bootstrap rollback and exact cleanup"; - progress.phase("prove all-agent managed bootstrap rollback and exact cleanup"); - await qualifyEveryRollback(rollbackHost, contracts); - await proveOwnedRuntimeInventoryClean(rollbackHost, cohort, providerStates); - const providerContainers = providerStates.map(({ authority }) => { - if (!authority) throw new Error("protected provider authority is missing from the receipt"); - return authority; - }); - await artifacts.writeJson("managed-image-protected-runtime-summary.json", { - agents: PROTECTED_MANAGED_IMAGE_AGENTS, - providerContainers, - providers: ["ollama", "vllm", "nim"], - rollbackAgents: PROTECTED_MANAGED_IMAGE_AGENTS, - }); - } catch (error) { - const detail = error instanceof Error ? error.message : String(error); - throw new Error(`protected managed-image runtime phase '${activePhase}' failed: ${detail}`, { - cause: error, - }); - } + progress.phase("prove all-agent managed bootstrap rollback and exact cleanup"); + await qualifyEveryRollback(rollbackHost, contracts); + await proveOwnedRuntimeInventoryClean(rollbackHost, cohort, vllmState.name, nimState.name); + const providerContainers = providerStates.map(({ authority }) => { + if (!authority) throw new Error("protected provider authority is missing from the receipt"); + return authority; + }); + await artifacts.writeJson("managed-image-protected-runtime-summary.json", { + agents: PROTECTED_MANAGED_IMAGE_AGENTS, + providerContainers, + providers: ["ollama", "vllm", "nim"], + rollbackAgents: PROTECTED_MANAGED_IMAGE_AGENTS, + }); } diff --git a/test/e2e/live/managed-image-protected-runtime.test.ts b/test/e2e/live/managed-image-protected-runtime.test.ts index 7911a67cc4d..0fbc5a94e83 100644 --- a/test/e2e/live/managed-image-protected-runtime.test.ts +++ b/test/e2e/live/managed-image-protected-runtime.test.ts @@ -8,7 +8,7 @@ import { qualifyProtectedManagedImageRuntime } from "./managed-image-protected-r import { protectedRuntimeDeadlines, PROTECTED_RUNTIME_CLEANUP_MS, -} from "./managed-image-protected-runtime-budget.ts"; +} from "./managed-image-protected-runtime-helpers.ts"; // Discovery runs outside the protected workflow. Execution below still requires // its real start timestamp before declaring or mutating a fixture. diff --git a/test/e2e/live/mcp-bridge-hermes-http.ts b/test/e2e/live/mcp-bridge-hermes-http.ts index fd3948b38ff..a2c54f983d0 100644 --- a/test/e2e/live/mcp-bridge-hermes-http.ts +++ b/test/e2e/live/mcp-bridge-hermes-http.ts @@ -35,13 +35,21 @@ export const HERMES_MCP_RESPONSE_FILE_BYTES = 65_536; export interface HermesMcpCommandResult { exitCode: number | null; - signal?: NodeJS.Signals | null; + signal: NodeJS.Signals | null; + timedOut: boolean; stdout: string; stderr: string; } export function isHermesGatewayDrainingResponse(result: HermesMcpCommandResult): boolean { - if (result.exitCode !== 0 || parseHttpStatus(result.stderr) !== 503) return false; + if ( + result.exitCode !== 0 || + result.signal !== null || + result.timedOut || + parseHttpStatus(result.stderr) !== 503 + ) { + return false; + } try { const body = JSON.parse(result.stdout) as { error?: { code?: unknown } }; return body.error?.code === "gateway_draining"; @@ -126,10 +134,12 @@ export function assertHermesMcpHttpResponse( explicitRedactionValues: Iterable, ): void { const status = parseHttpStatus(result.stderr); - if (result.exitCode !== 0) { - const transport = result.signal - ? `signal=${result.signal}` - : `exit=${result.exitCode ?? "unknown"}`; + if (result.exitCode !== 0 || result.signal !== null || result.timedOut) { + const transport = result.timedOut + ? "timeout" + : result.signal + ? `signal=${result.signal}` + : `exit=${result.exitCode ?? "unknown"}`; const detail = sanitizedPreview([result.stdout, result.stderr].filter(Boolean).join("\n"), [ ...explicitRedactionValues, ]); diff --git a/test/e2e/live/mcp-bridge-sandbox.ts b/test/e2e/live/mcp-bridge-sandbox.ts index b5f86593468..9235ae32eb3 100644 --- a/test/e2e/live/mcp-bridge-sandbox.ts +++ b/test/e2e/live/mcp-bridge-sandbox.ts @@ -12,8 +12,42 @@ import { discoverHostAddress } from "../fixtures/host-address.ts"; import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; +import { + buildMcpProviderRewriteAuthorization, + buildMcpCredentialHandleAuthorizationPattern, +} from "./mcp-provider-rewrite-probe.ts"; + const MCP_CURL_HTTP_CODE_MARKER = "NEMOCLAW_MCP_CURL_HTTP_CODE="; +export function buildMcpSecretAbsenceScript( + paths: readonly string[], + secrets: readonly string[], +): string { + if (paths.length === 0 || secrets.length === 0 || secrets.some((secret) => secret.length === 0)) { + throw new Error("MCP secret scan requires paths and nonempty credentials"); + } + return [ + "set -eu", + `for scan_path in ${paths.map(shellQuote).join(" ")}; do`, + ' if [ ! -e "$scan_path" ] && [ ! -L "$scan_path" ]; then', + ' scan_parent=$(dirname -- "$scan_path")', + ' [ -d "$scan_parent" ] && [ -r "$scan_parent" ] && [ -x "$scan_parent" ] || exit 2', + " continue", + " fi", + ' [ -e "$scan_path" ] || exit 2', + ` for scan_secret in ${secrets.map(shellQuote).join(" ")}; do`, + // Do not use grep -q: a match must not suppress a read error elsewhere in the tree. + ' if grep -RF -- "$scan_secret" "$scan_path" >/dev/null 2>&1; then', + " exit 1", + " else", + " scan_status=$?", + ' [ "$scan_status" -eq 1 ] || exit 2', + " fi", + " done", + "done", + ].join("\n"); +} + export type McpDnsRebindingAdapter = "openclaw-config" | "hermes-config" | "deepagents-config"; export type CapturedManagedMcpPolicy = { @@ -77,10 +111,7 @@ export async function captureManagedMcpPolicy( }; const networkPolicies = document.network_policies ?? {}; const policy = networkPolicies[options.policyKey]; - if (!policy) { - throw new Error(`${options.label}: managed MCP policy '${options.policyKey}' is absent`); - } - const endpoint = policy.endpoints?.[0]; + const endpoint = policy?.endpoints?.[0]; const expectedHost = new URL(options.url).hostname; if (endpoint?.host !== expectedHost) { throw new Error(`${options.label}: expected managed MCP host '${expectedHost}'`); @@ -128,51 +159,36 @@ export { setupDnsRebindingHostsFixture, } from "./dns-rebinding-hosts-fixture.ts"; -/** - * Accept the two fail-closed shapes OpenShell can expose for a denied HTTPS - * request: an L7 HTTP 403, or curl's exit 56 for a CONNECT-level proxy 403. - */ -export function isExpectedMcpCurlPolicyDenial( - result: Pick, -): boolean { - if (result.timedOut) return false; - - const httpCode = result.stdout.match( - new RegExp(`^${MCP_CURL_HTTP_CODE_MARKER}([0-9]{3})$`, "m"), - )?.[1]; - if (result.exitCode === 0) return httpCode === "403"; - - return ( - result.exitCode === 56 && - /curl:\s*\(56\)\s*CONNECT tunnel failed,\s*response 403/i.test(result.stderr) - ); -} +export { isExpectedMcpCurlPolicyDenial } from "../fixtures/curl-policy-denial.ts"; -/** - * Build an MCP request whose curl child retains the selected adapter runtime - * as an ancestor. OpenShell v0.0.106 attributes policy to /proc//exe and - * ancestors, so this exercises the same unavoidable Node/Python identity used - * by the corresponding adapter instead of an unrelated curl-only identity. - * - * Pinned upstream source contract: - * NVIDIA/OpenShell@c4b500a7de64d0b66e3ee8098f58d14299092162, - * crates/openshell-supervisor-network/src/proxy.rs:3070-3096 resolves once, - * proxy.rs:3121-3160 validates and retains that same address list, and - * proxy.rs:3193-3251 dials those addresses on the direct path used here. - */ +/** After removal, absent credentials cannot hide network denial: only policy denial evidence counts. + * A null adapter keeps curl alone; otherwise preserve the adapter's runtime ancestry. */ export function buildMcpDnsRebindingProbeScript( - adapter: McpDnsRebindingAdapter, + adapter: McpDnsRebindingAdapter | null, targetUrl: string, credentialKey: string, + afterRemoval = false, ): string { - const fileStem = `/tmp/nemoclaw-mcp-rebinding-${adapter}`; + const fileStem = `/tmp/nemoclaw-mcp-rebinding-${adapter ?? "curl-only"}`; const responsePath = `${fileStem}.body`; const stdoutPath = `${fileStem}.stdout`; const stderrPath = `${fileStem}.stderr`; - const body = JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/list" }); + // One initialize proves authenticated reachability without inventing a session. + // The caller separately verifies full tool discovery before this control. + const body = JSON.stringify({ + jsonrpc: "2.0", + id: 1, + method: "initialize", + params: { + protocolVersion: "2025-11-25", + capabilities: {}, + clientInfo: { name: "rebinding-probe", version: "1" }, + }, + }); const curlArgs = [ "curl", "-sS", + "--verbose", "--max-time", "30", "-o", @@ -185,26 +201,27 @@ export function buildMcpDnsRebindingProbeScript( "-H", "content-type: application/json", "-H", - `authorization: Bearer openshell:resolve:env:${credentialKey}`, + adapter === null + ? `authorization: Bearer openshell:resolve:env:${credentialKey}` + : "NEMOCLAW_MCP_AUTHORIZATION", "--data-binary", body, ]; const quotedCurl = curlArgs.map(shellQuote).join(" "); const runtimeCommand = (() => { switch (adapter) { + case null: + return `LC_ALL=C ${quotedCurl}`; case "openclaw-config": { - const runner = - 'const { spawnSync } = require("node:child_process"); const result = spawnSync(process.argv[1], process.argv.slice(2), { stdio: "inherit" }); process.exit(result.status ?? 1);'; + const runner = `const { spawnSync } = require("node:child_process"); const authorize = ${buildMcpProviderRewriteAuthorization.toString()}; const value = process.env[${JSON.stringify(credentialKey)}]; const header = value === undefined && ${afterRemoval} ? ${JSON.stringify(`Bearer openshell:resolve:env:${credentialKey}`)} : authorize(${JSON.stringify(credentialKey)}, value); if (!header) process.exit(2); const args = process.argv.slice(2).map(value => value === "NEMOCLAW_MCP_AUTHORIZATION" ? "authorization: " + header : value); const result = spawnSync(process.argv[1], args, { stdio: "inherit", env: { ...process.env, LC_ALL: "C" } }); process.exit(result.status ?? 1);`; return `nemoclaw-start node -e ${shellQuote(runner)} ${quotedCurl}`; } case "hermes-config": { - const runner = - "import subprocess, sys; raise SystemExit(subprocess.run(sys.argv[1:], check=False).returncode)"; + const runner = `import os, re, subprocess, sys; key = ${JSON.stringify(credentialKey)}; value = os.environ.get(key); header = "Bearer " + (value or ""); (re.fullmatch(${JSON.stringify(buildMcpCredentialHandleAuthorizationPattern(credentialKey))}, header) or (value is None and ${afterRemoval ? "True" : "False"})) or sys.exit(2); header = "Bearer openshell:resolve:env:" + key if value is None else header; args = ["authorization: " + header if value == "NEMOCLAW_MCP_AUTHORIZATION" else value for value in sys.argv[1:]]; raise SystemExit(subprocess.run(args, check=False, env={**os.environ, "LC_ALL": "C"}).returncode)`; return `/opt/hermes/.venv/bin/python -c ${shellQuote(runner)} ${quotedCurl}`; } case "deepagents-config": { - const runner = - "import subprocess, sys; raise SystemExit(subprocess.run(sys.argv[1:], check=False).returncode)"; + const runner = `import os, re, subprocess, sys; key = ${JSON.stringify(credentialKey)}; value = os.environ.get(key); header = "Bearer " + (value or ""); (re.fullmatch(${JSON.stringify(buildMcpCredentialHandleAuthorizationPattern(credentialKey))}, header) or (value is None and ${afterRemoval ? "True" : "False"})) or sys.exit(2); header = "Bearer openshell:resolve:env:" + key if value is None else header; args = ["authorization: " + header if value == "NEMOCLAW_MCP_AUTHORIZATION" else value for value in sys.argv[1:]]; raise SystemExit(subprocess.run(args, check=False, env={**os.environ, "LC_ALL": "C"}).returncode)`; return `/opt/venv/bin/python3 -c ${shellQuote(runner)} ${quotedCurl}`; } } @@ -219,7 +236,7 @@ export function buildMcpDnsRebindingProbeScript( "set -e", `cat ${shellQuote(responsePath)} 2>/dev/null || true`, `cat ${shellQuote(stdoutPath)} 2>/dev/null || true`, - `cat ${shellQuote(stderrPath)} >&2 2>/dev/null || true`, + `sed '/^> /d' ${shellQuote(stderrPath)} >&2 2>/dev/null || true`, 'exit "$probe_rc"', ].join("\n"); } diff --git a/test/e2e/live/mcp-bridge-servers.ts b/test/e2e/live/mcp-bridge-servers.ts index 12148027c2b..c01be24f35e 100644 --- a/test/e2e/live/mcp-bridge-servers.ts +++ b/test/e2e/live/mcp-bridge-servers.ts @@ -8,6 +8,7 @@ import http from "node:http"; import https from "node:https"; import os from "node:os"; +import type { ArtifactSink } from "../fixtures/artifacts.ts"; import type { CleanupRegistry } from "../fixtures/cleanup.ts"; import { closeServer, @@ -26,6 +27,58 @@ export interface StartedHttpServer { close(): Promise; } +interface CompatibleRequestDiagnostic { + receivedAtMs: number; + method: "GET" | "POST" | "other"; + route: "models" | "chat-completions" | "responses" | "other"; + authenticated: boolean; + bodyComplete: boolean; + stream: boolean | null; + responseStatus: number | null; + responseFinished: boolean; + responseClosed: boolean; +} + +export interface CompatibleMockServer extends StartedHttpServer { + diagnostics(): { + receivedCount: number; + retainedLimit: number; + requests: CompatibleRequestDiagnostic[]; + }; +} + +export async function withCompatibleMockFailureDiagnostics( + server: Pick, + artifacts: Pick, + operation: () => Promise, +): Promise { + const adminApprovalStartedAtMs = Date.now(); + const call = Promise.resolve().then(operation); + await call.catch(async () => { + let deadline: ReturnType | undefined; + try { + await Promise.allSettled([ + Promise.race([ + Promise.resolve().then(() => + artifacts.writeJson("admin-approval-compatible-requests.json", { + adminApprovalStartedAtMs, + capturedAtMs: Date.now(), + ...server.diagnostics(), + }), + ), + new Promise((resolve) => { + deadline = setTimeout(resolve, MCP_DIAGNOSTIC_PERSIST_TIMEOUT_MS); + }), + ]), + ]); + } finally { + clearTimeout(deadline); + } + }); + // Preserve the original approval result even when diagnostic publication fails. + await call; +} + export const FAKE_MCP_STATUS_RESULT_TOKEN = "MCP_STATUS_OK"; const MCP_DIAGNOSTIC_PERSIST_TIMEOUT_MS = 10_000; @@ -542,11 +595,45 @@ export async function startCompatibleMock(options: { resultToken: string; toolName: string; }; -}): Promise { +}): Promise { let selectedOpenClawToolName: string | undefined; + let receivedCount = 0; + const requestDiagnostics: CompatibleRequestDiagnostic[] = []; const server = http.createServer(async (req, res) => { const requestPath = new URL(req.url ?? "/", "http://compatible.mock").pathname; const auth = req.headers.authorization === `Bearer ${options.apiKey}`; + // Keep only fixed transport facts, never URLs, headers, prompts or response bodies. + const diagnostic: CompatibleRequestDiagnostic = { + receivedAtMs: Date.now(), + method: req.method === "GET" || req.method === "POST" ? req.method : "other", + route: ["/models", "/v1/models"].includes(requestPath) + ? "models" + : ["/chat/completions", "/v1/chat/completions"].includes(requestPath) + ? "chat-completions" + : ["/responses", "/v1/responses"].includes(requestPath) + ? "responses" + : "other", + authenticated: auth, + bodyComplete: req.complete, + stream: null, + responseStatus: null, + responseFinished: false, + responseClosed: false, + }; + receivedCount = Math.min(receivedCount + 1, 65_535); + requestDiagnostics.push(diagnostic); + if (requestDiagnostics.length > 64) requestDiagnostics.shift(); + req.once("end", () => { + diagnostic.bodyComplete = true; + }); + res.once("finish", () => { + diagnostic.responseStatus = res.statusCode; + diagnostic.responseFinished = true; + }); + res.once("close", () => { + diagnostic.responseStatus = res.headersSent ? res.statusCode : null; + diagnostic.responseClosed = true; + }); if (!auth) { jsonResponse(res, 401, { error: { message: "missing bearer credential" } }); return; @@ -569,6 +656,7 @@ export async function startCompatibleMock(options: { messages?: Array<{ role?: string; content?: unknown; tool_call_id?: string }>; tools?: Array<{ function?: { name?: string } }>; }; + diagnostic.stream = Boolean(body.stream); const visibleToolNames = new Set( (body.tools ?? []) .map((tool) => tool.function?.name) @@ -1067,6 +1155,11 @@ export async function startCompatibleMock(options: { return { port, close: () => closeServer(server), + diagnostics: () => ({ + receivedCount, + retainedLimit: 64, + requests: requestDiagnostics.map((request) => ({ ...request })), + }), }; } diff --git a/test/e2e/live/mcp-bridge-trusted-private.ts b/test/e2e/live/mcp-bridge-trusted-private.ts index e0c4a47f2fb..e4ae0266f96 100644 --- a/test/e2e/live/mcp-bridge-trusted-private.ts +++ b/test/e2e/live/mcp-bridge-trusted-private.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { waitForMcpCurlPolicyDenial } from "../fixtures/curl-policy-denial.ts"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import { MCP_PROBE_CONTROL_BEARER } from "../../../src/lib/actions/sandbox/mcp-bridge-resolution-probe.ts"; import type { ArtifactSink } from "../fixtures/artifacts.ts"; @@ -16,7 +17,6 @@ import { buildMcpDnsRebindingProbeScript, captureManagedMcpPolicy, hostPrivateAddressForSandbox, - isExpectedMcpCurlPolicyDenial, type McpDnsRebindingAdapter, remapDnsRebindingHostname, restoreDnsRebindingHostsFixture, @@ -106,7 +106,6 @@ export async function assertTrustedPrivateMcpRebindingDenied( }); const survivingPolicyBeforeAdd = survivingPolicyBeforeAddResult.policy; const trustedPrivateAddress = relay.address; - expect(trustedPrivateAddress).not.toBe(REBIND_PUBLIC_IP); await remapDnsRebindingHostname( host, options.sandboxName, @@ -214,7 +213,6 @@ export async function assertTrustedPrivateMcpRebindingDenied( url: rebindMcpUrl, }); expect(rebindingPolicy.policy.endpoints?.[0]).toMatchObject({ - host: REBIND_HOSTNAME, allowed_ips: [trustedPrivateAddress], }); await options.assertSecretAbsent( @@ -224,6 +222,19 @@ export async function assertTrustedPrivateMcpRebindingDenied( [REBIND_HOST_SECRET], `${options.artifactPrefix}-dns-rebinding-secret-absent-from-sandbox`, ); + const requestsBeforeControl = rebindMcp.requests.length; + const curlControl = await sandbox.execShell( + options.sandboxName, + trustedSandboxShellScript( + buildMcpDnsRebindingProbeScript(options.adapter, rebindMcpUrl, REBIND_CREDENTIAL_KEY), + ), + { + artifactName: `${options.artifactPrefix}-mcp-adapter-curl-control`, + env: buildAvailabilityProbeEnv(), + redactionValues: [REBIND_HOST_SECRET], + timeoutMs: 90_000, + }, + ); const requestsBeforeRebindingDenial = rebindMcp.requests.length; // OpenShell must retain the exact private address admitted at add time. A @@ -255,6 +266,17 @@ export async function assertTrustedPrivateMcpRebindingDenied( }, }); expect(driftInspection.trustedPrivateTarget.currentPins).toBeUndefined(); + const readDnsAudit = (suffix: string) => + sandbox.openshell( + ["logs", options.sandboxName, "-n", "500", "--source", "sandbox", "--since", "2m"], + { + artifactName: `${options.artifactPrefix}-mcp-rebinding-dns-audit-${suffix}`, + env: buildAvailabilityProbeEnv(), + redactionValues: [REBIND_HOST_SECRET], + timeoutMs: 30_000, + }, + ); + const dnsAuditBefore = await readDnsAudit("before"); const denial = await sandbox.execShell( options.sandboxName, trustedSandboxShellScript( @@ -267,8 +289,20 @@ export async function assertTrustedPrivateMcpRebindingDenied( timeoutMs: 90_000, }, ); + const policyDenied = await waitForMcpCurlPolicyDenial( + denial, + dnsAuditBefore, + readDnsAudit, + REBIND_HOSTNAME, + ); expect( - isExpectedMcpCurlPolicyDenial(denial), + policyDenied && + hostsFixture.server.answers() > 0 && + curlControl.exitCode === 0 && + !curlControl.timedOut && + curlControl.stdout.includes("NEMOCLAW_MCP_CURL_HTTP_CODE=200") && + requestsBeforeRebindingDenial === requestsBeforeControl + 1 && + rebindMcp.requests.at(-1)?.auth === `Bearer ${REBIND_HOST_SECRET}`, `${options.artifactPrefix} adapter identity must receive an OpenShell policy denial after rebinding\nstdout:\n${denial.stdout}\nstderr:\n${denial.stderr}`, ).toBe(true); expect( diff --git a/test/e2e/live/mcp-bridge.test.ts b/test/e2e/live/mcp-bridge.test.ts index b166e73e5b0..7a75b43e311 100644 --- a/test/e2e/live/mcp-bridge.test.ts +++ b/test/e2e/live/mcp-bridge.test.ts @@ -78,6 +78,7 @@ import { } from "./mcp-bridge-reliability.ts"; import { buildMcpDnsRebindingProbeScript, + buildMcpSecretAbsenceScript, expectExitNonZero, hostAddressForSandbox, isExpectedMcpCurlPolicyDenial, @@ -85,6 +86,7 @@ import { } from "./mcp-bridge-sandbox.ts"; import { startCompatibleMock, + withCompatibleMockFailureDiagnostics, startFakeMcpHttpsServer, startPublicMcpHttpsTunnel, } from "./mcp-bridge-servers.ts"; @@ -181,12 +183,7 @@ async function assertSecretAbsentFromSandbox( secrets: string[] = [HOST_SECRET], artifactName = "assert-secret-absent-from-sandbox", ): Promise { - const script = [ - "set -eu", - ...secrets.map( - (secret) => `! grep -R ${JSON.stringify(secret)} ${paths.join(" ")} 2>/dev/null`, - ), - ].join("\n"); + const script = buildMcpSecretAbsenceScript(paths, secrets); const result = await sandbox.execShell(sandboxName, trustedSandboxShellScript(script), { artifactName, env: buildAvailabilityProbeEnv(), @@ -272,7 +269,6 @@ async function assertConcurrentAddSerialized( server: CONCURRENT_SERVER_NAME, }); const status = statusObservation.result; - expectExitZero(status, `${options.artifactPrefix} concurrent add leaves one coherent bridge`); expect(JSON.parse(status.stdout)).toMatchObject({ server: CONCURRENT_SERVER_NAME, url: redactString(options.mcpUrl, [HOST_SECRET]), @@ -478,7 +474,7 @@ async function assertAdapterRequestDeniedAfterRemove( const denial = await sandbox.execShell( options.sandboxName, trustedSandboxShellScript( - buildMcpDnsRebindingProbeScript(options.adapter, options.mcpUrl, "FAKE_MCP_SECRET"), + buildMcpDnsRebindingProbeScript(options.adapter, options.mcpUrl, "FAKE_MCP_SECRET", true), ), { artifactName: `${options.artifactPrefix}-mcp-adapter-request-denied-after-remove`, @@ -520,9 +516,7 @@ async function assertRealAdapterToolCall( messages: [{ role: "user", content: prompt }], max_tokens: 256, }); - // ShellProbe redacts these values before it returns command output or writes - // artifacts. The HTTP assertion redacts them again before Vitest formats a - // bounded failure preview. + // ShellProbe and the HTTP assertion both redact secrets before reporting failure. const hermesRedactionValues = [ HOST_SECRET, ROTATED_HOST_SECRET, @@ -724,15 +718,12 @@ test( ...bridge, artifactName: "onboard-openclaw-mcp-bridge", }); - await approveOpenClawAdminScope( - host, - sandbox, - OPENCLAW_SANDBOX_NAME, - buildAvailabilityProbeEnv(), - [COMPATIBLE_KEY, HOST_SECRET], + await withCompatibleMockFailureDiagnostics(compatibleMock, artifacts, () => + approveOpenClawAdminScope(host, sandbox, OPENCLAW_SANDBOX_NAME, buildAvailabilityProbeEnv(), [ + COMPATIBLE_KEY, + HOST_SECRET, + ]), ); - // Prove raw OpenShell allowed_ips with curl and a /** grant before MCP - // mutation. The helper restores the exact base policy before returning. await runFullMcpBridgeE2eCoverage(mcpBridgeE2eScope, () => assertRawOpenShellAllowedIpsRebindingDenied({ artifacts, @@ -814,7 +805,6 @@ test( expect(aliasState.providerAbsent).toBe(true); expect(aliasState.policyAbsent).toBe(true); expect(aliasState.adapterAbsent).toBe(true); - // Exercise both native adapters with both credentials and policy entries present. const distinctMcp = await startFakeMcpHttpsServer({ ...MCP_SERVER_OPTIONS, secret: ROTATED_HOST_SECRET, @@ -956,23 +946,22 @@ test( ); expect(decoyMcp.requests).toHaveLength(0); - const deniedCurl = await sandbox.execShell( + const beforeCurlControl = fakeMcp.requests.length; + const curlControl = await sandbox.execShell( OPENCLAW_SANDBOX_NAME, trustedSandboxShellScript( - [ - "set -eu", - `body='{"jsonrpc":"2.0","id":1,"method":"tools/list"}'`, - "rm -f /tmp/nemoclaw-mcp-denied.out /tmp/nemoclaw-mcp-denied.err", - "set +e", - `code="$(curl -sS -o /tmp/nemoclaw-mcp-denied.out -w '%{http_code}' -X POST ${JSON.stringify(mcpUrl)} -H 'content-type: application/json' -H 'authorization: Bearer openshell:resolve:env:FAKE_MCP_SECRET' --data "$body" 2>/tmp/nemoclaw-mcp-denied.err)"`, - "curl_rc=$?", - "set -e", - "cat /tmp/nemoclaw-mcp-denied.out 2>/dev/null || true", - "cat /tmp/nemoclaw-mcp-denied.err >&2", - 'printf "NEMOCLAW_MCP_CURL_HTTP_CODE=%s\\n" "$code"', - 'exit "$curl_rc"', - ].join("\n"), + buildMcpDnsRebindingProbeScript("openclaw-config", mcpUrl, "FAKE_MCP_SECRET"), ), + { + artifactName: "mcp-allowlisted-ancestor-curl-control", + env: buildAvailabilityProbeEnv(), + timeoutMs: 60_000, + }, + ); + const curlControlRequests = fakeMcp.requests.slice(beforeCurlControl); + const deniedCurl = await sandbox.execShell( + OPENCLAW_SANDBOX_NAME, + trustedSandboxShellScript(buildMcpDnsRebindingProbeScript(null, mcpUrl, "FAKE_MCP_SECRET")), { artifactName: "mcp-non-allowlisted-binary-curl-denied", env: buildAvailabilityProbeEnv(), @@ -980,7 +969,13 @@ test( }, ); expect( - isExpectedMcpCurlPolicyDenial(deniedCurl), + isExpectedMcpCurlPolicyDenial(deniedCurl) && + curlControl.exitCode === 0 && + !curlControl.timedOut && + curlControl.stdout.includes("NEMOCLAW_MCP_CURL_HTTP_CODE=200") && + curlControlRequests.length === 1 && + curlControlRequests[0]?.auth === `Bearer ${HOST_SECRET}` && + fakeMcp.requests.length === beforeCurlControl + 1, `non-allowlisted curl must receive an OpenShell policy denial\nstdout:\n${deniedCurl.stdout}\nstderr:\n${deniedCurl.stderr}`, ).toBe(true); diff --git a/test/e2e/live/mcp-provider-rewrite-probe.ts b/test/e2e/live/mcp-provider-rewrite-probe.ts index b6b4088bc34..ca709ff8a79 100644 --- a/test/e2e/live/mcp-provider-rewrite-probe.ts +++ b/test/e2e/live/mcp-provider-rewrite-probe.ts @@ -1,11 +1,20 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { + isNetworkPolicyDenial, + NETWORK_POLICY_RESULT_PREFIX, +} from "../support/network-policy-probe.ts"; + export function buildMcpProviderRewriteAuthorization( credentialKey: string, runtimeValue: string | undefined, ): string | null { - if (!/^[A-Za-z_][A-Za-z0-9_]{0,127}$/u.test(credentialKey) || runtimeValue === undefined) { + if ( + !/^[A-Za-z_][A-Za-z0-9_]{0,127}$/u.test(credentialKey) || + runtimeValue === undefined || + /[\r\n]/u.test(runtimeValue) + ) { return null; } const escapedCredentialKey = credentialKey.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"); @@ -26,6 +35,8 @@ export function buildMcpCredentialHandleAuthorizationPattern(credentialKey: stri export const MCP_PROVIDER_REWRITE_PROBE_SOURCE = `const https = require("node:https"); const buildMcpProviderRewriteAuthorization = ${buildMcpProviderRewriteAuthorization.toString()}; +const NETWORK_POLICY_RESULT_PREFIX = ${JSON.stringify(NETWORK_POLICY_RESULT_PREFIX)}; +const isNetworkPolicyDenial = ${isNetworkPolicyDenial.toString()}; const url = new URL(process.argv[2]); const method = process.argv[3]; const expectation = process.argv[4]; @@ -44,11 +55,13 @@ const body = JSON.stringify({ }); const req = https.request({ hostname: url.hostname, + autoSelectFamily: false, port: url.port, path: url.pathname, method: "POST", headers: { "content-type": "application/json", + "mcp-protocol-version": "2025-11-25", "content-length": Buffer.byteLength(body), "authorization": authorization } @@ -59,15 +72,15 @@ const req = https.request({ res.on("end", () => { console.log(JSON.stringify({ status: res.statusCode, body: data })); const allowed = res.statusCode === 200 && data.includes('"result"'); - const denied = res.statusCode === 403; + const denied = isNetworkPolicyDenial({ exitCode: 0, stdout: NETWORK_POLICY_RESULT_PREFIX + JSON.stringify({ status: res.statusCode, response: data }) }); process.exit(expectation === "allow" ? (allowed ? 0 : 1) : (denied ? 0 : 1)); }); }); req.on("error", (error) => { console.error(error.message); - const strictDenied = expectation === "deny-strict" && /HTTP\\/1\\.[01] 403 Forbidden/.test(error.message); - strictDenied && console.log(JSON.stringify({ status: 403, error: error.message })); - process.exit(expectation === "deny" || strictDenied ? 0 : 1); + const record = NETWORK_POLICY_RESULT_PREFIX + JSON.stringify({ errorCode: error.code ?? null, syscall: error.syscall ?? null }); + console.log(record); + process.exit(expectation !== "allow" && isNetworkPolicyDenial({ exitCode: 0, stdout: record }) ? 0 : 1); }); req.end(body); `; diff --git a/test/e2e/live/messaging-compatible-endpoint.test.ts b/test/e2e/live/messaging-compatible-endpoint.test.ts index cf00187e7f3..ab2a0b03871 100644 --- a/test/e2e/live/messaging-compatible-endpoint.test.ts +++ b/test/e2e/live/messaging-compatible-endpoint.test.ts @@ -3,20 +3,25 @@ /** * - * This stays intentionally direct: the contract is the real - * Docker/OpenShell/nemoclaw boundary with a local OpenAI-compatible endpoint - * mock, Telegram messaging config, sandbox inference.local routing, and an - * OpenClaw agent turn through the compatible endpoint proxy path. + * Exercise a local OpenAI-compatible endpoint through OpenShell native mediation, + * Telegram configuration, and a real OpenClaw agent turn. */ import { randomUUID } from "node:crypto"; import fs from "node:fs"; import http from "node:http"; +import { customNativeProvider } from "../../../src/lib/inference/native-provider/custom.ts"; +import { shellQuote } from "../../../src/lib/core/shell-quote.ts"; +import { discoverHostInterfaceAddress } from "../fixtures/host-address.ts"; import { execTimeout, testTimeout } from "../../helpers/timeouts.ts"; import { resultText } from "../fixtures/clients/command.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; -import { type SandboxClient, validateSandboxName } from "../fixtures/clients/sandbox.ts"; +import { + type SandboxClient, + trustedSandboxShellScript, + validateSandboxName, +} from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { parseOpenClawAgentText } from "../fixtures/openclaw-agent-output.ts"; import { @@ -50,17 +55,8 @@ const TEST_TIMEOUT_MS = testTimeout(45 * 60_000); validateSandboxName(SANDBOX_NAME); -const HOP_BY_HOP_HEADERS = new Set([ - "proxy-authorization", - "proxy-connection", - "proxy-authenticate", - "connection", - "keep-alive", - "te", - "trailer", - "transfer-encoding", - "upgrade", -]); +// Native HTTP mediation may generate its own Connection or Transfer-Encoding headers. +const PROXY_HEADERS = new Set(["proxy-authorization", "proxy-connection", "proxy-authenticate"]); interface MockRequestLog { method: string; @@ -112,7 +108,7 @@ async function startCompatibleMock( const requestPath = new URL(req.url ?? "/", "http://compatible.mock").pathname; const auth = req.headers.authorization === `Bearer ${apiKey}` ? "ok" : "missing"; const hopHeaders = Object.keys(req.headers).filter((name) => - HOP_BY_HOP_HEADERS.has(name.toLowerCase()), + PROXY_HEADERS.has(name.toLowerCase()), ); if (req.method === "GET" && ["/v1/models", "/models"].includes(requestPath)) { @@ -274,11 +270,12 @@ async function sourceCliAvailable(host: HostCliClient): Promise { return result.exitCode === 0; } -function onboardEnv(endpointUrl: string): NodeJS.ProcessEnv { +export function onboardEnv(endpointUrl: string): NodeJS.ProcessEnv { return commandEnv({ COMPATIBLE_API_KEY: COMPATIBLE_KEY, DISCORD_BOT_TOKEN: undefined, NEMOCLAW_ENDPOINT_URL: endpointUrl, + NEMOCLAW_TRUSTED_PRIVATE_INFERENCE_HOSTS: new URL(endpointUrl).hostname, NEMOCLAW_MODEL: COMPAT_MODEL, NEMOCLAW_POLICY_MODE: "custom", NEMOCLAW_POLICY_PRESETS: "telegram", @@ -373,10 +370,14 @@ function openAiContent(raw: string): string { .join("\n"); } -async function assertOpenClawConfigShape(sandbox: SandboxClient): Promise { +export async function assertOpenClawConfigShape( + sandbox: SandboxClient, + endpointUrl: string, +): Promise { const script = String.raw` const fs = require("node:fs"); const model = process.argv[1]; +const endpointUrl = process.argv[2]; const cfg = JSON.parse(fs.readFileSync("/sandbox/.openclaw/openclaw.json", "utf8")); const providers = cfg.models?.providers ?? {}; const errors = []; @@ -389,11 +390,12 @@ const inference = providers.inference; if (!inference || typeof inference !== "object") { errors.push("models.providers.inference is missing"); } else { - if (inference.baseUrl !== "https://inference.local/v1") { + if (inference.baseUrl !== endpointUrl) { errors.push("inference baseUrl is " + JSON.stringify(inference.baseUrl)); } - if (inference.apiKey !== "unused") { - errors.push("inference apiKey is not the non-secret placeholder"); + if (!/^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_COMPATIBLE_API_KEY$/.test(inference.apiKey) + || inference.apiKey !== process.env.COMPATIBLE_API_KEY) { + errors.push("inference apiKey is not the sandbox credential reference"); } } const primary = cfg.agents?.defaults?.model?.primary; @@ -402,18 +404,21 @@ if (!cfg.channels?.telegram) errors.push("telegram channel config missing"); console.log(JSON.stringify({ provider_keys: providerKeys, inference_base: inference?.baseUrl, - inference_api_key: inference?.apiKey, primary, telegram_present: Boolean(cfg.channels?.telegram), errors, })); process.exit(errors.length ? 1 : 0); `; - const result = await sandbox.exec(SANDBOX_NAME, ["node", "-e", script, COMPAT_MODEL], { - artifactName: "openclaw-config-compatible-endpoint", - env: commandEnv(), - timeoutMs: 60_000, - }); + const result = await sandbox.exec( + SANDBOX_NAME, + ["node", "-e", script, COMPAT_MODEL, endpointUrl], + { + artifactName: "openclaw-config-compatible-endpoint", + env: commandEnv(), + timeoutMs: 60_000, + }, + ); expect(result.exitCode, resultText(result)).toBe(0); } @@ -448,7 +453,7 @@ sock.setTimeout(1000, () => finish("TIMEOUT", 1)); ); } -async function assertSandboxInference(sandbox: SandboxClient): Promise { +async function assertSandboxInference(sandbox: SandboxClient, endpointUrl: string): Promise { const payload = JSON.stringify({ model: COMPAT_MODEL, messages: [ @@ -456,21 +461,13 @@ async function assertSandboxInference(sandbox: SandboxClient): Promise { ], max_tokens: 32, }); - const response = await sandbox.exec( + const response = await sandbox.execShell( SANDBOX_NAME, - [ - "curl", - "-sS", - "--max-time", - "60", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", - payload, - ], + trustedSandboxShellScript( + `exec curl -fsS --max-time 60 ${shellQuote(endpointUrl + "/chat/completions")} -H 'Content-Type: application/json' -H "Authorization: Bearer \${COMPATIBLE_API_KEY:?}" --data-raw ${shellQuote(payload)}`, + ), { - artifactName: "sandbox-inference-local-compatible-chat", + artifactName: "sandbox-native-compatible-chat", env: commandEnv(), timeoutMs: 90_000, }, @@ -519,14 +516,14 @@ async function assertOpenClawAgentTurn( const newHopHeaderLogs = compatibleMock.hopHeaderLogs.slice(hopCountBefore); expect( newHopHeaderLogs.length, - "Mock logged no proxy_hop_headers line for the agent turn; agent did not reach /v1/chat/completions", + "The agent did not reach the mock /v1/chat/completions endpoint", ).toBeGreaterThan(0); const leaked = newHopHeaderLogs.flat().filter((name) => name.length > 0); - expect(leaked, `Proxy hop headers leaked to upstream: ${leaked.join(",")}`).toEqual([]); + expect(leaked, `Proxy headers leaked to upstream: ${leaked.join(",")}`).toEqual([]); } test( - "messaging compatible endpoint routes Telegram-enabled OpenClaw through inference.local", + "messaging compatible endpoint routes Telegram-enabled OpenClaw through native mediation", { timeout: TEST_TIMEOUT_MS, meta: { @@ -536,7 +533,7 @@ test( "confirm host reachability to the compatible endpoint", "onboard Telegram-enabled OpenClaw", "inspect the provider route and OpenClaw configuration", - "prove inference.local and agent traffic", + "prove native inference and agent traffic", "record authenticated traffic and proxy-header results", ], }, @@ -555,12 +552,12 @@ test( "local OpenAI-compatible mock endpoint starts and is reachable", "custom provider + Telegram onboard completes", "onboard runs the compatible endpoint sandbox smoke check", - "gateway registers compatible-endpoint provider", - "openclaw.json uses managed inference.local provider and Telegram config", + "gateway registers the endpoint-bound native provider", + "openclaw.json uses the native endpoint, a scoped credential reference, and Telegram config", "gateway stays up after Telegram provider initialization", - "sandbox inference.local chat completion reaches the mock with auth", + "sandbox native chat completion reaches the mock with auth", "OpenClaw agent turn completes through the compatible endpoint", - "http-proxy-fix.js strips RFC 7230 hop-by-hop proxy headers", + "native mediation does not forward proxy credentials or proxy connection headers", ], }); @@ -599,7 +596,11 @@ test( await compatibleMock.close(); }); - const endpointUrl = `http://host.openshell.internal:${new URL(compatibleMock.localBaseUrl).port}/v1`; + const inferenceHost = await discoverHostInterfaceAddress( + host, + "messaging-compatible-host-address", + ); + const endpointUrl = `http://${inferenceHost.address}:${new URL(compatibleMock.localBaseUrl).port}/v1`; progress.phase("confirm host reachability to the compatible endpoint"); const hostReachability = await host.command( "curl", @@ -621,20 +622,30 @@ test( progress.phase("onboard Telegram-enabled OpenClaw"); const { result: onboard, runner } = await runCompatibleOnboard(host, endpointUrl); expect(onboard.exitCode, resultText(onboard)).toBe(0); - expect(resultText(onboard)).toContain("Compatible endpoint responds through inference.local"); progress.phase("inspect the provider route and OpenClaw configuration"); - const provider = await host.command("openshell", ["provider", "get", "compatible-endpoint"], { - artifactName: "openshell-provider-get-compatible-endpoint", - env: commandEnv(), - timeoutMs: 30_000, + const nativeProvider = customNativeProvider({ + provider: "compatible-endpoint", + endpointUrl, + api: "openai-completions", + credentialEnv: "COMPATIBLE_API_KEY", + allowedIps: [inferenceHost.address], }); + const provider = await host.command( + host.openshellCommandPath, + ["provider", "get", nativeProvider.providerName], + { + artifactName: "openshell-provider-get-compatible-endpoint", + env: commandEnv(), + timeoutMs: 30_000, + }, + ); expect(provider.exitCode, resultText(provider)).toBe(0); - await assertOpenClawConfigShape(sandbox); - progress.phase("prove inference.local and agent traffic"); + await assertOpenClawConfigShape(sandbox, endpointUrl); + progress.phase("prove native inference and agent traffic"); await assertGatewayReady(sandbox); - await assertSandboxInference(sandbox); + await assertSandboxInference(sandbox, endpointUrl); await assertOpenClawAgentTurn(sandbox, compatibleMock); progress.phase("record authenticated traffic and proxy-header results"); @@ -671,9 +682,7 @@ test( authenticatedChatTraffic: compatibleMock.requests.some( (request) => request.path === "/v1/chat/completions" && request.auth === "ok", ), - proxyHopHeadersStripped: compatibleMock.hopHeaderLogs.every( - (headers) => headers.length === 0, - ), + proxyHeadersAbsent: compatibleMock.hopHeaderLogs.every((headers) => headers.length === 0), }, }); }, diff --git a/test/e2e/live/messaging-providers-helpers.ts b/test/e2e/live/messaging-providers-helpers.ts index fe991f4c408..47bfc1f5b5f 100644 --- a/test/e2e/live/messaging-providers-helpers.ts +++ b/test/e2e/live/messaging-providers-helpers.ts @@ -3,7 +3,6 @@ import crypto from "node:crypto"; import fs from "node:fs"; -import { isIPv4 } from "node:net"; import os from "node:os"; import path from "node:path"; @@ -46,7 +45,6 @@ export const LIVE_TIMEOUT_MS = 90 * 60_000; export const OPENSHELL_EXEC_ARGUMENT_LIMIT_BYTES = 32_768; export const FAKE_API_IMAGE = "node:24.22.1-trixie-slim@sha256:43b9ba1142b8dcf33fa8f922ca1be85714adc444e0586c7cf7726daaeabc1bea"; -const DEFAULT_OPENSHELL_DOCKER_NETWORK = "openshell-docker"; export const FAKE_API_PROXY_READINESS_PORT = 8079; export const FAKE_API_PROXY_SOURCE = String.raw` const net = require("node:net"); @@ -298,7 +296,9 @@ export function parseRuntimeProofPort(rawPort: string): number { } export function isUnresolvedPlaceholderRejection(text: string): boolean { - return /credential_injection_failed|unresolved credential placeholder/i.test(text); + return /credential_unavailable|credential_injection_failed|unresolved credential placeholder/i.test( + text, + ); } export function isNvidiaEndpointRateLimitFailure(text: string): boolean { @@ -1034,52 +1034,9 @@ export async function startFakeDockerApi( await fs.promises.rm(dir, { recursive: true, force: true }); }); - let proxyPublishAddress = "0.0.0.0"; - let proxyProbeAddress = "127.0.0.1"; - if (runtimeProvider.id === "docker") { - const openshellNetwork = - options.env.OPENSHELL_DOCKER_NETWORK_NAME ?? - process.env.OPENSHELL_DOCKER_NETWORK_NAME ?? - DEFAULT_OPENSHELL_DOCKER_NETWORK; - const openshellNetworkInspect = await runtimeProvider.command( - ["network", "inspect", openshellNetwork], - { - artifactName: `inspect-fake-${options.kind}-openshell-network`, - env: options.env, - redactionValues: options.redactionValues, - timeoutMs: 30_000, - }, - ); - expectExitZero(openshellNetworkInspect, "inspect OpenShell Docker network"); - let openshellNetworkRecords: unknown; - try { - openshellNetworkRecords = JSON.parse(openshellNetworkInspect.stdout); - } catch { - throw new Error("OpenShell Docker network inspection returned invalid JSON"); - } - const openshellBridgeAddresses = - Array.isArray(openshellNetworkRecords) && openshellNetworkRecords.length === 1 - ? (( - openshellNetworkRecords[0] as { - Driver?: unknown; - IPAM?: { Config?: Array<{ Gateway?: unknown }> }; - } - ).IPAM?.Config?.flatMap((entry) => - typeof entry.Gateway === "string" && isIPv4(entry.Gateway) ? [entry.Gateway] : [], - ) ?? []) - : []; - const openshellBridgeAddress = - openshellBridgeAddresses.length === 1 ? openshellBridgeAddresses[0] : undefined; - if ( - (openshellNetworkRecords as Array<{ Driver?: unknown }> | undefined)?.[0]?.Driver !== - "bridge" || - typeof openshellBridgeAddress !== "string" - ) { - throw new Error("OpenShell Docker network must expose exactly one IPv4 bridge gateway"); - } - proxyPublishAddress = openshellBridgeAddress; - proxyProbeAddress = openshellBridgeAddress; - } + // The Docker supervisor uses host networking and pins its host alias to loopback. + const proxyPublishAddress = runtimeProvider.id === "docker" ? "127.0.0.1" : "0.0.0.0"; + const proxyProbeAddress = "127.0.0.1"; const networkCreate = await runtimeProvider.command( ["network", "create", "--internal", network], @@ -1320,10 +1277,6 @@ export async function applyRestRewritePolicy( SANDBOX_NAME, "--add-endpoint", `host.openshell.internal:${api.port}:read-write:rest:enforce:request-body-credential-rewrite,allowed-ip=10.0.0.0/8,allowed-ip=172.16.0.0/12,allowed-ip=192.168.0.0/16`, - "--add-allow", - `host.openshell.internal:${api.port}:GET:/**`, - "--add-allow", - `host.openshell.internal:${api.port}:POST:/**`, "--binary", "/usr/local/bin/node", "--binary", @@ -1366,10 +1319,6 @@ export async function applyWebSocketRewritePolicy( SANDBOX_NAME, "--add-endpoint", `host.openshell.internal:${api.port}:read-write:websocket:enforce:websocket-credential-rewrite,allowed-ip=10.0.0.0/8,allowed-ip=172.16.0.0/12,allowed-ip=192.168.0.0/16`, - "--add-allow", - `host.openshell.internal:${api.port}:GET:/**`, - "--add-allow", - `host.openshell.internal:${api.port}:WEBSOCKET_TEXT:/**`, "--binary", "/usr/local/bin/node", "--binary", @@ -1407,6 +1356,7 @@ async function bindFixturePolicyEndpoint( await rebindFixtureProviderPolicyEndpoint(host, SANDBOX_NAME, { artifactName: `bind-${api.kind}-${protocol}-credential`, credentialEnv: credentialKey, + allowMethods: protocol === "rest" ? ["GET", "POST"] : ["GET", "WEBSOCKET_TEXT"], endpoint: { host: "host.openshell.internal", port: api.port, diff --git a/test/e2e/live/messaging-providers.test.ts b/test/e2e/live/messaging-providers.test.ts index eff303c3c08..738470dbbe7 100644 --- a/test/e2e/live/messaging-providers.test.ts +++ b/test/e2e/live/messaging-providers.test.ts @@ -542,16 +542,10 @@ process.exit(Array.isArray(channels) && channels.some((c) => c?.channelId === "w !Object.prototype.hasOwnProperty.call(discordAccount, "token"), "M8: Discord token is omitted from openclaw.json", ); - const expectedManagedProxyHost = nonEmpty(state.env.NEMOCLAW_PROXY_HOST) ?? "10.200.0.1"; - const expectedManagedProxyPort = nonEmpty(state.env.NEMOCLAW_PROXY_PORT) ?? "3128"; - const expectedManagedProxy = `http://${expectedManagedProxyHost}:${expectedManagedProxyPort}`; - const discordAccountProxy = accountString(discordAccount, "proxy"); - const managedProxyUrl = typeof config.proxy?.proxyUrl === "string" ? config.proxy.proxyUrl : ""; check( - discordAccountProxy === "" && - config.proxy?.enabled === true && - managedProxyUrl === expectedManagedProxy, - `M9b: Discord relies on OpenClaw managed proxy config, with no per-account loopback proxy (account.proxy='${discordAccountProxy}', proxy.proxyUrl='${managedProxyUrl}')`, + !Object.prototype.hasOwnProperty.call(discordAccount, "proxy") && + !Object.prototype.hasOwnProperty.call(config, "proxy"), + "M9b: Discord configuration omits account and global proxies", ); check(accountBool(telegramAccount, "enabled") === true, "M10: Telegram account is enabled"); check(accountBool(discordAccount, "enabled") === true, "M11: Discord account is enabled"); @@ -729,17 +723,6 @@ req.setTimeout(15000, () => { req.destroy(); console.log("TIMEOUT"); }); "M13-policy: live policy contains Discord endpoints and Node binaries", ); - const proxyEnv = await sandboxOutput( - sandbox, - 'printf "HTTPS_PROXY=%s\\nhttps_proxy=%s\\nNO_PROXY=%s\\nno_proxy=%s\\n" "$HTTPS_PROXY" "$https_proxy" "$NO_PROXY" "$no_proxy"', - "proxy-env-messaging-providers", - redactionValues, - ); - check( - /10\.200\.0\.1:3128/.test(proxyEnv), - "M13-proxy: sandbox uses the OpenShell gateway proxy", - ); - const discordReach = await sandboxOutput( sandbox, `node - <<'NODE' @@ -777,7 +760,7 @@ NODE`, redactionValues, ); if (discordReach.includes("api:HTTP_") && discordReach.includes("cdn:HTTP_")) { - check(true, "M13: Node.js reached Discord API and CDN through proxy"); + check(true, "M13: Node.js reached Discord API and CDN"); } else if ( /TIMEOUT|ENETUNREACH|EHOSTUNREACH|ETIMEDOUT|ECONNRESET|socket hang up|network/i.test( discordReach, diff --git a/test/e2e/live/model-router-provider-routed-inference.test.ts b/test/e2e/live/model-router-provider-routed-inference.test.ts index 0596663422f..30a75fe0c02 100644 --- a/test/e2e/live/model-router-provider-routed-inference.test.ts +++ b/test/e2e/live/model-router-provider-routed-inference.test.ts @@ -13,7 +13,7 @@ import { } from "./model-router-provider-routed-inference-helpers.ts"; // Focused direct CLI/sandbox test: the contract is the real provider-routed -// onboard boundary plus one ordinary sandbox inference.local completion. +// onboard boundary plus one ordinary sandbox native provider completion. const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? "e2e-model-router"; const ONBOARD_TIMEOUT_MS = execTimeout(25 * 60_000); @@ -21,14 +21,14 @@ const ONBOARD_TIMEOUT_MS = execTimeout(25 * 60_000); process.env.NEMOCLAW_CLI_BIN ??= CLI_ENTRYPOINT; test( - "model-router provider-routed onboard returns an inference.local completion", + "model-router provider-routed onboard returns a native provider completion", { meta: { e2ePhases: [ "confirm routed-provider prerequisites", "clear the previous routed-provider sandbox", "onboard the routed provider", - "request a routed inference.local completion", + "request a routed native provider completion", "record the routed inference contract result", ], }, @@ -49,7 +49,7 @@ test( "the selected runtime is available before onboarding", "NVIDIA_API_KEY is present and nvapi-prefixed, then staged for the router's NVIDIA_INFERENCE_API_KEY credential", "nemoclaw onboard --fresh completes with NEMOCLAW_PROVIDER=routed", - "sandbox inference.local returns a valid chat completion through the routed provider", + "the attached native provider returns a valid chat completion through the routed provider", ], }); @@ -90,17 +90,18 @@ test( await host.recordSandboxContainerDiscoveryOnFailure(SANDBOX_NAME, onboard.exitCode, [apiKey]); expect(onboard.exitCode, resultText(onboard)).toBe(0); - progress.phase("request a routed inference.local completion"); + progress.phase("request a routed native provider completion"); await runtime.expectInferenceLocalChatCompletion( { sandboxName: SANDBOX_NAME }, { - artifactName: "sandbox-inference-local-routed-completion", - curlMaxTimeSeconds: 90, - maxTokens: 128, + modelRouter: true, model: "nvidia-routed", prompt: "Reply with a short greeting.", - redactionValues: [apiKey], + maxTokens: 128, + curlMaxTimeSeconds: 90, timeoutMs: 120_000, + redactionValues: [apiKey], + artifactName: "sandbox-native-routed-completion", }, ); diff --git a/test/e2e/live/network-policy.test.ts b/test/e2e/live/network-policy.test.ts index 0b2aa1a700a..14a90592f5b 100644 --- a/test/e2e/live/network-policy.test.ts +++ b/test/e2e/live/network-policy.test.ts @@ -23,8 +23,8 @@ import { CLI_ENTRYPOINT } from "../fixtures/paths.ts"; import { ensureConfiguredRuntimeProviderAvailable } from "../fixtures/runtime-provider.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; import { - buildNetworkPolicyCurlProbe, - parseNetworkPolicyCurlOutput, + buildNetworkPolicyRequest, + isNetworkPolicyDenial, } from "../support/network-policy-probe.ts"; import { runRestrictedOnboardWithRetry } from "./restricted-onboard-helpers.ts"; @@ -100,15 +100,20 @@ async function probeUrl( sandbox: SandboxClient, url: string, artifactName: string, -): Promise { - const result = await sandboxBash(sandbox, buildNetworkPolicyCurlProbe(url), artifactName); - return text(result); +): Promise { + return sandbox.exec(SANDBOX_NAME, buildNetworkPolicyRequest(url), { + artifactName, + env: baseEnv(), + timeoutMs: SANDBOX_EXEC_TIMEOUT_MS, + }); } async function startMarkerServer( marker: string, -): Promise<{ port: number; close: () => Promise }> { +): Promise<{ port: number; readonly requests: number; close: () => Promise }> { + let requests = 0; const server = createServer((_req, res) => { + requests += 1; res.writeHead(200, { "content-type": "text/plain; charset=utf-8" }); res.end(`${marker}\n`); }); @@ -124,6 +129,9 @@ async function startMarkerServer( let closed = false; return { port: address.port, + get requests() { + return requests; + }, close: async () => { if (closed) return; closed = true; @@ -186,8 +194,8 @@ function fail(detail) { const configPath = process.env.OPENCLAW_CONFIG_PATH || "/sandbox/.openclaw/openclaw.json"; const config = JSON.parse(fs.readFileSync(configPath, "utf8")); -if (config?.tools?.web?.fetch?.useTrustedEnvProxy !== true) { - fail("tools.web.fetch.useTrustedEnvProxy must be enabled"); +if (config?.tools?.web?.fetch?.useTrustedEnvProxy !== false) { + fail("tools.web.fetch.useTrustedEnvProxy must be disabled for native egress"); } const distDir = "/usr/local/lib/node_modules/openclaw/dist"; @@ -230,16 +238,22 @@ function summary(value) { function errorChainDetail(error) { const details = []; + let nativeDenied = false; + const isConnectDenial = (error, depth = 0) => depth < 8 && error && typeof error === "object" && ( + (error.code === "EACCES" && error.syscall === "connect") || + (Array.isArray(error.errors) && error.errors.length > 0 && error.errors.every((child) => isConnectDenial(child, depth + 1))) + ); const seen = new Set(); let current = error; for (let depth = 0; depth < 8 && current !== undefined && !seen.has(current); depth += 1) { const detail = current && (current.stack || current.message) ? current.stack || current.message : current; details.push(String(detail)); + nativeDenied ||= isConnectDenial(current); seen.add(current); current = current && typeof current === "object" ? current.cause : undefined; } - return details.join("\nCaused by: "); + return { detail: details.join("\nCaused by: "), nativeDenied }; } const approved = await webFetch.execute("e2e-approved-host-gateway", { @@ -265,11 +279,11 @@ try { } fail("E2E_FAIL_DENIED_PORT_UNEXPECTED_SUCCESS: " + deniedText); } catch (error) { - const detail = errorChainDetail(error); + const { detail, nativeDenied } = errorChainDetail(error); if (/E2E_FAIL_DENIED_PORT_|SsrFBlockedError|Blocked hostname|private\/internal\/special-use/i.test(detail)) { throw error; } - if (!/Web fetch failed \(403\)|\b403\b/i.test(detail)) { + if (!nativeDenied && !(/\b403\b/u.test(detail) && /policy[_ ]denied|not allowed by any policy/iu.test(detail))) { throw error; } console.log("E2E_WEB_FETCH_DENIED_OK " + detail.split("\n")[0].slice(0, 300)); @@ -411,7 +425,7 @@ test( "https://example.com/", "network-policy-default-denial", ); - expect(defaultDenied).toMatch(/\b403\b/); + expect(isNetworkPolicyDenial(defaultDenied), text(defaultDenied)).toBe(true); const approvedMarker = "NEMOCLAW_HOST_GATEWAY_ALLOWED"; const deniedMarker = "NEMOCLAW_HOST_GATEWAY_DENIED_PORT"; @@ -440,17 +454,21 @@ test( `http://host.openshell.internal:${approvedServer.port}/`, "network-policy-approved-host-gateway-port", ); - expect(approved).toContain(approvedMarker); + expect( + approved.exitCode === 0 && !approved.timedOut && approved.stdout.includes(approvedMarker), + text(approved), + ).toBe(true); const denied = await probeUrl( sandbox, `http://host.openshell.internal:${deniedServer.port}/`, "network-policy-denied-host-gateway-port", ); - const deniedResult = parseNetworkPolicyCurlOutput(denied); const deniedEvidence = "shell/network-policy-denied-host-gateway-result.json"; - await artifacts.writeJson(deniedEvidence, deniedResult ?? { response: denied, status: null }); - expect(denied).not.toContain(deniedMarker); - expect(deniedResult?.status, denied).toBe(403); + await artifacts.writeJson(deniedEvidence, denied); + expect(!text(denied).includes(deniedMarker) && deniedServer.requests === 0, text(denied)).toBe( + true, + ); + expect(isNetworkPolicyDenial(denied), text(denied)).toBe(true); progress.phase("prove the installed OpenClaw web_fetch path obeys the host-gateway policy"); const webFetch = await sandboxBash( @@ -461,7 +479,10 @@ NEMOCLAW_WEB_FETCH_PROBE`, "network-policy-openclaw-web-fetch", ); const webFetchText = text(webFetch); - expect(webFetch.exitCode, webFetchText).toBe(0); + expect( + webFetch.exitCode === 0 && !webFetch.timedOut && deniedServer.requests === 0, + webFetchText, + ).toBe(true); const startTimeAfterDeny = await readSandboxStartTime( sandbox, "network-policy-start-time-after-deny-probe", diff --git a/test/e2e/live/onboard-policy-preset-sequencing.test.ts b/test/e2e/live/onboard-policy-preset-sequencing.test.ts index 184cda002ec..faf1f0577a6 100644 --- a/test/e2e/live/onboard-policy-preset-sequencing.test.ts +++ b/test/e2e/live/onboard-policy-preset-sequencing.test.ts @@ -9,6 +9,7 @@ import { resultText } from "../fixtures/clients/command.ts"; import { validateSandboxName } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { startFakeOpenAiCompatibleServer } from "../fixtures/fake-openai-compatible.ts"; +import { buildSandboxInferenceChatCommand } from "../fixtures/inference-adapter.ts"; import { resolveVerifiedCloudflaredBinary } from "./cloudflared-prerequisite.ts"; import { startPublicMcpHttpsTunnel } from "./mcp-bridge-servers.ts"; import { readRegistrySandboxEntry } from "../fixtures/phases/state-validation.ts"; @@ -29,8 +30,8 @@ import { driveInteractiveCommand } from "./onboard-interactive-pty.ts"; // // This mock-provider variant follows the reporter's journey // (`nemoclaw onboard` with "Other OpenAI-compatible endpoint"). It selects -// Personal so the same journey also proves the hosted custom endpoint remains -// on inference.local without acquiring a native provider attachment. +// Personal so the same journey also proves the hosted custom endpoint works +// through its sandbox-attached native provider. const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? "e2e-policy-order"; validateSandboxName(SANDBOX_NAME); @@ -60,7 +61,7 @@ test( "start the hosted custom compatible-endpoint fixture", "drive the interactive onboard wizard through a real PTY", "confirm every ordered onboarding step appears in order", - "confirm Personal keeps the hosted custom endpoint on inference.local", + "confirm Personal permits the hosted custom native endpoint", ], }, }, @@ -169,7 +170,7 @@ test( searchFrom = index + marker.length; }); - progress.phase("confirm Personal keeps the hosted custom endpoint on inference.local"); + progress.phase("confirm Personal permits the hosted custom native endpoint"); const policyList = await host.command("node", [CLI_ENTRYPOINT, SANDBOX_NAME, "policy-list"], { artifactName: "policy-list-personal-compatible-endpoint", env: buildAvailabilityProbeEnv(), @@ -180,27 +181,24 @@ test( expect(policyListText).toMatch(/●\s+personal-open-internet\s+\[/u); const registered = readRegistrySandboxEntry(SANDBOX_NAME); - expect(recordedNativeProviderAttachment(registered)).toBeUndefined(); + expect(recordedNativeProviderAttachment(registered)).toMatchObject({ + endpointUrl: `${tunnel.origin}/v1`, + }); const chat = await sandbox.exec( SANDBOX_NAME, - [ - "curl", - "-fsS", - "--max-time", - "60", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", + buildSandboxInferenceChatCommand( JSON.stringify({ model: MODEL, messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], max_tokens: 50, }), - ], + "compatible-endpoint", + `${tunnel.origin}/v1`, + 60, + ), { - artifactName: "personal-compatible-endpoint-inference-local-chat", + artifactName: "personal-compatible-endpoint-native-chat", env: buildAvailabilityProbeEnv(), redactionValues: [apiKey], timeoutMs: 90_000, diff --git a/test/e2e/live/onboard-repair.test.ts b/test/e2e/live/onboard-repair.test.ts index 72fa64c8f53..cab10d78eae 100644 --- a/test/e2e/live/onboard-repair.test.ts +++ b/test/e2e/live/onboard-repair.test.ts @@ -6,6 +6,7 @@ import { withPodmanOwnerDiagnostic, captureBoundedPodmanOwnerDiagnostic, } from "../fixtures/podman-owner-diagnostic"; +import { registeredCompatibleChatCommand } from "../fixtures/registered-inference.ts"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -160,6 +161,7 @@ test( }); const fake = await startFakeOpenAiCompatibleServer({ + additionalModels: ["changed-test-model"], host: "0.0.0.0", progress, publicHost: "host.openshell.internal", @@ -399,21 +401,15 @@ test( const requestOffset = fake.requests().length; const sourceChat = await sandbox.exec( SANDBOX_NAME, - [ - "curl", - "-fsS", - "--max-time", - "60", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", + registeredCompatibleChatCommand( + SANDBOX_NAME, + "test-model", JSON.stringify({ model: "test-model", messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], max_tokens: 50, }), - ], + ), { artifactName: "phase-3-source-inference-after-rejected-backup", env: env(), @@ -476,21 +472,15 @@ test( const replacementRequestOffset = fake.requests().length; const replacementChat = await sandbox.exec( SANDBOX_NAME, - [ - "curl", - "-fsS", - "--max-time", - "60", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", + registeredCompatibleChatCommand( + SANDBOX_NAME, + "changed-test-model", JSON.stringify({ model: "changed-test-model", messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], max_tokens: 50, }), - ], + ), { artifactName: "phase-3-replacement-inference-after-successful-recreate", env: env(), diff --git a/test/e2e/live/onboard-resume.test.ts b/test/e2e/live/onboard-resume.test.ts index e4f79204543..a08bf817818 100644 --- a/test/e2e/live/onboard-resume.test.ts +++ b/test/e2e/live/onboard-resume.test.ts @@ -1,11 +1,15 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { captureSandboxFailureDiagnostics } from "../fixtures/sandbox-failure-diagnostics.ts"; +import { + captureLifecycleInstallFailureDiagnostics, + captureSandboxFailureDiagnostics, +} from "../fixtures/sandbox-failure-diagnostics.ts"; import { withPodmanOwnerDiagnostic, captureBoundedPodmanOwnerDiagnostic, } from "../fixtures/podman-owner-diagnostic"; +import { registeredCompatibleChatCommand } from "../fixtures/registered-inference.ts"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -24,7 +28,7 @@ import { prepareOnboardSandboxes } from "../fixtures/onboard-precleanup.ts"; import { assertCleanupSucceededOrAbsent } from "../fixtures/cleanup-resources.ts"; import { resultText } from "../fixtures/clients/command.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; -import { trustedSandboxShellScript, validateSandboxName } from "../fixtures/clients/sandbox.ts"; +import { validateSandboxName } from "../fixtures/clients/sandbox.ts"; import { cleanupCorporateCaFixture, corporateCaMergeProbeScript, @@ -95,9 +99,9 @@ interface SessionStateComplete { } interface SessionStateRetryableFailure { - status: "failed" | "in_progress"; + status: "failed"; resumable: true; - machine: { state: "failed" | "post_verify" }; + machine: { state: "failed" }; } interface MutableSessionState extends Record { @@ -210,7 +214,7 @@ test( "resume sandbox recreation filters stale extra providers while preserving live attachments", "resume proves recreated sandbox provider attachments are selectively reconciled", "host trust-store anchor corporate CA source is baked and merged after resume", - "an unreachable committed route pauses at final verification and completes after repair", + "an unreachable committed route leaves a resumable failure and completes after repair", "non-recreate resume retains the Ready sandbox, dashboard port, and exact forward listener", "implicit resume is detected and --fresh suppresses that auto-resume", ], @@ -375,6 +379,13 @@ test( redactionValues: [FAKE_COMPATIBLE_AUTH_VALUE], timeoutMs: execTimeout(ONBOARD_FINAL_HANDOFF_COMMAND_TIMEOUT_MS), }); + await captureLifecycleInstallFailureDiagnostics(host, runtimeProvider, firstRun, { + sandboxName: SANDBOX_NAME, + gatewayName: "nemoclaw", + artifactPrefix: "phase-2-initial-onboard-failure", + env: buildAvailabilityProbeEnv(), + redactionValues: [FAKE_COMPATIBLE_AUTH_VALUE], + }); const firstText = `${firstRun.stdout}\n${firstRun.stderr}`; // Assertion: interrupted-exit-1. @@ -401,16 +412,16 @@ test( // must inject the stored compatible-endpoint credential upstream; this POST // is the positive auth proof and is deliberately newer than fixture startup // and the direct readiness fetch excluded by onboardingRequestOffset. - const inferenceAfterInterrupt = await sandbox.execShell( + const inferenceAfterInterrupt = await sandbox.exec( SANDBOX_NAME, - trustedSandboxShellScript( - `curl -fsS --max-time 60 https://inference.local/v1/chat/completions -H 'Content-Type: application/json' --data '${JSON.stringify( - { - model: FAKE_COMPATIBLE_MODEL, - messages: [{ role: "user", content: "reply with OK" }], - max_tokens: 8, - }, - )}'`, + registeredCompatibleChatCommand( + SANDBOX_NAME, + FAKE_COMPATIBLE_MODEL, + JSON.stringify({ + model: FAKE_COMPATIBLE_MODEL, + messages: [{ role: "user", content: "reply with OK" }], + max_tokens: 8, + }), ), { artifactName: "phase-2-authenticated-inference-post", @@ -423,9 +434,6 @@ test( `${inferenceAfterInterrupt.stdout}\n${inferenceAfterInterrupt.stderr}`, ).toBe(0); - // Assertion: session-file-present. - expect(fs.existsSync(SESSION_FILE)).toBe(true); - // Assertion: session-file-interrupted-state. const interrupted = readSession(SESSION_FILE); await artifacts.writeJson( @@ -578,12 +586,11 @@ test( ).toBe(true); // Assertion: registry-has-sandbox. - expect(fs.existsSync(REGISTRY_FILE)).toBe(true); const registry = JSON.parse(fs.readFileSync(REGISTRY_FILE, "utf8")) as unknown; expect(containsExactJsonToken(registry, SANDBOX_NAME)).toBe(true); // ────────────────────────────────────────────────────────────────── - // Phase 3.5: a committed route that goes offline leaves final + // Phase 3.5: a committed route that goes offline leaves native inference // verification retryable; restoring the same endpoint lets a later resume // re-probe and complete without recreating the sandbox. // ────────────────────────────────────────────────────────────────── @@ -633,26 +640,22 @@ test( `${unavailableResumeRun.exitCode !== 0}:${listenerBeforeRouteFailure.valid}:${listenerAfterRouteFailure.valid}:${listenerBeforeRouteFailure.identity === listenerAfterRouteFailure.identity}`, `${unavailableResumeText}\n${listenerBeforeRouteFailure.output}\n${listenerAfterRouteFailure.output}`, ).toBe("true:true:true:true"); - expect( - hasHermesApi || - unavailableResumeText.includes("Compatible endpoint sandbox smoke check failed"), - ).toBe(true); - expect(unavailableResumeText).toContain("inference.local"); + expect(unavailableResumeText).toContain( + "Provider-neutral sandbox inference smoke check failed.", + ); expect(unavailableResumeText).not.toContain( `Deleting and recreating sandbox '${SANDBOX_NAME}'`, ); const paused = readSession(SESSION_FILE); - const expectedPausedStatus = hasHermesApi ? "in_progress" : "failed"; - const expectedPausedMachineState = hasHermesApi ? "post_verify" : "failed"; await artifacts.writeJson("phase-3-5-session-route-unavailable.json", { status: paused.status, resumable: paused.resumable, machineState: paused.machine.state, }); - expect(paused.status).toBe(expectedPausedStatus); + expect(paused.status).toBe("failed"); expect(paused.resumable).toBe(true); - expect(paused.machine.state).toBe(expectedPausedMachineState); + expect(paused.machine.state).toBe("failed"); fake = await startFakeOpenAiCompatibleServer({ apiKey: FAKE_COMPATIBLE_AUTH_VALUE, diff --git a/test/e2e/live/openclaw-agent-assertion.ts b/test/e2e/live/openclaw-agent-assertion.ts index 3626108f4d9..d2fe6bbb827 100644 --- a/test/e2e/live/openclaw-agent-assertion.ts +++ b/test/e2e/live/openclaw-agent-assertion.ts @@ -42,8 +42,9 @@ export interface OpenClawAgentAssertionEvidence { export interface OpenClawAgentAssertionOptions { apiKey: string; + collectToolEvidenceOnFailure?: boolean; expected: string; - finalReplyContract?: "unavailable-search"; + finalReplyContract?: "unavailable-search" | "exact-token"; label: string; prompt: string; persistCommandArtifacts?: boolean; @@ -146,6 +147,14 @@ export async function runOpenClawAgentAssertion( reply, response: combined, }); + if ( + classification.passed && + args.finalReplyContract === "exact-token" && + (reply.trim() !== args.expected || finalVisibleReply.trim() !== args.expected) + ) { + classification = { passed: false, failureClass: "deterministic" }; + lastFailure = "payload and final visible reply must contain only the expected token"; + } if ( classification.passed && args.finalReplyContract === "unavailable-search" && @@ -212,7 +221,10 @@ export async function runOpenClawAgentAssertion( } const validation = await validateOpenClawAgentAttemptEvidence({ classification, - collectToolEvidence: args.publicFetchExpectation !== undefined && !classification.passed, + collectToolEvidence: + (args.publicFetchExpectation !== undefined || + args.collectToolEvidenceOnFailure === true) && + !classification.passed, label: args.label, recordToolEvidenceReductionFailure: async (failure) => { await recordAttemptEvidence( @@ -223,9 +235,20 @@ export async function runOpenClawAgentAssertion( recordToolEvidence: async (toolEvidence) => { await recordAttemptEvidence( `actions/${args.label}-attempt-${attempt}-reduced.json`, - (args.toolEvidence?.project ?? projectPersonalPublicFetchToolEvidenceArtifact)( - toolEvidence, - ), + args.toolEvidence?.project(toolEvidence) ?? + (args.publicFetchExpectation + ? projectPersonalPublicFetchToolEvidenceArtifact(toolEvidence) + : { + schemaVersion: 1, + errorCount: toolEvidence.errors.length, + finalStatusCount: toolEvidence.finalStatuses.length, + toolCallCount: toolEvidence.toolCalls.length, + toolExecutionCount: toolEvidence.toolExecutions.length, + toolResultCount: toolEvidence.toolResults.length, + execCalls: toolEvidence.toolCalls.filter(({ name }) => name === "exec").length, + execResults: toolEvidence.toolResults.filter(({ name }) => name === "exec") + .length, + }), ); }, reduceToolEvidence, diff --git a/test/e2e/live/openclaw-discord-pairing.test.ts b/test/e2e/live/openclaw-discord-pairing.test.ts index 0a828fccbdb..17575299324 100644 --- a/test/e2e/live/openclaw-discord-pairing.test.ts +++ b/test/e2e/live/openclaw-discord-pairing.test.ts @@ -113,7 +113,7 @@ test( expectExitZero(provider, "Discord provider exists"); const configScript = - "import json; cfg=json.load(open('/sandbox/.openclaw/openclaw.json')); account=(cfg.get('channels',{}).get('discord',{}).get('accounts',{}).get('default') or {}); proxy=cfg.get('proxy') or {}; print(json.dumps({'hasToken': 'token' in account, 'dmPolicy': account.get('dmPolicy',''), 'allowFrom': account.get('allowFrom', []), 'accountProxy': account.get('proxy',''), 'managedProxy': proxy.get('proxyUrl','')}))"; + "import json; cfg=json.load(open('/sandbox/.openclaw/openclaw.json')); account=(cfg.get('channels',{}).get('discord',{}).get('accounts',{}).get('default') or {}); print(json.dumps({'hasToken': 'token' in account, 'dmPolicy': account.get('dmPolicy',''), 'allowFrom': account.get('allowFrom', []), 'hasAccountProxy': 'proxy' in account, 'hasGlobalProxy': 'proxy' in cfg}))"; const config = await sandboxSh( sandbox, SANDBOX_NAME, @@ -128,13 +128,13 @@ test( hasToken: boolean; dmPolicy: string; allowFrom: string[]; - accountProxy: string; - managedProxy: string; + hasAccountProxy: boolean; + hasGlobalProxy: boolean; }; expect(configSummary.hasToken, "Discord config must omit the token field").toBe(false); expect(configSummary.dmPolicy).not.toBe("allowlist"); - expect(configSummary.accountProxy, "Discord account proxy").toBe(""); - expect(configSummary.managedProxy, "OpenClaw managed proxy").toMatch(/^http:\/\//); + expect(configSummary.hasAccountProxy, "Discord config must omit the account proxy").toBe(false); + expect(configSummary.hasGlobalProxy, "OpenClaw config must omit the global proxy").toBe(false); await assertOpenClawStateRoot(sandbox, SANDBOX_NAME, "discord", redactions); diff --git a/test/e2e/live/openclaw-inference-switch-helpers.ts b/test/e2e/live/openclaw-inference-switch-helpers.ts index 9a6aaf87226..5bc91db7ab8 100644 --- a/test/e2e/live/openclaw-inference-switch-helpers.ts +++ b/test/e2e/live/openclaw-inference-switch-helpers.ts @@ -4,6 +4,7 @@ // Fixture and response helpers shared by the openclaw-inference-switch live // target and fast e2e-support tests without gating on NEMOCLAW_RUN_LIVE_E2E=1. +import fs from "node:fs"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -14,6 +15,46 @@ import { type FakeOpenAiCompatibleServerOptions, } from "../fixtures/fake-openai-compatible.ts"; +// PID/start time survives execve. Native RPC status includes the authenticated hello boot ID. +export async function readOpenClawBootId( + sandbox: SandboxClient, + sandboxName: string, + options: { env: NodeJS.ProcessEnv; artifactName: string }, +): Promise { + const result = await sandbox.exec( + sandboxName, + [ + "openclaw", + "gateway", + "status", + "--port", + "18789", + "--require-rpc", + "--json", + "--timeout", + "10000", + ], + { ...options, timeoutMs: 20_000 }, + ); + // Launcher diagnostics use stderr even when the authenticated RPC succeeds. + if (result.exitCode !== 0 || result.timedOut || result.signal !== null) return null; + try { + const document = JSON.parse(result.stdout) as { + rpc?: { ok?: unknown; kind?: unknown; server?: { bootId?: unknown } }; + } | null; + const rpc = document?.rpc; + const bootId = rpc?.server?.bootId; + return rpc?.ok === true && + rpc.kind === "read" && + typeof bootId === "string" && + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/iu.test(bootId) + ? bootId + : null; + } catch { + return null; + } +} + export interface OpenClawPostSwitchInferenceAttempt { exitCode: number | null; httpStatus: string; @@ -159,9 +200,10 @@ export const MOCK_BASELINE_MODEL = "openclaw-switch-baseline-model"; export function startMockOpenClawBaselineProvider( progress: FakeOpenAiCompatibleServerOptions["progress"], publicHost = "host.openshell.internal", + apiKey = MOCK_BASELINE_API_KEY, ) { return startFakeOpenAiCompatibleServer({ - apiKey: MOCK_BASELINE_API_KEY, + apiKey, chatContent: "PONG", host: "0.0.0.0", model: MOCK_BASELINE_MODEL, @@ -215,24 +257,29 @@ export async function readOpenClawConfig( artifactName: string; nativeNvidia?: boolean; nativeCustomApi?: string; + registeredProvider?: string; + credentialEnv?: string; }, ): Promise { return sandbox.exec( sandboxName, - options.nativeNvidia || options.nativeCustomApi + options.nativeNvidia || options.nativeCustomApi || options.registeredProvider ? [ "node", "-e", [ 'const fs = require("node:fs");', 'const config = JSON.parse(fs.readFileSync("/sandbox/.openclaw/openclaw.json", "utf8"));', - `const provider = config.models?.providers?.${options.nativeCustomApi === "anthropic-messages" ? "anthropic" : "inference"};`, - // A fresh exec receives the newly attached provider handle. Keep - // both values inside the sandbox because ShellProbe redacts the - // credential-shaped config field before returning stdout. - `const credentialEnv = ${JSON.stringify(options.nativeCustomApi ? (options.nativeCustomApi === "anthropic-messages" ? "COMPATIBLE_ANTHROPIC_API_KEY" : "COMPATIBLE_API_KEY") : "NVIDIA_INFERENCE_API_KEY")};`, + `const provider = config.models?.providers?.[${JSON.stringify(options.registeredProvider ?? (options.nativeCustomApi === "anthropic-messages" ? "anthropic" : "inference"))}];`, + // OpenShell retains scoped handles across same-provider revisions. + // Check their form here; the caller also proves provider attachment + // and authenticated inference through the actual OpenClaw gateway. + // Keep both values inside the sandbox before artifact redaction. + `const credentialEnv = ${JSON.stringify(options.credentialEnv ?? (options.nativeCustomApi ? (options.nativeCustomApi === "anthropic-messages" ? "COMPATIBLE_ANTHROPIC_API_KEY" : "COMPATIBLE_API_KEY") : "NVIDIA_INFERENCE_API_KEY"))};`, "const handle = process.env[credentialEnv];", - 'require("node:assert/strict").ok(new RegExp("^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_" + credentialEnv + "$").test(handle ?? "") && provider?.apiKey === handle, "native credential reference mismatch");', + 'const scoped = new RegExp("^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_" + credentialEnv + "$");', + 'require("node:assert/strict").ok(typeof handle === "string" && scoped.test(handle), "native exec credential is not scoped");', + 'require("node:assert/strict").ok(typeof provider?.apiKey === "string" && scoped.test(provider.apiKey), "native config credential is not scoped");', "delete provider.apiKey;", "process.stdout.write(JSON.stringify(config));", ].join(" "), @@ -241,3 +288,88 @@ export async function readOpenClawConfig( { artifactName: options.artifactName, env: options.env, timeoutMs: options.timeoutMs }, ); } + +export function httpStatusFromResponse(response: string): string { + return ( + response + .split("\n") + .filter((line) => line.startsWith("__NEMOCLAW_HTTP_STATUS__=")) + .at(-1) + ?.replace("__NEMOCLAW_HTTP_STATUS__=", "") + .trim() ?? "" + ); +} + +export function httpBodyFromResponse(response: string): string { + return response + .split("\n") + .filter((line) => !line.startsWith("__NEMOCLAW_HTTP_STATUS__=")) + .join("\n"); +} + +// Keep ownership records until the CLI teardown and its runtime absence check finish. +// Cache failure as well as success: cleanup registration must not repeat a failed destroy. +export function createOpenClawInferenceSwitchCleanup(input: { + host: HostCliClient; + verifyGatewayRemoved: () => Promise; + sandboxName: string; + home: string; + env: NodeJS.ProcessEnv; + retain: boolean; +}): () => Promise { + let cleanup: Promise | undefined; + return () => { + cleanup ??= input.retain + ? Promise.resolve() + : (async () => { + await input.host.cleanupSandbox(input.sandboxName, { + cleanupGateway: true, + artifactName: "cleanup-inference-switch-sandbox-and-gateway", + env: input.env, + timeoutMs: 120_000, + }); + await input.verifyGatewayRemoved(); + fs.rmSync(input.home, { recursive: true, force: true }); + })(); + return cleanup; + }; +} + +interface ChatCompletionResponse { + choices?: Array<{ + message?: { + content?: unknown; + reasoning_content?: unknown; + reasoning?: unknown; + }; + text?: unknown; + }>; +} + +interface AnthropicResponse { + content?: Array<{ text?: unknown }>; +} + +export function parseChatContent(raw: string): string { + const response = JSON.parse(raw) as ChatCompletionResponse; + return (response.choices ?? []) + .map((choice) => { + const message = choice.message ?? {}; + if (typeof message.content === "string") return message.content; + if (typeof message.reasoning_content === "string") return message.reasoning_content; + if (typeof message.reasoning === "string") return message.reasoning; + if (typeof choice.text === "string") return choice.text; + return ""; + }) + .join("\n") + .trim(); +} + +export function parseAnthropicContent(raw: string): string { + const response = JSON.parse(raw) as AnthropicResponse; + return (response.content ?? []) + .map((part) => (typeof part.text === "string" ? part.text : "")) + .filter(Boolean) + .join(" ") + .trim(); +} diff --git a/test/e2e/live/openclaw-inference-switch.test.ts b/test/e2e/live/openclaw-inference-switch.test.ts index e7f17fad91c..a01af102fd0 100644 --- a/test/e2e/live/openclaw-inference-switch.test.ts +++ b/test/e2e/live/openclaw-inference-switch.test.ts @@ -1,8 +1,13 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -// Hosted switching and native local routing share the OpenClaw live contract. - +import { + readRegisteredLocalRoute, + operatorInferenceSwitchFixture, + OPERATOR_SWITCH_PROVIDER, + OPERATOR_SWITCH_API_KEY, + OPERATOR_SWITCH_CREDENTIAL_ENV, +} from "../fixtures/registered-inference.ts"; import fs from "node:fs"; import type { Server } from "node:http"; import type { AddressInfo } from "node:net"; @@ -56,8 +61,14 @@ import { import { runBoundedRetry } from "../../../tools/e2e/retry-evidence.mts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; import { + createOpenClawInferenceSwitchCleanup, + parseChatContent, + parseAnthropicContent, getRouteOutput, + httpStatusFromResponse, + httpBodyFromResponse, readOpenClawConfig, + readOpenClawBootId, agentReplyContainsToken, classifyExhaustedPostSwitchEvidence, classifyOpenClawPostSwitchInferenceAttempt, @@ -107,6 +118,7 @@ const SWITCH_PROVIDER = process.env.NEMOCLAW_SWITCH_PROVIDER ?? PUBLIC_NVIDIA_SW const SWITCH_MODEL = process.env.NEMOCLAW_SWITCH_MODEL ?? PUBLIC_NVIDIA_SWITCH_MODEL; const SWITCH_INFERENCE_API = process.env.NEMOCLAW_SWITCH_INFERENCE_API ?? "openai-completions"; const SWITCH_MOCK_ANTHROPIC = process.env.NEMOCLAW_SWITCH_MOCK_ANTHROPIC ?? "0"; +const SWITCH_REGISTERED = SWITCH_PROVIDER === OPERATOR_SWITCH_PROVIDER; const SWITCH_NATIVE_CUSTOM = process.env.NEMOCLAW_SWITCH_NATIVE_CUSTOM === "1"; const SWITCH_MOCK_PORT = parsePortEnv("NEMOCLAW_SWITCH_MOCK_PORT", 0); const TEST_TIMEOUT_MS = testTimeout(75 * 60_000); @@ -117,28 +129,10 @@ const AGENT_TIMEOUT_MS = 150_000; validateSandboxName(SANDBOX_NAME); -interface ChatCompletionResponse { - choices?: Array<{ - message?: { - content?: unknown; - reasoning_content?: unknown; - reasoning?: unknown; - }; - text?: unknown; - }>; -} - -interface AnthropicResponse { - content?: Array<{ text?: unknown }>; -} - -interface SandboxRegistry { - sandboxes?: Record< - string, - { - nativeNvidiaProviderAttachment?: unknown; - } - >; +interface MockAnthropicProvider { + endpointUrl: string; + requests(): readonly MockAnthropicRequest[]; + close(): Promise; } function parsePortEnv(name: string, fallback: number): number { @@ -201,15 +195,6 @@ function writeCustomOpenClawDockerfile(home: string): string { return dockerfilePath; } -async function bestEffortStateReset(run: () => Promise): Promise { - try { - await run(); - } catch { - // Cleanup probes are intentionally best-effort so early setup failures do - // not hide the primary assertion or install failure. - } -} - async function runNemoclaw( host: HostCliClient, home: string, @@ -245,40 +230,6 @@ async function sandboxShell( }); } -async function resetOpenClawInferenceSwitchState( - host: HostCliClient, - sandbox: SandboxClient, - home: string, - artifactPrefix: string, -): Promise { - await bestEffortStateReset(() => - runNemoclaw(host, home, [SANDBOX_NAME, "destroy", "--yes"], { - artifactName: `${artifactPrefix}-nemoclaw-destroy-openclaw-inference-switch`, - timeoutMs: 120_000, - }), - ); - await bestEffortStateReset(() => - sandbox.openshell(["sandbox", "delete", SANDBOX_NAME], { - artifactName: `${artifactPrefix}-openshell-sandbox-delete-openclaw-inference-switch`, - env: commandEnv(home), - timeoutMs: 60_000, - }), - ); - await bestEffortStateReset(() => - sandbox.openshell(["gateway", "destroy", "-g", "nemoclaw"], { - artifactName: `${artifactPrefix}-openshell-gateway-destroy-openclaw-inference-switch`, - env: commandEnv(home), - timeoutMs: 120_000, - }), - ); -} - -interface MockAnthropicProvider { - endpointUrl: string; - requests(): readonly MockAnthropicRequest[]; - close(): Promise; -} - function closeServer(server: Server): Promise { return new Promise((resolve, reject) => { server.close((error) => (error ? reject(error) : resolve())); @@ -355,29 +306,22 @@ async function prepareCompatibleAnthropicSwitchBinding( return binding; } -async function openclawGatewayPid(sandbox: SandboxClient, home: string): Promise { - const result = await sandboxShell( - sandbox, - home, - 'ps -eo pid=,comm=,args= 2>/dev/null | awk \'$2 != "sh" && $2 != "bash" && $2 != "awk" && $0 ~ /openclaw/ && $0 ~ /gateway run/ { print $1; exit }\' || true', - { - artifactName: "openclaw-gateway-pid", - timeoutMs: 30_000, - }, - ); - return result.stdout.trim(); -} - -async function verifyNativeNvidiaAttachmentAfterSwitch( +async function verifyNativeAttachmentAfterSwitch( home: string, sandbox: SandboxClient, + operator?: Awaited>, + operatorId?: string, ): Promise { + let observed: unknown; + let expected: object; switch (SWITCH_PROVIDER) { case PUBLIC_NVIDIA_SWITCH_PROVIDER: { const registryPath = path.join(home, ".nemoclaw", "sandboxes.json"); - const registry = JSON.parse(fs.readFileSync(registryPath, "utf8")) as SandboxRegistry; - expect( - await readPublicNvidiaSwitchAttachmentEvidence({ + const registry = JSON.parse(fs.readFileSync(registryPath, "utf8")) as { + sandboxes?: Record; + }; + observed = { + evidence: await readPublicNvidiaSwitchAttachmentEvidence({ artifactName: "native-nvidia-provider-attachment-after-switch", env: commandEnv(home), logicalProvider: SWITCH_PROVIDER, @@ -385,27 +329,37 @@ async function verifyNativeNvidiaAttachmentAfterSwitch( sandbox, sandboxName: SANDBOX_NAME, }), - ).toBe(PUBLIC_NVIDIA_SWITCH_ATTACHMENT_EVIDENCE); + }; + expected = { evidence: PUBLIC_NVIDIA_SWITCH_ATTACHMENT_EVIDENCE }; + break; } + case OPERATOR_SWITCH_PROVIDER: + observed = await operator!.evidence(); + expected = operator!.expected(operatorId!); + break; + default: + return; } + expect(observed).toMatchObject(expected); } -// Source tests own serialized config and registry/session field values. This -// live owner retains the actual configuration read and real gateway requests. +// Validate the sandbox credential before the real gateway request. async function recordOpenClawConfig( sandbox: SandboxClient, home: string, expected: { - model: string; inferenceApi: string; artifactName: string; nativeNvidia?: boolean; nativeCustomEndpoint?: string; + registered?: boolean; }, ): Promise { const configResult = await readOpenClawConfig(sandbox, SANDBOX_NAME, { nativeNvidia: expected.nativeNvidia, nativeCustomApi: expected.nativeCustomEndpoint ? expected.inferenceApi : undefined, + registeredProvider: expected.registered ? SWITCH_PROVIDER : undefined, + credentialEnv: expected.registered ? OPERATOR_SWITCH_CREDENTIAL_ENV : undefined, artifactName: expected.artifactName, env: commandEnv(home), timeoutMs: COMMAND_TIMEOUT_MS, @@ -413,48 +367,6 @@ async function recordOpenClawConfig( expect(configResult.exitCode, resultText(configResult)).toBe(0); } -function httpStatusFromResponse(response: string): string { - return ( - response - .split("\n") - .filter((line) => line.startsWith("__NEMOCLAW_HTTP_STATUS__=")) - .at(-1) - ?.replace("__NEMOCLAW_HTTP_STATUS__=", "") - .trim() ?? "" - ); -} - -function httpBodyFromResponse(response: string): string { - return response - .split("\n") - .filter((line) => !line.startsWith("__NEMOCLAW_HTTP_STATUS__=")) - .join("\n"); -} - -function parseChatContent(raw: string): string { - const response = JSON.parse(raw) as ChatCompletionResponse; - return (response.choices ?? []) - .map((choice) => { - const message = choice.message ?? {}; - if (typeof message.content === "string") return message.content; - if (typeof message.reasoning_content === "string") return message.reasoning_content; - if (typeof message.reasoning === "string") return message.reasoning; - if (typeof choice.text === "string") return choice.text; - return ""; - }) - .join("\n") - .trim(); -} - -function parseAnthropicContent(raw: string): string { - const response = JSON.parse(raw) as AnthropicResponse; - return (response.content ?? []) - .map((part) => (typeof part.text === "string" ? part.text : "")) - .filter(Boolean) - .join(" ") - .trim(); -} - async function checkSandboxInference( sandbox: SandboxClient, artifacts: ArtifactSink, @@ -471,11 +383,12 @@ async function checkSandboxInference( : { model: SWITCH_MODEL, messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], - max_tokens: 100, + // Hosted reasoning models share this budget with their final answer. + max_tokens: 1024, }; const payloadB64 = Buffer.from(JSON.stringify(payload), "utf8").toString("base64"); const curlCommand = nativeCustomEndpoint - ? `curl -sS -o "$tmp" -w '%{http_code}' --max-time 90 ${shellQuote(`${nativeCustomEndpoint}/v1/messages`)} -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01' -H "$AUTH_HEADER" --data-binary @/tmp/nemoclaw-switch-payload.json` + ? `curl -sS -o "$tmp" -w '%{http_code}' --max-time 90 ${shellQuote(`${nativeCustomEndpoint}${SWITCH_REGISTERED ? "/chat/completions" : "/v1/messages"}`)} -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01' -H "$AUTH_HEADER" --data-binary @/tmp/nemoclaw-switch-payload.json` : SWITCH_PROVIDER === PUBLIC_NVIDIA_SWITCH_PROVIDER ? `curl -sS -o "$tmp" -w '%{http_code}' --max-time 90 ${NVIDIA_HOSTED_NATIVE_ENDPOINT}/chat/completions -H 'Content-Type: application/json' -H "$AUTH_HEADER" --data-binary @/tmp/nemoclaw-switch-payload.json` : SWITCH_INFERENCE_API === "anthropic-messages" @@ -486,8 +399,10 @@ async function checkSandboxInference( ...(nativeCustomEndpoint ? [ nativeHostedAuthHeaderScript({ - credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", - api: "anthropic-messages", + credentialEnv: SWITCH_REGISTERED + ? OPERATOR_SWITCH_CREDENTIAL_ENV + : "COMPATIBLE_ANTHROPIC_API_KEY", + api: SWITCH_REGISTERED ? "openai-completions" : "anthropic-messages", }), ] : SWITCH_PROVIDER === PUBLIC_NVIDIA_SWITCH_PROVIDER @@ -591,6 +506,7 @@ async function checkOpenClawGatewayInference( inferenceApi: string; artifactName: string; phase: string; + provider?: string; }, mockProvider: MockAnthropicProvider | undefined, baselineProvider?: Pick, @@ -682,7 +598,8 @@ exit "$rc" request.model === expected.model, ); const expectedOpenClawProvider = - expected.inferenceApi === "anthropic-messages" ? "anthropic" : "inference"; + expected.provider ?? + (expected.inferenceApi === "anthropic-messages" ? "anthropic" : "inference"); if ( result.exitCode === 0 && modelRun?.provider === expectedOpenClawProvider && @@ -710,11 +627,6 @@ exit "$rc" ); } -// The pure reply-matching and mock-baseline-config assertions that previously -// lived here as test(...) blocks (which only run under the opt-in live lane) -// are covered in the fast e2e-support project instead: -// test/e2e/support/openclaw-inference-switch-helpers.test.ts. - function isExternalProviderValidationFailure(text: string): boolean { return ( /NVIDIA Endpoints endpoint validation failed/i.test(text) && @@ -733,6 +645,8 @@ async function runOpenClawInferenceSetWithRetry( const attempts = inferenceSetAttemptCount(process.env.NEMOCLAW_SWITCH_SET_ATTEMPTS); const compatibleCredentialEnv = (() => { switch (SWITCH_PROVIDER) { + case OPERATOR_SWITCH_PROVIDER: + return OPERATOR_SWITCH_CREDENTIAL_ENV; case "compatible-endpoint": return "COMPATIBLE_API_KEY"; case "compatible-anthropic-endpoint": @@ -770,7 +684,7 @@ async function runOpenClawInferenceSetWithRetry( runNemoclaw(host, home, args, { artifactName: `nemoclaw-inference-set-${attempt}`, env: { - ...compatibleAnthropicSwitchEnv(switchBinding), + ...compatibleAnthropicSwitchEnv(SWITCH_REGISTERED ? null : switchBinding), ...(publicNvidiaApiKey ? { NVIDIA_INFERENCE_API_KEY: publicNvidiaApiKey } : {}), }, redactionValues, @@ -786,7 +700,6 @@ test( meta: { e2ePhases: [ "confirm the selected runtime and choose the baseline provider", - "clear existing inference-switch state", "onboard baseline OpenClaw", "prove the selected route after onboarding", "prove the selected route after gateway restart", @@ -799,7 +712,17 @@ test( ], }, }, - async ({ artifacts, cleanup, host, progress, runtimeProvider, sandbox, secrets, skip }) => { + async ({ + artifacts, + cleanup, + gateway, + host, + progress, + runtimeProvider, + sandbox, + secrets, + skip, + }) => { await artifacts.target.declare({ id: "openclaw-inference-switch", boundary: "install-sh-openclaw-inference-set-and-live-gateway-model-run", @@ -817,7 +740,7 @@ test( "when staged, the authenticated baseline fixture receives each selected-model OpenClaw gateway request", "nemoclaw inference set switches the running sandbox route", "OpenClaw gateway is supervisor-restarted after every changed inference configuration", - "NemoClaw reports the switched provider/model", + "OpenClaw consumes the selected provider and model through its native attachment", "sandbox inference returns PONG from the switched model", "OpenClaw gateway model inference answers through the switched route without agent tools", ], @@ -833,14 +756,19 @@ test( ? requirePublicNvidiaSwitchKey(secrets.required("NVIDIA_API_KEY")) : null; const useMockBaseline = - SWITCH_PROVIDER === "compatible-anthropic-endpoint" && SWITCH_MOCK_ANTHROPIC === "1"; + SWITCH_REGISTERED || + (SWITCH_PROVIDER === "compatible-anthropic-endpoint" && SWITCH_MOCK_ANTHROPIC === "1"); const baselineProvider: FakeOpenAiCompatibleServer | undefined = useMockBaseline ? await startMockOpenClawBaselineProvider(progress) : undefined; - cleanup.trackDisposable("close baseline inference provider", async () => { - await baselineProvider?.close(); + let operatorProvider: FakeOpenAiCompatibleServer | undefined; + cleanup.trackDisposable("close inference providers", async () => { + await Promise.all([baselineProvider?.close(), operatorProvider?.close()]); }); - const cloudflaredBin = SWITCH_NATIVE_CUSTOM + operatorProvider = SWITCH_REGISTERED + ? await startMockOpenClawBaselineProvider(progress, undefined, OPERATOR_SWITCH_API_KEY) + : undefined; + const cloudflaredBin = useMockBaseline ? await resolveVerifiedCloudflaredBinary(cleanup, host) : undefined; // Hosted endpoint qualification must cross a network boundary. The runner's @@ -861,6 +789,21 @@ test( readinessStatus: 200, }) : undefined; + const operatorTunnel = operatorProvider + ? await startPublicMcpHttpsTunnel({ + cloudflaredBin, + cleanup, + label: "OpenClaw operator inference", + progress, + server: { + port: Number(new URL(operatorProvider.baseUrl).port), + close: () => operatorProvider!.close(), + }, + originProtocol: "http", + readinessPath: "/v1/models", + readinessStatus: 200, + }) + : undefined; const baseline = baselineProvider ? mockBaselineInference( baselineTunnel ? `${baselineTunnel.origin}/v1` : baselineProvider.baseUrl, @@ -875,40 +818,30 @@ test( : process.env, ); const apiKey = baseline.apiKey; - const redactionValues = [apiKey, publicApiKey].filter( + const redactionValues = [apiKey, publicApiKey, OPERATOR_SWITCH_API_KEY].filter( (value): value is string => typeof value === "string", ); const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openclaw-switch-home-")); let mockProvider: MockAnthropicProvider | undefined; - cleanup.trackDisposable( - `remove OpenClaw inference switch test home for ${SANDBOX_NAME}`, - () => { - fs.rmSync(home, { recursive: true, force: true }); - }, - ); - const imageArgs = - runtimeProvider.id === "podman" ? [] : ["--from", writeCustomOpenClawDockerfile(home)]; - cleanup.trackGateway(host, "nemoclaw", { - artifactName: "cleanup-openshell-gateway-destroy-openclaw-inference-switch", + const cleanupState = createOpenClawInferenceSwitchCleanup({ + host, + verifyGatewayRemoved: () => + gateway.expectRemoved("nemoclaw", { + artifactName: "cleanup-inference-switch-gateway-absent", + env: commandEnv(home), + }), + sandboxName: SANDBOX_NAME, + home, env: commandEnv(home), - timeoutMs: 120_000, + retain: process.env.NEMOCLAW_E2E_KEEP_SANDBOX === "1", }); - cleanup.trackDisposable(`delete OpenShell sandbox ${SANDBOX_NAME}`, () => - sandbox.cleanupSandbox(SANDBOX_NAME, { - artifactName: "cleanup-openshell-sandbox-delete-openclaw-inference-switch", - env: commandEnv(home), - timeoutMs: 60_000, - }), - ); - cleanup.trackSandbox(host, SANDBOX_NAME, { - artifactName: "cleanup-nemoclaw-destroy-openclaw-inference-switch", - env: commandEnv(home), - timeoutMs: 120_000, + cleanup.trackDisposable("clean up inference-switch sandbox and gateway", cleanupState); + cleanup.trackDisposable("close switched Anthropic provider", async () => { + await mockProvider?.close(); }); - - progress.phase("clear existing inference-switch state"); - await resetOpenClawInferenceSwitchState(host, sandbox, home, "pre-cleanup"); + const imageArgs = + runtimeProvider.id === "podman" ? [] : ["--from", writeCustomOpenClawDockerfile(home)]; progress.phase("onboard baseline OpenClaw"); const onboard = await host.nemoclaw(["onboard", "--non-interactive", "--yes", ...imageArgs], { @@ -949,7 +882,6 @@ test( ); const verifyInitialRoute = async (artifactSuffix: string): Promise => { await recordOpenClawConfig(sandbox, home, { - model: baseline.model, inferenceApi: "openai-completions", artifactName: `read-openclaw-initial-route-${artifactSuffix}`, nativeCustomEndpoint: baselineTunnel || publicApiKey ? baseline.endpointUrl : undefined, @@ -1008,13 +940,25 @@ test( endpointUrl: mockProvider.endpointUrl, }); } - // Only the explicit Anthropic bridge supplies endpoint metadata. The - // compatible baseline reuses its registered OpenShell provider, while the - // public NVIDIA provider has no caller-supplied endpoint identity. + const operator = SWITCH_REGISTERED + ? await operatorInferenceSwitchFixture(host, { + home, + env: commandEnv(home), + endpointUrl: `${operatorTunnel!.origin}/v1`, + credentialValue: OPERATOR_SWITCH_API_KEY, + sandboxName: SANDBOX_NAME, + model: SWITCH_MODEL, + }) + : undefined; + cleanup.trackDisposable("remove fixture-owned operator provider", async () => { + await operator?.cleanup(); + }); + const operatorId = await operator?.create(); const switchBinding = - SWITCH_PROVIDER === "compatible-anthropic-endpoint" + operator ?? + (SWITCH_PROVIDER === "compatible-anthropic-endpoint" ? await prepareCompatibleAnthropicSwitchBinding(host, home, mockProvider) - : null; + : null); switchBinding && redactionValues.push(switchBinding.credentialValue); progress.phase("switch the route and verify restart semantics"); @@ -1026,7 +970,14 @@ test( redactionValues, false, ); - const pidBefore = await openclawGatewayPid(sandbox, home); + const pidBefore = await gateway.resolveGatewayIdentity( + { sandboxName: SANDBOX_NAME }, + { env: commandEnv(home), artifactName: "gateway-identity-before-inference-switch" }, + ); + const bootBefore = await readOpenClawBootId(sandbox, SANDBOX_NAME, { + env: commandEnv(home), + artifactName: "gateway-boot-before-inference-switch", + }); const switchResult = await runOpenClawInferenceSetWithRetry( host, home, @@ -1037,39 +988,40 @@ test( ); expect(switchResult.exitCode, resultText(switchResult)).toBe(0); - const pidAfter = await openclawGatewayPid(sandbox, home); - const gatewayPidStable = pidBefore && pidAfter ? pidBefore === pidAfter : null; - if (gatewayPidStable !== null) { - expect( - gatewayPidStable, - `OpenClaw gateway process did not change after the config switch (${pidBefore} -> ${pidAfter})`, - ).toBe(false); - } + const pidAfter = await gateway.resolveGatewayIdentity( + { sandboxName: SANDBOX_NAME }, + { env: commandEnv(home), artifactName: "gateway-identity-after-inference-switch" }, + ); + const bootAfter = await readOpenClawBootId(sandbox, SANDBOX_NAME, { + env: commandEnv(home), + artifactName: "gateway-boot-after-inference-switch", + }); + expect( + Boolean(pidBefore && pidAfter && bootBefore && bootAfter && bootBefore !== bootAfter), + "OpenClaw needs valid supervisor identities and a new authenticated boot after the config switch", + ).toBe(true); progress.phase("inspect route configuration and recorded state"); - const route = await getRouteOutput(host, { + await getRouteOutput(host, { env: commandEnv(home), timeoutMs: COMMAND_TIMEOUT_MS, }); - expect(route.exitCode, resultText(route)).toBe(0); - const routeState = JSON.parse(route.stdout) as { provider?: unknown; model?: unknown }; - expect(routeState.provider).toBe(SWITCH_PROVIDER); - expect(routeState.model).toBe(SWITCH_MODEL); await recordOpenClawConfig(sandbox, home, { - model: SWITCH_MODEL, inferenceApi: SWITCH_INFERENCE_API, artifactName: "read-openclaw-config-after-inference-switch", nativeNvidia: SWITCH_PROVIDER === PUBLIC_NVIDIA_SWITCH_PROVIDER, - nativeCustomEndpoint: SWITCH_NATIVE_CUSTOM ? switchBinding!.endpointUrl : undefined, + registered: SWITCH_REGISTERED, + nativeCustomEndpoint: + SWITCH_NATIVE_CUSTOM || SWITCH_REGISTERED ? switchBinding!.endpointUrl : undefined, }); - await verifyNativeNvidiaAttachmentAfterSwitch(home, sandbox); + await verifyNativeAttachmentAfterSwitch(home, sandbox, operator, operatorId); progress.phase("prove sandbox and OpenClaw gateway inference"); const inference = await checkSandboxInference( sandbox, artifacts, home, - SWITCH_NATIVE_CUSTOM ? switchBinding!.endpointUrl : undefined, + SWITCH_NATIVE_CUSTOM || SWITCH_REGISTERED ? switchBinding!.endpointUrl : undefined, ); if (inference !== "ok") { await artifacts.target.complete({ @@ -1090,8 +1042,10 @@ test( inferenceApi: SWITCH_INFERENCE_API, artifactName: "openclaw-gateway-inference-after-switch", phase: "prove sandbox and OpenClaw gateway inference", + provider: SWITCH_REGISTERED ? SWITCH_PROVIDER : undefined, }, mockProvider, + operatorProvider, ); if (gatewayInference !== "ok") { await artifacts.target.complete({ @@ -1103,13 +1057,49 @@ test( skip(gatewayInference.skipped); } - progress.phase("apply sandbox retention and record the result"); - if (process.env.NEMOCLAW_E2E_KEEP_SANDBOX !== "1") { - await resetOpenClawInferenceSwitchState(host, sandbox, home, "final"); - const registryPath = path.join(home, ".nemoclaw", "sandboxes.json"); - const registryText = fs.existsSync(registryPath) ? fs.readFileSync(registryPath, "utf8") : ""; - expect(registryText).not.toContain(`"${SANDBOX_NAME}"`); + switch (SWITCH_PROVIDER) { + case OPERATOR_SWITCH_PROVIDER: { + const replacementId = await operator!.replace(); + await artifacts.writeJson("operator-provider-replacement.json", { + originalProviderId: operatorId, + replacementProviderId: replacementId, + }); + const beforeDenied = operatorProvider!.requests().length; + const denied = await host.nemoclaw( + [ + "inference", + "set", + "--provider", + SWITCH_PROVIDER, + "--model", + SWITCH_MODEL, + "--sandbox", + SANDBOX_NAME, + "--endpoint-url", + operator!.endpointUrl, + "--credential-env", + OPERATOR_SWITCH_CREDENTIAL_ENV, + "--inference-api", + "openai-completions", + ], + { + artifactName: "operator-replaced-identity-denial", + env: commandEnv(home), + redactionValues, + timeoutMs: COMMAND_TIMEOUT_MS, + }, + ); + expect(denied.exitCode).not.toBe(0); + expect(resultText(denied)).toContain( + "Recorded registered inference provider identity or profile changed", + ); + expect(operatorProvider!.requests()).toHaveLength(beforeDenied); + await operator!.cleanup(); + break; + } } + progress.phase("apply sandbox retention and record the result"); + await cleanupState(); await artifacts.target.complete({ id: "openclaw-inference-switch", @@ -1123,7 +1113,9 @@ test( baselineGatewayReachedFixture: baselineProvider ? true : null, inferenceSetCompleted: switchResult.exitCode === 0, gatewayRestartExpected: true, - gatewayPidStable, + gatewayBootChanged: bootBefore !== bootAfter, + gatewayProcessBefore: pidBefore, + gatewayProcessAfter: pidAfter, routeChecked: true, inferenceLocalPong: true, inferenceLocalModelMatched: true, @@ -1143,10 +1135,10 @@ test( "onboard native OpenClaw owner", "rebuild native OpenClaw owner", "restart native OpenClaw owner", - "onboard shared-route sibling", - "record shared route before model switch", + "onboard native-route sibling", + "record sibling route before model switch", "switch native OpenClaw model", - "verify shared route unchanged", + "verify sibling route unchanged", "request inference through OpenClaw", "check credential isolation", "verify owner reachability", @@ -1187,7 +1179,7 @@ test( timeoutMs: 120_000, }); expect(start.exitCode, resultText(start)).toBe(0); - progress.phase("onboard shared-route sibling"); + progress.phase("onboard native-route sibling"); const sibling = inferenceSandboxName("e2e-oc-peer"); const peerEndpoint = await startFakeOpenAiCompatibleServer({ apiKey, @@ -1219,12 +1211,12 @@ test( ONBOARD_FINAL_HANDOFF_COMMAND_TIMEOUT_MS, ); expectOnboardSuccess(peer, `${sibling} onboard`); - progress.phase("record shared route before model switch"); - const sharedBefore = await sandbox.openshell(["inference", "get", "-g", "nemoclaw"], { - artifactName: `${owner}-shared-route-before-switch`, - env: buildAvailabilityProbeEnv(), - timeoutMs: 30_000, - }); + progress.phase("record sibling route before model switch"); + const sharedBefore = await readRegisteredLocalRoute( + sandbox, + sibling, + `${owner}-sibling-route-before-switch`, + ); expect(sharedBefore.exitCode, resultText(sharedBefore)).toBe(0); progress.phase("switch native OpenClaw model"); const switched = await host.nemoclaw( @@ -1237,12 +1229,12 @@ test( }, ); expect(switched.exitCode, resultText(switched)).toBe(0); - progress.phase("verify shared route unchanged"); - const sharedAfter = await sandbox.openshell(["inference", "get", "-g", "nemoclaw"], { - artifactName: `${owner}-shared-route-after-switch`, - env: buildAvailabilityProbeEnv(), - timeoutMs: 30_000, - }); + progress.phase("verify sibling route unchanged"); + const sharedAfter = await readRegisteredLocalRoute( + sandbox, + sibling, + `${owner}-sibling-route-after-switch`, + ); expect(sharedAfter.exitCode).toBe(sharedBefore.exitCode); expect(sharedAfter.stdout).toBe(sharedBefore.stdout); progress.phase("request inference through OpenClaw"); diff --git a/test/e2e/live/openclaw-stopped-recovery.ts b/test/e2e/live/openclaw-stopped-recovery.ts index 83b3a5c233c..f48c9b107ca 100644 --- a/test/e2e/live/openclaw-stopped-recovery.ts +++ b/test/e2e/live/openclaw-stopped-recovery.ts @@ -2,6 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import assert from "node:assert/strict"; +import { parseStrictOpenShellSandboxListJson } from "../../../src/lib/adapters/openshell/sandbox-identity.ts"; import { shellQuote } from "../../../src/lib/core/shell-quote.ts"; import { waitUntilAsync } from "../../../src/lib/core/wait.ts"; import type { ArtifactSink } from "../fixtures/artifacts.ts"; @@ -35,7 +36,7 @@ export async function proveStoppedDockerAgentRecovery( } assert(sandboxName.startsWith("e2e-"), "Container disruption requires a test-owned sandbox name"); const gateway = process.env.OPENSHELL_GATEWAY ?? "nemoclaw"; - const env = buildAvailabilityProbeEnv(); + const env = { ...buildAvailabilityProbeEnv(), OPENSHELL_GATEWAY: gateway }; let observation = 0; const readSource = async (): Promise<{ id: string; phase: string }> => { const result = await sandbox.openshell(["sandbox", "list", "-g", gateway, "-o", "json"], { @@ -44,13 +45,13 @@ export async function proveStoppedDockerAgentRecovery( timeoutMs: 10_000, }); assertExitZero(result, "observe the test-owned sandbox lifecycle"); - const rows = JSON.parse(result.stdout) as Array<{ id: string; name: string; phase: string }>; + const rows = parseStrictOpenShellSandboxListJson(result.stdout); + assert(rows, "Stopped recovery requires a complete, valid sandbox inventory"); const selected = rows.filter((row) => row.name === sandboxName); assert.equal(selected.length, 1); return selected[0]!; }; const source = await readSource(); - assert(typeof source.id === "string" && /^[A-Za-z0-9._-]{1,512}$/u.test(source.id)); assert.equal(source.phase, "Ready"); const container = await runtime.resolveSandboxResourceHandle(sandboxName, { artifactName: `${prefix}-stopped-source-container`, @@ -84,25 +85,19 @@ export async function proveStoppedDockerAgentRecovery( const writeMarkers = `umask 077; printf '%s' ${shellQuote(marker)} > ${shellQuote(markerPath)} && printf '%s' ${shellQuote(unknownRootMarker)} > ${shellQuote(unknownRootMarkerPath)}`; // #11165: retain agent-owned state while the managed login profile is broken. // The ordinary healthy rebuild above this proof remains a separate control. - const prepared = + const breakProfile = agentName === "langchain-deepagents-code" - ? await runtime.execSandboxAsRoot( - sandboxName, - [ - "/bin/sh", - "-c", - `${writeMarkers} && chown --reference=/sandbox/.deepagents/.state ${shellQuote(markerPath)} ${shellQuote(unknownRootMarkerPath)} && rm -f /sandbox/.bash_profile && ln -s /sandbox/hostile-env.sh /sandbox/.bash_profile && sync`, - ], - { - artifactName: `${prefix}-stopped-source-write-marker`, - sanitizeEnvironment: true, - timeoutMs: 10_000, - }, - ) - : await sandbox.exec(sandboxName, ["sh", "-c", `${writeMarkers} && sync`], { - artifactName: `${prefix}-stopped-source-write-marker`, - env, - }); + ? " && rm -f /sandbox/.bash_profile && ln -s /sandbox/hostile-env.sh /sandbox/.bash_profile" + : ""; + const prepared = await sandbox.exec( + sandboxName, + ["sh", "-c", `${writeMarkers}${breakProfile} && sync`], + { + artifactName: `${prefix}-stopped-source-write-marker`, + env, + ...(agentName === "langchain-deepagents-code" ? { timeoutMs: 10_000 } : {}), + }, + ); assertExitZero(prepared, "prepare the stopped-recovery source and marker"); assertExitZero( await runtime.command(["kill", container], { diff --git a/test/e2e/live/openshell-allowed-ips-rebinding.ts b/test/e2e/live/openshell-allowed-ips-rebinding.ts index 7c909cc6fba..08e46462c9f 100644 --- a/test/e2e/live/openshell-allowed-ips-rebinding.ts +++ b/test/e2e/live/openshell-allowed-ips-rebinding.ts @@ -16,7 +16,7 @@ import type { HostCliClient } from "../fixtures/clients/host.ts"; import { type SandboxClient, trustedSandboxShellScript } from "../fixtures/clients/sandbox.ts"; import { expect } from "../fixtures/e2e-test.ts"; import { discoverHostAddress } from "../fixtures/host-address.ts"; -import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; +import { waitForMcpCurlPolicyDenial } from "../fixtures/curl-policy-denial.ts"; import { type DnsRebindingHostsFixture, remapDnsRebindingHostname, @@ -81,6 +81,7 @@ export function parseRawOpenShellAllowedIpsRebindingEndpoint( export function buildRawOpenShellAllowedIpsRebindingPolicy( basePolicyYaml: string, port: number, + pinnedIp = RAW_OPENSHELL_REBIND_PINNED_IP, ): string { const policy = parseRawPolicy(basePolicyYaml); policy.network_policies = { @@ -94,8 +95,10 @@ export function buildRawOpenShellAllowedIpsRebindingPolicy( path: "/mcp", protocol: "mcp", enforcement: "enforce", - allowed_ips: [RAW_OPENSHELL_REBIND_PINNED_IP], + allowed_ips: [pinnedIp], mcp: { + // OpenShell 0.1.3 includes this default in policy readback. + versions: ["2025-11-25"], max_body_bytes: 4096, strict_tool_names: true, allow_all_known_mcp_methods: false, @@ -112,33 +115,31 @@ export function buildRawOpenShellAllowedIpsRebindingPolicy( } /** - * Exercise OpenShell directly with a raw MCP request and require an exact 403. + * Exercise OpenShell directly; the caller verifies an HTTP, connect, or audited DNS denial. * This intentionally bypasses every NemoClaw MCP command and agent adapter. * - * Pinned resolve-validate-connect implementation: - * https://github.com/NVIDIA/OpenShell/blob/3dee5570a46076a57a3b056f35f35ebc0861ac85/crates/openshell-supervisor-network/src/proxy.rs#L2648-L2674 - * resolves once, #L2699-L2739 plus #L2794-L2803 validate and return that - * address list, and #L4093-L4100 plus #L4340-L4342 carry that same list through - * the explicit HTTP-forward connection path used by this probe. + * OpenShell 0.1.3 also reports native TCP destination denial as connect EACCES; + * preserve curl's errno diagnostic rather than requiring the old proxy's 403. */ -export function buildRawOpenShellAllowedIpsRebindingProbeScript(targetUrl: string): string { +export function buildRawOpenShellAllowedIpsRebindingProbeScript( + targetUrl: string, + outputDirectory = "/tmp", +): string { const body = JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/list" }); - const responsePath = "/tmp/nemoclaw-raw-openshell-rebinding.body"; - const stderrPath = "/tmp/nemoclaw-raw-openshell-rebinding.stderr"; + const responsePath = path.join(outputDirectory, "nemoclaw-raw-openshell-rebinding.body"); + const stderrPath = path.join(outputDirectory, "nemoclaw-raw-openshell-rebinding.stderr"); return [ "set -u", `rm -f ${shellQuote(responsePath)} ${shellQuote(stderrPath)}`, `body=${shellQuote(body)}`, "set +e", - `status="$(curl -sS --max-time 30 -o ${shellQuote(responsePath)} -w '%{http_code}' -X POST ${shellQuote(targetUrl)} -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' --data-binary "$body" 2>${shellQuote(stderrPath)})"`, + `status="$(LC_ALL=C curl -sS --verbose --max-time 30 -o ${shellQuote(responsePath)} -w '%{http_code}' -X POST ${shellQuote(targetUrl)} -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -H 'mcp-protocol-version: 2025-11-25' --data-binary "$body" 2>${shellQuote(stderrPath)})"`, "curl_rc=$?", "set -e", `cat ${shellQuote(responsePath)} 2>/dev/null || true`, `cat ${shellQuote(stderrPath)} >&2 2>/dev/null || true`, `printf '${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}%s\\n' "$status"`, - 'if [ "$curl_rc" -eq 0 ] && [ "$status" = "403" ]; then exit 0; fi', - 'if [ "$curl_rc" -ne 0 ]; then exit "$curl_rc"; fi', - "exit 1", + 'exit "$curl_rc"', ].join("\n"); } @@ -196,7 +197,6 @@ export async function assertRawOpenShellAllowedIpsRebindingDenied(options: { let policyMutationAttempted = false; try { const reboundAddress = await hostAddressForSandbox(options.host); - expect(reboundAddress).not.toBe(RAW_OPENSHELL_REBIND_PINNED_IP); expect( isPrivateIp(reboundAddress), `${reboundAddress} must be a private rebinding target`, @@ -211,8 +211,8 @@ export async function assertRawOpenShellAllowedIpsRebindingDenied(options: { options.host, options.sandboxName, hostsFixture, - RAW_OPENSHELL_REBIND_PINNED_IP, - "raw-openshell-rebinding-map-public-pin", + reboundAddress, + "raw-openshell-rebinding-map-control-pin", ); const basePolicy = await options.sandbox.openshell( @@ -240,11 +240,36 @@ export async function assertRawOpenShellAllowedIpsRebindingDenied(options: { ); policyMutationAttempted = true; + const controlPolicyApplied = await setPolicyDocument( + options.sandboxName, + buildRawOpenShellAllowedIpsRebindingPolicy(basePolicyYaml, server.port, reboundAddress), + { nonFatal: true, operation: "allow the raw OpenShell rebinding control" }, + ); + await new Promise((resolve) => setTimeout(resolve, options.policySettleMs)); + const targetUrl = `http://${RAW_OPENSHELL_REBIND_HOSTNAME}:${server.port}/mcp`; + const control = await options.sandbox.execShell( + options.sandboxName, + trustedSandboxShellScript(buildRawOpenShellAllowedIpsRebindingProbeScript(targetUrl)), + { artifactName: "raw-openshell-rebinding-allowed-control", env, timeoutMs: 60_000 }, + ); + const requestsBeforeDenial = server.requestCount(); + await remapDnsRebindingHostname( + options.host, + options.sandboxName, + hostsFixture, + RAW_OPENSHELL_REBIND_PINNED_IP, + "raw-openshell-rebinding-map-public-pin", + ); expect( - await setPolicyDocument(options.sandboxName, fs.readFileSync(policyPath, "utf8"), { + (await setPolicyDocument(options.sandboxName, fs.readFileSync(policyPath, "utf8"), { nonFatal: true, operation: "run the raw OpenShell allowed_ips rebinding proof", - }), + })) && + controlPolicyApplied && + control.exitCode === 0 && + !control.timedOut && + control.stdout.includes(`${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}200`) && + requestsBeforeDenial === 1, "raw-openshell-rebinding-policy-set", ).toBe(true); await new Promise((resolve) => setTimeout(resolve, options.policySettleMs)); @@ -261,7 +286,6 @@ export async function assertRawOpenShellAllowedIpsRebindingDenied(options: { const effectiveEndpoint = parseRawOpenShellAllowedIpsRebindingEndpoint(effectivePolicy.stdout); expect(effectiveEndpoint).toMatchObject({ allowed_ips: [RAW_OPENSHELL_REBIND_PINNED_IP], - host: RAW_OPENSHELL_REBIND_HOSTNAME, port: server.port, protocol: "mcp", }); @@ -274,22 +298,33 @@ export async function assertRawOpenShellAllowedIpsRebindingDenied(options: { "raw-openshell-rebinding-map-private-unpinned", ); - const targetUrl = `http://${RAW_OPENSHELL_REBIND_HOSTNAME}:${server.port}/mcp`; + const readDnsAudit = (suffix: string) => + options.sandbox.openshell( + ["logs", options.sandboxName, "-n", "500", "--source", "sandbox", "--since", "2m"], + { artifactName: `raw-openshell-rebinding-dns-audit-${suffix}`, env, timeoutMs: 30_000 }, + ); + const dnsAuditBefore = await readDnsAudit("before"); const denial = await options.sandbox.execShell( options.sandboxName, trustedSandboxShellScript(buildRawOpenShellAllowedIpsRebindingProbeScript(targetUrl)), { - artifactName: "raw-openshell-rebinding-exact-403", + artifactName: "raw-openshell-rebinding-policy-denial", env, timeoutMs: 60_000, }, ); - expect(denial.exitCode, resultText(denial)).toBe(0); - expect(denial.stdout).toContain(`${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}403`); + const policyDenied = await waitForMcpCurlPolicyDenial( + denial, + dnsAuditBefore, + readDnsAudit, + RAW_OPENSHELL_REBIND_HOSTNAME, + RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER, + ); + expect(policyDenied && hostsFixture.server.answers() > 0, resultText(denial)).toBe(true); expect( server.requestCount(), - "raw OpenShell allowed_ips denial must record zero upstream requests", - ).toBe(0); + "raw OpenShell allowed_ips denial must record no additional upstream requests", + ).toBe(requestsBeforeDenial); } finally { try { if (policyMutationAttempted && basePolicyPath) { diff --git a/test/e2e/live/openshell-credential-generation-window.test.ts b/test/e2e/live/openshell-credential-generation-window.test.ts index 21ba9ea7a93..96229d277a5 100644 --- a/test/e2e/live/openshell-credential-generation-window.test.ts +++ b/test/e2e/live/openshell-credential-generation-window.test.ts @@ -5,6 +5,8 @@ import { buildMcpCredentialDetachedCommand, buildMcpCredentialRevisionObservationCommand, } from "../../../src/lib/actions/sandbox/mcp-bridge-provider-readiness.ts"; +import { normalizeNativeLocalProviderAttachment } from "../../../src/lib/inference/native-local/profile.ts"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; import { parseOpenShellSandboxId } from "../../../src/lib/adapters/openshell/sandbox-identity.ts"; import { approveOpenClawAdminScope } from "./openclaw-admin-scope.ts"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; @@ -13,17 +15,13 @@ import { assertExitZero as expectExitZero, resultText } from "../fixtures/client import type { HostCliClient } from "../fixtures/clients/host.ts"; import { type SandboxClient, trustedSandboxShellScript } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; -import { - HOSTED_INFERENCE_CREDENTIAL_ENV, - HOSTED_INFERENCE_PROVIDER_NAME, -} from "../fixtures/hosted-inference.ts"; import { MCP_BRIDGE_TEST_CREDENTIALS } from "../fixtures/mcp-bridge-credentials.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; -import { hostAddressForSandbox } from "./mcp-bridge-sandbox.ts"; import { prepareOwnedSandboxForOnboard } from "../fixtures/owned-sandbox-cleanup.ts"; import { type FakeMcpHttpsServer, startCompatibleMock, + withCompatibleMockFailureDiagnostics, startFakeMcpHttpsServer, startPublicMcpHttpsTunnel, } from "./mcp-bridge-servers.ts"; @@ -209,6 +207,7 @@ async function readSandboxFile( } async function waitForReadyRevision(sandbox: SandboxClient): Promise { + let ready = ""; await expect .poll( async () => { @@ -217,7 +216,8 @@ async function waitForReadyRevision(sandbox: SandboxClient): Promise { CREDENTIAL_WINDOW_PATHS.ready, "credential-window-old-child-ready-poll", ); - return result.exitCode === 0 ? result.stdout.trim() : ""; + ready = result.exitCode === 0 ? result.stdout.trim() : ""; + return ready; }, { interval: 500, @@ -226,13 +226,7 @@ async function waitForReadyRevision(sandbox: SandboxClient): Promise { }, ) .toMatch(/^\{"revision":"v[0-9]{1,20}"\}$/u); - const result = await readSandboxFile( - sandbox, - CREDENTIAL_WINDOW_PATHS.ready, - "credential-window-old-child-ready", - ); - expectExitZero(result, "read old credential-window child revision"); - return (JSON.parse(result.stdout) as { revision: string }).revision; + return (JSON.parse(ready) as { revision: string }).revision; } async function waitForAcknowledgement( @@ -298,7 +292,7 @@ async function updateProviderCredential( ): Promise { const args = credentialEnv === CREDENTIAL_WINDOW_ENV_NAME - ? buildCredentialWindowProviderUpdateArgs(providerName, expiresAtMs, secret.length === 0) + ? buildCredentialWindowProviderUpdateArgs(providerName, expiresAtMs) : [ "provider", "update", @@ -366,7 +360,7 @@ test( "prove a retained credential generation expires", "prove an expired inference credential cannot delete the source sandbox", "rotate beyond the retained generation window", - "prove key and bridge removal revoke access", + "prove current credential expiry and bridge removal revoke access", "re-add the bridge and keep the old process revoked", "rebuild the sandbox and confirm credential reuse", "remove the MCP bridge and audit denied requests", @@ -385,11 +379,11 @@ test( await artifacts.target.declare({ id: "openshell-credential-generation-window", contracts: [ - "OpenShell f27ff150 retained credential generations", + "OpenShell 0.1.7 ordinary static credential generations", "NemoClaw MCP detach, restart, and rebuild lifecycle", "NemoClaw rejects an expired selected inference credential before rebuild deletion", ], - sourceRevision: "3dee5570a46076a57a3b056f35f35ebc0861ac85", + sourceRevision: "bdb956f2cd65a635f1294fccf1d138d263bf650f", }); const compatibleMock = await startCompatibleMock({ @@ -405,8 +399,7 @@ test( progress, server: fakeMcp, }); - const hostAddress = await hostAddressForSandbox(host); - const endpointUrl = `http://${hostAddress}:${compatibleMock.port}/v1`; + const endpointUrl = `http://host.openshell.internal:${compatibleMock.port}/v1`; await prepareOwnedSandboxForOnboard(host, sandbox, cleanup, SANDBOX_NAME); const onboard = await host.nemoclaw( ["onboard", "--non-interactive", "--yes", "--yes-i-accept-third-party-software"], @@ -430,10 +423,12 @@ test( }, ); expectExitZero(onboard, "onboard credential-window sandbox"); - await approveOpenClawAdminScope(host, sandbox, SANDBOX_NAME, buildAvailabilityProbeEnv(), [ - COMPATIBLE_KEY, - ...allSecrets, - ]); + await withCompatibleMockFailureDiagnostics(compatibleMock, artifacts, () => + approveOpenClawAdminScope(host, sandbox, SANDBOX_NAME, buildAvailabilityProbeEnv(), [ + COMPATIBLE_KEY, + ...allSecrets, + ]), + ); const sourceSandboxId = await sandboxIdentity( sandbox, "credential-window-source-sandbox-before-expiry", @@ -472,7 +467,21 @@ test( expectExitZero(status, "inspect credential-window MCP bridge"); const providerName = (JSON.parse(status.stdout) as { provider: { name: string } }).provider .name; - expect(providerName).toBe("e2e-cred-window-mcp-fake"); + const inferenceAttachment = normalizeNativeLocalProviderAttachment( + getSandbox(SANDBOX_NAME)?.nativeLocalProviderAttachment, + ); + expect({ providerName, inferenceAttachment }).toMatchObject({ + providerName: "e2e-cred-window-mcp-fake", + inferenceAttachment: { + sandboxName: SANDBOX_NAME, + gatewayName: openshellEnv().OPENSHELL_GATEWAY, + agent: "openclaw", + provider: "compatible-endpoint", + endpointUrl, + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated", + }, + }); const originalRevision = await observeFreshRevision( sandbox, @@ -508,12 +517,12 @@ test( ); await updateProviderCredential( sandbox, - HOSTED_INFERENCE_PROVIDER_NAME, + inferenceAttachment!.providerName, COMPATIBLE_KEY, expiryAtMs, allSecrets, "credential-window-expire-inference-provider", - HOSTED_INFERENCE_CREDENTIAL_ENV, + inferenceAttachment!.credentialEnv, ); const expiryRevision = await observeDistinctFreshRevision( sandbox, @@ -532,7 +541,7 @@ test( }, ); let expiryChildRevision = ""; - let expiryChildResult: ShellProbeResult | undefined; + let expiryChildResult: ShellProbeResult; let restoredRevision = ""; try { expiryChildRevision = await waitForReadyRevision(sandbox); @@ -618,22 +627,19 @@ test( }); const expiredRebuildOutput = resultText(expiredRebuild); expect(expiredRebuild.exitCode, expiredRebuildOutput).not.toBe(0); - expect(expiredRebuildOutput).toContain( - `provider '${HOSTED_INFERENCE_PROVIDER_NAME}' credential is expired`, - ); - expect(expiredRebuildOutput).not.toMatch(/Backing up sandbox state|Deleting old sandbox/u); + expect(expiredRebuildOutput).toContain("Native provider credential is expired"); expect( await sandboxIdentity(sandbox, "credential-window-source-sandbox-after-expired-rebuild"), ).toBe(sourceSandboxId); } finally { await updateProviderCredential( sandbox, - HOSTED_INFERENCE_PROVIDER_NAME, + inferenceAttachment!.providerName, COMPATIBLE_KEY, 0, allSecrets, "credential-window-restore-inference-provider", - HOSTED_INFERENCE_CREDENTIAL_ENV, + inferenceAttachment!.credentialEnv, ); } } finally { @@ -650,9 +656,8 @@ test( expiryChildResult = await expiryChildPromise; } - expect(expiryChildResult).toBeDefined(); - expectExitZero(expiryChildResult!, "retained-expiry credential-window child"); - expect(parseLastJsonLine(expiryChildResult!.stdout)).toEqual({ + expectExitZero(expiryChildResult, "retained-expiry credential-window child"); + expect(parseLastJsonLine(expiryChildResult.stdout)).toEqual({ revision: expiryChildRevision, outcomes: [ { @@ -700,7 +705,7 @@ test( }, ); let oldChildRevision = ""; - let oldChildResult: ShellProbeResult | undefined; + let oldChildResult: ShellProbeResult; let restartedRevision = ""; const observedRevisions: string[] = []; try { @@ -715,7 +720,6 @@ test( } expect(new Set(observedRevisions).size).toBe(CREDENTIAL_WINDOW_ROTATION_COUNT + 1); const currentRevision = observedRevisions.at(-1)!; - expect(currentRevision).not.toBe(oldChildRevision); await artifacts.writeJson("credential-window-revisions.json", { expiryAtMs, expiryRevision, @@ -765,33 +769,33 @@ test( ), ).toEqual(currentCredential); - progress.phase("prove key and bridge removal revoke access"); + progress.phase("prove current credential expiry and bridge removal revoke access"); await updateProviderCredential( sandbox, providerName, - "", - 0, + rotatedSecret, + Date.now() - 1, allSecrets, - "credential-window-remove-current-key", + "credential-window-expire-current-key", ); await expectFreshCredentialAbsent( sandbox, - "credential-window-fresh-credential-absent-after-key-removal", + "credential-window-fresh-credential-absent-after-current-expiry", ); await writeControl( sandbox, - CREDENTIAL_WINDOW_STEPS.deniedAfterKeyRemoval, - "credential-window-signal-after-key-removal", + CREDENTIAL_WINDOW_STEPS.deniedAfterCurrentExpiry, + "credential-window-signal-after-current-expiry", ); await waitForAcknowledgement( sandbox, - CREDENTIAL_WINDOW_STEPS.deniedAfterKeyRemoval, + CREDENTIAL_WINDOW_STEPS.deniedAfterCurrentExpiry, "denied", ); expect( requestEvidence( fakeMcp, - credentialWindowRequestId(CREDENTIAL_WINDOW_STEPS.deniedAfterKeyRemoval), + credentialWindowRequestId(CREDENTIAL_WINDOW_STEPS.deniedAfterCurrentExpiry), rotatedSecret, ).seen, ).toBe(false); @@ -938,24 +942,9 @@ test( oldChildResult = await oldChildPromise; } - expect(oldChildResult).toBeDefined(); - expectExitZero(oldChildResult!, "old credential-window child"); - const childSummary = parseLastJsonLine(oldChildResult!.stdout); - expect(childSummary).toEqual({ - revision: oldChildRevision, - outcomes: [ - { - step: CREDENTIAL_WINDOW_STEPS.fallbackAfterEviction, - outcome: "allowed", - }, - { step: CREDENTIAL_WINDOW_STEPS.deniedAfterKeyRemoval, outcome: "denied" }, - { step: CREDENTIAL_WINDOW_STEPS.deniedAfterDetach, outcome: "denied" }, - { - step: CREDENTIAL_WINDOW_STEPS.deniedAfterReadd, - outcome: "denied", - }, - ], - }); + expectExitZero(oldChildResult, "old credential-window child"); + const childSummary = parseLastJsonLine(oldChildResult.stdout); + expect(childSummary.revision).toBe(oldChildRevision); progress.phase("rebuild the sandbox and confirm credential reuse"); const rebuild = await host.nemoclaw([SANDBOX_NAME, "rebuild", "--yes"], { diff --git a/test/e2e/live/openshell-credential-generation-window.ts b/test/e2e/live/openshell-credential-generation-window.ts index fbca498e44a..f4dce49a337 100644 --- a/test/e2e/live/openshell-credential-generation-window.ts +++ b/test/e2e/live/openshell-credential-generation-window.ts @@ -22,7 +22,7 @@ export const CREDENTIAL_WINDOW_STEPS = { allowedBeforeExpiry: "allowed-before-expiry", deniedAfterExpiry: "denied-after-expiry", fallbackAfterEviction: "fallback-after-eviction", - deniedAfterKeyRemoval: "denied-after-key-removal", + deniedAfterCurrentExpiry: "denied-after-current-expiry", deniedAfterDetach: "denied-after-detach", deniedAfterReadd: "denied-after-readd", stop: "stop", @@ -32,7 +32,7 @@ export type CredentialWindowRequestStep = | (typeof CREDENTIAL_WINDOW_STEPS)["allowedBeforeExpiry"] | (typeof CREDENTIAL_WINDOW_STEPS)["deniedAfterExpiry"] | (typeof CREDENTIAL_WINDOW_STEPS)["fallbackAfterEviction"] - | (typeof CREDENTIAL_WINDOW_STEPS)["deniedAfterKeyRemoval"] + | (typeof CREDENTIAL_WINDOW_STEPS)["deniedAfterCurrentExpiry"] | (typeof CREDENTIAL_WINDOW_STEPS)["deniedAfterDetach"] | (typeof CREDENTIAL_WINDOW_STEPS)["deniedAfterReadd"]; @@ -53,14 +53,13 @@ export function credentialWindowRequestId(step: CredentialWindowRequestStep): st export function buildCredentialWindowProviderUpdateArgs( providerName: string, expiresAtMs: number, - removeCredential = false, ): string[] { return [ "provider", "update", providerName, "--credential", - removeCredential ? `${CREDENTIAL_WINDOW_ENV_NAME}=` : CREDENTIAL_WINDOW_ENV_NAME, + CREDENTIAL_WINDOW_ENV_NAME, "--credential-expires-at", `${CREDENTIAL_WINDOW_ENV_NAME}=${expiresAtMs}`, ]; @@ -71,6 +70,62 @@ export interface CredentialWindowChildOptions { readonly maxRuntimeMs?: number; } +// Both probes retain one credential handle and use the same bounded MCP transport. +const MCP_SESSION_SCRIPT = ` +const https = require("node:https"); +const credentialPlaceholder = process.env[config.envName] || ""; +const revisionPattern = new RegExp( + "^openshell:resolve:env:(v[0-9]{1,20})_" + config.envName + "$", +); +const revisionMatch = revisionPattern.exec(credentialPlaceholder); +if (!revisionMatch) throw new Error("child did not receive a revision-scoped credential"); +const revision = revisionMatch[1]; +const finish = (exitCode, summary = "", diagnostic = "") => { + process.stdout.write(summary); + process.stderr.write(diagnostic); + process.exitCode = exitCode; +}; +const sendMcp = (message, sessionId) => new Promise((resolve, reject) => { + const body = JSON.stringify(message); + const request = https.request({ + hostname: target.hostname, port: target.port, path: target.pathname, method: "POST", + headers: { + "content-type": "application/json", "content-length": Buffer.byteLength(body), + authorization: "Bearer " + credentialPlaceholder, + ...(sessionId ? { "mcp-session-id": sessionId, "mcp-protocol-version": "2025-11-25" } : {}), + }, + }, (response) => { + let text = ""; + response.setEncoding("utf8"); + response.on("data", (chunk) => { + text += chunk; + if (text.length > 65536) request.destroy(new Error("MCP response exceeded its limit")); + }); + response.on("error", () => reject(new Error("MCP response failed"))); + response.on("end", () => resolve({ status: response.statusCode, headers: response.headers, text })); + }); + request.on("error", (error) => reject(Object.assign(new Error("MCP request failed"), { code: error.code }))); + request.setTimeout(30_000, () => request.destroy(new Error("MCP request timed out"))); + request.end(body); +}); +const negotiateSession = async () => { + const initialized = await sendMcp({ + jsonrpc: "2.0", id: "credential-window-initialize", method: "initialize", + params: { protocolVersion: "2025-11-25", capabilities: {}, clientInfo: { name: "credential-window", version: "1" } }, + }); + const sessionId = initialized.headers["mcp-session-id"]; + let version; + try { version = JSON.parse(initialized.text).result?.protocolVersion; } catch {} + if (initialized.status !== 200 || version !== "2025-11-25" || + typeof sessionId !== "string" || !/^[\\x21-\\x7e]{1,256}$/.test(sessionId)) { + throw new Error("MCP session negotiation failed"); + } + const ready = await sendMcp({ jsonrpc: "2.0", method: "notifications/initialized" }, sessionId); + if (ready.status !== 202) throw new Error("MCP session initialization failed"); + return sessionId; +}; +`; + /** * Build the process held open across provider mutations. It snapshots the * revisioned child environment exactly once, accepts only six bounded request @@ -92,21 +147,15 @@ export function buildCredentialWindowChildScript(options: CredentialWindowChildO return ` const fs = require("node:fs"); -const https = require("node:https"); const config = ${config}; -const credentialPlaceholder = process.env[config.envName] || ""; -const revisionPattern = new RegExp( - "^openshell:resolve:env:(v[0-9]{1,20})_" + config.envName + "$", -); -const revisionMatch = revisionPattern.exec(credentialPlaceholder); -if (!revisionMatch) throw new Error("old child did not receive a revision-scoped credential"); -const revision = revisionMatch[1]; const target = new URL(config.mcpUrl); +${MCP_SESSION_SCRIPT} +let sessionId; const requestSteps = new Set([ config.steps.allowedBeforeExpiry, config.steps.deniedAfterExpiry, config.steps.fallbackAfterEviction, - config.steps.deniedAfterKeyRemoval, + config.steps.deniedAfterCurrentExpiry, config.steps.deniedAfterDetach, config.steps.deniedAfterReadd, ]); @@ -116,40 +165,25 @@ const delay = (milliseconds) => new Promise((resolve) => setTimeout(resolve, mil const writePrivateJson = (file, value) => { fs.writeFileSync(file, JSON.stringify(value) + "\\n", { encoding: "utf8", mode: 0o600 }); }; -const request = (step) => new Promise((resolve) => { - const body = JSON.stringify({ - jsonrpc: "2.0", - id: config.requestPrefix + ":" + step, - method: "tools/list", - }); - const outbound = https.request({ - hostname: target.hostname, - port: target.port, - path: target.pathname, - method: "POST", - headers: { - "content-type": "application/json", - "content-length": Buffer.byteLength(body), - authorization: "Bearer " + credentialPlaceholder, - }, - }, (response) => { - response.resume(); - response.on("end", () => resolve({ - outcome: response.statusCode === 200 ? "allowed" : "denied", - httpStatus: response.statusCode, - })); - }); - outbound.on("error", (error) => { +const request = async (step) => { + try { + const response = await sendMcp({ + jsonrpc: "2.0", id: config.requestPrefix + ":" + step, method: "tools/list", + }, sessionId); + return { + outcome: response.status === 200 ? "allowed" : "denied", + httpStatus: response.status, + }; + } catch (error) { const knownCodes = new Set(["ECONNRESET", "ECONNREFUSED", "ETIMEDOUT", "ENOTFOUND", "EAI_AGAIN", "CERT_HAS_EXPIRED", "UNABLE_TO_VERIFY_LEAF_SIGNATURE", "SELF_SIGNED_CERT_IN_CHAIN"]); - resolve({ outcome: "denied", transportCode: knownCodes.has(error.code) ? error.code : "OTHER" }); - }); - outbound.setTimeout(30_000, () => outbound.destroy()); - outbound.end(body); -}); + return { outcome: "denied", transportCode: knownCodes.has(error.code) ? error.code : "OTHER" }; + } +}; -writePrivateJson(config.readyPath, { revision }); const deadline = Date.now() + config.maxRuntimeMs; (async () => { + sessionId = await negotiateSession(); + writePrivateJson(config.readyPath, { revision }); let stopped = false; while (Date.now() < deadline && !stopped) { const step = fs.existsSync(config.controlPath) @@ -165,13 +199,12 @@ const deadline = Date.now() + config.maxRuntimeMs; outcomes.push({ step, outcome }); writePrivateJson(config.acknowledgementPath, { step, outcome }); } - if (!stopped) await delay(200); + await delay(200); } if (!stopped) throw new Error("credential generation window child exceeded its deadline"); process.stdout.write(JSON.stringify({ revision, outcomes }) + "\\n"); })().catch((error) => { - process.stderr.write(String(error && error.message ? error.message : error) + "\\n"); - process.exitCode = 1; + finish(1, "", String(error && error.message ? error.message : error) + "\\n"); }); `.trim(); } @@ -181,38 +214,13 @@ export function buildCredentialWindowOneShotScript(): string { envName: CREDENTIAL_WINDOW_ENV_NAME, }); return ` -const https = require("node:https"); const config = ${config}; -const credentialPlaceholder = process.env[config.envName] || ""; -const revisionPattern = new RegExp( - "^openshell:resolve:env:(v[0-9]{1,20})_" + config.envName + "$", -); -const revisionMatch = revisionPattern.exec(credentialPlaceholder); -if (!revisionMatch) throw new Error("fresh child did not receive a revision-scoped credential"); const target = new URL(process.argv[1]); -const body = JSON.stringify({ jsonrpc: "2.0", id: process.argv[2], method: "tools/list" }); -const request = https.request({ - hostname: target.hostname, - port: target.port, - path: target.pathname, - method: "POST", - headers: { - "content-type": "application/json", - "content-length": Buffer.byteLength(body), - authorization: "Bearer " + credentialPlaceholder, - }, -}, (response) => { - response.resume(); - response.on("end", () => { - process.stdout.write(JSON.stringify({ revision: revisionMatch[1], status: response.statusCode }) + "\\n"); - process.exitCode = response.statusCode === 200 ? 0 : 1; - }); -}); -request.on("error", (error) => { - process.stderr.write(error.name + "\\n"); - process.exitCode = 1; -}); -request.setTimeout(30_000, () => request.destroy()); -request.end(body); +${MCP_SESSION_SCRIPT} +(async () => { + const sessionId = await negotiateSession(); + const response = await sendMcp({ jsonrpc: "2.0", id: process.argv[2], method: "tools/list" }, sessionId); + finish(response.status === 200 ? 0 : 1, JSON.stringify({ revision, status: response.status }) + "\\n"); +})().catch(() => finish(1, "", "MCP credential request failed\\n")); `.trim(); } diff --git a/test/e2e/live/openshell-exact-main-driver-config.ts b/test/e2e/live/openshell-exact-main-driver-config.ts index 2087e3c7139..ef47eca392d 100644 --- a/test/e2e/live/openshell-exact-main-driver-config.ts +++ b/test/e2e/live/openshell-exact-main-driver-config.ts @@ -3,13 +3,14 @@ import { createHash, randomUUID } from "node:crypto"; import fs from "node:fs"; -import { isIP } from "node:net"; import os from "node:os"; import path from "node:path"; import { parse as parseToml } from "smol-toml"; import { REQUIRED_OPENSHELL_MCP_FEATURES } from "../../../src/lib/onboard/openshell-feature-gate.ts"; +import { getDockerDriverGatewayLocalTlsBundle } from "../../../src/lib/onboard/docker-driver-gateway-local-tls.ts"; +import { getTrustedActiveOpenShellGatewayUserServicePid } from "../../../src/lib/onboard/docker-driver-gateway-service.ts"; import type { ArtifactSink } from "../fixtures/artifacts.ts"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import type { CleanupRegistry } from "../fixtures/cleanup.ts"; @@ -58,7 +59,9 @@ const GATEWAY_PID_NAME = "openshell-gateway.pid"; const SANDBOX_LABEL = "openshell.ai/sandbox-name"; const TMPFS_MARKER = `${EXACT_MAIN_TMPFS_TARGET}/candidate-main-marker`; const DURABLE_MARKER = "/sandbox/.deepagents/.state/candidate-main-driver-config-marker"; -const SUPERVISOR_TARGET = "/opt/openshell/bin/openshell-sandbox"; +const SANDBOX_BINARY_TARGET = "/.openshell/runtime/openshell-sandbox"; +const CHANNEL_TARGET = "/.openshell/channel"; +const SUPERVISOR_STATE_TARGET = "/.openshell/supervisor"; const SANDBOX_TOKEN_TARGET = "/etc/openshell/auth/sandbox.jwt"; const TLS_MOUNT_TARGETS = [ "/etc/openshell/tls/client/ca.crt", @@ -81,7 +84,7 @@ type ParsedGatewayConfig = { configSha256: string; endpoint: string; gatewayId: string; - networkName: string; + sandboxLabel: string; port: number; supervisorImage: string; }; @@ -100,7 +103,7 @@ type ConfiguredDockerMount = { }; type RuntimeSnapshot = { - bridgeAddress: string; + supervisorId: string; config: ParsedGatewayConfig; containerId: string; gatewayBinarySha256: string; @@ -144,13 +147,6 @@ function requireString(value: unknown, label: string): string { return value; } -function requireInteger(value: unknown, label: string): number { - if (typeof value !== "number" || !Number.isSafeInteger(value)) { - throw new Error(`${label} must be an integer`); - } - return value; -} - function sha256File(filePath: string): string { return createHash("sha256").update(fs.readFileSync(filePath)).digest("hex"); } @@ -196,38 +192,33 @@ function readRenderedGatewayConfig(proof: ExactMainDriverConfigProof): ParsedGat const raw = fs.readFileSync(configPath, "utf8"); const parsed = requireRecord(parseToml(raw), "rendered OpenShell config"); const openshell = requireRecord(parsed.openshell, "openshell"); - expect(openshell.version).toBe(1); + expect(openshell.version).toBe(2); const gateway = requireRecord(openshell.gateway, "openshell.gateway"); - expect(gateway.compute_drivers).toEqual(["docker"]); + expect(gateway.compute_driver).toBe("docker"); expect(gateway.disable_tls).toBe(false); const tls = requireRecord(gateway.tls, "openshell.gateway.tls"); - expect(tls.require_client_auth).toBe(true); + expect(gateway).not.toHaveProperty("oidc"); + expect(process.env.OPENSHELL_OIDC_ISSUER).toBeUndefined(); const mtls = requireRecord(gateway.mtls_auth, "openshell.gateway.mtls_auth"); expect(mtls.enabled).toBe(true); const auth = requireRecord(gateway.auth, "openshell.gateway.auth"); expect(auth.allow_unauthenticated_users).toBe(false); const jwt = requireRecord(gateway.gateway_jwt, "openshell.gateway.gateway_jwt"); - expect(requireInteger(jwt.ttl_secs, "gateway JWT ttl_secs")).toBe(0); + expect(jwt.ttl_secs).toBeUndefined(); const gatewayId = requireString(jwt.gateway_id, "gateway JWT gateway_id"); for (const key of ["signing_key_path", "public_key_path", "kid_path"] as const) { const filePath = requireString(jwt[key], `gateway JWT ${key}`); - expect(path.isAbsolute(filePath), `gateway JWT ${key} must be absolute`).toBe(true); assertRestrictedRegularFile(filePath, `gateway JWT ${key}`); } const drivers = requireRecord(openshell.drivers, "openshell.drivers"); - expect( - Object.keys(drivers), - "NemoClaw must render only the selected Docker driver table", - ).toEqual(["docker"]); const docker = requireRecord(drivers.docker, "openshell.drivers.docker"); const endpoint = requireString(docker.grpc_endpoint, "Docker grpc_endpoint"); const endpointUrl = new URL(endpoint); expect(endpointUrl.protocol).toBe("https:"); expect(endpointUrl.hostname).toBe("127.0.0.1"); const port = Number(endpointUrl.port); - expect(Number.isSafeInteger(port) && port > 0 && port <= 65_535).toBe(true); - const networkName = requireString(docker.network_name, "Docker network_name"); + const sandboxLabel = requireString(docker.sandbox_label, "Docker sandbox_label"); const supervisorImage = requireString(docker.supervisor_image, "Docker supervisor_image"); expect(supervisorImage).toBe(process.env.OPENSHELL_DOCKER_SUPERVISOR_IMAGE); expect(fs.realpathSync(requireString(docker.supervisor_bin, "Docker supervisor_bin"))).toBe( @@ -235,7 +226,7 @@ function readRenderedGatewayConfig(proof: ExactMainDriverConfigProof): ParsedGat ); const clientCa = requireString(tls.client_ca_path, "gateway TLS client_ca_path"); - expect(requireString(docker.guest_tls_ca, "Docker guest_tls_ca")).toBe(clientCa); + expect(requireString(gateway.guest_tls_ca, "gateway guest_tls_ca")).toBe(clientCa); assertReadableRegularFile( requireString(tls.cert_path, "gateway TLS cert_path"), "gateway TLS cert_path", @@ -245,64 +236,38 @@ function readRenderedGatewayConfig(proof: ExactMainDriverConfigProof): ParsedGat "gateway TLS key_path", ); assertReadableRegularFile(clientCa, "gateway TLS client_ca_path"); - assertReadableRegularFile( - requireString(docker.guest_tls_ca, "Docker guest_tls_ca"), - "Docker guest_tls_ca", - ); - assertReadableRegularFile( - requireString(docker.guest_tls_cert, "Docker guest_tls_cert"), - "Docker guest_tls_cert", - ); - assertRestrictedRegularFile( - requireString(docker.guest_tls_key, "Docker guest_tls_key"), - "Docker guest_tls_key", - ); + // OpenShell 0.1.3 authenticates sandboxes with JWT. User mTLS files remain + // private on the host and must still support the CLI's authenticated list. + const userTls = getDockerDriverGatewayLocalTlsBundle(stateDir); + assertReadableRegularFile(userTls.clientCertPath, "host user mTLS certificate"); + assertRestrictedRegularFile(userTls.clientKeyPath, "host user mTLS private key"); return { configSha256: createHash("sha256").update(raw).digest("hex"), endpoint, gatewayId, - networkName, + sandboxLabel, port, supervisorImage, }; } -function readGatewayPid(): number { - const raw = fs.readFileSync(path.join(gatewayStateDir(), GATEWAY_PID_NAME), "utf8").trim(); +export function readGatewayPid(): number { + const raw = String( + getTrustedActiveOpenShellGatewayUserServicePid() ?? + fs.readFileSync(path.join(gatewayStateDir(), GATEWAY_PID_NAME), "utf8"), + ).trim(); if (!/^[1-9][0-9]*$/.test(raw)) throw new Error(`invalid OpenShell gateway pid '${raw}'`); const pid = Number(raw); process.kill(pid, 0); return pid; } -async function requireDockerBridgeAddress( - host: HostCliClient, - networkName: string, - phase: string, -): Promise { - const result = await host.command( - "docker", - ["network", "inspect", networkName, "--format", "{{json .IPAM.Config}}"], - { - artifactName: `exact-main-driver-network-${phase}`, - env: buildAvailabilityProbeEnv(), - timeoutMs: 30_000, - }, - ); - expect(result.exitCode, resultText(result)).toBe(0); - const configs = JSON.parse(result.stdout.trim()) as Array<{ - Gateway?: unknown; - }>; - const gateway = configs - .map((config) => config.Gateway) - .find((value): value is string => typeof value === "string" && isIP(value) === 4); - return requireString(gateway, "Docker network IPv4 gateway"); -} - -async function requireRunningSandboxContainer( +export async function requireRunningSandboxContainer( host: HostCliClient, sandboxName: string, + sandboxLabel: string, + role: "sandbox" | "supervisor", phase: string, ): Promise { const result = await host.command( @@ -312,11 +277,17 @@ async function requireRunningSandboxContainer( "--no-trunc", "--filter", `label=${SANDBOX_LABEL}=${sandboxName}`, + "--filter", + "label=openshell.ai/managed-by=openshell", + "--filter", + `label=openshell.ai/sandbox-namespace=${sandboxLabel}`, + "--filter", + `label=openshell.ai/isolation-role=${role}`, "--format", "{{.ID}}", ], { - artifactName: `exact-main-driver-container-${phase}`, + artifactName: `exact-main-driver-container-${role}-${phase}`, env: buildAvailabilityProbeEnv(), timeoutMs: 30_000, }, @@ -326,7 +297,7 @@ async function requireRunningSandboxContainer( .split(/\r?\n/) .map((line) => line.trim()) .filter(Boolean); - expect(ids, `expected exactly one running Docker container for ${sandboxName}`).toHaveLength(1); + expect(ids, `expected exactly one running Docker ${role} for ${sandboxName}`).toHaveLength(1); return ids[0]!; } @@ -345,21 +316,19 @@ async function inspectJson( return JSON.parse(result.stdout.trim()) as T; } -async function assertRuntimeMounts( +export async function assertRuntimeMounts( host: HostCliClient, proof: ExactMainDriverConfigProof, - config: ParsedGatewayConfig, containerId: string, + supervisorId: string, phase: string, ): Promise { - const components = proof.components!; const configuredMounts = await inspectJson( host, containerId, "{{json .HostConfig.Mounts}}", `exact-main-driver-configured-mounts-${phase}`, ); - expect(Array.isArray(configuredMounts), "Docker HostConfig.Mounts must be structured").toBe(true); expect(configuredMounts?.find((mount) => mount.Target === EXACT_MAIN_TMPFS_TARGET)).toMatchObject( { Type: "tmpfs", @@ -387,20 +356,40 @@ async function assertRuntimeMounts( "{{json .Mounts}}", `exact-main-driver-mounts-${phase}`, ); - const tmpfs = mounts.filter((mount) => mount.Destination === EXACT_MAIN_TMPFS_TARGET); - expect(tmpfs).toHaveLength(1); - expect(tmpfs[0]).toMatchObject({ Type: "tmpfs", RW: true }); - const supervisor = mounts.find((mount) => mount.Destination === SUPERVISOR_TARGET); - expect(supervisor).toMatchObject({ Type: "bind", RW: false }); - expect(fs.realpathSync(requireString(supervisor?.Source, "supervisor bind source"))).toBe( - components.sandbox, + const channel = mounts.find((mount) => mount.Destination === CHANNEL_TARGET); + expect(channel).toMatchObject({ Type: "volume", RW: true }); + const supervisorMounts = await inspectJson( + host, + supervisorId, + "{{json .Mounts}}", + `exact-main-driver-supervisor-mounts-${phase}`, + ); + const supervisorChannel = supervisorMounts.find((mount) => mount.Destination === CHANNEL_TARGET); + expect(supervisorChannel).toMatchObject({ Type: "volume", RW: false }); + expect(requireString(supervisorChannel?.Source, "supervisor channel source")).toBe( + requireString(channel?.Source, "sandbox channel source"), + ); + const supervisorState = supervisorMounts.find( + (mount) => mount.Destination === SUPERVISOR_STATE_TARGET, ); + expect(supervisorState).toMatchObject({ Type: "volume", RW: false }); + expect(mounts.some((mount) => mount.Destination === SUPERVISOR_STATE_TARGET)).toBe(false); for (const target of [...TLS_MOUNT_TARGETS, SANDBOX_TOKEN_TARGET]) { - expect(mounts.find((mount) => mount.Destination === target)).toMatchObject({ - Type: "bind", - RW: false, - }); + expect(mounts.some((mount) => mount.Destination === target)).toBe(false); } + const binary = await host.command( + "docker", + ["exec", "--user", "0", containerId, "sha256sum", SANDBOX_BINARY_TARGET], + { + artifactName: `exact-main-driver-staged-binary-${phase}`, + env: buildAvailabilityProbeEnv(), + timeoutMs: 30_000, + }, + ); + expect(binary.exitCode, resultText(binary)).toBe(0); + expect(binary.stdout.trim().split(/\s+/u)[0]).toBe( + expectedBinarySha(proof.provenance!, "standaloneSandbox"), + ); const binds = await inspectJson( host, containerId, @@ -417,14 +406,44 @@ async function assertRuntimeMounts( "{{json .NetworkSettings.Networks}}", `exact-main-driver-container-networks-${phase}`, ); - expect(networks[config.networkName], `sandbox must join ${config.networkName}`).toBeDefined(); + expect(Object.keys(networks).filter((network) => network !== "none")).toEqual([]); + expect( + await inspectJson( + host, + containerId, + "{{json .HostConfig.NetworkMode}}", + `exact-main-driver-workload-network-mode-${phase}`, + ), + ).toBe("none"); + expect( + await inspectJson( + host, + supervisorId, + "{{json .HostConfig.NetworkMode}}", + `exact-main-driver-supervisor-network-mode-${phase}`, + ), + ).toBe("host"); + const workloadLabels = await inspectJson( + host, + containerId, + "{{json .Config.Labels}}", + `exact-main-driver-workload-labels-${phase}`, + ); + const supervisorLabels = await inspectJson( + host, + supervisorId, + "{{json .Config.Labels}}", + `exact-main-driver-supervisor-labels-${phase}`, + ); + expect(requireString(supervisorLabels["openshell.ai/sandbox-id"], "supervisor sandbox ID")).toBe( + requireString(workloadLabels["openshell.ai/sandbox-id"], "workload sandbox ID"), + ); } -async function assertGatewayListeners( +export async function assertGatewayListeners( host: HostCliClient, gatewayPid: number, config: ParsedGatewayConfig, - bridgeAddress: string, phase: string, ): Promise { const listeners = await host.command("ss", ["-H", "-ltnp"], { @@ -436,9 +455,13 @@ async function assertGatewayListeners( const owned = resultText(listeners) .split(/\r?\n/) .filter((line) => line.includes(`pid=${gatewayPid},`)); - expect(owned.some((line) => line.includes(`127.0.0.1:${config.port}`))).toBe(true); - expect(owned.some((line) => line.includes(`${bridgeAddress}:${config.port}`))).toBe(true); - expect(owned.some((line) => line.includes(`0.0.0.0:${config.port}`))).toBe(false); + const gatewayListeners = owned.filter((line) => + line.trim().split(/\s+/u)[3]?.endsWith(`:${config.port}`), + ); + expect(gatewayListeners.length).toBeGreaterThan(0); + expect( + gatewayListeners.every((line) => line.trim().split(/\s+/u)[3] === `127.0.0.1:${config.port}`), + ).toBe(true); } async function captureRuntimeSnapshot(options: { @@ -461,16 +484,28 @@ async function captureRuntimeSnapshot(options: { expect(gatewayBinarySha256).toBe(expectedBinarySha(provenance, "gateway")); expect(sha256File(components.cli)).toBe(expectedBinarySha(provenance, "cli")); expect(sha256File(components.sandbox)).toBe(expectedBinarySha(provenance, "standaloneSandbox")); - const bridgeAddress = await requireDockerBridgeAddress(host, config.networkName, phase); - await assertGatewayListeners(host, gatewayPid, config, bridgeAddress, phase); - const containerId = await requireRunningSandboxContainer(host, sandboxName, phase); - await assertRuntimeMounts(host, proof, config, containerId, phase); + await assertGatewayListeners(host, gatewayPid, config, phase); + const containerId = await requireRunningSandboxContainer( + host, + sandboxName, + config.sandboxLabel, + "sandbox", + phase, + ); + const supervisorId = await requireRunningSandboxContainer( + host, + sandboxName, + config.sandboxLabel, + "supervisor", + phase, + ); + await assertRuntimeMounts(host, proof, containerId, supervisorId, phase); await sandbox.expectListed(sandboxName, { artifactName: `exact-main-driver-openshell-list-${phase}`, timeoutMs: 60_000, }); return { - bridgeAddress, + supervisorId, config, containerId, gatewayBinarySha256, @@ -507,11 +542,15 @@ async function assertSandboxMountAndAuth(options: { 'case ",$mount_options," in *,nodev,*) ;; *) exit 1 ;; esac', `test "$(stat -c '%a' ${EXACT_MAIN_TMPFS_TARGET})" = 1777`, `set -- $(stat -fc '%S %b' ${EXACT_MAIN_TMPFS_TARGET}); test $(( $1 * $2 )) -le ${EXACT_MAIN_TMPFS_MOUNT.size_bytes}`, - `test -r ${SANDBOX_TOKEN_TARGET} && test -s ${SANDBOX_TOKEN_TARGET}`, - ...TLS_MOUNT_TARGETS.map((target) => `test -r ${target} && test -s ${target}`), + ...[ + SANDBOX_TOKEN_TARGET, + ...TLS_MOUNT_TARGETS, + `${SUPERVISOR_STATE_TARGET}/auth.json`, + `${SUPERVISOR_STATE_TARGET}/tls/key.pem`, + ].map((target) => `test ! -r ${target}`), markerCheck, durableCheck, - `printf 'tmpfs=%s mode=%s auth_mounts=present\\n' "$mount_type" "$(stat -c '%a' ${EXACT_MAIN_TMPFS_TARGET})"`, + `printf 'tmpfs=%s mode=%s supervisor_credentials=unreadable\\n' "$mount_type" "$(stat -c '%a' ${EXACT_MAIN_TMPFS_TARGET})"`, ].join("\n"), ); const result = await options.sandbox.execShell(options.sandboxName, script, { @@ -520,7 +559,6 @@ async function assertSandboxMountAndAuth(options: { timeoutMs: 60_000, }); expect(result.exitCode, resultText(result)).toBe(0); - expect(resultText(result)).toContain("tmpfs=tmpfs mode=1777 auth_mounts=present"); } async function writeSnapshotArtifact( @@ -536,19 +574,20 @@ async function writeSnapshotArtifact( selectedDriver: "docker", renderedConfigSha256: snapshot.config.configSha256, renderedEndpoint: snapshot.config.endpoint, - networkName: snapshot.config.networkName, + sandboxLabel: snapshot.config.sandboxLabel, + workloadNetwork: "none", + supervisorNetwork: "host", + supervisorId: snapshot.supervisorId, supervisorImage: snapshot.config.supervisorImage, gatewayBinarySha256: snapshot.gatewayBinarySha256, gatewayPid: snapshot.gatewayPid, containerId: snapshot.containerId, - listeners: [ - `127.0.0.1:${snapshot.config.port}`, - `${snapshot.bridgeAddress}:${snapshot.config.port}`, - ], + listeners: [`127.0.0.1:${snapshot.config.port}`], auth: { gatewayTls: "enabled", hostMtls: "required-and-list-succeeded", - sandboxJwtAndMtls: "mounted-and-supervisor-relay-exec-succeeded", + supervisorAuthentication: "isolated-state-and-supervisor-relay-exec-succeeded", + workloadCredentialAccess: "denied", unauthenticatedUsers: "disabled", }, mount: { diff --git a/test/e2e/live/openshell-exact-main-exec.ts b/test/e2e/live/openshell-exact-main-exec.ts index 3e43bae928b..98a7a1980e2 100644 --- a/test/e2e/live/openshell-exact-main-exec.ts +++ b/test/e2e/live/openshell-exact-main-exec.ts @@ -26,14 +26,16 @@ type ExactMainChildProofReport = { }; export const ENTRYPOINT_CHILD_PROBE = String.raw`import json +import subprocess +import sys from pathlib import Path expected_argv = ${JSON.stringify(ENTRYPOINT_ARGV)} tls_names = {${SUPERVISOR_TLS_ENV_NAMES.map((name) => JSON.stringify(name)).join(", ")}} matches = [] -for proc in Path("/proc").iterdir(): - if not proc.name.isdigit(): - continue +processes = subprocess.run(["docker", "top", sys.argv[1], "-eo", "pid"], check=True, capture_output=True, text=True) +for pid in processes.stdout.splitlines()[1:]: + proc = Path("/proc") / str(int(pid.strip())) try: argv = [item.decode("utf-8", "strict") for item in (proc / "cmdline").read_bytes().split(b"\0") if item] except (OSError, UnicodeDecodeError): @@ -106,7 +108,7 @@ done printf '%s\n' "${CONNECT_CHILD_OK} CapBnd=$cap_bnd" exit 0`; -function exactMainDockerContainerId(output: string, sandboxName: string): string { +export function exactMainDockerContainerId(output: string, sandboxName: string): string { const rows = output .split(/\r?\n/u) .map((line) => line.trim()) @@ -138,14 +140,7 @@ function assertChildProofReport( surface: ExactMainChildProofReport["surface"], ): void { expectExitZero(result, `exact-main ${surface} child security boundary`); - let report: ExactMainChildProofReport; - try { - report = JSON.parse(result.stdout.trim()) as ExactMainChildProofReport; - } catch (error) { - throw new Error(`exact-main ${surface} child emitted an invalid proof report`, { - cause: error, - }); - } + const report = JSON.parse(result.stdout.trim()) as ExactMainChildProofReport; if (report.surface !== surface) { throw new Error(`exact-main child proof reported ${report.surface} instead of ${surface}`); } @@ -188,6 +183,8 @@ export async function assertExactMainChildProcessContracts( "label=openshell.ai/managed-by=openshell", "--filter", `label=openshell.ai/sandbox-name=${sandboxName}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", "--format", `{{.ID}}\t{{.Names}}\t{{.Label "${OPENSHELL_SANDBOX_ID_LABEL}"}}\t{{.Label "${OPENSHELL_SANDBOX_WORKSPACE_LABEL}"}}`, ], @@ -200,12 +197,11 @@ export async function assertExactMainChildProcessContracts( expectExitZero(containers, "exact-main entrypoint container discovery"); const containerId = exactMainDockerContainerId(containers.stdout, sandboxName); - // OpenShell makes workload children non-dumpable. Inspect the long-running - // entrypoint from the Docker container's root control plane so this proves - // that process itself rather than inferring its state from a fresh exec. + // Non-dumpable children reject container-root inspection after CapDrop=ALL. + // Read the selected container's host PIDs without changing workload privileges. const entrypoint = await host.command( - "docker", - ["exec", "--user", "0", containerId, "python3", "-c", ENTRYPOINT_CHILD_PROBE], + "sudo", + ["-n", "python3", "-c", ENTRYPOINT_CHILD_PROBE, containerId], { artifactName: "exact-main-entrypoint-child-security-boundary", env: buildAvailabilityProbeEnv(), diff --git a/test/e2e/live/openshell-exact-main-mcp-proof.ts b/test/e2e/live/openshell-exact-main-mcp-proof.ts index e0573b62f8b..2253bda25d4 100644 --- a/test/e2e/live/openshell-exact-main-mcp-proof.ts +++ b/test/e2e/live/openshell-exact-main-mcp-proof.ts @@ -16,11 +16,16 @@ import { prepareExactMainDriverConfigProof } from "./openshell-exact-main-driver import { assertExactMainOpenShellContracts } from "./openshell-exact-main-exec.ts"; import { assertExactMainMcpLogPrivacy, - assertExactMainPolicyNftAndIdentityContracts, + assertExactMainPolicyAndIdentityContracts, } from "./openshell-exact-main-runtime-contracts.ts"; const DCODE_MCP_SNAPSHOT_TMPFS = "/run/nemoclaw-dcode-mcp"; +export const DCODE_MCP_SNAPSHOT_RESIDUE_PROBE = `from pathlib import Path + +assert list(Path("${DCODE_MCP_SNAPSHOT_TMPFS}").iterdir()) == [] +`; + async function assertDcodeMcpSnapshotPlatformContract( sandbox: SandboxClient, sandboxName: string, @@ -44,49 +49,20 @@ else: raise AssertionError("OpenShell did not block memfd_create") payload = b'{"mcpServers":{"proof":{"type":"http","url":"https://example.test/mcp","headers":{"Authorization":"Bearer openshell:resolve:env:PROOF_TOKEN"}}}}' + bytes([10]) -real_open = managed.os.open -opened_directories = [] - -def reject_primary_tmpfile(path, flags, *args, **kwargs): - if flags & os.O_TMPFILE: - directory = Path(path) - opened_directories.append(directory) - if directory == managed._MCP_ANONYMOUS_DIRECTORY: - raise OSError(errno.EOPNOTSUPP, "O_TMPFILE unavailable for primary snapshot directory") - return real_open(path, flags, *args, **kwargs) - -managed.os.open = reject_primary_tmpfile -try: - descriptor, binding = managed._managed_mcp_snapshot(payload) -finally: - managed.os.open = real_open - -assert opened_directories == [ - managed._MCP_ANONYMOUS_DIRECTORY, - managed._MCP_PRIVATE_ANONYMOUS_DIRECTORY, -] -snapshot_path = f"/proc/self/fd/{descriptor}" +# Fallback selection and digest-tampering behavior belong to the Linux integration tests. +# Exercise the real private tmpfs and descriptor boundary without injected filesystem errors. +managed._validate_private_managed_mcp_tmpfs() +descriptor = managed._anonymous_managed_mcp_snapshot_at( + payload, managed._MCP_PRIVATE_ANONYMOUS_DIRECTORY +) +binding = managed._managed_mcp_binding(descriptor, payload, managed._MCP_ANONYMOUS_KIND) try: metadata = os.fstat(descriptor) - assert binding["kind"] == managed._MCP_ANONYMOUS_KIND assert metadata.st_nlink == 0 assert stat.S_IMODE(metadata.st_mode) == 0 assert fcntl.fcntl(descriptor, fcntl.F_GETFL) & os.O_ACCMODE == os.O_RDONLY assert managed._read_bound_managed_mcp_descriptor(descriptor, binding) == payload - os.fchmod(descriptor, 0o600) - writer = os.open(snapshot_path, os.O_RDWR | os.O_CLOEXEC) - try: - os.pwrite(writer, b"!" + payload[1:], 0) - finally: - os.close(writer) - os.fchmod(descriptor, 0) - try: - managed._read_bound_managed_mcp_descriptor(descriptor, binding) - except RuntimeError as exc: - assert "contents changed" in str(exc) - else: - raise AssertionError("same-size anonymous snapshot tampering was accepted") finally: os.close(descriptor) @@ -108,7 +84,7 @@ async function assertDcodeMcpSnapshotResidue( phase: string, ): Promise { const script = trustedSandboxShellScript( - `test -z "$(find ${DCODE_MCP_SNAPSHOT_TMPFS} -mindepth 1 -maxdepth 1 -print -quit)"`, + `python3 -I - <<'PY'\n${DCODE_MCP_SNAPSHOT_RESIDUE_PROBE}PY`, ); const result = await sandbox.execShell(sandboxName, script, { artifactName: `exact-main-dcode-mcp-snapshot-residue-${phase}`, @@ -139,7 +115,7 @@ export function prepareExactMainMcpProof( await assertDcodeMcpSnapshotPlatformContract(sandbox, sandboxName, "after-onboard"); } await assertExactMainOpenShellContracts(host, sandboxName); - await assertExactMainPolicyNftAndIdentityContracts({ + await assertExactMainPolicyAndIdentityContracts({ artifacts, cleanup, host, diff --git a/test/e2e/live/openshell-exact-main-runtime-contracts.ts b/test/e2e/live/openshell-exact-main-runtime-contracts.ts index a4afeebcac1..ae3864c69e5 100644 --- a/test/e2e/live/openshell-exact-main-runtime-contracts.ts +++ b/test/e2e/live/openshell-exact-main-runtime-contracts.ts @@ -2,6 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 import path from "node:path"; +import { createServer } from "node:net"; +import { createSocket } from "node:dgram"; import YAML from "yaml"; @@ -19,12 +21,12 @@ import { trustedSandboxShellScript, } from "../fixtures/clients/sandbox.ts"; import { expect } from "../fixtures/e2e-test.ts"; -import { MCP_BRIDGE_TEST_CREDENTIALS } from "../fixtures/mcp-bridge-credentials.ts"; + +import { exactMainDockerContainerId } from "./openshell-exact-main-exec.ts"; const EXACT_MAIN_POLICY_KEY = "exact_main_live_exe_identity"; -const LIVE_EXE_PATH = "/tmp/nemoclaw-exact-main-live-exe/live-bash"; -const NETNS_PROXY_HOST = "10.200.0.1"; -const NETNS_PROXY_PORT = 3128; +const LIVE_EXE_PATH = "/tmp/nemoclaw-exact-main-live-exe/live-python"; +const CONTROLLED_HOST = "host.openshell.internal"; const POLICY_TIMEOUT_MS = 120_000; const SANDBOX_CONTAINER_LABEL = "openshell.ai/sandbox-name"; @@ -40,14 +42,6 @@ export interface ExactMainPolicyStatus { version: number; } -export interface ExactMainNftInspection { - chainPolicy: string; - loopbackAcceptIndex: number; - proxyAcceptIndex: number; - rejectIndexes: Record<"ipv4Tcp" | "ipv4Udp" | "ipv6Tcp" | "ipv6Udp", number>; - ruleCount: number; -} - function isRecord(value: unknown): value is JsonRecord { return typeof value === "object" && value !== null && !Array.isArray(value); } @@ -97,146 +91,29 @@ export function parseExactMainPolicyStatus(raw: string): ExactMainPolicyStatus { }; } -function containsObject(value: unknown, predicate: (candidate: JsonRecord) => boolean): boolean { - if (Array.isArray(value)) return value.some((entry) => containsObject(entry, predicate)); - if (!isRecord(value)) return false; - if (predicate(value)) return true; - return Object.values(value).some((entry) => containsObject(entry, predicate)); -} - -function hasVerdict(rule: JsonRecord, verdict: "accept" | "reject"): boolean { - return containsObject(rule, (candidate) => Object.hasOwn(candidate, verdict)); -} - -function hasMatch(rule: JsonRecord, selector: JsonRecord, right: string | number): boolean { - return containsObject( - rule, - (candidate) => - isRecord(candidate.match) && - containsObject(candidate.match, (nested) => - Object.entries(selector).every(([key, value]) => nested[key] === value), - ) && - candidate.match.right === right, - ); -} - -function nftRuleMatchesReject( - rule: JsonRecord, - family: "ipv4" | "ipv6", - protocol: "tcp" | "udp", -): boolean { - const rejectType = family === "ipv4" ? "icmp" : "icmpv6"; - return ( - hasVerdict(rule, "reject") && - hasMatch(rule, { key: "l4proto" }, protocol) && - containsObject( - rule, - (candidate) => - isRecord(candidate.reject) && - candidate.reject.type === rejectType && - candidate.reject.expr === "port-unreachable", - ) - ); -} - -export function inspectExactMainNftRuleset(raw: string): ExactMainNftInspection { +export function inspectExactMainIsolation(raw: string, containerId: string): JsonRecord { const parsed: unknown = JSON.parse(raw); - if (!isRecord(parsed) || !Array.isArray(parsed.nftables)) { - throw new Error("nft -j output must contain an nftables array"); - } - const entries = parsed.nftables.filter(isRecord); - const table = entries - .map((entry) => entry.table) - .find( - (candidate) => - isRecord(candidate) && candidate.family === "inet" && candidate.name === "openshell_bypass", - ); - if (!table) throw new Error("missing inet openshell_bypass nft table"); - const chain = entries - .map((entry) => entry.chain) - .find( - (candidate) => - isRecord(candidate) && - candidate.family === "inet" && - candidate.table === "openshell_bypass" && - candidate.name === "output", - ); - if (!isRecord(chain)) throw new Error("missing openshell_bypass output chain"); - if ( - chain.type !== "filter" || - chain.hook !== "output" || - chain.prio !== 0 || - chain.policy !== "accept" - ) { - throw new Error( - `unexpected openshell_bypass chain contract: ${JSON.stringify({ - hook: chain.hook, - policy: chain.policy, - prio: chain.prio, - type: chain.type, - })}`, - ); - } - - const rules = entries - .map((entry) => entry.rule) - .filter( - (candidate): candidate is JsonRecord => - isRecord(candidate) && - candidate.family === "inet" && - candidate.table === "openshell_bypass" && - candidate.chain === "output", - ); - const proxyAcceptIndex = rules.findIndex( - (rule) => - hasVerdict(rule, "accept") && - hasMatch(rule, { protocol: "ip", field: "daddr" }, NETNS_PROXY_HOST) && - hasMatch(rule, { protocol: "tcp", field: "dport" }, NETNS_PROXY_PORT), - ); - const loopbackAcceptIndex = rules.findIndex( - (rule) => hasVerdict(rule, "accept") && hasMatch(rule, { key: "oifname" }, "lo"), - ); - if (proxyAcceptIndex < 0) throw new Error("missing exact proxy IPv4 accept rule"); - if (loopbackAcceptIndex < 0) throw new Error("missing loopback accept rule"); - - const rejectKinds = [ - ["ipv4Tcp", "ipv4", "tcp"], - ["ipv6Tcp", "ipv6", "tcp"], - ["ipv4Udp", "ipv4", "udp"], - ["ipv6Udp", "ipv6", "udp"], - ] as const; - const rejectIndexes = {} as ExactMainNftInspection["rejectIndexes"]; - for (const [key, family, protocol] of rejectKinds) { - const matches = rules - .map((rule, index) => ({ index, matches: nftRuleMatchesReject(rule, family, protocol) })) - .filter((entry) => entry.matches); - if (matches.length !== 1) { - throw new Error( - `expected exactly one ${family} ${protocol} reject rule, got ${matches.length}`, - ); - } - rejectIndexes[key] = matches[0]?.index ?? -1; - } - - const firstReject = Math.min(...Object.values(rejectIndexes)); - if (proxyAcceptIndex >= firstReject || loopbackAcceptIndex >= firstReject) { - throw new Error("nft accept rules must precede every required reject rule"); + if (!Array.isArray(parsed) || parsed.length !== 1 || !isRecord(parsed[0])) { + throw new Error("expected exactly one workload inspection"); } + const container = parsed[0]; + const config = container.HostConfig; if ( - !( - rejectIndexes.ipv4Tcp < rejectIndexes.ipv6Tcp && - rejectIndexes.ipv6Tcp < rejectIndexes.ipv4Udp && - rejectIndexes.ipv4Udp < rejectIndexes.ipv6Udp - ) + container.Id !== containerId || + !isRecord(config) || + config.NetworkMode !== "none" || + !Array.isArray(config.CapDrop) || + !config.CapDrop.includes("ALL") || + (config.CapAdd !== null && (!Array.isArray(config.CapAdd) || config.CapAdd.length !== 0)) || + config.Privileged !== false ) { - throw new Error("required nft rejects are not in IPv4/IPv6 TCP then IPv4/IPv6 UDP order"); + throw new Error("workload does not preserve native network/capability isolation"); } return { - chainPolicy: "accept", - loopbackAcceptIndex, - proxyAcceptIndex, - rejectIndexes, - ruleCount: rules.length, + containerId, + networkMode: config.NetworkMode, + capDrop: config.CapDrop, + privileged: false, }; } @@ -260,60 +137,35 @@ function buildIdentityPolicy(basePolicyYaml: string, mcpUrl: string): string { return YAML.stringify(parsed); } -const LIVE_EXE_CONNECT_SCRIPT = String.raw`set -eu -endpoint_host=$1 -endpoint_port=$2 -first_result=$3 -second_result=$4 -trigger=$5 -proxy_url=$(printenv HTTPS_PROXY 2>/dev/null || printenv https_proxy 2>/dev/null || printenv HTTP_PROXY 2>/dev/null || printenv http_proxy 2>/dev/null || true) -[ -n "$proxy_url" ] || { printf '%s\n' 'proxy environment missing' >&2; exit 1; } -proxy_authority=$(printf '%s' "$proxy_url" | sed -E 's#^[^:]+://##; s#/.*$##') -proxy_host=$(printf '%s' "$proxy_authority" | cut -d: -f1) -proxy_port=$(printf '%s' "$proxy_authority" | cut -d: -f2) -[ -n "$proxy_host" ] && [ "$proxy_port" -ge 1 ] 2>/dev/null -connect_status() { - status=000 - if exec 7<>"/dev/tcp/$proxy_host/$proxy_port"; then - printf 'CONNECT %s:%s HTTP/1.1\r\nHost: %s:%s\r\nConnection: close\r\n\r\n' "$endpoint_host" "$endpoint_port" "$endpoint_host" "$endpoint_port" >&7 - IFS=' ' read -r -t 30 _ status _ <&7 || status=000 - exec 7<&- - exec 7>&- - fi - printf '%s\n' "$status" -} -connect_until_allowed() { - result_path=$1 - allowed_status=000 - allowed_attempt=0 - while [ "$allowed_attempt" -lt 50 ]; do - allowed_attempt=$((allowed_attempt + 1)) - allowed_status=$(connect_status) - [ "$allowed_status" = 200 ] && break - sleep 0.1 - done - printf '%s\n' "$allowed_status" > "$result_path" -} -connect_until_allowed "$first_result" -IFS= read -r _ < "$trigger" -connect_until_allowed "$second_result"`; - -const LIVE_EXE_ONESHOT_SCRIPT = String.raw`set -eu -endpoint_host=$1 -endpoint_port=$2 -proxy_url=$(printenv HTTPS_PROXY 2>/dev/null || printenv https_proxy 2>/dev/null || printenv HTTP_PROXY 2>/dev/null || printenv http_proxy 2>/dev/null || true) -[ -n "$proxy_url" ] || { printf '%s\n' '000'; exit 0; } -proxy_authority=$(printf '%s' "$proxy_url" | sed -E 's#^[^:]+://##; s#/.*$##') -proxy_host=$(printf '%s' "$proxy_authority" | cut -d: -f1) -proxy_port=$(printf '%s' "$proxy_authority" | cut -d: -f2) -status=000 -if exec 7<>"/dev/tcp/$proxy_host/$proxy_port"; then - printf 'CONNECT %s:%s HTTP/1.1\r\nHost: %s:%s\r\nConnection: close\r\n\r\n' "$endpoint_host" "$endpoint_port" "$endpoint_host" "$endpoint_port" >&7 - IFS=' ' read -r -t 30 _ status _ <&7 || status=000 - exec 7<&- - exec 7>&- -fi -printf '%s\n' "$status"`; +const LIVE_EXE_CONNECT_SCRIPT = String.raw`import errno +import socket +import sys +import time + +host, port = sys.argv[1], int(sys.argv[2]) +def connect_status(): + try: + with socket.create_connection((host, port), timeout=3) as stream: + stream.sendall(b"NEMOCLAW_IDENTITY_CONTROL") + return "allowed" if stream.recv(128) == b"NEMOCLAW_IDENTITY_CONTROL" else "invalid-echo" + except OSError as exc: + return "EACCES" if exc.errno == errno.EACCES else f"unexpected-{exc.errno}" +def publish(result_path): + status = "unattempted" + for _ in range(50): + status = connect_status() + if status == "allowed": + break + time.sleep(0.1) + with open(result_path, "w", encoding="utf-8") as output: + output.write(status) +if len(sys.argv) == 3: + print(connect_status()) +else: + publish(sys.argv[3]) + with open(sys.argv[5], encoding="utf-8") as trigger: + trigger.readline() + publish(sys.argv[4])`; export function buildExactMainLiveExeIdentityScript(mcpUrl: string): string { const endpoint = new URL(mcpUrl); @@ -331,17 +183,17 @@ export function buildExactMainLiveExeIdentityScript(mcpUrl: string): string { 'trap \'if [ -n "${old_pid:-}" ]; then kill "$old_pid" 2>/dev/null || true; fi; rm -rf "$root"\' EXIT', 'rm -rf "$root"', 'mkdir -p "$root"', - "bash_source=$(command -v bash)", - '[ -n "$bash_source" ]', - 'cp "$bash_source" "$live"', + "python_source=$(command -v python3)", + '[ -n "$python_source" ]', + 'cp "$python_source" "$live"', 'chmod 0755 "$live"', 'first="$root/old-first.status"', 'second="$root/old-second.status"', 'trigger="$root/old-second.trigger"', 'mkfifo "$trigger"', `worker_script=${shellQuote(LIVE_EXE_CONNECT_SCRIPT)}`, - `oneshot_script=${shellQuote(LIVE_EXE_ONESHOT_SCRIPT)}`, - `"$live" -c "$worker_script" exact-main-old ${shellQuote(endpoint.hostname)} ${endpointPort} "$first" "$second" "$trigger" &`, + `oneshot_script=${shellQuote(LIVE_EXE_CONNECT_SCRIPT)}`, + `"$live" -c "$worker_script" ${shellQuote(endpoint.hostname)} ${endpointPort} "$first" "$second" "$trigger" &`, "old_pid=$!", "wait_for_result() {", " result_path=$1", @@ -358,8 +210,8 @@ export function buildExactMainLiveExeIdentityScript(mcpUrl: string): string { "}", 'wait_for_result "$first"', "old_first=$(tr -d '\\r\\n' < \"$first\")", - '[ "$old_first" = 200 ] || { printf \'expected initial live-exe CONNECT 200, got %s\\n\' "$old_first" >&2; exit 1; }', - 'replacement="$root/live-bash.next"', + '[ "$old_first" = allowed ] || { printf \'expected initial live-exe broker connection allowed, got %s\\n\' "$old_first" >&2; exit 1; }', + 'replacement="$root/live-python.next"', 'cp "$live" "$replacement"', "printf '\\nNEMOCLAW_EXACT_MAIN_REPLACEMENT\\n' >> \"$replacement\"", 'chmod 0755 "$replacement"', @@ -371,70 +223,63 @@ export function buildExactMainLiveExeIdentityScript(mcpUrl: string): string { "printf 'go\\n' > \"$trigger\"", 'wait_for_result "$second"', "old_second=$(tr -d '\\r\\n' < \"$second\")", - '[ "$old_second" = 200 ] || { printf \'expected retained live-exe CONNECT 200, got %s\\n\' "$old_second" >&2; exit 1; }', + '[ "$old_second" = allowed ] || { printf \'expected retained live-exe broker connection allowed, got %s\\n\' "$old_second" >&2; exit 1; }', 'wait "$old_pid"', 'old_pid=""', - `new_status=$("$live" -c "$oneshot_script" exact-main-new ${shellQuote(endpoint.hostname)} ${endpointPort} | tr -d '\\r\\n')`, + `new_status=$("$live" -c "$oneshot_script" ${shellQuote(endpoint.hostname)} ${endpointPort} | tr -d '\\r\\n')`, "printf 'NEMOCLAW_LIVE_EXE_OLD_FIRST=%s\\n' \"$old_first\"", "printf 'NEMOCLAW_LIVE_EXE_OLD_SECOND=%s\\n' \"$old_second\"", "printf 'NEMOCLAW_LIVE_EXE_NEW=%s\\n' \"$new_status\"", "printf 'NEMOCLAW_LIVE_EXE_OLD_LINK=%s\\n' \"$old_exe\"", "printf 'NEMOCLAW_LIVE_EXE_OLD_HASH=%s\\n' \"$old_hash\"", "printf 'NEMOCLAW_LIVE_EXE_NEW_HASH=%s\\n' \"$new_hash\"", - '[ "$new_status" = 403 ] || { printf \'expected replacement live-exe CONNECT 403, got %s\\n\' "$new_status" >&2; exit 1; }', + '[ "$new_status" = EACCES ] || { printf \'expected replacement live-exe broker EACCES, got %s\\n\' "$new_status" >&2; exit 1; }', ].join("\n"); } -const BYPASS_SERVER_CODE = String.raw`import os -import selectors -import socket -import sys - -ready_path = sys.argv[1] -bind_host = sys.argv[2] -tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -tcp.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -tcp.bind((bind_host, 0)) -port = tcp.getsockname()[1] -tcp.listen(8) -udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) -udp.bind((bind_host, port)) -selector = selectors.DefaultSelector() -selector.register(tcp, selectors.EVENT_READ, "tcp") -selector.register(udp, selectors.EVENT_READ, "udp") -with open(ready_path, "w", encoding="utf-8") as ready: - ready.write(f"{os.getpid()} {port}\n") - ready.flush() -while True: - for key, _ in selector.select(timeout=30): - if key.data == "tcp": - client, _ = tcp.accept() - with client: - client.settimeout(2) - payload = client.recv(4096) - client.sendall(payload or b"NEMOCLAW_TCP_ECHO") - else: - payload, address = udp.recvfrom(4096) - udp.sendto(payload or b"NEMOCLAW_UDP_ECHO", address)`; - -const START_BYPASS_SERVER_SCRIPT = String.raw`ready=$1 -log=$2 -server_code=$3 -bind_host=$4 -rm -f "$ready" "$log" -nohup python3 -c "$server_code" "$ready" "$bind_host" >"$log" 2>&1 /dev/null; then - cat "$log" >&2 || true - exit 1 - fi - attempts=$((attempts + 1)) - [ "$attempts" -lt 100 ] || { cat "$log" >&2 || true; exit 1; } - sleep 0.1 -done -cat "$ready"`; +async function startControlledListener(cleanup: CleanupRegistry) { + const observed = { tcp: 0, udp: 0 }; + const tcp = createServer((socket) => { + observed.tcp += 1; + socket.setTimeout(3_000, () => socket.destroy()); + socket.on("error", () => socket.destroy()); + socket.on("data", (data) => socket.end(data)); + }); + const udp = createSocket("udp4"); + let closed = false; + const close = async () => { + if (closed) return; + await Promise.all([ + new Promise((resolve, reject) => + tcp.close((error) => (error ? reject(error) : resolve())), + ), + new Promise((resolve) => udp.close(() => resolve())), + ]); + closed = true; + }; + await new Promise((resolve, reject) => { + tcp.once("error", reject); + tcp.listen(0, "127.0.0.1", resolve); + }); + const address = tcp.address(); + if (!address || typeof address === "string") throw new Error("listener has no TCP address"); + try { + await new Promise((resolve, reject) => { + udp.once("error", reject); + udp.bind(address.port, "127.0.0.1", resolve); + }); + } catch (error) { + await new Promise((resolve) => tcp.close(() => resolve())); + udp.close(); + throw error; + } + udp.on("message", (data, peer) => { + observed.udp += 1; + udp.send(data, peer.port, peer.address); + }); + cleanup.add("stop exact-main controlled TCP/UDP listener", close); + return { port: address.port, observed, close }; +} export const DIRECT_BYPASS_PROBE_CODE = String.raw`import errno import json @@ -480,7 +325,7 @@ tcp_error = result["tcp"]["error"] or {} udp_error = result["udp"]["error"] or {} if result["tcp"]["connected"] or result["udp"]["echoed"]: raise SystemExit(1) -denial_errnos = {errno.ECONNREFUSED, errno.EPERM} +denial_errnos = {errno.EACCES} if tcp_error.get("errno") not in denial_errnos or udp_error.get("errno") not in denial_errnos: raise SystemExit(1) if result["tcp"]["elapsedMs"] >= 2500 or result["udp"]["elapsedMs"] >= 2500: @@ -500,57 +345,39 @@ async function findSandboxContainer( "label=openshell.ai/managed-by=openshell", "--filter", `label=${SANDBOX_CONTAINER_LABEL}=${sandboxName}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", "--format", - "{{.ID}}", + '{{.ID}}\t{{.Names}}\t{{.Label "openshell.ai/sandbox-id"}}\t{{.Label "openshell.ai/sandbox-workspace"}}', ], { artifactName, env: buildAvailabilityProbeEnv(), timeoutMs: 30_000 }, ); expectExitZero(result, `locate OpenShell sandbox container ${sandboxName}`); - const ids = result.stdout.trim().split(/\s+/u).filter(Boolean); - expect(ids, resultText(result)).toHaveLength(1); - return ids[0] ?? ""; + return exactMainDockerContainerId(result.stdout, sandboxName); } -export function buildExactMainNftInspectionScript(): string { - return [ - "nft_path=", - "for candidate in /usr/sbin/nft /sbin/nft /usr/bin/nft; do", - ' if [ -x "$candidate" ]; then nft_path="$candidate"; break; fi', - "done", - "[ -n \"$nft_path\" ] || { printf 'nft executable not found\\n' >&2; exit 127; }", - "active_namespace_count=0", - "namespace_path=", - "for namespace_candidate in /var/run/netns/sandbox-*; do", - ' [ -e "$namespace_candidate" ] || continue', - " namespace=${namespace_candidate##*/}", - ' [ -n "$(ip netns pids "$namespace" 2>/dev/null)" ] || continue', - " active_namespace_count=$((active_namespace_count + 1))", - ' namespace_path="$namespace_candidate"', - "done", - 'if [ "$active_namespace_count" -gt 1 ]; then', - " printf 'expected at most one active sandbox netns, got %s\\n' \"$active_namespace_count\" >&2", - " ip netns list >&2 || true", - " exit 1", - "fi", - 'if [ "$active_namespace_count" -eq 1 ]; then', - ' exec nsenter --net="$namespace_path" -- "$nft_path" -j list table inet openshell_bypass', - "fi", - 'exec "$nft_path" -j list table inet openshell_bypass', - ].join("\n"); -} - -async function inspectNftRules( +async function inspectIsolation( host: HostCliClient, containerId: string, artifactName: string, -): Promise<{ inspection: ExactMainNftInspection; raw: string }> { +): Promise { const result = await host.command( "docker", - ["exec", "--user", "0", containerId, "sh", "-ceu", buildExactMainNftInspectionScript()], - { artifactName, env: buildAvailabilityProbeEnv(), timeoutMs: 30_000 }, + [ + "inspect", + "--format", + '{"Id":{{json .Id}},"HostConfig":{"NetworkMode":{{json .HostConfig.NetworkMode}},"CapDrop":{{json .HostConfig.CapDrop}},"CapAdd":{{json .HostConfig.CapAdd}},"Privileged":{{json .HostConfig.Privileged}}}}', + containerId, + ], + { + artifactName, + env: buildAvailabilityProbeEnv(), + timeoutMs: 30_000, + persistArtifacts: false, + }, ); - expectExitZero(result, "inspect exact-main nft bypass rules"); - return { inspection: inspectExactMainNftRuleset(result.stdout), raw: result.stdout }; + expectExitZero(result, "inspect exact-main native workload isolation"); + return inspectExactMainIsolation(`[${result.stdout.trim()}]`, containerId); } async function readPolicyStatus( @@ -571,10 +398,11 @@ async function readPolicyStatus( return parseExactMainPolicyStatus(result.stdout); } -async function waitForRestartedPolicy( +export async function waitForRestartedPolicy( sandbox: SandboxClient, sandboxName: string, expected: ExactMainPolicyStatus, + expectedRevision: ExactMainPolicyStatus, ): Promise<{ effective: ExactMainPolicyStatus; revision: ExactMainPolicyStatus }> { const deadline = Date.now() + POLICY_TIMEOUT_MS; let last = "no status response"; @@ -594,7 +422,7 @@ async function waitForRestartedPolicy( ); if ( revision.status === "loaded" && - revision.hash === expected.hash && + revision.hash === expectedRevision.hash && revision.activeVersion === expected.version && effective.status === "effective" && effective.hash === expected.hash && @@ -639,117 +467,51 @@ async function waitForInitialPolicyAcknowledgement( throw new Error(`missing exact initial policy acknowledgement ${marker}\n${last}`); } -async function stopBypassServer( - host: HostCliClient, - containerId: string, - pid: number, - readyPath: string, - logPath: string, -): Promise { - const result = await host.command( - "docker", - [ - "exec", - "--user", - "0", - containerId, - "sh", - "-ceu", - 'kill "$1" 2>/dev/null || true; rm -f "$2" "$3"', - "exact-main-bypass-cleanup", - String(pid), - readyPath, - logPath, - ], - { - artifactName: "exact-main-bypass-listener-cleanup", - env: buildAvailabilityProbeEnv(), - timeoutMs: 30_000, - }, - ); - expectExitZero(result, "stop exact-main controlled bypass listener"); -} - async function assertDirectBypassDenied(options: { - cleanup: CleanupRegistry; host: HostCliClient; - containerId: string; sandbox: SandboxClient; sandboxName: string; - version: number; + listener: Awaited>; }): Promise { - const readyPath = `/tmp/nemoclaw-exact-main-bypass-${options.version}.ready`; - const logPath = `/tmp/nemoclaw-exact-main-bypass-${options.version}.log`; - const start = await options.host.command( - "docker", - [ - "exec", - "--user", - "0", - options.containerId, - "sh", - "-ceu", - START_BYPASS_SERVER_SCRIPT, - "exact-main-bypass-listener", - readyPath, - logPath, - BYPASS_SERVER_CODE, - NETNS_PROXY_HOST, - ], + const control = () => + options.host.command( + "python3", + [ + "-c", + String.raw`import socket, sys +port = int(sys.argv[1]) +with socket.create_connection(("127.0.0.1", port), timeout=3) as tcp: + tcp.sendall(b"host-control") + assert tcp.recv(128) == b"host-control" +with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as udp: + udp.settimeout(3) + udp.sendto(b"host-control", ("127.0.0.1", port)) + assert udp.recv(128) == b"host-control" +`, + String(options.listener.port), + ], + { artifactName: "exact-main-listener-positive-control", timeoutMs: 30_000 }, + ); + expectExitZero(await control(), "controlled TCP/UDP listeners echo before denial probe"); + const before = { ...options.listener.observed }; + const probe = await options.sandbox.exec( + options.sandboxName, + ["python3", "-c", DIRECT_BYPASS_PROBE_CODE, CONTROLLED_HOST, String(options.listener.port)], { - artifactName: "exact-main-bypass-listener-start", - env: buildAvailabilityProbeEnv(), + artifactName: "exact-main-direct-tcp-udp-bypass-denied", + env: sandboxAccessEnv(), timeoutMs: 30_000, }, ); - expectExitZero(start, "start controlled TCP/UDP bypass listener"); - const match = start.stdout.trim().match(/^(\d+)\s+(\d+)$/u); - if (!match) throw new Error(`unexpected bypass listener identity: ${start.stdout}`); - const pid = Number(match[1]); - const port = Number(match[2]); - let listenerRunning = true; - const cleanupListener = async () => { - if (!listenerRunning) return; - await stopBypassServer(options.host, options.containerId, pid, readyPath, logPath); - listenerRunning = false; - }; - options.cleanup.add("stop exact-main controlled TCP/UDP bypass listener", cleanupListener); - try { - const probe = await options.sandbox.exec( - options.sandboxName, - ["python3", "-c", DIRECT_BYPASS_PROBE_CODE, NETNS_PROXY_HOST, String(port)], - { - artifactName: "exact-main-direct-tcp-udp-bypass-denied", - env: sandboxAccessEnv(), - timeoutMs: 30_000, - }, - ); - if (probe.exitCode !== 0) { - await Promise.allSettled([ - Promise.resolve().then(() => - options.sandbox.openshell( - ["logs", options.sandboxName, "-n", "500", "--since", "2m", "--source", "all"], - { - artifactName: "exact-main-post-restart-failure-logs", - captureLimitBytes: 32_768, - env: sandboxAccessEnv(), - redactionValues: Object.values(MCP_BRIDGE_TEST_CREDENTIALS), - timeoutMs: 30_000, - }, - ), - ), - ]); - } - expectExitZero(probe, "deny direct IPv4 TCP and UDP bypass to controlled listeners"); - const parsed: unknown = JSON.parse(probe.stdout); - if (!isRecord(parsed)) throw new Error("direct bypass probe must return a JSON object"); - return parsed; - } finally { - await cleanupListener(); - } + expectExitZero(probe, "native broker rejects unauthorized TCP and non-DNS UDP"); + const parsed: unknown = JSON.parse(probe.stdout); + if (!isRecord(parsed)) throw new Error("direct bypass probe must return a JSON object"); + expect(options.listener.observed).toEqual(before); + expectExitZero(await control(), "controlled TCP/UDP listeners remain reachable after denial"); + return parsed; } -export async function assertExactMainPolicyNftAndIdentityContracts(options: { +export async function assertExactMainPolicyAndIdentityContracts(options: { artifacts: ArtifactSink; cleanup: CleanupRegistry; host: HostCliClient; @@ -767,6 +529,8 @@ export async function assertExactMainPolicyNftAndIdentityContracts(options: { expectExitZero(base, "capture exact-main base policy"); const basePolicyYaml = parseOpenShellPolicy(base.stdout).yamlBody; + const listener = await startControlledListener(options.cleanup); + const controlledUrl = `http://${CONTROLLED_HOST}:${listener.port}`; let restoreRequired = false; const restorePolicy = async () => { if (!restoreRequired) return; @@ -791,6 +555,7 @@ export async function assertExactMainPolicyNftAndIdentityContracts(options: { }; options.cleanup.add("restore exact-main base policy", restorePolicy); + let primaryFailure: unknown; try { const before = await readPolicyStatus( options.sandbox, @@ -802,7 +567,7 @@ export async function assertExactMainPolicyNftAndIdentityContracts(options: { expect( await setPolicyDocument( options.sandboxName, - buildIdentityPolicy(basePolicyYaml, options.mcpUrl), + buildIdentityPolicy(basePolicyYaml, controlledUrl), { nonFatal: true, operation: "apply the exact-main live-exe identity policy", @@ -828,23 +593,24 @@ export async function assertExactMainPolicyNftAndIdentityContracts(options: { ); expect(revision).toMatchObject({ activeVersion: effective.version, - hash: effective.hash, sandbox: options.sandboxName, status: "loaded", version: effective.version, }); + // Effective policy includes attached provider rules. Its hash and the + // stored base revision hash each remain stable across the restart. const containerId = await findSandboxContainer( options.host, options.sandboxName, "exact-main-sandbox-container-before-restart", ); - const beforeRestartNft = await inspectNftRules( + const beforeRestartIsolation = await inspectIsolation( options.host, containerId, - "exact-main-nft-rules-before-restart", + "exact-main-native-isolation-before-restart", ); - // OpenShell 0.0.116 treats an unexpected main-process exit as terminal Error. + // OpenShell treats an unexpected main-process exit as terminal Error. // Its lifecycle commands retain the container and wait for Stopped/Ready. for (const operation of ["stop", "start"]) { const result = await options.sandbox.openshell(["sandbox", operation, options.sandboxName], { @@ -864,29 +630,22 @@ export async function assertExactMainPolicyNftAndIdentityContracts(options: { options.sandbox, options.sandboxName, effective, + revision, ); const acknowledgement = await waitForInitialPolicyAcknowledgement( options.sandbox, options.sandboxName, effective.version, ); - const afterRestartNft = await inspectNftRules( + const afterRestartIsolation = await inspectIsolation( options.host, restartedContainerId, - "exact-main-nft-rules-after-restart", + "exact-main-native-isolation-after-restart", ); - const bypass = await assertDirectBypassDenied({ - cleanup: options.cleanup, - containerId: restartedContainerId, - host: options.host, - sandbox: options.sandbox, - sandboxName: options.sandboxName, - version: effective.version, - }); const liveExe = await options.sandbox.execShell( options.sandboxName, - trustedSandboxShellScript(buildExactMainLiveExeIdentityScript(options.mcpUrl)), + trustedSandboxShellScript(buildExactMainLiveExeIdentityScript(controlledUrl)), { artifactName: "exact-main-live-proc-exe-identity", env: sandboxAccessEnv(), @@ -894,21 +653,34 @@ export async function assertExactMainPolicyNftAndIdentityContracts(options: { }, ); expectExitZero(liveExe, "prove live /proc//exe identity across on-disk replacement"); - expect(liveExe.stdout).toContain("NEMOCLAW_LIVE_EXE_OLD_FIRST=200"); - expect(liveExe.stdout).toContain("NEMOCLAW_LIVE_EXE_OLD_SECOND=200"); - expect(liveExe.stdout).toContain("NEMOCLAW_LIVE_EXE_NEW=403"); - expect(liveExe.stdout).toMatch(/NEMOCLAW_LIVE_EXE_OLD_LINK=.*live-bash \(deleted\)/u); + expect(listener.observed.tcp).toBe(2); + expect(listener.observed.udp).toBe(0); + expect(liveExe.stdout).toContain("NEMOCLAW_LIVE_EXE_NEW=EACCES"); + expect(liveExe.stdout).toMatch(/NEMOCLAW_LIVE_EXE_OLD_LINK=.*live-python \(deleted\)/u); - await options.artifacts.writeJson("exact-main-policy-nft-identity-summary.json", { + const bypass = await assertDirectBypassDenied({ ...options, listener }); + await options.artifacts.writeJson("exact-main-policy-native-identity-summary.json", { acknowledgement, bypass, - nftAfterRestart: afterRestartNft.inspection, - nftBeforeRestart: beforeRestartNft.inspection, + isolationAfterRestart: afterRestartIsolation, + isolationBeforeRestart: beforeRestartIsolation, policyAfterRestart: restartedPolicy, policyHotUpdate: { effective, revision }, }); + } catch (error) { + primaryFailure = error; + throw error; } finally { - await restorePolicy(); + const cleanupResults = await Promise.allSettled([restorePolicy(), listener.close()]); + const failures = cleanupResults.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + if (failures.length) { + throw new AggregateError( + primaryFailure === undefined ? failures : [primaryFailure, ...failures], + "Exact-main proof cleanup failed", + ); + } } } diff --git a/test/e2e/live/openshell-gateway-auth-probe.ts b/test/e2e/live/openshell-gateway-auth-probe.ts index 8945de8a18a..00a0460f653 100644 --- a/test/e2e/live/openshell-gateway-auth-probe.ts +++ b/test/e2e/live/openshell-gateway-auth-probe.ts @@ -2,13 +2,10 @@ // SPDX-License-Identifier: Apache-2.0 import { spawnSync } from "node:child_process"; -import { createPrivateKey, sign as signPayload } from "node:crypto"; -import fs from "node:fs"; import path from "node:path"; import { getDockerDriverGatewayLocalTlsBundle } from "../../../src/lib/onboard/docker-driver-gateway-local-tls"; -const SANDBOX_JWT_SUBJECT_PREFIX = "spiffe://openshell/sandbox/"; export const DOCKER_GRPC_PROBE_IMAGE = "node:24.22.1-trixie-slim@sha256:43b9ba1142b8dcf33fa8f922ca1be85714adc444e0586c7cf7726daaeabc1bea"; const CONTAINER_PROBE_CA_PATH = "/tmp/nemoclaw-probe-ca.crt"; @@ -24,6 +21,7 @@ export type GatewayAuthProbeResult = { export type SandboxTokenContainerProbeOptions = { authorization?: string; dockerBin: string; + grpcPath?: "/openshell.v1.OpenShell/Health"; hostGatewayIp?: string; networkName: string; payload: Buffer; @@ -54,38 +52,6 @@ export function getSandboxConfigRequest(sandboxId: string): Buffer { return Buffer.concat([Buffer.from([(1 << 3) | 2]), varint(bytes.length), bytes]); } -function parseTomlString(toml: string, key: string): string { - const match = toml.match(new RegExp(`^${key} = "([^"]+)"$`, "m")); - if (!match?.[1]) throw new Error(`missing TOML key ${key}`); - return match[1]; -} - -function base64UrlJson(value: unknown): string { - return Buffer.from(JSON.stringify(value), "utf-8").toString("base64url"); -} - -export function mintSandboxJwt(options: { configPath: string; sandboxId: string }): string { - const toml = fs.readFileSync(options.configPath, "utf-8"); - const signingKeyPath = parseTomlString(toml, "signing_key_path"); - const kid = fs.readFileSync(parseTomlString(toml, "kid_path"), "utf-8").trim(); - const gatewayId = parseTomlString(toml, "gateway_id"); - const now = Math.floor(Date.now() / 1000); - const identity = `openshell-gateway:${gatewayId}`; - const header = base64UrlJson({ alg: "EdDSA", kid, typ: "JWT" }); - const payload = base64UrlJson({ - aud: identity, - exp: now + 3600, - iat: now, - iss: identity, - sandbox_id: options.sandboxId, - sub: `${SANDBOX_JWT_SUBJECT_PREFIX}${options.sandboxId}`, - }); - const signingInput = `${header}.${payload}`; - const privateKey = createPrivateKey(fs.readFileSync(signingKeyPath, "utf-8")); - const signature = signPayload(null, Buffer.from(signingInput), privateKey).toString("base64url"); - return `${signingInput}.${signature}`; -} - function containerProbeNetworkArgs( networkName: string, useHostNetwork: boolean, @@ -189,7 +155,7 @@ export function buildSandboxTokenContainerProbeInvocation( "--volume", `${path.resolve(bundle.clientKeyPath)}:${CONTAINER_PROBE_CLIENT_KEY_PATH}:ro`, "--env", - "PROBE_GRPC_PATH=/openshell.v1.OpenShell/GetSandboxConfig", + `PROBE_GRPC_PATH=${options.grpcPath ?? "/openshell.v1.OpenShell/GetSandboxConfig"}`, "--env", `PROBE_GATEWAY_PORT=${String(options.port)}`, "--env", diff --git a/test/e2e/live/openshell-gateway-auth-source-contract-helpers.ts b/test/e2e/live/openshell-gateway-auth-source-contract-helpers.ts index 43e217ffeaf..e717c2deac0 100644 --- a/test/e2e/live/openshell-gateway-auth-source-contract-helpers.ts +++ b/test/e2e/live/openshell-gateway-auth-source-contract-helpers.ts @@ -20,19 +20,117 @@ import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import type { CleanupRegistry } from "../fixtures/cleanup.ts"; import type { HostCliClient } from "../fixtures/clients/index.ts"; import { expect } from "../fixtures/e2e-test.ts"; -import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; +import { + exactGatewayRelease, + OPENSHELL_V017_QUALIFICATION, +} from "../fixtures/openshell-v0116-qualification.ts"; import { spawnObservedChild } from "../fixtures/observed-child-process.ts"; import type { TestProgress } from "../fixtures/progress.ts"; +import { redactString } from "../fixtures/redaction.ts"; import { DOCKER_GRPC_PROBE_IMAGE, getSandboxConfigRequest, - mintSandboxJwt, runSandboxTokenContainerProbe, } from "./openshell-gateway-auth-probe.ts"; import { verifyOpenShellTlsServerNameSourceBoundary } from "./openshell-v0116-tls-server-name-source.ts"; export { buildSandboxTokenContainerProbeInvocation } from "./openshell-gateway-auth-probe.ts"; +// Only reads the launch bundle produced inside this test's private state directory. +// A handwritten legacy JWT cannot establish a persisted sandbox session. +export function createGatewayAuthSandbox(options: { + cleanup: Pick; + configPath: string; + port: number; + stateDir: string; +}): { name: string; token: string } { + const { cleanup, configPath, port, stateDir } = options; + const env = { + PATH: process.env.PATH, + XDG_CONFIG_HOME: path.join(stateDir, "client-config"), + XDG_STATE_HOME: path.join(stateDir, "runtime-state"), + XDG_DATA_HOME: path.join(stateDir, "client-data"), + OPENSHELL_GATEWAY: "auth-contract", + OPENSHELL_LOCAL_TLS_DIR: path.join(stateDir, "tls"), + OPENSHELL_COLOR: "never", + }; + const cli = (args: string[]): string => { + const result = spawnSync("openshell", args, { + env, + encoding: "utf8", + timeout: 120_000, + killSignal: "SIGKILL", + maxBuffer: 1024 * 1024, + }); + // Keep stdout private; redact stderr before exposing bounded failure diagnostics. + expect( + result.status, + `OpenShell auth fixture ${args[0]} ${args[1]} failed (${result.status}): ${redactString(result.stderr ?? result.error?.message ?? "").slice(0, 4096)}`, + ).toBe(0); + return result.stdout; + }; + cli(["gateway", "add", `https://127.0.0.1:${port}`, "--local", "--name", "auth-contract"]); + const name = "auth-contract"; + const policyPath = path.join(stateDir, "sandbox-policy.yaml"); + fs.writeFileSync( + policyPath, + `version: 1 +filesystem_policy: + include_workdir: true + read_only: [/usr, /bin, /lib, /lib64, /etc, /proc, /dev/urandom] + read_write: [/tmp, /dev/null] +landlock: + compatibility: best_effort +process: + run_as_user: "65532" + run_as_group: "65532" +network_policies: {} +`, + { mode: 0o600 }, + ); + // Registered before creation so a partial create is reconciled before gateway teardown. + cleanup.add("delete OpenShell auth contract sandbox", () => { + cli(["sandbox", "delete", name]); + }); + const created = JSON.parse( + cli([ + "sandbox", + "create", + "--name", + name, + "--from", + DOCKER_GRPC_PROBE_IMAGE, + "--policy", + policyPath, + "--no-auto-providers", + "--detach", + "--no-tty", + "-o", + "json", + "--", + "/bin/sh", + "-c", + "sleep 600", + ]), + ) as { id?: string; name?: string; phase?: string }; + expect(created.id).toMatch(/^[a-f0-9-]{36}$/u); + const namespace = fs + .readFileSync(configPath, "utf8") + .match(/^sandbox_label = "([a-zA-Z0-9_.-]+)"$/mu)?.[1]; + if (!namespace) throw new Error("OpenShell auth fixture sandbox namespace is missing"); + const bundlePath = path.join( + env.XDG_STATE_HOME, + "openshell", + "docker-sandbox-tokens", + namespace, + created.id!, + "supervisor-auth.json", + ); + const bundle = JSON.parse(fs.readFileSync(bundlePath, "utf8")) as { gateway_token?: string }; + expect(typeof bundle.gateway_token === "string" && bundle.gateway_token.length > 0).toBe(true); + return { name, token: bundle.gateway_token! }; +} + type SkipFn = (message?: string) => void; type ScenarioFixtures = { @@ -442,9 +540,7 @@ async function runOpenShellGatewayAuthSourceContractScenarioUnchecked( const version = run(gatewayBin, ["--version"]); expect(version.status, commandOutput(version)).toBe(0); - expect(commandOutput(version)).toContain( - process.env.NEMOCLAW_CANDIDATE_VERSION || OPENSHELL_V0116_QUALIFICATION.version, - ); + expect(exactGatewayRelease(commandOutput(version))).toBe(OPENSHELL_V017_QUALIFICATION.version); await requireDockerDaemon({ dockerBin, host, skip }); @@ -479,13 +575,14 @@ async function runOpenShellGatewayAuthSourceContractScenarioUnchecked( OPENSHELL_BIND_ADDRESS: "127.0.0.1", OPENSHELL_DB_URL: `sqlite:${path.join(stateDir, "openshell.db")}`, OPENSHELL_DOCKER_NETWORK_NAME: networkName, - OPENSHELL_DOCKER_SUPERVISOR_IMAGE: OPENSHELL_V0116_QUALIFICATION.supervisorImage, + OPENSHELL_DOCKER_SUPERVISOR_IMAGE: OPENSHELL_V017_QUALIFICATION.supervisorImage, OPENSHELL_DRIVERS: "docker", OPENSHELL_GRPC_ENDPOINT: `https://127.0.0.1:${port}`, OPENSHELL_LOCAL_TLS_DIR: certBundle.localTlsDir, OPENSHELL_SERVER_PORT: String(port), OPENSHELL_SSH_GATEWAY_HOST: "127.0.0.1", OPENSHELL_SSH_GATEWAY_PORT: String(port), + XDG_STATE_HOME: path.join(stateDir, "runtime-state"), }; const launch = dependencies.buildDockerDriverGatewayLaunch({ env: { @@ -500,15 +597,13 @@ async function runOpenShellGatewayAuthSourceContractScenarioUnchecked( requiredGlibcVersions: [], stateDir, }); - expect(launch.mode).toBe("host"); - expect(launch.env.OPENSHELL_DISABLE_GATEWAY_AUTH).toBeUndefined(); await artifacts.writeJson("scenario.json", { contracts: [ "NemoClaw-generated OPENSHELL_GATEWAY_CONFIG enables local mTLS and sandbox JWT auth", "inherited OPENSHELL_DISABLE_GATEWAY_AUTH is scrubbed before launch", "no-token Docker-origin access to user-callable gateway APIs is rejected or unreachable", - "mTLS-only Docker-origin access without sandbox JWT does not return sandbox config", + "mTLS-only access without a sandbox identity or workspace scope does not return config", "valid sandbox JWT access from Docker origin to sandbox-allowlisted APIs reaches OpenShell auth", "a sandbox JWT minted for one sandbox cannot access another sandbox config", ], @@ -562,8 +657,13 @@ async function runOpenShellGatewayAuthSourceContractScenarioUnchecked( expect(noTokenProbeWasRejected(noToken), commandOutput(noToken)).toBe(true); const configPath = String(launch.env.OPENSHELL_GATEWAY_CONFIG || ""); - expect(configPath).toBe(path.join(stateDir, "openshell-gateway.toml")); - const sandboxId = "sandbox-auth-contract"; + const { name: sandboxId, token: sandboxToken } = createGatewayAuthSandbox({ + cleanup, + configPath, + port, + stateDir, + }); + registerSandboxJwtArtifactRedaction(artifacts, sandboxToken); const mtlsOnlyContainerCall = runSandboxTokenContainerProbe({ dockerBin, networkName, @@ -580,8 +680,6 @@ async function runOpenShellGatewayAuthSourceContractScenarioUnchecked( ).toBe(true); progress.phase("probe sandbox JWT authorization boundaries"); - const sandboxToken = mintSandboxJwt({ configPath, sandboxId }); - registerSandboxJwtArtifactRedaction(artifacts, sandboxToken); const sandboxCall = await callGrpc({ authorization: `Bearer ${sandboxToken}`, path: "/openshell.v1.OpenShell/GetSandboxConfig", @@ -591,8 +689,7 @@ async function runOpenShellGatewayAuthSourceContractScenarioUnchecked( }); await artifacts.writeJson("sandbox-jwt-probe.json", sandboxCall); expect(sandboxCall.httpStatus, JSON.stringify(sandboxCall)).toBe(200); - expect(sandboxCall.grpcStatus, JSON.stringify(sandboxCall)).toBeDefined(); - expect(["7", "16"]).not.toContain(sandboxCall.grpcStatus); + expect(sandboxCall.grpcStatus, JSON.stringify(sandboxCall)).toBe("0"); const sandboxContainerCall = runSandboxTokenContainerProbe({ authorization: `Bearer ${sandboxToken}`, @@ -608,24 +705,23 @@ async function runOpenShellGatewayAuthSourceContractScenarioUnchecked( expect(sandboxContainerCall.status, commandOutput(sandboxContainerCall)).toBe(0); const sandboxContainerResult = JSON.parse(sandboxContainerCall.stdout.trim()) as GrpcResult; expect(sandboxContainerResult.httpStatus, JSON.stringify(sandboxContainerResult)).toBe(200); - expect(sandboxContainerResult.grpcStatus, JSON.stringify(sandboxContainerResult)).toBeDefined(); - expect(["7", "16"]).not.toContain(sandboxContainerResult.grpcStatus); + expect(sandboxContainerResult.grpcStatus, JSON.stringify(sandboxContainerResult)).toBe("0"); const crossSandboxContainerCall = runSandboxTokenContainerProbe({ authorization: `Bearer ${sandboxToken}`, dockerBin, networkName, - payload: getSandboxConfigRequest("sandbox-auth-contract-other"), + payload: getSandboxConfigRequest("auth-other"), port, stateDir, useHostNetwork, }); await artifacts.writeJson("cross-sandbox-jwt-container-probe.json", crossSandboxContainerCall); skipUnavailableProbeImage(crossSandboxContainerCall, skip); - expect( - probeDidNotReturnSandboxConfig(crossSandboxContainerCall), - commandOutput(crossSandboxContainerCall), - ).toBe(true); + expect(crossSandboxContainerCall.status, commandOutput(crossSandboxContainerCall)).toBe(0); + const crossSandboxResult = JSON.parse(crossSandboxContainerCall.stdout.trim()) as GrpcResult; + expect(crossSandboxResult.httpStatus, JSON.stringify(crossSandboxResult)).toBe(200); + expect(crossSandboxResult.grpcStatus, JSON.stringify(crossSandboxResult)).toBe("7"); } finally { await artifacts.writeText("openshell-gateway.log", gatewayLog); } diff --git a/test/e2e/live/openshell-gateway-auth-source-contract.test.ts b/test/e2e/live/openshell-gateway-auth-source-contract.test.ts index 9c75df2ae11..6e139585c27 100644 --- a/test/e2e/live/openshell-gateway-auth-source-contract.test.ts +++ b/test/e2e/live/openshell-gateway-auth-source-contract.test.ts @@ -4,12 +4,11 @@ import { buildDockerDriverGatewayLaunch } from "../../../dist/lib/onboard/docker-driver-gateway-launch"; import { ensureDockerDriverGatewayLocalTlsBundle } from "../../../dist/lib/onboard/docker-driver-gateway-local-tls"; import { test } from "../fixtures/e2e-test.ts"; -import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; +import { OPENSHELL_V017_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; import { runOpenShellGatewayAuthSourceContractScenario } from "./openshell-gateway-auth-source-contract-helpers.ts"; const LIVE_TIMEOUT_MS = 8 * 60_000; -const OPENSHELL_GATEWAY_AUTH_CONTRACT_VERSION = - process.env.NEMOCLAW_CANDIDATE_VERSION ?? OPENSHELL_V0116_QUALIFICATION.version; +const OPENSHELL_GATEWAY_AUTH_CONTRACT_VERSION = OPENSHELL_V017_QUALIFICATION.version; test( `OpenShell ${OPENSHELL_GATEWAY_AUTH_CONTRACT_VERSION} Docker-driver gateway auth uses NemoClaw mTLS plus sandbox JWT`, diff --git a/test/e2e/live/openshell-gateway-upgrade-helpers.ts b/test/e2e/live/openshell-gateway-upgrade-helpers.ts index fbe0caf3bb2..64dfcd14d11 100644 --- a/test/e2e/live/openshell-gateway-upgrade-helpers.ts +++ b/test/e2e/live/openshell-gateway-upgrade-helpers.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { stripVTControlCharacters } from "node:util"; import { resultText, shellQuote } from "../fixtures/clients/command.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; @@ -32,6 +33,194 @@ const MANAGED_IMAGE_QUALIFICATION_ENV_KEYS = [ ] as const; export const GATEWAY_UPGRADE_INSTALL_TIMEOUT_MS = 35 * 60_000; +/** The pinned legacy CLI validates the SSH process against its sandbox and port. */ +export function legacyGatewayForwardIdentity( + listing: Pick, + listener: Pick, + sandboxName: string, + port: string, +): string | null { + if ( + [listing, listener].some((result) => result.exitCode !== 0 || result.signal || result.timedOut) + ) + return null; + const rows = stripVTControlCharacters(listing.stdout) + .trim() + .split(/\r?\n/) + .map((line) => line.trim().split(/\s+/)) + .filter((fields) => fields[0] === sandboxName && fields[2] === port); + if (rows.length !== 1) return null; + const row = rows[0]!; + const pid = row[3] ?? ""; + if ( + row.length !== 5 || + row[4] !== "running" || + !/^[1-9]\d*$/.test(pid) || + !Number.isSafeInteger(Number(pid)) + ) + return null; + const listeners = [...new Set(listener.stdout.trim().split(/\s+/))]; + return listeners.length === 1 && listeners[0] === pid ? row.join("\t") : null; +} + +export async function captureLegacyGatewayForward( + host: Pick, + sandboxName: string, + port: string, + env: NodeJS.ProcessEnv, + phase: string, +): Promise<{ identity: string | null; output: string }> { + const [listing, listener] = await Promise.all([ + host.command(host.openshellCommandPath, ["forward", "list"], { + artifactName: `${phase}-dashboard-forward-list`, + env, + timeoutMs: 15_000, + }), + host.command("lsof", ["-ti", `:${port}`, "-sTCP:LISTEN"], { + artifactName: `${phase}-dashboard-forward-listener`, + env, + timeoutMs: 15_000, + }), + ]); + return { + identity: legacyGatewayForwardIdentity(listing, listener, sandboxName, port), + output: [listing, listener].map(resultText).join("\n"), + }; +} + +type ForwardObservation = Pick; + +function listenerPid(observation: ForwardObservation): string | null { + const pids = [...new Set(observation.stdout.trim().split(/\s+/))]; + return observation.exitCode === 0 && + !observation.signal && + !observation.timedOut && + pids.length === 1 && + /^[1-9]\d*$/.test(pids[0]!) + ? pids[0]! + : null; +} + +/** The v0.0.123 fixture launches ForwardTcp directly, outside the SSH forward registry. */ +export function legacyGatewayServiceForwardIdentity( + probes: readonly ForwardObservation[], + sandboxName: string, + port: string, +): string | null { + if ( + probes.length !== 8 || + probes.some((probe) => probe.exitCode !== 0 || probe.signal || probe.timedOut) + ) + return null; + const [ + before, + command, + statBefore, + executable, + expectedExecutable, + commandLine, + statAfter, + after, + ] = probes; + const pid = listenerPid(before!); + if (!pid || listenerPid(after!) !== pid) return null; + const processStart = (raw: string): string | null => { + const end = raw.lastIndexOf(") "); + if (!raw.startsWith(`${pid} (`) || end < 0) return null; + const fields = raw + .slice(end + 2) + .trim() + .split(/\s+/); + return fields[0] !== "Z" && /^[1-9]\d*$/.test(fields[19] ?? "") ? fields[19]! : null; + }; + const start = processStart(statBefore!.stdout); + const binary = executable!.stdout.trim(); + const commandPath = command!.stdout.trim(); + const argv = commandLine!.stdout.split("\0"); + if (argv.at(-1) === "") argv.pop(); + const expected = [ + commandPath, + "--gateway", + "nemoclaw", + "--workspace", + "default", + "forward", + "service", + sandboxName, + "--target-port", + port, + "--target-host", + "127.0.0.1", + "--local", + `127.0.0.1:${port}`, + ]; + if ( + !start || + processStart(statAfter!.stdout) !== start || + !commandPath.startsWith("/") || + !binary.startsWith("/") || + binary !== expectedExecutable!.stdout.trim() || + argv.length !== expected.length || + argv.some((value, index) => value !== expected[index]) + ) + return null; + return `${pid}\t${start}\t${binary}\t${JSON.stringify(argv)}`; +} + +export async function captureLegacyGatewayServiceForward( + host: Pick, + sandboxName: string, + port: string, + env: NodeJS.ProcessEnv, + phase: string, +): Promise<{ identity: string | null; output: string }> { + const probe = (suffix: string, command: string, args: string[]) => + host.command(command, args, { + artifactName: `${phase}-dashboard-service-${suffix}`, + env, + timeoutMs: 15_000, + }); + const [before, command] = await Promise.all([ + probe("listener-before", "lsof", ["-ti", `:${port}`, "-sTCP:LISTEN"]), + probe("command", "which", [host.openshellCommandPath]), + ]); + const pid = listenerPid(before); + const commandPath = command.stdout.trim(); + if ( + !pid || + command.exitCode !== 0 || + command.signal || + command.timedOut || + !commandPath.startsWith("/") + ) { + return { identity: null, output: [before, command].map(resultText).join("\n") }; + } + const statBefore = await probe("stat-before", "cat", [`/proc/${pid}/stat`]); + const [executable, expectedExecutable, commandLine] = await Promise.all([ + probe("executable", "readlink", ["-f", `/proc/${pid}/exe`]), + probe("expected-executable", "readlink", ["-f", commandPath]), + probe("argv", "cat", [`/proc/${pid}/cmdline`]), + ]); + const [statAfter, after] = await Promise.all([ + probe("stat-after", "cat", [`/proc/${pid}/stat`]), + probe("listener-after", "lsof", ["-ti", `:${port}`, "-sTCP:LISTEN"]), + ]); + const probes = [ + before, + command, + statBefore, + executable, + expectedExecutable, + commandLine, + statAfter, + after, + ]; + return { + identity: legacyGatewayServiceForwardIdentity(probes, sandboxName, port), + output: probes.map(resultText).join("\n"), + }; +} + /** Apply path spelling at the installer boundary, after login-shell startup. */ export function gatewayUpgradeInstallerCommand(args: readonly string[], home?: string): string { const prefix = home @@ -124,6 +313,13 @@ function gatewayServiceWasReplaced(evidence: GatewayUpgradeServiceEvidence): boo return before !== null && after !== null && before !== after; } +/** A denied cutover must leave the same active gateway service in place. */ +export function gatewayServiceWasPreserved(evidence: GatewayUpgradeServiceEvidence): boolean { + const before = activeServiceInvocation(evidence.before, evidence.expectedFragmentPath); + const after = activeServiceInvocation(evidence.after, evidence.expectedFragmentPath); + return before !== null && before === after; +} + export async function captureGatewayUpgradeFailureDiagnostics( exitCode: number | null, capture: (() => Promise) | undefined, diff --git a/test/e2e/live/openshell-gateway-upgrade.test.ts b/test/e2e/live/openshell-gateway-upgrade.test.ts index 5b60e53310f..d310700cc66 100644 --- a/test/e2e/live/openshell-gateway-upgrade.test.ts +++ b/test/e2e/live/openshell-gateway-upgrade.test.ts @@ -4,9 +4,9 @@ /** * Installs one reviewed historical NemoClaw/OpenShell gateway, creates a real * OpenClaw sandbox, seeds durable workspace state, and runs the current - * installer upgrade path with the current managed agent image. The survivor - * must remain usable. Fixture-declared stopped sandboxes must preserve their - * workspace state and stopped phase after native state restoration. + * installer upgrade path. A 0.0.x gateway with registered sandboxes must + * remain usable when the 0.1.x cutover is refused. Fixture-declared stopped + * sandboxes must retain their workspace state and stopped phase. */ import { createHash } from "node:crypto"; @@ -18,6 +18,7 @@ import { removeReviewedNpmArchive, } from "../../../scripts/lib/reviewed-npm-archive.mts"; import { shellQuote } from "../../../src/lib/core/shell-quote"; +import { SUPPORTED_OPENSHELL_FALLBACK_VERSION } from "../../../src/lib/onboard/openshell-version"; import { REVIEWED_GATEWAY_UPGRADE_FIXTURE, REVIEWED_GATEWAY_UPGRADE_FIXTURES, @@ -40,11 +41,12 @@ import { REPO_ROOT } from "../fixtures/paths.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; import { captureGatewayUpgradeFailureDiagnostics, + captureLegacyGatewayForward, + captureLegacyGatewayServiceForward, currentGatewayUpgradeInstallerArgs, currentNemoclawUpgradeRef, gatewayCredentialNonExposureScript, - gatewayUpgradeRecoverySucceeded, - gatewayUpgradeAgentResponseIsSuccessful, + gatewayServiceWasPreserved, gatewayUpgradeInstallerCommand, type GatewayUpgradeServiceEvidence, captureGatewayUpgradeService, @@ -87,7 +89,8 @@ const OLD_INSTALLER_SHA256 = const OLD_OPENSHELL_VERSION = process.env.NEMOCLAW_OLD_OPENSHELL_VERSION ?? REVIEWED_GATEWAY_UPGRADE_FIXTURE_FOR_REF.openShellVersion; -const CURRENT_OPENSHELL_VERSION = process.env.NEMOCLAW_CURRENT_OPENSHELL_VERSION ?? "0.0.116"; +const CURRENT_OPENSHELL_VERSION = + process.env.NEMOCLAW_CURRENT_OPENSHELL_VERSION ?? SUPPORTED_OPENSHELL_FALLBACK_VERSION; const OLD_SANDBOX_BASE_IMAGE_REF = process.env.NEMOCLAW_OLD_SANDBOX_BASE_IMAGE_REF ?? REVIEWED_GATEWAY_UPGRADE_FIXTURE_FOR_REF.sandboxBaseImageRef; @@ -120,6 +123,9 @@ const DASHBOARD_PORT = "18789"; const GATEWAY_CREDENTIAL = "nemoclaw-gateway-upgrade-fixture-key"; const VERIFY_SYSTEMD_UPGRADE = OLD_NEMOCLAW_COMMIT === REVIEWED_GATEWAY_REGISTRATION_UPGRADE_FIXTURE.nemoclawCommit; +const captureFixtureGatewayForward = VERIFY_SYSTEMD_UPGRADE + ? captureLegacyGatewayServiceForward + : captureLegacyGatewayForward; const TEST_TIMEOUT_MS = REVIEWED_GATEWAY_UPGRADE_FIXTURE_FOR_REF.additionalStoppedSandboxes > 0 ? 110 * 60_000 @@ -224,43 +230,25 @@ type CredentialBoundaryEvidence = { async function collectOpenClawCredentialBoundary( host: HostCliClient, fake: FakeOpenAiCompatibleServer, - phase: "legacy" | "upgraded", + phase: "legacy" | "preserved", ): Promise { const secretNonExposure = await runInSurvivorSandbox( host, gatewayCredentialNonExposureScript(GATEWAY_CREDENTIAL), { artifactName: `state-upgrade-${phase}-secret-non-exposure`, - currentCli: phase === "upgraded", }, ); const requestOffset = fake.requests().length; const sessionId = `e2e-state-upgrade-${phase}`; const agentCommand = `agent --agent main --json --thinking off --session-id ${shellQuote(sessionId)} -m ${shellQuote("Reply with only: ok")}`; - // OpenClaw 2026.9.1's local convenience command requests operator.admin. - // The same native agent RPC needs operator.write for this ordinary message. - const rpcParams = JSON.stringify({ - message: "Reply with only: ok", - agentId: "main", - sessionKey: `agent:main:${sessionId}`, - thinking: "off", - idempotencyKey: sessionId, - cleanupBundleMcpOnRunEnd: true, - }); const agentOptions = { artifactName: `state-upgrade-${phase}-agent`, redactionValues: [GATEWAY_CREDENTIAL], timeoutMs: 120_000, }; - const agent = - phase === "upgraded" - ? await bash( - host, - `nemoclaw ${shellQuote(SURVIVOR_SANDBOX)} exec -- openclaw gateway call agent --json --expect-final --timeout 120000 --params ${shellQuote(rpcParams)}`, - agentOptions, - ) - : await runInSurvivorSandbox(host, `openclaw ${agentCommand}`, agentOptions); + const agent = await runInSurvivorSandbox(host, `openclaw ${agentCommand}`, agentOptions); const requests = fake .requests() .slice(requestOffset) @@ -272,7 +260,6 @@ async function collectOpenClawCredentialBoundary( valid: secretNonExposure.exitCode === 0 && agent.exitCode === 0 && - (phase !== "upgraded" || gatewayUpgradeAgentResponseIsSuccessful(agent.stdout)) && parseOpenClawAgentText(agent.stdout).trim().toLowerCase() === "ok" && requests.length > 0 && requests.every((request) => request.auth === "ok" && request.authorizationSent === true), @@ -568,11 +555,11 @@ async function writeSurvivorMarker(host: HostCliClient): Promise { ); } -async function installCurrentNemoclawUpgrade( +async function attemptCurrentNemoclawUpgrade( host: HostCliClient, artifacts: ArtifactSink, fakeBaseUrl: string, -): Promise { +): Promise { const currentRef = currentNemoclawUpgradeRef(process.env); const currentBaseEnv = liveEnv({ GITHUB_TOKEN: process.env.GITHUB_TOKEN ?? "", @@ -596,109 +583,105 @@ async function installCurrentNemoclawUpgrade( ["COMPATIBLE_API_KEY"], ); const redactionValues = [GATEWAY_CREDENTIAL, process.env.GITHUB_TOKEN ?? ""].filter(Boolean); - await runInstallerPayload( + const result = await bash( host, - `current-${currentRef.slice(0, 12)}`, - currentGatewayUpgradeInstallerArgs(path.join(REPO_ROOT, "scripts", "install.sh")), - artifacts, - "current-install.log", - currentEnv, + gatewayUpgradeInstallerCommand( + currentGatewayUpgradeInstallerArgs(path.join(REPO_ROOT, "scripts", "install.sh")), + VERIFY_SYSTEMD_UPGRADE ? `${os.homedir()}//` : undefined, + ), { - ...(VERIFY_SYSTEMD_UPGRADE ? { home: `${os.homedir()}//` } : {}), - onFailure: async () => { - await Promise.allSettled([ - bash(host, `nemoclaw ${shellQuote(SURVIVOR_SANDBOX)} doctor`, { - artifactName: "current-install-failure-doctor", - env: currentEnv, - redactionValues, - timeoutMs: 120_000, - }), - bash( - host, - `container_id="$(docker ps -aq --filter ${shellQuote(`label=openshell.ai/sandbox-name=${SURVIVOR_SANDBOX}`)} | head -n 1)" -start_log="$(mktemp)" -trap 'rm -f -- "$start_log"' EXIT -docker cp "$container_id:/tmp/nemoclaw-start.log" "$start_log" -tail -n 500 "$start_log"`, - { - artifactName: "current-install-failure-start-log", - env: currentEnv, - redactionValues, - timeoutMs: 30_000, - }, - ), - ]); - }, + artifactName: `current-${currentRef.slice(0, 12)}-installer`, + captureLimitBytes: 1024 * 1024, + env: currentEnv, redactionValues, + timeoutMs: GATEWAY_UPGRADE_INSTALL_TIMEOUT_MS, }, ); + artifacts.addRedactionValues(redactionValues); + await artifacts.writeText("current-install.log", resultText(result)); const openshellVersion = await bash( host, - `openshell --version | grep -F -- ${shellQuote(CURRENT_OPENSHELL_VERSION)}`, + `openshell --version | grep -F -- ${shellQuote(OLD_OPENSHELL_VERSION)}`, { - artifactName: "current-openshell-version", + artifactName: "preserved-openshell-version", redactionValues, timeoutMs: 30_000, }, ); - expectExitZero(openshellVersion, "current openshell --version"); + expectExitZero(openshellVersion, "preserved openshell --version"); + return result; } -async function assertSurvivorSandboxAfterUpgrade( +async function assertLegacyGatewayPreserved( host: HostCliClient, + attempt: ShellProbeResult, + legacyForward: Awaited>, service?: GatewayUpgradeServiceEvidence, ): Promise { - await waitForSandboxPhase(host, SURVIVOR_SANDBOX, "Ready", "post-upgrade"); + await waitForSandboxPhase(host, SURVIVOR_SANDBOX, "Ready", "after-refusal"); const stateChecks = [ await bash( host, - `nemoclaw ${shellQuote(SURVIVOR_SANDBOX)} exec -- grep -Fx -- ${shellQuote(SURVIVOR_MARKER)} ${shellQuote(SURVIVOR_MARKER_PATH)}`, - { artifactName: "post-upgrade-survivor-marker", timeoutMs: 60_000 }, + `openshell sandbox exec --name ${shellQuote(SURVIVOR_SANDBOX)} -- grep -Fx -- ${shellQuote(SURVIVOR_MARKER)} ${shellQuote(SURVIVOR_MARKER_PATH)}`, + { artifactName: "preserved-survivor-marker", timeoutMs: 60_000 }, ), ]; for (const sandboxName of ADDITIONAL_STOPPED_SANDBOXES) { - await waitForSandboxPhase(host, sandboxName, "Stopped", `post-upgrade-${sandboxName}`); - await bash(host, `openshell sandbox start -g nemoclaw ${shellQuote(sandboxName)}`, { - artifactName: `post-upgrade-start-${sandboxName}`, - timeoutMs: 120_000, - }); - await waitForSandboxPhase(host, sandboxName, "Ready", `post-upgrade-${sandboxName}`); - + await waitForSandboxPhase(host, sandboxName, "Stopped", `after-refusal-${sandboxName}`); + const start = await bash( + host, + `openshell sandbox start -g nemoclaw ${shellQuote(sandboxName)}`, + { + artifactName: `preserved-start-${sandboxName}`, + timeoutMs: 120_000, + }, + ); + stateChecks.push(start); + await waitForSandboxPhase(host, sandboxName, "Ready", `preserved-${sandboxName}`); const marker = `gateway-upgrade-stopped-${sandboxName}`; stateChecks.push( await bash( host, - `nemoclaw ${shellQuote(sandboxName)} exec -- grep -Fx -- ${shellQuote(marker)} ${shellQuote(STOPPED_SANDBOX_MARKER_PATH)}`, - { artifactName: `post-upgrade-marker-${sandboxName}`, timeoutMs: 60_000 }, + `openshell sandbox exec --name ${shellQuote(sandboxName)} -- grep -Fx -- ${shellQuote(marker)} ${shellQuote(STOPPED_SANDBOX_MARKER_PATH)}`, + { artifactName: `preserved-marker-${sandboxName}`, timeoutMs: 60_000 }, ), ); - await bash(host, `openshell sandbox stop -g nemoclaw ${shellQuote(sandboxName)}`, { - artifactName: `post-upgrade-stop-${sandboxName}`, + const stop = await bash(host, `openshell sandbox stop -g nemoclaw ${shellQuote(sandboxName)}`, { + artifactName: `preserved-stop-${sandboxName}`, timeoutMs: 120_000, }); - await waitForSandboxPhase(host, sandboxName, "Stopped", `post-upgrade-${sandboxName}`); + stateChecks.push(stop); + await waitForSandboxPhase(host, sandboxName, "Stopped", `restored-${sandboxName}`); } - const recover = await bash(host, `nemoclaw ${shellQuote(SURVIVOR_SANDBOX)} recover`, { - artifactName: "post-upgrade-forward-recovery", - }); - - const forward = await host.inspectOpenShellForwardListener(DASHBOARD_PORT, SURVIVOR_SANDBOX, { - artifactName: "post-upgrade-dashboard-forward", - env: liveEnv(), - }); + const forward = await captureFixtureGatewayForward( + host, + SURVIVOR_SANDBOX, + DASHBOARD_PORT, + liveEnv(), + "preserved", + ); expect( - gatewayUpgradeRecoverySucceeded(recover, forward, stateChecks, service), - `${stateChecks.map(resultText).join("\n")}\n${resultText(recover)}\n${forward.output}\n${JSON.stringify(service)}`, + attempt.exitCode !== 0 && + !attempt.timedOut && + !attempt.signal && + resultText(attempt).includes("OpenShell 0.1.x requires sandbox recreation") && + resultText(attempt).includes("existing gateway and sandboxes were left unchanged") && + LEGACY_SANDBOXES.every((name) => resultText(attempt).includes(name)) && + stateChecks.every((result) => result.exitCode === 0) && + legacyForward.identity !== null && + forward.identity === legacyForward.identity && + (!service || gatewayServiceWasPreserved(service)), + `${resultText(attempt)}\n${stateChecks.map(resultText).join("\n")}\n${legacyForward.output}\n${forward.output}\n${JSON.stringify(service)}`, ).toBe(true); } const runOpenShellGatewayUpgrade = test.skipIf(process.platform !== "linux"); runOpenShellGatewayUpgrade( - "openshell-gateway-upgrade: restores gateway registration and all sandbox state (#11898)", + "openshell-gateway-upgrade: refuses the 0.1.x cutover and preserves registered sandboxes (#12558)", { timeout: TEST_TIMEOUT_MS, meta: { @@ -707,9 +690,9 @@ runOpenShellGatewayUpgrade( "install pinned legacy NemoClaw and its sandbox", "create and stop the fixture-declared additional sandboxes", "verify legacy credential custody and write durable workspace state", - "upgrade to the current OpenShell gateway", - "verify preserved workspace state and every recovered stopped sandbox", - "verify upgraded credential custody", + "attempt the 0.1.x cutover and require refusal", + "verify the legacy gateway and sandbox state remain intact", + "verify preserved credential custody", ], }, }, @@ -720,16 +703,15 @@ runOpenShellGatewayUpgrade( boundary: [ `real old install.sh fetched from ${OLD_NEMOCLAW_REF}`, "real Docker/OpenShell gateway, dashboard forward, and fixture-declared OpenClaw sandboxes", - "exact-name confirmation for the known-managed legacy fixture", - "current scripts/install.sh gateway upgrade path", - "equivalent current-owned dashboard forward after upgrade", + "current scripts/install.sh refusal despite legacy upgrade flags", + "existing dashboard forward and gateway service after refusal", "gateway credential absent from sandbox environment and managed OpenClaw files", - "authenticated OpenClaw turns before and after upgrade", - "durable workspace restore and survivor discovery through the current CLI", - "stopped-sandbox phase restoration and current lifecycle usability", + "authenticated OpenClaw turns before and after refusal", + "durable workspace state remains in the legacy sandbox", + "fixture-declared stopped sandboxes retain their phase", ...(VERIFY_SYSTEMD_UPGRADE ? [ - "real systemd user-service retirement and restart with duplicate-slash HOME and XDG_CONFIG_HOME", + "same active systemd user service after refusal with duplicate-slash HOME and XDG_CONFIG_HOME", ] : []), ], @@ -822,22 +804,31 @@ runOpenShellGatewayUpgrade( VERIFY_SYSTEMD_UPGRADE, ); - progress.phase("upgrade to the current OpenShell gateway"); - await installCurrentNemoclawUpgrade(host, artifacts, fake.baseUrl); - const upgradedService = await captureGatewayUpgradeService( + progress.phase("attempt the 0.1.x cutover and require refusal"); + const legacyForward = await captureFixtureGatewayForward( + host, + SURVIVOR_SANDBOX, + DASHBOARD_PORT, + liveEnv(), + "legacy", + ); + const attempt = await attemptCurrentNemoclawUpgrade(host, artifacts, fake.baseUrl); + const preservedService = await captureGatewayUpgradeService( host, - "upgraded", + "preserved", liveEnv(), VERIFY_SYSTEMD_UPGRADE, ); - progress.phase("verify preserved workspace state and every recovered stopped sandbox"); - await assertSurvivorSandboxAfterUpgrade( + progress.phase("verify the legacy gateway and sandbox state remain intact"); + await assertLegacyGatewayPreserved( host, + attempt, + legacyForward, VERIFY_SYSTEMD_UPGRADE ? { before: legacyService, - after: upgradedService, + after: preservedService, expectedFragmentPath: fs.realpathSync( path.join( os.homedir(), @@ -850,13 +841,13 @@ runOpenShellGatewayUpgrade( } : undefined, ); - progress.phase("verify upgraded credential custody"); - const upgradedCredentialBoundary = await collectOpenClawCredentialBoundary( + progress.phase("verify preserved credential custody"); + const preservedCredentialBoundary = await collectOpenClawCredentialBoundary( host, fake, - "upgraded", + "preserved", ); - const credentialBoundaries = [legacyCredentialBoundary, upgradedCredentialBoundary]; + const credentialBoundaries = [legacyCredentialBoundary, preservedCredentialBoundary]; expect( credentialBoundaries.every((evidence) => evidence.valid), credentialBoundaries.map((evidence) => evidence.diagnostic).join("\n"), diff --git a/test/e2e/live/openshell-v0116-tls-server-name-source.ts b/test/e2e/live/openshell-v0116-tls-server-name-source.ts index 9a3e9b342ba..83ddb64bb05 100644 --- a/test/e2e/live/openshell-v0116-tls-server-name-source.ts +++ b/test/e2e/live/openshell-v0116-tls-server-name-source.ts @@ -3,7 +3,7 @@ import { createHash } from "node:crypto"; -import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; +import { OPENSHELL_V017_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; const MAX_SOURCE_BYTES = 2 * 1024 * 1024; const TLS_SERVER_NAME_REMOVE = "remove(openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME);"; @@ -96,6 +96,144 @@ export const OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES: readonly OpenShellTlsServe }, ]); +// Preserve the historical checks above. The split supervisor uses a protected +// child-environment filter; Podman mediates user variables and VM guests receive +// only driver-owned boot metadata. Bind each reviewed implementation and test. +export const OPENSHELL_V012_TLS_SERVER_NAME_SOURCES: readonly OpenShellTlsServerNameSource[] = + Object.freeze([ + { + ...OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES[0]!, + blobSha: "1f4c91f5e4eae31603f49df11955b37abe9710df", + checks: [ + { + category: "driver", + driver: "docker", + orderedTokens: [ + "fn docker_child_environment(sandbox: &DriverSandbox)", + "environment.extend(spec.environment.clone());", + "openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME,", + "environment.remove(protected);", + ], + }, + ], + }, + { + ...OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES[1]!, + blobSha: "c5036f45473c3b1af179c60fa41e3c48d1c7c4c2", + checks: [ + { + category: "regression", + driver: "docker", + orderedTokens: [ + "fn docker_child_environment_strips_supervisor_control_keys()", + "openshell_core::sandbox_env::GATEWAY_TLS_SERVER_NAME,", + '.insert(key.to_string(), "spoofed".to_string());', + "let env = docker_child_environment(&sandbox);", + 'assert!(!env.values().any(|value| value == "spoofed"));', + ], + }, + ], + }, + { + ...OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES[2]!, + blobSha: "bb4ee88c9f7010835a56e75a570785b6dfed09b7", + checks: [ + { + category: "driver", + driver: "podman", + orderedTokens: [ + "user_env.insert(k.clone(), v.clone());", + "env.insert(openshell_core::sandbox_env::USER_ENVIRONMENT.into(), json);", + `env.${TLS_SERVER_NAME_REMOVE}`, + ], + }, + OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES[2]!.checks[1]!, + ], + }, + { + ...OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES[3]!, + blobSha: "8cdabb22c5bd087178dd43345980452047248598", + checks: [ + { + category: "driver", + driver: "vm", + orderedTokens: [ + "fn build_guest_environment(sandbox: &Sandbox, config: &VmDriverConfig)", + "let mut environment: HashMap = HashMap::new();", + "let mut pairs = environment.into_iter().collect::>();", + "fn build_guest_environment_keeps_user_values_in_child_channel()", + 'assert!(!env.iter().any(|entry| entry.starts_with("LD_PRELOAD=")));', + ], + }, + OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES[3]!.checks[1]!, + ], + }, + ]); + +// OpenShell 0.1.3 retains these environment boundaries and regression predicates. +// Bind them to the new release blobs without replacing historical source evidence. +export const OPENSHELL_V013_TLS_SERVER_NAME_SOURCES: readonly OpenShellTlsServerNameSource[] = + Object.freeze([ + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[0]!, + blobSha: "7a8935ff2908580e2fbadacb01fd78bbef05e93d", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[1]!, + blobSha: "3d8968f2dbcb35922a6f90a8f3282fdce0c77528", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[2]!, + blobSha: "3091ac978c8c2d9ea481b4f7bf2d405458f04c01", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[3]!, + blobSha: "ab1aca902bc1a705366685750ecb32332077452a", + }, + ]); + +// OpenShell 0.1.5 preserves the reviewed environment filtering and regression tests. +export const OPENSHELL_V015_TLS_SERVER_NAME_SOURCES: readonly OpenShellTlsServerNameSource[] = + Object.freeze([ + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[0]!, + blobSha: "059ba1535cb438215d23693926f0ef57129716ff", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[1]!, + blobSha: "0159ecd99f5e2a0aead65feaa7cde65b8af6d886", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[2]!, + blobSha: "9c3841d000a73646539aa48a31bf12d2b1bd3ef1", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[3]!, + blobSha: "65c0c82a5bb24316d8620a80c78137d5ce754f75", + }, + ]); + +// OpenShell 0.1.7 preserves the reviewed environment filtering and regression tests. +export const OPENSHELL_V017_TLS_SERVER_NAME_SOURCES: readonly OpenShellTlsServerNameSource[] = + Object.freeze([ + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[0]!, + blobSha: "6b761f2f599e49fd8fae5e47239465f92cdb3a92", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[1]!, + blobSha: "92d20739f35ce7f27326df3c54c8e90de6596707", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[2]!, + blobSha: "2f606dfa6e26f1667d6412db86de3b05cc763f77", + }, + { + ...OPENSHELL_V012_TLS_SERVER_NAME_SOURCES[3]!, + blobSha: "f7946c34c29ee75f49fb5a131ab873f3e91683c5", + }, + ]); + function gitBlobSha(source: string): string { const content = Buffer.from(source, "utf8"); const header = Buffer.from(`blob ${String(content.byteLength)}\0`, "utf8"); @@ -153,7 +291,11 @@ type VerificationResult = { export async function verifyOpenShellTlsServerNameSourceBoundary( fetchSource: typeof fetch = fetch, - reviewedSources: readonly OpenShellTlsServerNameSource[] = OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES, + reviewedSources: readonly OpenShellTlsServerNameSource[] = OPENSHELL_V017_TLS_SERVER_NAME_SOURCES, + qualification: Readonly<{ + sourceRevision: string; + version: string; + }> = OPENSHELL_V017_QUALIFICATION, ): Promise<{ drivers: VerificationResult[]; regressions: VerificationResult[]; @@ -165,7 +307,7 @@ export async function verifyOpenShellTlsServerNameSourceBoundary( for (const reviewedSource of reviewedSources) { const url = `https://raw.githubusercontent.com/NVIDIA/OpenShell/` + - `${OPENSHELL_V0116_QUALIFICATION.sourceRevision}/${reviewedSource.path}`; + `${qualification.sourceRevision}/${reviewedSource.path}`; const response = await fetchSource(url, { redirect: "error", signal: AbortSignal.timeout(30_000), @@ -190,7 +332,7 @@ export async function verifyOpenShellTlsServerNameSourceBoundary( return { drivers, regressions, - sourceRevision: OPENSHELL_V0116_QUALIFICATION.sourceRevision, - version: OPENSHELL_V0116_QUALIFICATION.version, + sourceRevision: qualification.sourceRevision, + version: qualification.version, }; } diff --git a/test/e2e/live/package-database-read-only.ts b/test/e2e/live/package-database-read-only.ts index 450bfa63164..c289507849e 100644 --- a/test/e2e/live/package-database-read-only.ts +++ b/test/e2e/live/package-database-read-only.ts @@ -27,7 +27,8 @@ export async function expectPackageDatabaseReadOnly( [ "sh", "-c", - 'set -eu; probe="$1"; test ! -e "$probe"; install -o sandbox -g sandbox -m 600 /dev/null "$probe"', + // Workload containers drop CAP_CHOWN; writable mode isolates Landlock from DAC. + 'set -eu; probe="$1"; test ! -e "$probe"; install -m 666 /dev/null "$probe"', "sh", sentinel, ], diff --git a/test/e2e/live/pi-agent-qualification.test.ts b/test/e2e/live/pi-agent-qualification.test.ts index 5a34e38000b..777e5c37087 100644 --- a/test/e2e/live/pi-agent-qualification.test.ts +++ b/test/e2e/live/pi-agent-qualification.test.ts @@ -31,6 +31,10 @@ import { } from "../fixtures/clients/sandbox.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { assertNoDockerfileBuild, createDockerBuildGuard } from "../fixtures/docker-build-guard.ts"; +import { + cleanupOwnedSandboxAfterOnboard, + cleanupRegisteredSandbox, +} from "../fixtures/owned-sandbox-cleanup.ts"; import { REPO_ROOT } from "../fixtures/paths.ts"; import type { LifecyclePhaseFixture } from "../fixtures/phases/lifecycle.ts"; import type { TestProgress } from "../fixtures/progress.ts"; @@ -245,7 +249,7 @@ async function persistentStateInventory( const result = await execPiShell( sandbox, trustedSandboxShellScript( - `${stateFixture}{ [ ! -f /sandbox/.pi/agent/settings.json ] || sha256sum /sandbox/.pi/agent/settings.json; find /sandbox/.pi/agent/sessions -type f -name '*.jsonl' -exec sha256sum {} +; find /sandbox/.pi/agent -maxdepth 1 -type f -name trust.json -exec grep -F '${PI_TRUST_PRESERVATION_MARKER}' {} +; } | sort`, + `set -euo pipefail; ${stateFixture}{ [ ! -f /sandbox/.pi/agent/settings.json ] || sha256sum /sandbox/.pi/agent/settings.json; find /sandbox/.pi/agent/sessions -type f -name '*.jsonl' -exec sha256sum {} +; find /sandbox/.pi/agent -maxdepth 1 -type f -name trust.json -exec sed -n '/${PI_TRUST_PRESERVATION_MARKER}/p' {} +; } | sort`, ), { artifactName: `pi-${phase}-persistent-state-inventory`, env, timeoutMs: 30_000 }, ); @@ -311,7 +315,16 @@ test( ], }, }, - async ({ artifacts, cleanup, host, inference, lifecycle, progress, sandbox }) => { + async ({ + artifacts, + cleanup, + host, + inference, + lifecycle, + progress, + runtimeProvider, + sandbox, + }) => { const platform = qualificationPlatform( process.arch, process.env.NEMOCLAW_PI_QUALIFICATION_PLATFORM, @@ -339,9 +352,20 @@ test( cleanup.trackDisposable("remove Pi Docker build guard", guard.dispose); cleanup.trackGateway(host, GATEWAY, { env, timeoutMs: 60_000 }); cleanup.trackDisposable("remove Pi OpenShell sandbox", () => - sandbox.cleanupSandbox(SANDBOX_NAME, { env, timeoutMs: 60_000 }), + cleanupOwnedSandboxAfterOnboard(sandbox, runtimeProvider, SANDBOX_NAME, { + env, + timeoutMs: 60_000, + }), + ); + cleanup.trackDisposable(`destroy sandbox ${SANDBOX_NAME}`, () => + cleanupRegisteredSandbox(SANDBOX_NAME, () => + host.cleanupSandbox(SANDBOX_NAME, { + env, + cleanupGateway: true, + timeoutMs: 5 * 60_000, + }), + ), ); - cleanup.trackSandbox(host, SANDBOX_NAME, { env, timeoutMs: 5 * 60_000 }); await artifacts.target.declare({ id: "pi-agent-qualification", diff --git a/test/e2e/live/podman-cpu-lifecycle-artifacts.ts b/test/e2e/live/podman-cpu-lifecycle-artifacts.ts index 6d26c3d3d44..bed87700c6d 100644 --- a/test/e2e/live/podman-cpu-lifecycle-artifacts.ts +++ b/test/e2e/live/podman-cpu-lifecycle-artifacts.ts @@ -10,6 +10,7 @@ import { redactString } from "../fixtures/redaction.ts"; // failed build. The focused support test binds these allowlist keys to the // production Podman ownership constants. const PODMAN_MANAGED_LABEL = "openshell.managed"; +const PODMAN_ISOLATION_ROLE_LABEL = "openshell.ai/isolation-role"; const PODMAN_SANDBOX_ID_LABEL = "openshell.ai/sandbox-id"; const PODMAN_SANDBOX_NAME_LABEL = "openshell.ai/sandbox-name"; const PODMAN_SANDBOX_NAMESPACE_LABEL = "openshell.ai/sandbox-namespace"; @@ -35,6 +36,8 @@ type JsonRecord = Record; export interface PodmanContainerArtifactSummary { readonly labels: Readonly>; readonly state: { + readonly exitCode: number | null; + readonly oomKilled: boolean | null; readonly paused: boolean | null; readonly running: boolean | null; readonly status: string | null; @@ -90,6 +93,11 @@ export function sanitizePodmanInspectArtifact(output: string): PodmanContainerAr return Object.freeze({ labels: Object.freeze({ [PODMAN_MANAGED_LABEL]: labels[PODMAN_MANAGED_LABEL] === "true" ? "true" : null, + [PODMAN_ISOLATION_ROLE_LABEL]: + labels[PODMAN_ISOLATION_ROLE_LABEL] === "sandbox" || + labels[PODMAN_ISOLATION_ROLE_LABEL] === "supervisor" + ? labels[PODMAN_ISOLATION_ROLE_LABEL] + : null, [PODMAN_SANDBOX_ID_LABEL]: identityValue(labels[PODMAN_SANDBOX_ID_LABEL]), [PODMAN_SANDBOX_NAME_LABEL]: identityValue(labels[PODMAN_SANDBOX_NAME_LABEL]), [PODMAN_SANDBOX_NAMESPACE_LABEL]: identityValue(labels[PODMAN_SANDBOX_NAMESPACE_LABEL], true), @@ -100,6 +108,14 @@ export function sanitizePodmanInspectArtifact(output: string): PodmanContainerAr [AGENT_LABEL]: identityValue(labels[AGENT_LABEL]), }), state: Object.freeze({ + exitCode: + typeof state.ExitCode === "number" && + Number.isInteger(state.ExitCode) && + state.ExitCode >= 0 && + state.ExitCode <= 255 + ? state.ExitCode + : null, + oomKilled: booleanValue(state.OOMKilled), paused: booleanValue(state.Paused), running: booleanValue(state.Running), status: stateValue(state.Status), diff --git a/test/e2e/live/podman-cpu-lifecycle-helpers.ts b/test/e2e/live/podman-cpu-lifecycle-helpers.ts index 8c9ddddb541..e011e08311e 100644 --- a/test/e2e/live/podman-cpu-lifecycle-helpers.ts +++ b/test/e2e/live/podman-cpu-lifecycle-helpers.ts @@ -7,6 +7,7 @@ import path from "node:path"; import type { ContainerEngine } from "../../../src/lib/adapters/container-engine"; import { + PODMAN_ISOLATION_ROLE_LABEL, PODMAN_MANAGED_LABEL, PODMAN_SANDBOX_CONTAINER_PREFIX, PODMAN_SANDBOX_ID_LABEL, @@ -19,7 +20,7 @@ import { import { redactFull } from "../../../src/lib/security/redact"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import { expect } from "../fixtures/e2e-test.ts"; -import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; +import { OPENSHELL_V017_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; import { spawnObservedChild } from "../fixtures/observed-child-process.ts"; import type { TestProgress } from "../fixtures/progress.ts"; import { @@ -35,7 +36,7 @@ import { export const ARTIFACT_DIR = process.env.E2E_ARTIFACT_DIR ?? ""; export const GATEWAY_NAME = "nemoclaw-18080"; -export const OPENSHELL_VERSION = OPENSHELL_V0116_QUALIFICATION.version; +export const OPENSHELL_VERSION = OPENSHELL_V017_QUALIFICATION.version; export const SOCKET_PATH = process.env.E2E_PODMAN_SOCKET ?? ""; const FULL_CONTAINER_ID = /^[0-9a-f]{64}$/u; @@ -50,6 +51,7 @@ interface ManagedContainerInspect { Cmd: string[]; Entrypoint: string | string[]; Labels: Record; + User: string; }; Id: string; Name: string; @@ -139,7 +141,7 @@ export async function startPinnedGateway( artifactDir = ARTIFACT_DIR, ): Promise { const child = spawnObservedChild(gatewayBin, [], { - activityLabel: "command: pinned OpenShell 0.0.116 Podman gateway", + activityLabel: `command: pinned OpenShell ${OPENSHELL_VERSION} Podman gateway`, progress, spawn: { env: { ...process.env, ...gatewayEnv }, @@ -147,7 +149,8 @@ export async function startPinnedGateway( }, }); let output = ""; - const recordOutput = (chunk: unknown) => { + let importantOutput = ""; + const recordOutput = (chunk: string) => { output = `${output}${String(chunk)}`.slice(-MAX_GATEWAY_DIAGNOSTIC_CHARS * 2); if (!artifactDir) return; fs.mkdirSync(artifactDir, { recursive: true, mode: 0o700 }); @@ -157,8 +160,39 @@ export async function startPinnedGateway( { encoding: "utf-8", mode: 0o600 }, ); }; - child.stdout?.on("data", recordOutput); - child.stderr?.on("data", recordOutput); + for (const stream of [child.stdout, child.stderr]) { + // Keep warnings and errors when frequent successful RPC logs replace the tail. + // Buffer each stream independently so split credentials are redacted together. + let pendingLine = ""; + let oversized = false; + const recordLines = (chunk: string) => { + for (const part of chunk.split(/(?<=\n)/u)) { + if (!oversized) pendingLine += part; + if (pendingLine.length > MAX_GATEWAY_DIAGNOSTIC_CHARS) { + pendingLine = ""; + oversized = true; + } + if (!part.endsWith("\n")) continue; + const line = stripAnsi(pendingLine); + if (artifactDir && !oversized && /^(?:\S+\s+)?(?:WARN|ERROR)\s/u.test(line)) { + importantOutput = gatewayDiagnostic(`${importantOutput}${line}`); + fs.writeFileSync( + path.join(artifactDir, "openshell-podman-gateway-errors.log"), + importantOutput, + { encoding: "utf-8", mode: 0o600 }, + ); + } + pendingLine = ""; + oversized = false; + } + }; + stream?.setEncoding("utf8"); + stream?.on("data", (chunk: string) => { + recordOutput(chunk); + recordLines(chunk); + }); + stream?.once("end", () => recordLines("\n")); + } const deadline = Date.now() + 30_000; while (Date.now() < deadline) { @@ -245,6 +279,8 @@ export function exactContainerId(engine: ContainerEngine, sandboxName: string): `label=${PODMAN_SANDBOX_NAME_LABEL}=${sandboxName}`, "--filter", `label=${PODMAN_SANDBOX_WORKSPACE_LABEL}=${PODMAN_SANDBOX_WORKSPACE}`, + "--filter", + `label=${PODMAN_ISOLATION_ROLE_LABEL}=sandbox`, ]); expect(result).toMatchObject({ status: 0, stderr: "" }); const rows = result.stdout @@ -271,16 +307,14 @@ export function inspectContainer( const labels = entry.Config.Labels; const sandboxId = labels[PODMAN_SANDBOX_ID_LABEL]; expect(entry.Id).toBe(containerId); - expect(entry.Id).toMatch(FULL_CONTAINER_ID); expect(sandboxId).toBeTruthy(); expect(entry.Name).toBe(`${PODMAN_SANDBOX_CONTAINER_PREFIX}${sandboxName}-${sandboxId}`); expect(labels).toMatchObject({ - [PODMAN_MANAGED_LABEL]: "true", - [PODMAN_SANDBOX_NAME_LABEL]: sandboxName, [PODMAN_SANDBOX_NAMESPACE_LABEL]: PODMAN_SANDBOX_NAMESPACE, - [PODMAN_SANDBOX_WORKSPACE_LABEL]: PODMAN_SANDBOX_WORKSPACE, }); - expect(entry.Config.Cmd).toEqual(["--workdir", "/sandbox"]); + // OpenShell 0.1.3 sets the policy identity in OCI User, not launcher arguments. + expect(entry.Config.User).toBe("1000:1000"); + expect(entry.Config.Cmd).toEqual(["--bootstrap", "/.openshell/channel/sandbox/bootstrap.json"]); const entrypoint = Array.isArray(entry.Config.Entrypoint) ? entry.Config.Entrypoint : [entry.Config.Entrypoint]; diff --git a/test/e2e/live/podman-cpu-lifecycle.test.ts b/test/e2e/live/podman-cpu-lifecycle.test.ts index ea333b8e81e..e6616acec04 100644 --- a/test/e2e/live/podman-cpu-lifecycle.test.ts +++ b/test/e2e/live/podman-cpu-lifecycle.test.ts @@ -33,7 +33,10 @@ import type { SandboxEntry } from "../../../src/lib/state/registry"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import { expect, test } from "../fixtures/e2e-test.ts"; import { REPO_ROOT } from "../fixtures/paths.ts"; -import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; +import { + exactGatewayRelease, + OPENSHELL_V017_QUALIFICATION, +} from "../fixtures/openshell-v0116-qualification.ts"; import { consumeNativeRuntimeCandidateEvidence, type NativeRuntimeCandidateEvidence, @@ -64,11 +67,11 @@ const BASE_IMAGE = // Keep the rootless proof on the immutable sandbox-base from the NemoClaw // v0.0.89 fixture, which runs OpenShell v0.0.85. Unlike a minimal Ubuntu // image, it includes the `ip` binary needed before workload startup and - // exercises the v0.0.85 image-to-v0.0.116 supervisor compatibility boundary. + // exercises the v0.0.85 image-to-v0.1.7 supervisor compatibility boundary. "ghcr.io/nvidia/nemoclaw/sandbox-base@sha256:3265d482f67c9d81ee3a59b0bbad5eb5ea6c705fea81ece8ae888ed12794f7f1"; const ACTIVATION_POLICY = path.join(REPO_ROOT, "test/e2e/live/podman-cpu-lifecycle-policy.yaml"); const GATEWAY_PORT = 18_080; -const SUPERVISOR_IMAGE = OPENSHELL_V0116_QUALIFICATION.supervisorImage; +const SUPERVISOR_IMAGE = OPENSHELL_V017_QUALIFICATION.supervisorImage; const E2E_PHASES = [ "consume exact candidate prerequisites", "pin the exact rootless Podman endpoint", @@ -120,7 +123,6 @@ test( progress.phase("pin the exact rootless Podman endpoint"); expect(process.platform).toBe("linux"); - expect(process.getuid?.()).not.toBe(0); expect(ARTIFACT_DIR).not.toBe(""); let runtimeEngines = engines(); const bundle = createPodmanRuntimeProviderBundle({ engines: runtimeEngines }); @@ -150,10 +152,12 @@ test( const sandboxBin = executableOnPath("openshell-sandbox"); for (const component of [openshellBin, gatewayBin, sandboxBin]) { expect( - await runCommand(shellProbe, component, ["--version"], { - artifactName: `podman-lifecycle-version-${path.basename(component)}`, - }), - ).toContain(OPENSHELL_VERSION); + exactGatewayRelease( + await runCommand(shellProbe, component, ["--version"], { + artifactName: `podman-lifecycle-version-${path.basename(component)}`, + }), + ), + ).toBe(OPENSHELL_VERSION); } const uid = process.getuid?.() ?? -1; @@ -292,6 +296,7 @@ exit 1 "--label", `nemoclaw.agent=${agent}`, "--no-tty", + "--detach", "--", "/bin/sh", "-lc", @@ -330,6 +335,18 @@ exit 1 ).toBe(agent); const activated = inspectContainer(runtimeEngines.sandboxLifecycle, sandboxName); expect(activated.State).toMatchObject({ Paused: false, Running: true, Status: "running" }); + expect( + await runCommand( + shellProbe, + openshellBin, + ["sandbox", "exec", "--name", sandboxName, "-g", GATEWAY_NAME, "--", "pwd"], + { + artifactName: `podman-lifecycle-workspace-${agent}`, + env: cliEnv, + timeoutMs: 10_000, + }, + ), + ).toBe("/sandbox"); } expect( diff --git a/test/e2e/live/podman-portable-uninstall.test.ts b/test/e2e/live/podman-portable-uninstall.test.ts index b747b88f986..62aad8368dc 100644 --- a/test/e2e/live/podman-portable-uninstall.test.ts +++ b/test/e2e/live/podman-portable-uninstall.test.ts @@ -60,6 +60,7 @@ function sandboxCreateArgs(): string[] { "--policy", path.join(REPO_ROOT, "test/e2e/live/podman-cpu-lifecycle-policy.yaml"), "--no-tty", + "--detach", "--", "/bin/sh", "-lc", diff --git a/test/e2e/live/portable-profile-gateway-proof.ts b/test/e2e/live/portable-profile-gateway-proof.ts index 1f658e27bf5..a39c7b70517 100644 --- a/test/e2e/live/portable-profile-gateway-proof.ts +++ b/test/e2e/live/portable-profile-gateway-proof.ts @@ -2,11 +2,57 @@ // SPDX-License-Identifier: Apache-2.0 import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import type { HermesPortableLifecycleCommandResult } from "../../../src/lib/onboard/experimental/hermes-portable-lifecycle.ts"; + +import { redactFullWithUrls } from "../../../src/lib/security/redact.ts"; import { spawnObservedChild } from "../fixtures/observed-child-process.ts"; import type { TestProgress } from "../fixtures/progress.ts"; import { stripAnsi } from "./json-envelope.ts"; +/** Capture failed fixture containers before the owning fixture removes its store. */ +export function recordPortableSandboxDiagnostics( + socketPath: string, + sandboxName: string, + failureStatus: number | null, +): void { + if (failureStatus === 0) return; + const capture = (args: string[]) => + spawnSync("podman", ["--url", `unix://${socketPath}`, ...args], { + env: process.env, + encoding: "utf-8", + stdio: ["ignore", "pipe", "pipe"], + timeout: 10_000, + killSignal: "SIGKILL", + maxBuffer: 512 * 1024, + }); + const listed = capture([ + "ps", + "--all", + "--no-trunc", + "--filter", + `label=openshell.ai/sandbox-name=${sandboxName}`, + "--format", + "{{.ID}}", + ]); + console.error( + redactFullWithUrls( + `Sandbox diagnostic inventory (status ${String(listed.status)}):\n${listed.stdout ?? ""}${listed.stderr ?? ""}${listed.error?.message ?? ""}`, + ), + ); + if (listed.status !== 0 || listed.error) return; + for (const id of String(listed.stdout).trim().split(/\s+/u)) { + if (!/^[a-f0-9]{64}$/u.test(id)) continue; + const logs = capture(["logs", id]); + console.error( + redactFullWithUrls( + `Sandbox container ${id} logs (status ${String(logs.status)}):\n${logs.stdout ?? ""}${logs.stderr ?? ""}${logs.error?.message ?? ""}`, + ), + ); + } +} + /** Verify that the pinned gateway accepts the generated rootless Podman configuration. */ export async function verifyPinnedPodmanGatewayStarts( gatewayBin: string, @@ -56,5 +102,37 @@ export async function verifyPinnedPodmanGatewayStarts( ); } finally { child.kill("SIGTERM"); + console.error(`Pinned OpenShell gateway diagnostics:\n${redactFullWithUrls(output)}`); } } + +export function captureOpenShell( + executablePath: string, + env: NodeJS.ProcessEnv, + args: readonly string[], + timeoutMs = 30_000, +): HermesPortableLifecycleCommandResult { + const result = spawnSync(executablePath, [...args], { + encoding: "utf-8", + env, + killSignal: "SIGKILL", + maxBuffer: 512 * 1024, + stdio: ["ignore", "pipe", "pipe"], + timeout: + timeoutMs <= 10_000 + ? 10_000 + : timeoutMs <= 30_000 + ? 30_000 + : timeoutMs <= 40_000 + ? 40_000 + : timeoutMs <= 60_000 + ? 60_000 + : 240_000, + }); + return { + status: result.status, + stdout: result.stdout ?? "", + stderr: result.stderr ?? "", + ...(result.error ? { error: result.error } : {}), + }; +} diff --git a/test/e2e/live/portable-profile-rootless-linux.test.ts b/test/e2e/live/portable-profile-rootless-linux.test.ts index ea8c8c4aff5..eaa84e6e2ae 100644 --- a/test/e2e/live/portable-profile-rootless-linux.test.ts +++ b/test/e2e/live/portable-profile-rootless-linux.test.ts @@ -8,6 +8,8 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; +import { redactFullWithUrls } from "../../../src/lib/security/redact.ts"; + import * as importedGatewayEnv from "../../../src/lib/onboard/docker-driver-gateway-env.ts"; import * as importedGatewayLocalTls from "../../../src/lib/onboard/docker-driver-gateway-local-tls.ts"; import * as importedBuildContextStage from "../../../src/lib/onboard/build-context-stage.ts"; @@ -17,6 +19,7 @@ import * as importedSandboxPrebuild from "../../../src/lib/onboard/sandbox-prebu import * as importedBuildContext from "../../../src/lib/sandbox/build-context.ts"; import { capturePodmanSocketAuthority } from "../../../src/lib/adapters/podman/index.ts"; import { captureHermesPortableOpenShellExecutableAuthority } from "../../../src/lib/adapters/openshell/resolve-shared.ts"; +import { parseStrictOpenShellSandboxListJson } from "../../../src/lib/adapters/openshell/sandbox-identity.ts"; import { OPENSHELL_HEAVY_TIMEOUT_MS } from "../../../src/lib/adapters/openshell/timeouts.ts"; import { loadAgent } from "../../../src/lib/agent/defs.ts"; import { @@ -62,9 +65,9 @@ import type { SandboxEntry } from "../../../src/lib/state/registry/types.ts"; import { retryUntil } from "../../../src/lib/core/retry.ts"; import { streamSandboxCreate } from "../../../src/lib/sandbox/create-stream.ts"; import { test } from "../fixtures/e2e-test.ts"; +import { OPENSHELL_V017_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; import type { ArtifactSink } from "../fixtures/artifacts.ts"; import { recordPortableHermesGatewayProbeFailure } from "../fixtures/portable-hermes-probe-diagnostics.ts"; -import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; import { capturePortableRegistryProcessBoundary, collectPortableRegistryReadinessFailure, @@ -91,13 +94,13 @@ import { runPortableE2eWithCleanup, } from "../fixtures/portable-sandbox-build-diagnostics.ts"; import type { TestProgress } from "../fixtures/progress.ts"; +import { runSandboxTokenContainerProbe } from "./openshell-gateway-auth-probe.ts"; import type { ShellProbe } from "../fixtures/shell-probe.ts"; import { - getSandboxConfigRequest, - mintSandboxJwt, - runSandboxTokenContainerProbe, -} from "./openshell-gateway-auth-probe.ts"; -import { verifyPinnedPodmanGatewayStarts } from "./portable-profile-gateway-proof.ts"; + captureOpenShell, + recordPortableSandboxDiagnostics, + verifyPinnedPodmanGatewayStarts, +} from "./portable-profile-gateway-proof.ts"; import { proveHermesPortableOnboardingTransaction, withAmbientPodmanHost, @@ -380,44 +383,13 @@ function selectInstallerPodmanRuntime(repoRoot: string): string { return run("bash", ["-c", script]); } -function captureOpenShell( - executablePath: string, - env: NodeJS.ProcessEnv, - args: readonly string[], - timeoutMs = 30_000, -): HermesPortableLifecycleCommandResult { - const result = spawnSync(executablePath, [...args], { - encoding: "utf-8", - env, - killSignal: "SIGKILL", - maxBuffer: 512 * 1024, - stdio: ["ignore", "pipe", "pipe"], - timeout: - timeoutMs <= 10_000 - ? 10_000 - : timeoutMs <= 30_000 - ? 30_000 - : timeoutMs <= 40_000 - ? 40_000 - : timeoutMs <= 60_000 - ? 60_000 - : 240_000, - }); - return { - status: result.status, - stdout: result.stdout ?? "", - stderr: result.stderr ?? "", - ...(result.error ? { error: result.error } : {}), - }; -} - function requireOpenShellResult( result: HermesPortableLifecycleCommandResult, label: string, ): string { assert.ok( result.status === 0 && !result.error, - `${label} failed: ${String(result.error?.message ?? result.stderr ?? result.stdout)}`, + `${label} failed (status ${String(result.status)}): ${redactFullWithUrls(String(result.error?.message || result.stderr || result.stdout))}`, ); return String(result.stdout).trim(); } @@ -550,7 +522,7 @@ function waitForOpenShellSandboxAbsent( 10_000, ); try { - const sandboxes = JSON.parse(String(result.stdout)) as unknown; + const sandboxes = parseStrictOpenShellSandboxListJson(String(result.stdout)); return { absent: result.status === 0 && @@ -663,6 +635,7 @@ async function proveHistoricalHermesPortableLifecycle(input: { "--policy", qualifiedPolicyPath, "--no-tty", + "--detach", "--", ...startupArgv, ]; @@ -693,10 +666,17 @@ async function proveHistoricalHermesPortableLifecycle(input: { observedCreatePhase === "Error" ? "OpenShell Hermes sandbox entered Error during creation." : null, - readyCheckOutputPatterns: [/Setting up NemoClaw/u], + // Detached create has no startup stream; stopping an attached client + // also terminates its canonical main process. + readyCheckOutputPatterns: [], waitForReadyTermination: true, }, ); + recordPortableSandboxDiagnostics( + input.runtimeAuthority.socketPath, + sandboxName, + createResult.status, + ); requireOpenShellResult( { status: createResult.status, @@ -847,13 +827,30 @@ async function proveHistoricalHermesPortableLifecycle(input: { }, } satisfies Parameters[1]; const upgradeResult = await startSandbox(sandboxName, publicStartDeps); - const firstStop = await withMcpLifecycleLock( + console.log("Historical Hermes public start:", JSON.stringify(upgradeResult)); + recordPortableSandboxDiagnostics( + input.runtimeAuthority.socketPath, sandboxName, - () => - stopHermesPortableSandboxLifecycle(sandboxName, context, () => undefined, lifecycleDeps), - { stateDir: path.join(receiptStateDir, "state") }, + upgradeResult.exitCode, + ); + const firstStop = + upgradeResult.exitCode === 0 + ? await withMcpLifecycleLock( + sandboxName, + () => + stopHermesPortableSandboxLifecycle( + sandboxName, + context, + () => undefined, + lifecycleDeps, + ), + { stateDir: path.join(receiptStateDir, "state") }, + ) + : null; + assert.ok( + firstStop?.kind === "stopped", + `Historical Hermes public start or first stop failed: upgrade=${JSON.stringify(upgradeResult)} stop=${firstStop?.kind ?? "not attempted"}`, ); - assert.equal(firstStop.kind, "stopped", "Historical Hermes receipt did not stop"); const firstStoppedContainerStatus = waitForReceiptOwnedContainerExit(activeContainerId); const firstTerminalPhase = waitForOpenShellTerminalPhase( input.openshellBin, @@ -1278,7 +1275,7 @@ async function main( gatewayPort: 8080, stateDir, podmanSocketPath: `${runtimeDir}/podman/podman.sock`, - getDockerSupervisorImage: () => OPENSHELL_V0116_QUALIFICATION.supervisorImage, + getDockerSupervisorImage: () => OPENSHELL_V017_QUALIFICATION.supervisorImage, resolveSandboxBin: () => sandboxBin, }); assert.equal(gatewayEnv.OPENSHELL_GRPC_ENDPOINT, `https://${PORTABLE_HOST_GATEWAY_IP}:8080`); @@ -1325,30 +1322,24 @@ async function main( waitForOpenShellGateway(openshellBin, openshellClientEnv); progress.phase("verify distinct same-network routes"); - const sandboxId = "portable-e2e"; - const sandboxToken = mintSandboxJwt({ - configPath: gatewayEnv.OPENSHELL_GATEWAY_CONFIG, - sandboxId, - }); - artifacts.addRedactionValues([sandboxToken]); const gatewayProbe = runSandboxTokenContainerProbe({ - authorization: `Bearer ${sandboxToken}`, dockerBin: "podman", + grpcPath: "/openshell.v1.OpenShell/Health", hostGatewayIp: PORTABLE_HOST_GATEWAY_IP, networkName: PORTABLE_DOCKER_NETWORK_NAME, - payload: getSandboxConfigRequest(sandboxId), + payload: Buffer.alloc(0), port: 8080, stateDir, }); - await recordPortableHermesGatewayProbeFailure(artifacts, sandboxToken, gatewayProbe); + // The Health probe uses mTLS and does not mint a sandbox bearer token. + await recordPortableHermesGatewayProbeFailure(artifacts, "", gatewayProbe); assert.equal(gatewayProbe.status, 0, "The authenticated same-network gateway probe failed."); const gatewayResult = JSON.parse(gatewayProbe.stdout) as { grpcStatus?: string; httpStatus?: number; }; assert.equal(gatewayResult.httpStatus, 200); - assert.ok(gatewayResult.grpcStatus); - assert.ok(!["7", "16"].includes(gatewayResult.grpcStatus)); + assert.equal(gatewayResult.grpcStatus, "0"); const registryRouteProof = [ `exec 3<>/dev/tcp/${PORTABLE_REGISTRY_IP}/5000`, diff --git a/test/e2e/live/runtime-identity-oauth-server.ts b/test/e2e/live/runtime-identity-oauth-server.ts index 2d4edbf5cf9..3daa1d4a353 100644 --- a/test/e2e/live/runtime-identity-oauth-server.ts +++ b/test/e2e/live/runtime-identity-oauth-server.ts @@ -16,6 +16,8 @@ import { import type { StartedHttpServer } from "./mcp-bridge-servers.ts"; export interface RuntimeIdentityTokenRequest { + readonly sequence: number; + readonly receivedAt: string; readonly method: string; readonly path: string; readonly grantTypeOk: boolean; @@ -33,19 +35,12 @@ export interface RuntimeIdentityResourceRequest { } export interface RuntimeIdentityOAuthServer extends StartedHttpServer { + readonly certificate: Buffer; tokenRequests(): readonly RuntimeIdentityTokenRequest[]; resourceRequests(): readonly RuntimeIdentityResourceRequest[]; secretValues(): readonly string[]; } -function requireTcpPort(server: https.Server): number { - const address = server.address(); - if (!address || typeof address === "string") { - throw new Error("runtime identity OAuth fixture did not bind to a TCP port"); - } - return address.port; -} - function generateEphemeralTlsMaterial(): { dir: string; cert: Buffer; @@ -67,6 +62,8 @@ function generateEphemeralTlsMaterial(): { "1", "-subj", "/CN=nemoclaw-runtime-identity-e2e", + "-addext", + "subjectAltName=IP:127.0.0.1", "-keyout", keyPath, "-out", @@ -130,6 +127,7 @@ export async function startRuntimeIdentityOAuthServer(options: { let currentRefreshToken = refreshTokens[0]; let currentAccessToken: string | undefined; let issueCount = 0; + let tokenRequestSequence = 0; const resourcePath = options.resourcePath ?? "/resource"; const tokenPath = options.tokenPath ?? "/oauth/token"; @@ -145,6 +143,10 @@ export async function startRuntimeIdentityOAuthServer(options: { } if (req.method === "POST" && requestPath === tokenPath) { + // Capture arrival before reading the body so a slow request cannot appear + // to start after a later blueprint operation. Never retain the raw form. + const receivedAt = new Date().toISOString(); + const sequence = ++tokenRequestSequence; const body = new URLSearchParams(await readRequestBody(req)); const grantTypeOk = body.get("grant_type") === "refresh_token"; const clientIdOk = body.get("client_id") === options.clientId; @@ -153,6 +155,8 @@ export async function startRuntimeIdentityOAuthServer(options: { const accepted = grantTypeOk && clientIdOk && refreshTokenOk && clientSecretOk; const nextVersion = accepted && issueCount < accessTokens.length ? issueCount + 1 : null; tokenRequests.push({ + sequence, + receivedAt, method: req.method, path: requestPath, grantTypeOk, @@ -215,9 +219,10 @@ export async function startRuntimeIdentityOAuthServer(options: { writeJsonResponse(res, 404, { error: "not found" }); }); - await listenOnRandomPort(server); + const port = await listenOnRandomPort(server); return { - port: requireTcpPort(server), + port, + certificate: tls.cert, tokenRequests: () => tokenRequests, resourceRequests: () => resourceRequests, secretValues: () => [options.clientId, options.clientSecret, ...accessTokens, ...refreshTokens], diff --git a/test/e2e/live/sandbox-survival.test.ts b/test/e2e/live/sandbox-survival.test.ts index b091494edce..f190528e438 100644 --- a/test/e2e/live/sandbox-survival.test.ts +++ b/test/e2e/live/sandbox-survival.test.ts @@ -115,6 +115,7 @@ test( }) => { const hosted = requireHostedInferenceConfig(secrets); const apiKey = hosted.apiKey; + const sandboxEnv = { ...sandboxAccessEnv(), OPENSHELL_GATEWAY: "nemoclaw" }; await artifacts.target.declare({ id: "sandbox-survival", @@ -143,7 +144,7 @@ test( ], { artifactName: "pre-cleanup-openshell-delete-sandbox-survival", - env: buildAvailabilityProbeEnv(), + env: sandboxEnv, timeoutMs: SANDBOX_SURVIVAL_OPENSHELL_DELETE_TIMEOUT_MS, }, ); @@ -189,6 +190,7 @@ test( ); const sandboxCleanupOptions = { artifactName: "cleanup-openshell-delete-sandbox-survival", + env: sandboxEnv, redactionValues: [apiKey], }; let sandboxDeleted = false; @@ -199,7 +201,7 @@ test( host, { artifactName: "cleanup-probe-openshell-sandbox-survival", - env: buildAvailabilityProbeEnv(), + env: sandboxCleanupOptions.env, redactionValues: sandboxCleanupOptions.redactionValues, timeoutMs: 30_000, }, @@ -232,7 +234,7 @@ test( const execShell = (script: string, artifactName: string) => sandbox.exec(SANDBOX_NAME, ["sh", "-lc", script], { artifactName, - env: sandboxAccessEnv(), + env: sandboxEnv, timeoutMs: SANDBOX_SURVIVAL_NATIVE_READINESS.timeoutMs, }); @@ -253,7 +255,7 @@ test( progress.phase("stop the sandbox through OpenShell"); const stop = await sandbox.openshell(["sandbox", "stop", "-g", "nemoclaw", SANDBOX_NAME], { artifactName: "openshell-stop-sandbox-survival", - env: buildAvailabilityProbeEnv(), + env: sandboxEnv, timeoutMs: SANDBOX_SURVIVAL_SANDBOX_LIFECYCLE_TIMEOUT_MS, }); assertExitZero(stop, "OpenShell sandbox stop"); @@ -261,7 +263,7 @@ test( progress.phase("start the sandbox through OpenShell"); const start = await sandbox.openshell(["sandbox", "start", "-g", "nemoclaw", SANDBOX_NAME], { artifactName: "openshell-start-sandbox-survival", - env: buildAvailabilityProbeEnv(), + env: sandboxEnv, timeoutMs: SANDBOX_SURVIVAL_SANDBOX_LIFECYCLE_TIMEOUT_MS, }); assertExitZero(start, "OpenShell sandbox start"); @@ -269,6 +271,7 @@ test( progress.phase("recheck native agent readiness, state, and plugin execution"); await lifecycle.waitForSandboxReadyAfterGatewayRestart(instance, { artifactNamePrefix: "post-openshell-start-ready", + env: sandboxEnv, attempts: SANDBOX_SURVIVAL_LIFECYCLE_READINESS.attempts, delayMs: SANDBOX_SURVIVAL_LIFECYCLE_READINESS.delayMs, timeoutMs: SANDBOX_SURVIVAL_LIFECYCLE_READINESS.timeoutMs, @@ -279,19 +282,12 @@ test( markers, "post-openshell-start-marker-read", ); - await approveOpenClawAdminScope( - host, - sandbox, - SANDBOX_NAME, - { ...buildAvailabilityProbeEnv(), ...hosted.env }, - [apiKey], - false, - ); + await approveOpenClawAdminScope(host, sandbox, SANDBOX_NAME, sandboxEnv, [apiKey], false); const pluginOutcome = await nativePluginAgentInvocationOutcome(sandbox, { sandboxName: SANDBOX_NAME, version: "v1", artifactName: "post-openshell-start-native-plugin-invoke", - env: sandboxAccessEnv(), + env: sandboxEnv, turnTimeoutMs: SANDBOX_SURVIVAL_NATIVE_PLUGIN_TURN_TIMEOUT_MS, audit: SANDBOX_SURVIVAL_NATIVE_PLUGIN_AUDIT, }); @@ -302,13 +298,13 @@ test( progress.phase("destroy the sandbox"); await sandbox.cleanupSandbox(SANDBOX_NAME, { artifactName: "final-openshell-delete-sandbox-survival", - env: buildAvailabilityProbeEnv(), + env: sandboxEnv, timeoutMs: SANDBOX_SURVIVAL_FINAL_DESTROY_TIMEOUT_MS, }); sandboxDeleted = true; const postDestroyList = await sandbox.list({ artifactName: "post-destroy-openshell-sandbox-list", - env: buildAvailabilityProbeEnv(), + env: sandboxEnv, timeoutMs: SANDBOX_SURVIVAL_POST_DESTROY_LIST_TIMEOUT_MS, }); const destroyedAtEnd = !outputContainsSandbox(postDestroyList, SANDBOX_NAME); diff --git a/test/e2e/live/sessions-agents-cli.test.ts b/test/e2e/live/sessions-agents-cli.test.ts index c7b5fc55ba6..9b2e28334af 100644 --- a/test/e2e/live/sessions-agents-cli.test.ts +++ b/test/e2e/live/sessions-agents-cli.test.ts @@ -28,6 +28,7 @@ import { parseJsonFromText } from "./json-envelope.ts"; const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? "e2e-sessions-cli"; const TEST_AGENT_ID = process.env.NEMOCLAW_E2E_AGENT_ID ?? "work"; +const TEST_SESSION_KEY = `agent:${TEST_AGENT_ID}:e2e-delete`; const ONBOARD_TIMEOUT_MS = execTimeout(40 * 60_000); const AGENT_TURN_TIMEOUT_MS = 5 * 60_000; const GATEWAY_RPC_TIMEOUT_MS = 120_000; @@ -203,13 +204,6 @@ function agentEntries(envelope: unknown): JsonRecord[] { : []; } -function firstSessionKey(envelope: unknown): string | undefined { - for (const entry of sessionEntries(envelope)) { - if (typeof entry.key === "string" && entry.key.length > 0) return entry.key; - } - return undefined; -} - async function expectJsonCommand( host: HostCliClient, args: string[], @@ -395,7 +389,19 @@ test( progress.phase("seed and delete the secondary-agent session"); const workSeed = await runNemoclaw( host, - [SANDBOX_NAME, "exec", "--", "openclaw", "agent", "--agent", TEST_AGENT_ID, "-m", "ping"], + [ + SANDBOX_NAME, + "exec", + "--", + "openclaw", + "agent", + "--agent", + TEST_AGENT_ID, + "--session-key", + TEST_SESSION_KEY, + "-m", + "ping", + ], hosted, { artifactName: "seed-work-agent-session", @@ -410,15 +416,16 @@ test( hosted, "tc-sess-05-work-agent-sessions-json", ); - const sessionKey = firstSessionKey(workSessions); - expect( - sessionKey, - `expected a session key for agent '${TEST_AGENT_ID}' after seed prompt`, - ).toBeTruthy(); + const sessionKey = sessionEntries(workSessions).find( + (entry) => entry.key === TEST_SESSION_KEY, + )?.key; + expect(sessionKey, `expected the seeded session '${TEST_SESSION_KEY}' after seed prompt`).toBe( + TEST_SESSION_KEY, + ); const deleteSession = await runNemoclaw( host, - [SANDBOX_NAME, "sessions", "delete", sessionKey!, "--json"], + [SANDBOX_NAME, "sessions", "delete", TEST_SESSION_KEY, "--json"], hosted, { artifactName: "tc-sess-05-sessions-delete-json", timeoutMs: GATEWAY_RPC_TIMEOUT_MS }, ); diff --git a/test/e2e/live/token-rotation.test.ts b/test/e2e/live/token-rotation.test.ts index 7aa9b788ad7..ff6aadeee7e 100644 --- a/test/e2e/live/token-rotation.test.ts +++ b/test/e2e/live/token-rotation.test.ts @@ -17,6 +17,7 @@ import { expect, test } from "../fixtures/e2e-test.ts"; import { startFakeOpenAiCompatibleServer } from "../fixtures/fake-openai-compatible.ts"; import { captureOpenClawOnboardFailure } from "../fixtures/openclaw-onboard-diagnostics.ts"; import { CLI_ENTRYPOINT, REPO_ROOT } from "../fixtures/paths.ts"; +import type { LifecyclePhaseFixture } from "../fixtures/phases/lifecycle.ts"; // Keep this free-standing and direct: the contract is the real CLI + // OpenShell/provider boundary for messaging credential reuse/rotation, not the @@ -263,21 +264,19 @@ async function deleteSandboxIfOpenshellExists( ); } -async function destroyGatewayIfOpenshellExists( +async function cleanupGateway( host: import("../fixtures/clients/host.ts").HostCliClient, + lifecycle: LifecyclePhaseFixture, artifactName: string, ): Promise { - await host.command( - "bash", - [ - "-lc", - "if command -v openshell >/dev/null 2>&1; then openshell gateway destroy -g nemoclaw; fi", - ], - { - artifactName, - env: buildAvailabilityProbeEnv(), - timeoutMs: 60_000, - }, + const env = buildAvailabilityProbeEnv(); + const options = { env, redactionValues: redactionValues() }; + await lifecycle.stopGatewayRuntime({ env, userServiceMode: "permanent" }); + await cleanupWhenOpenShellAvailable( + host, + { artifactName: `${artifactName}-probe`, ...options, timeoutMs: 30_000 }, + () => + host.cleanupGatewayRegistration("nemoclaw", { artifactName, ...options, timeoutMs: 60_000 }), ); } @@ -295,7 +294,7 @@ test( ], }, }, - async ({ artifacts, cleanup, host, progress, runtimeProvider, sandbox }) => { + async ({ artifacts, cleanup, host, lifecycle, progress, runtimeProvider, sandbox }) => { assertTokenPairsDiffer(); await runtimeProvider.requireAvailable({ @@ -348,17 +347,8 @@ test( }; cleanup.trackGateway( { - cleanupGatewayRegistration: (name: string) => - cleanupWhenOpenShellAvailable( - host, - { - artifactName: "cleanup-probe-openshell-gateway-token-rotation", - env: gatewayCleanupOptions.env, - redactionValues: gatewayCleanupOptions.redactionValues, - timeoutMs: 30_000, - }, - () => host.cleanupGatewayRegistration(name, gatewayCleanupOptions), - ), + cleanupGatewayRegistration: () => + cleanupGateway(host, lifecycle, gatewayCleanupOptions.artifactName), }, "nemoclaw", gatewayCleanupOptions, @@ -415,10 +405,7 @@ test( host, "pre-cleanup-openshell-sandbox-delete-token-rotation", ); - await destroyGatewayIfOpenshellExists( - host, - "pre-cleanup-openshell-gateway-destroy-token-rotation", - ); + await cleanupGateway(host, lifecycle, "pre-cleanup-openshell-gateway-destroy-token-rotation"); progress.phase("install the sandbox and confirm provider hashes"); const first = await runInstall(host, fakeOpenAI.baseUrl, TOKEN_A, { diff --git a/test/e2e/mock-parity.json b/test/e2e/mock-parity.json index 6f9d7c3eaf6..a8dad5646d0 100644 --- a/test/e2e/mock-parity.json +++ b/test/e2e/mock-parity.json @@ -34,9 +34,11 @@ { "live": "test/e2e/live/portable-profile-rootless-linux.test.ts", "liveSources": [ + "test/e2e/live/portable-profile-gateway-proof.ts", "test/e2e/fixtures/portable-profile-rootless-finalization-helpers.ts", "test/e2e/fixtures/portable-registry-readiness.ts", - "test/e2e/fixtures/portable-sandbox-build-diagnostics.ts" + "test/e2e/fixtures/portable-sandbox-build-diagnostics.ts", + "test/e2e/fixtures/portable-hermes-transaction-live.ts" ], "fast": [ "src/lib/actions/sandbox/doctor-flow.test.ts", @@ -49,9 +51,11 @@ "src/lib/onboard/experimental/portable-host-preparation.test.ts", "src/lib/onboard/sandbox-prebuild.test.ts", "test/agents/hermes/hermes-start.test.ts", - "test/e2e/support/portable-registry-readiness.test.ts", "test/e2e/support/portable-profile-rootless-runtime-workflow.test.ts", - "test/e2e/support/portable-sandbox-build-diagnostics.test.ts" + "test/e2e/support/portable-profile-gateway-proof.test.ts", + "test/e2e/support/portable-registry-readiness.test.ts", + "test/e2e/support/portable-sandbox-build-diagnostics.test.ts", + "test/e2e/support/portable-hermes-transaction-live.test.ts" ] }, { @@ -95,10 +99,7 @@ }, { "live": "test/e2e/live/managed-image-protected-runtime.test.ts", - "liveSources": [ - "test/e2e/live/managed-image-protected-runtime-helpers.ts", - "test/e2e/live/managed-image-protected-runtime-budget.ts" - ], + "liveSources": ["test/e2e/live/managed-image-protected-runtime-helpers.ts"], "fast": [ "test/e2e/support/gpu-ollama-cleanup.test.ts", "src/lib/inference/nim.test.ts", @@ -150,7 +151,10 @@ }, { "live": "test/e2e/live/podman-cpu-lifecycle.test.ts", - "liveSources": ["test/e2e/live/podman-cpu-lifecycle-helpers.ts"], + "liveSources": [ + "test/e2e/live/podman-cpu-lifecycle-artifacts.ts", + "test/e2e/live/podman-cpu-lifecycle-helpers.ts" + ], "fast": [ "src/lib/adapters/container-engine.test.ts", "src/lib/adapters/podman/index.test.ts", @@ -160,7 +164,9 @@ "src/lib/onboard/runtime-provider/podman.test.ts", "src/lib/onboard/runtime-provider/podman-preflight.test.ts", "src/lib/onboard/runtime-provider/podman.test.ts", - "test/e2e/support/podman-cpu-proof-workflow.test.ts" + "test/e2e/support/podman-cpu-proof-workflow.test.ts", + "test/e2e/support/podman-cpu-lifecycle-artifacts.test.ts", + "test/e2e/support/podman-cpu-lifecycle-helpers.test.ts" ] }, { @@ -205,12 +211,20 @@ { "live": "test/e2e/live/inference-routing.test.ts", "liveSources": [ + "test/e2e/fixtures/curl-policy-denial.ts", + "test/e2e/fixtures/registered-inference.ts", + "test/e2e/live/cloudflared-prerequisite.ts", + "test/e2e/live/https-pin-compatible-server.ts", + "test/e2e/live/runtime-identity-oauth-server.ts", "test/e2e/live/inference-routing-helpers.ts", "test/e2e/live/openclaw-admin-scope.ts", "test/e2e/fixtures/admin-approval-connect.ts", - "test/e2e/fixtures/admin-approval-connect.sh" + "test/e2e/fixtures/admin-approval-connect.sh", + "test/e2e/live/dns-rebinding-hosts-fixture.ts", + "test/e2e/fixtures/dns-rebinding-server.ts" ], "fast": [ + "test/e2e/support/mcp-bridge-sandbox.test.ts", "src/lib/inference/native-local.test.ts", "src/lib/inference/local-vllm-auth.test.ts", "src/lib/inference/serving/vllm-host-local-probe.test.ts", @@ -222,12 +236,16 @@ "nemoclaw/src/blueprint/runtime-identity.test.ts", "src/lib/onboard/inference-providers/compatible-endpoint-gateway-route.test.ts", "test/onboarding/blueprint-runtime-identity-lifecycle.test.ts", + "test/e2e/support/cloudflared-prerequisite.test.ts", + "test/e2e/support/e2e-http-protocol.test.ts", "test/e2e/support/hosted-inference.test.ts", "test/e2e/support/onboard-timeout-contract.test.ts", "test/onboarding/onboard-inference-gateway-scope.test.ts", "test/e2e/support/managed-image-activation-diagnostics.test.ts", "test/e2e/support/hosted-provider-smoke-selection.test.ts", - "test/security/admin-approval-helper.test.ts" + "test/security/admin-approval-helper.test.ts", + "test/e2e/support/registered-inference.test.ts", + "test/e2e/support/dns-rebinding-server.test.ts" ] }, { @@ -299,6 +317,7 @@ { "live": "test/e2e/live/bootstrap-install-smoke.test.ts", "fast": [ + "test/e2e/support/full-e2e-inference-probe.test.ts", "test/automation/e2e/brev-launchable-ci-cpu-checksum.test.ts", "test/e2e/support/e2e-live-target-gating.test.ts", "test/e2e/support/e2e-progress-fixture.test.ts", @@ -314,6 +333,7 @@ { "live": "test/e2e/live/launchable-smoke.test.ts", "fast": [ + "test/e2e/support/full-e2e-inference-probe.test.ts", "test/automation/e2e/brev-launchable-ci-cpu-checksum.test.ts", "test/e2e/support/e2e-live-target-gating.test.ts", "test/e2e/support/e2e-progress-fixture.test.ts", @@ -363,12 +383,16 @@ "fast": [ "test/e2e/support/openshell-gateway-auth-contract-workflow-boundary.test.ts", "test/e2e/support/openshell-gateway-auth-source-contract-helpers.test.ts", + "test/e2e/support/openshell-gateway-auth-sandbox.test.ts", "test/e2e/support/openshell-v0116-tls-server-name-source.test.ts" ] }, { "live": "test/e2e/live/external-gateway-health.test.ts", - "liveSources": ["test/e2e/live/external-gateway-health-helpers.ts"], + "liveSources": [ + "test/e2e/live/external-gateway-health-helpers.ts", + "test/e2e/fixtures/external-gateway-health-config.ts" + ], "fast": [ "test/e2e/support/external-gateway-health-helpers.test.ts", "test/e2e/support/external-gateway-health-process.test.ts", @@ -508,6 +532,7 @@ "live": "test/e2e/live/agent-turn-latency.test.ts", "liveSources": ["test/e2e/live/agent-turn-latency-helpers.ts"], "fast": [ + "test/e2e/support/agent-turn-latency-route.test.ts", "test/e2e/support/agent-turn-latency-progress.test.ts", "test/e2e/support/e2e-cleanup-resources.test.ts", "test/e2e/support/e2e-clients.test.ts", @@ -632,7 +657,10 @@ }, { "live": "test/e2e/live/double-onboard.test.ts", + "liveSources": ["test/e2e/fixtures/curl-policy-denial.ts"], "fast": [ + "test/e2e/support/mcp-bridge-sandbox.test.ts", + "test/e2e/support/registered-inference.test.ts", "test/cli/sandbox-status-text.test.ts", "src/lib/actions/uninstall/run-plan-agent-state-volume.test.ts", "src/lib/onboard/created-sandbox-finalization.test.ts", @@ -721,13 +749,15 @@ "liveSources": [ "test/e2e/live/hermes-cli-adapter-live.ts", "test/e2e/live/hermes-e2e-phases.ts", - "test/e2e/live/hermes-skill-lifecycle.ts" + "test/e2e/live/hermes-skill-lifecycle.ts", + "test/e2e/live/package-database-read-only.ts" ], "fast": [ "src/lib/actions/sandbox/skill-install.test.ts", "test/agents/hermes/hermes-image-build-probes.test.ts", "test/agents/hermes/hermes-start.test.ts", "test/e2e/support/hermes-cli-adapter-live.test.ts", + "test/e2e/support/package-database-read-only.test.ts", "test/e2e/support/hermes-acp-live.test.ts", "test/e2e/support/managed-image-cohort-contract.test.ts", "test/e2e/support/managed-image-receipt.test.ts", @@ -797,6 +827,7 @@ { "live": "test/e2e/live/mcp-bridge.test.ts", "liveSources": [ + "test/e2e/fixtures/curl-policy-denial.ts", "test/e2e/fixtures/owned-sandbox-cleanup.ts", "test/e2e/fixtures/routed-private-relay.ts", "test/e2e/live/dns-rebinding-hosts-fixture.ts", @@ -815,13 +846,16 @@ "test/e2e/live/mcp-provider-rewrite-probe.ts", "test/e2e/live/openshell-allowed-ips-rebinding.ts", "test/e2e/live/openshell-driver-config-test-wrapper.ts", + "test/e2e/live/openshell-exact-main-driver-config.ts", "test/e2e/live/openshell-exact-main-exec.ts", + "test/e2e/live/openshell-exact-main-mcp-proof.ts", "test/e2e/live/openshell-exact-main-runtime-contracts.ts", "test/e2e/live/openclaw-stopped-recovery.ts", "test/e2e/live/openclaw-admin-scope.ts", "test/e2e/fixtures/admin-approval-connect.ts", "test/e2e/fixtures/admin-approval-connect.sh", - "test/e2e/fixtures/sandbox-failure-diagnostics.ts" + "test/e2e/fixtures/sandbox-failure-diagnostics.ts", + "test/e2e/fixtures/dns-rebinding-server.ts" ], "fast": [ "src/lib/actions/sandbox/mcp-bridge-state.test.ts", @@ -852,7 +886,9 @@ "src/lib/onboard/runtime-provider/docker-stopped-state-capture.test.ts", "test/state/snapshot.test.ts", "test/e2e/support/managed-image-activation-diagnostics.test.ts", - "test/security/admin-approval-helper.test.ts" + "test/e2e/support/openclaw-stopped-recovery.test.ts", + "test/security/admin-approval-helper.test.ts", + "test/e2e/support/dns-rebinding-server.test.ts" ] }, { @@ -862,6 +898,7 @@ "test/e2e/support/e2e-cleanup-resources.test.ts", "test/e2e/support/e2e-clients.test.ts", "test/e2e/support/messaging-compatible-endpoint-helpers.test.ts", + "test/e2e/support/messaging-compatible-native.test.ts", "test/e2e/support/openclaw-agent-output.test.ts" ] }, @@ -899,7 +936,8 @@ "fast": [ "test/e2e/support/e2e-cleanup-resources.test.ts", "test/e2e/support/e2e-clients.test.ts", - "test/e2e/support/model-router-provider-routed-inference.test.ts" + "test/e2e/support/model-router-provider-routed-inference.test.ts", + "test/e2e/support/e2e-phase-runtime.test.ts" ] }, { @@ -1046,7 +1084,8 @@ "test/e2e/support/e2e-clients.test.ts", "test/e2e/support/native-lifecycle-migration-contract.test.ts", "src/lib/onboard/runtime-provider/docker-stopped-state-capture.test.ts", - "test/e2e/support/managed-image-activation-diagnostics.test.ts" + "test/e2e/support/managed-image-activation-diagnostics.test.ts", + "test/e2e/support/openclaw-stopped-recovery.test.ts" ], "liveSources": [ "test/e2e/live/openclaw-stopped-recovery.ts", diff --git a/test/e2e/support/agent-turn-latency-progress.test.ts b/test/e2e/support/agent-turn-latency-progress.test.ts index 6e2f3aba151..3a6a7ff3d92 100644 --- a/test/e2e/support/agent-turn-latency-progress.test.ts +++ b/test/e2e/support/agent-turn-latency-progress.test.ts @@ -20,7 +20,6 @@ import { cleanupTurnSandboxes, hermesTurnCommand, installSandbox, - route, turnLatencyInstallAttemptCount, } from "../live/agent-turn-latency-helpers.ts"; @@ -130,32 +129,6 @@ it.each([ } }); -it.each(["openclaw", "hermes"] as const)( - "reads the %s native route from sandbox status without the host credential", - async (agent) => { - const calls: Array<{ args: string[]; options: Record }> = []; - const host = { - nemoclaw: async (args: string[], options: Record) => { - calls.push({ args, options }); - return successfulProbe(); - }, - } as unknown as HostCliClient; - - await route(host, "e2e-turn-lat", agent, fakeInference(), `${agent}-route`); - - expect(calls).toHaveLength(1); - expect(calls[0].args).toEqual(["e2e-turn-lat", "status"]); - expect(calls[0].options).toMatchObject({ - artifactName: `${agent}-route`, - env: { NEMOCLAW_SANDBOX_NAME: "e2e-turn-lat" }, - }); - expect((calls[0].options.env as NodeJS.ProcessEnv).NEMOCLAW_AGENT).toBe( - agent === "hermes" ? "hermes" : undefined, - ); - expect(JSON.stringify(calls[0].options)).not.toContain("secret-api-key"); - }, -); - describe("live test progress", () => { afterEach(() => { vi.useRealTimers(); diff --git a/test/e2e/support/agent-turn-latency-route.test.ts b/test/e2e/support/agent-turn-latency-route.test.ts new file mode 100644 index 00000000000..31c762c60a7 --- /dev/null +++ b/test/e2e/support/agent-turn-latency-route.test.ts @@ -0,0 +1,238 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import { runInNewContext } from "node:vm"; +import { expect, it, vi } from "vitest"; +import { customNativeProvider } from "../../../src/lib/inference/native-provider/custom.ts"; +import { + NVIDIA_HOSTED_NATIVE_ENDPOINT, + NVIDIA_HOSTED_NATIVE_PROFILE_ID, + NVIDIA_HOSTED_NATIVE_PROVIDER, +} from "../../../src/lib/inference/native-nvidia/contract.ts"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; +import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; +import { route, type AgentTurnInference } from "../live/agent-turn-latency-helpers.ts"; + +vi.mock("../../../src/lib/state/registry.ts", () => ({ getSandbox: vi.fn() })); + +const custom = customNativeProvider({ + provider: "compatible-endpoint", + endpointUrl: "https://inference-api.nvidia.com/v1", + api: "openai-completions", + credentialEnv: "SOURCE_KEY", + allowedIps: ["8.8.8.8"], +}); +const providers = [ + { + name: "compatible-endpoint", + endpoint: custom.endpoint, + key: custom.credentialEnv, + receipt: { + schemaVersion: 1 as const, + profileId: custom.profileId, + providerName: custom.providerName, + providerId: "owned-provider", + endpointUrl: custom.endpointUrl, + allowedIps: custom.allowedIps, + custom: custom.custom, + }, + }, + { + name: "nvidia-prod", + endpoint: NVIDIA_HOSTED_NATIVE_ENDPOINT, + key: "NVIDIA_INFERENCE_API_KEY", + receipt: { + schemaVersion: 1 as const, + profileId: NVIDIA_HOSTED_NATIVE_PROFILE_ID, + providerName: NVIDIA_HOSTED_NATIVE_PROVIDER, + providerId: "owned-provider", + }, + }, +]; + +const cases = ["openclaw", "hermes"].flatMap((agent) => + providers.map((provider) => ({ agent: agent as "openclaw" | "hermes", provider })), +); + +it.each(cases)("observes the live $agent route for $provider.name", async ({ agent, provider }) => { + const fixture = harness(agent, provider); + await expect(fixture.read()).resolves.toEqual({ + provider: provider.name, + model: "selected-model", + }); + expect(fixture.openshell.mock.calls.map(([args]) => args)).toEqual([ + ["provider", "get", "-g", "recorded-gateway", provider.receipt.providerName], + ["sandbox", "provider", "list", "-g", "recorded-gateway", "latency-sandbox"], + ]); + const options = [...fixture.openshell.mock.calls, ...fixture.exec.mock.calls].map((call) => + call.at(-1), + ); + expect(options).toEqual( + Array(3).fill( + expect.objectContaining({ + timeoutMs: 30_000, + redactionValues: ["host-secret"], + env: expect.objectContaining({ OPENSHELL_GATEWAY: "recorded-gateway" }), + }), + ), + ); + expect(JSON.stringify(options)).not.toContain('"COMPATIBLE_API_KEY":"host-secret"'); + expect(fixture.inference.env).not.toHaveBeenCalled(); + expect(fixture.projection()).not.toContain("config-secret"); +}); + +it.each( + cases.flatMap((item) => + [ + "provider-id", + "profile", + "credential", + "config-key", + "attachment", + "provider-failed", + "attachment-failed", + "config-failed", + "endpoint", + "api", + "agent-provider", + "receipt", + ].map((drift) => ({ ...item, drift })), + ), +)("rejects $drift drift for $agent with $provider.name", async ({ agent, provider, drift }) => { + await expect(harness(agent, provider, drift).read()).resolves.toBeNull(); +}); + +it.each(cases)( + "returns the observed $agent model instead of the requested model", + async ({ agent, provider }) => { + await expect(harness(agent, provider, "model").read()).resolves.toEqual({ + provider: provider.name, + model: "different-model", + }); + }, +); + +function harness(agent: "openclaw" | "hermes", provider: (typeof providers)[number], drift = "") { + vi.mocked(getSandbox).mockReturnValue({ + name: "latency-sandbox", + agent: agent === "openclaw" ? null : agent, + gatewayName: "recorded-gateway", + provider: provider.name, + ...(drift === "receipt" + ? {} + : provider.name === "nvidia-prod" + ? { nativeNvidiaProviderAttachment: provider.receipt } + : { nativeHostedProviderAttachment: provider.receipt }), + } as NonNullable>); + const inference: AgentTurnInference = { + mode: provider.name === "nvidia-prod" ? "public-nvidia" : "internal-nvidia", + provider: provider.name === "nvidia-prod" ? "nvidia" : "custom", + model: "selected-model", + expectedRouteProvider: provider.name, + env: vi.fn(() => ({ COMPATIBLE_API_KEY: "host-secret" })), + redactionValues: () => ["host-secret"], + }; + const openshell = vi + .fn() + .mockResolvedValueOnce({ + exitCode: drift === "provider-failed" ? 1 : 0, + stdout: [ + `Name: ${provider.receipt.providerName}`, + `Id: ${drift === "provider-id" ? "foreign-provider" : "owned-provider"}`, + `Type: ${drift === "profile" ? "foreign-profile" : provider.receipt.profileId}`, + "Resource version: 1", + `Credential keys: ${drift === "credential" ? "WRONG_KEY" : provider.key}`, + `Config keys: ${drift === "config-key" ? "BASE_URL" : ""}`, + ].join("\n"), + }) + .mockResolvedValueOnce({ + exitCode: drift === "attachment-failed" ? 1 : 0, + stdout: drift === "attachment" ? "another-provider" : provider.receipt.providerName, + }); + const model = drift === "model" ? "different-model" : "selected-model"; + const endpoint = drift === "endpoint" ? "https://wrong.example/v1" : provider.endpoint; + const config = + agent === "openclaw" + ? { + agents: { + defaults: { + model: { + primary: `${drift === "agent-provider" ? "foreign" : "inference"}/${model}`, + }, + }, + }, + models: { + providers: { + inference: { + baseUrl: endpoint, + api: drift === "api" ? "openai-responses" : "openai-completions", + apiKey: "config-secret", + }, + }, + }, + } + : { + model: { + default: model, + provider: drift === "agent-provider" ? "foreign" : "custom", + base_url: endpoint, + api_key: "config-secret", + ...(drift === "api" ? { api_mode: "codex_responses" } : {}), + }, + }; + let projection = ""; + const exec = vi.fn(async (_name, command) => { + const project = { + openclaw: () => { + runInNewContext(command[2], { + require: () => ({ readFileSync: () => JSON.stringify(config) }), + process: { + argv: ["node", command[3]], + stdout: { + write: (value: string) => { + projection = value; + }, + }, + }, + }); + }, + hermes: () => { + expect(command[0]).toBe("/opt/hermes/.venv/bin/python3"); + const result = spawnSync( + "python3", + [ + "-c", + [ + "import builtins,io,json,sys,types", + "config=json.loads(sys.stdin.read())", + "builtins.open=lambda filename: io.StringIO(json.dumps(config))", + "sys.modules['yaml']=types.SimpleNamespace(safe_load=json.load)", + "exec(sys.argv[1])", + ].join("\n"), + command[2], + ], + { input: JSON.stringify(config), encoding: "utf8" }, + ); + expect(result.status, result.stderr).toBe(0); + projection = result.stdout; + }, + }; + project[agent](); + return { exitCode: drift === "config-failed" ? 1 : 0, stdout: projection }; + }); + return { + inference, + openshell, + exec, + projection: () => projection, + read: () => + route( + { openshell, exec } as unknown as SandboxClient, + "latency-sandbox", + agent, + inference, + "latency-route", + ), + }; +} diff --git a/test/e2e/support/base-image-publication-inputs.test.ts b/test/e2e/support/base-image-publication-inputs.test.ts new file mode 100644 index 00000000000..dbf216b54d9 --- /dev/null +++ b/test/e2e/support/base-image-publication-inputs.test.ts @@ -0,0 +1,44 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { + baseImageInputsChanged, + parseBaseImagePushPaths, +} from "../../../tools/e2e/base-image-publication.mts"; + +const BASE_IMAGE_WORKFLOW_SOURCE = fs.readFileSync( + path.resolve(import.meta.dirname, "../../../.github/workflows/base-image.yaml"), + "utf8", +); + +describe("base-image publication inputs", () => { + it("publishes after a root package manifest changes", () => { + const reviewedPaths = parseBaseImagePushPaths(BASE_IMAGE_WORKFLOW_SOURCE); + + expect(reviewedPaths).toEqual(expect.arrayContaining(["package.json", "package-lock.json"])); + expect(baseImageInputsChanged(["package.json"], reviewedPaths)).toBe(true); + expect(baseImageInputsChanged(["package-lock.json"], reviewedPaths)).toBe(true); + }); + + it.each([ + [ + "src/lib/inference/native-provider/catalog.ts", + "src/lib/inference/native-provider/catalog.ts", + ], + ["src/lib/adapters/container-engine.ts", "src/lib/adapters/container-engine.ts"], + ["src/lib/adapters/podman/**", "src/lib/adapters/podman/index.ts"], + [ + "src/lib/agent/deep-agents-code-runtime-identity.json", + "src/lib/agent/deep-agents-code-runtime-identity.json", + ], + ["test/e2e/fixtures/docker-build-guard.ts", "test/e2e/fixtures/docker-build-guard.ts"], + ])("publishes after managed-image input %s changes", (publisherPath, changedPath) => { + const reviewedPaths = parseBaseImagePushPaths(BASE_IMAGE_WORKFLOW_SOURCE); + + expect(reviewedPaths).toContain(publisherPath); + expect(baseImageInputsChanged([changedPath], reviewedPaths)).toBe(true); + }); +}); diff --git a/test/e2e/support/base-image-publication.test.ts b/test/e2e/support/base-image-publication.test.ts index be8c9383e14..454a1bed2e7 100644 --- a/test/e2e/support/base-image-publication.test.ts +++ b/test/e2e/support/base-image-publication.test.ts @@ -9,7 +9,6 @@ import path from "node:path"; import { describe, expect, it, vi } from "vitest"; import { - baseImageInputsChanged, collectPaginated, expandBaseImagePushPaths, type FirstParentHistory, @@ -37,10 +36,6 @@ const RUN_URL_ROOT = "https://github.com/NVIDIA/NemoClaw/actions/runs"; const RUN_URL = `https://github.com/NVIDIA/NemoClaw/actions/runs/${RUN_ID}`; const MANAGED_IMAGE_PROMOTION_JOB = "Publish complete managed images / Promote complete multi-platform managed image cohort"; -const BASE_IMAGE_WORKFLOW_SOURCE = fs.readFileSync( - path.resolve(import.meta.dirname, "../../../.github/workflows/base-image.yaml"), - "utf8", -); const WORKFLOW_SOURCE = `on: push: branches: [main] @@ -176,29 +171,6 @@ function successfulManualJobs(): Record[] { } describe("base-image publication evidence", () => { - it("publishes after a root package manifest changes", () => { - const reviewedPaths = parseBaseImagePushPaths(BASE_IMAGE_WORKFLOW_SOURCE); - - expect(reviewedPaths).toEqual(expect.arrayContaining(["package.json", "package-lock.json"])); - expect(baseImageInputsChanged(["package.json"], reviewedPaths)).toBe(true); - expect(baseImageInputsChanged(["package-lock.json"], reviewedPaths)).toBe(true); - }); - - it.each([ - ["src/lib/adapters/container-engine.ts", "src/lib/adapters/container-engine.ts"], - ["src/lib/adapters/podman/**", "src/lib/adapters/podman/index.ts"], - [ - "src/lib/agent/deep-agents-code-runtime-identity.json", - "src/lib/agent/deep-agents-code-runtime-identity.json", - ], - ["test/e2e/fixtures/docker-build-guard.ts", "test/e2e/fixtures/docker-build-guard.ts"], - ])("publishes after managed-image input %s changes", (publisherPath, changedPath) => { - const reviewedPaths = parseBaseImagePushPaths(BASE_IMAGE_WORKFLOW_SOURCE); - - expect(reviewedPaths).toContain(publisherPath); - expect(baseImageInputsChanged([changedPath], reviewedPaths)).toBe(true); - }); - it.each(["push", "workflow_dispatch"])("accepts %s publication preflight events", (eventName) => { expect(isBaseImagePublicationEvent(eventName)).toBe(true); }); diff --git a/test/e2e/support/brave-search-isolation.test.ts b/test/e2e/support/brave-search-isolation.test.ts index ef4d037b8a2..76da15932a8 100644 --- a/test/e2e/support/brave-search-isolation.test.ts +++ b/test/e2e/support/brave-search-isolation.test.ts @@ -76,6 +76,39 @@ describe("Brave runtime credential boundary probes", () => { expect(result.status).toBe(97); }); + it.each([false, true])( + "reaps its child with group signals denied and raw credential %s", + (raw) => { + const harness = String.raw`import json, os, pathlib, subprocess, sys +from unittest.mock import patch +probe = sys.argv.pop(1) +raw = sys.argv.pop(1) == "true" +sys.argv = [sys.argv[0], sys.executable, "-c", "import time; time.sleep(30)"] +scope = {} +def read(path): + return b"node\0/app/openclaw.mjs\0agent\0" if path.name == "cmdline" else ( + b"BRAVE_API_KEY=synthetic-raw-key" if raw else b"") +try: + with patch.object(os, "killpg", side_effect=PermissionError()), patch.object(pathlib.Path, "read_bytes", read): + try: + exec(probe, scope) + except SystemExit as result: + print(json.dumps({"status": result.code, "reaped": scope["child"].returncode is not None})) +finally: + child = scope.get("child") + if child is not None and child.poll() is None: + child.kill() + child.wait(timeout=5) +`; + const result = spawnSync("python3", ["-c", harness, BRAVE_AGENT_BOUNDARY, String(raw)], { + encoding: "utf8", + timeout: 15_000, + }); + expect(result.status, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toEqual({ status: raw ? 98 : 0, reaped: true }); + }, + ); + it.each(["FileNotFoundError", "ProcessLookupError", "PermissionError"])( "fails closed without output when the agent environment raises %s", (errorType) => { @@ -104,12 +137,6 @@ with patch.object(pathlib.Path, "read_bytes", read): it.each([ ["node\0/app/openclaw/openclaw.mjs\0agent\0", "", 0], - [ - "node\0/app/openclaw/openclaw.mjs\0agent\0", - "BRAVE_API_KEY=openshell:resolve:env:v1_BRAVE_API_KEY", - 0, - ], - ["node\0/app/openclaw/openclaw.mjs\0agent\0", "BRAVE_API_KEY=synthetic-raw-key", 98], ["sh\0-c\0openclaw agent\0", "", 97], ["openclaw-gateway\0", "", 97], ])("inspects only the selected agent child: %s / %s", (command, environment, status) => { diff --git a/test/e2e/support/candidate-openshell-trust.test.ts b/test/e2e/support/candidate-openshell-trust.test.ts index 9b725b58257..f51d57c192a 100644 --- a/test/e2e/support/candidate-openshell-trust.test.ts +++ b/test/e2e/support/candidate-openshell-trust.test.ts @@ -2,12 +2,13 @@ // SPDX-License-Identifier: Apache-2.0 import { spawnSync } from "node:child_process"; -import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { mkdtempSync, mkdirSync, readFileSync, writeFileSync, rmSync } from "node:fs"; import { dirname, join } from "node:path"; import { tmpdir } from "node:os"; import { afterEach, describe, expect, it, vi } from "vitest"; import { installerReleaseTsv, + readOpenShellTrustRegistry, resolveOpenShellTrustRegistry, validateInstallerSources, } from "../../../scripts/checks/extract-installer-pins.mts"; @@ -35,6 +36,32 @@ afterEach(() => ); describe("reviewed candidate OpenShell trust", () => { + it.each([ + "0b9c65bcc255d97d2fbfef638fbb4b374d7617fb2aaaf6cb76b898ded74e498f", + "65da7bf3c9a930e11f4028b808d34cdbebc63d1a30672e98e1f2650f9d9a269d", + ])("retains previous installer identity %s while revoking its execution", (previousTemplate) => { + const registry = readOpenShellTrustRegistry(); + const previous = structuredClone(registry); + previous.revokedTemplateSha256 = []; + const previousRelease = previous.releases.find((release) => release.version === "0.1.3")!; + previousRelease.installerTemplateSha256 = previousRelease.installerTemplateSha256.slice( + 0, + previousRelease.installerTemplateSha256.indexOf(previousTemplate) + 1, + ); + const releases = resolveOpenShellTrustRegistry(registry, previous); + expect( + releases.find((release) => release.version === "0.1.3")?.revokedTemplateSha256, + ).toContain(previousTemplate); + const root = new URL("../../../", import.meta.url).pathname; + const sources = Object.fromEntries( + Object.entries(CANDIDATE_OPENSHELL_SOURCES).map(([key, path]) => [ + key, + readFileSync(join(root, path), "utf8"), + ]), + ) as Record; + expect(validateInstallerSources(sources, releases).releaseVersion).toBe("0.1.7"); + }); + it.each( (["installer", "version"] as const).flatMap((consumer) => (["older", "empty", "absent"] as const).map((scenario) => ({ consumer, scenario })), diff --git a/test/e2e/support/candidate-openshell-version.test.ts b/test/e2e/support/candidate-openshell-version.test.ts index 3658d7157f7..37308917120 100644 --- a/test/e2e/support/candidate-openshell-version.test.ts +++ b/test/e2e/support/candidate-openshell-version.test.ts @@ -223,6 +223,30 @@ describe("reviewed candidate MCP runtime evidence", () => { }); }); + it("binds OpenShell 0.1.7 to the verified upstream release artifacts", () => { + expect(reviewedOpenShellMcpRuntime("0.1.7")).toEqual({ + supervisorImage: + "ghcr.io/nvidia/openshell/supervisor@sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06", + provenance: { + schemaVersion: 1, + sourceRepository: "NVIDIA/OpenShell", + releaseTag: "v0.1.7", + sourceSha: "5d25211b25a8fc887e8d5c606d57d6ab22f1d76c", + artifacts: { + cli: { + binarySha256: "718f40674738206a84bc3a967966225ff47dfed9ba5738099d56777faa4af793", + }, + gateway: { + binarySha256: "0b7cb54fd8558bfad4c57f556f7fc1a20ef8cd94ebb11f4f40a9a90b74b4ad17", + }, + standaloneSandbox: { + binarySha256: "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21", + }, + }, + }, + }); + }); + it.each( [ ["0.0.116", "mcp-bridge", "mcp-bridge-deepagents"], @@ -237,6 +261,8 @@ describe("reviewed candidate MCP runtime evidence", () => { ["0.1.3", "openshell-credential-generation-window", "openshell-credential-generation-window"], ["0.1.5", "mcp-bridge", "mcp-bridge-deepagents"], ["0.1.5", "openshell-credential-generation-window", "openshell-credential-generation-window"], + ["0.1.7", "mcp-bridge", "mcp-bridge-deepagents"], + ["0.1.7", "openshell-credential-generation-window", "openshell-credential-generation-window"], ].flatMap(([version, job, artifact]) => [ [version, job, artifact, "local"], [version, job, artifact, "path"], @@ -287,7 +313,7 @@ describe("reviewed candidate MCP runtime evidence", () => { (job, artifact, role, binary) => { const directory = mkdtempSync(join(tmpdir(), "nemoclaw-mcp-provenance-")); try { - const fixture = prepareRuntimeFixture(directory, "0.1.5", job); + const fixture = prepareRuntimeFixture(directory, "0.1.7", job); writeFileSync(join(fixture.bin, binary), "altered binary"); const result = fixture.run(); expect(result.status).not.toBe(0); diff --git a/test/e2e/support/cloudflared-prerequisite.test.ts b/test/e2e/support/cloudflared-prerequisite.test.ts index e3c24173c95..3d1982cebe7 100644 --- a/test/e2e/support/cloudflared-prerequisite.test.ts +++ b/test/e2e/support/cloudflared-prerequisite.test.ts @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import { createHash } from "node:crypto"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; @@ -8,6 +9,7 @@ import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; import { CleanupRegistry } from "../fixtures/cleanup.ts"; +import { cloudflaredCommandFixture } from "../fixtures/cloudflared-command.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import { readReviewedCloudflaredPin, @@ -15,7 +17,64 @@ import { } from "../live/cloudflared-prerequisite.ts"; describe("reviewed cloudflared prerequisite (#6141)", () => { - afterEach(() => vi.unstubAllEnvs()); + afterEach(() => { + vi.unstubAllEnvs(); + vi.restoreAllMocks(); + }); + + it.each([ + { failure: "digest", error: "package SHA256 mismatch", calls: 1 }, + { failure: "extract", error: "dpkg-deb failed", calls: 2 }, + { failure: "execute", error: "failed while preparing cloudflared", calls: 3 }, + { failure: "version", error: "unexpected cloudflared version", calls: 3 }, + { failure: "none", error: null, calls: 3 }, + ])("verifies package bytes and executable identity with $failure failure", async (scenario) => { + const bytes = Buffer.from("synthetic reviewed package"); + const digest = createHash("sha256").update(bytes).digest("hex"); + vi.spyOn(fs, "readFileSync").mockReturnValueOnce( + JSON.stringify({ + jobs: { + run: { + steps: [ + { + name: "Install reviewed cloudflared", + env: { CLOUDFLARED_VERSION: "2026.6.1", CLOUDFLARED_DEB_SHA256: digest }, + }, + ], + }, + }, + }), + ); + const fixture = cloudflaredCommandFixture(scenario.failure, bytes); + const command = vi.fn(fixture.command); + const cleanup = new CleanupRegistry(); + try { + const result = resolveVerifiedCloudflaredBinary( + cleanup, + { command } as unknown as HostCliClient, + { platform: "linux", arch: "x64" }, + ); + await (scenario.error + ? expect(result).rejects.toThrow(scenario.error) + : result.then((binary) => { + expect(binary).toBe( + path.join(path.dirname(fixture.packagePath()), "extracted/usr/bin/cloudflared"), + ); + expect(command).toHaveBeenNthCalledWith( + 3, + binary, + ["--version"], + expect.objectContaining({ + artifactName: "inference-routing-cloudflared-verify-version", + }), + ); + })); + expect(command).toHaveBeenCalledTimes(scenario.calls); + } finally { + await cleanup.runAll(); + } + expect(fs.existsSync(path.dirname(fixture.packagePath()))).toBe(false); + }); it("reads the reviewed version and digest from the exact workflow", () => { expect(readReviewedCloudflaredPin()).toEqual({ diff --git a/test/e2e/support/common-egress-agent-helpers.test.ts b/test/e2e/support/common-egress-agent-helpers.test.ts index 0ffcdbcafa5..0c6a4395115 100644 --- a/test/e2e/support/common-egress-agent-helpers.test.ts +++ b/test/e2e/support/common-egress-agent-helpers.test.ts @@ -8,6 +8,7 @@ import { join } from "node:path"; import { DatabaseSync } from "node:sqlite"; import { describe, expect, it, vi } from "vitest"; +import { nestedFetchEvidence } from "../../helpers/openclaw-nested-fetch-evidence.ts"; import { agentReplyContainsToken, @@ -494,6 +495,35 @@ describe("common-egress agent parsing and classification helpers", () => { }); }); + it.each([ + "valid", + "wrong run", + "wrong parent", + "wrong generated ID", + "wrong wrapper target", + "tool failure", + "missing error flag", + "missing result", + "omitted result", + "oversized request", + "wrong URL", + "wrong role", + "context-visible record", + "duplicate record", + "hidden forbidden tool", + "parent outside snapshot", + ])("validates nested web_fetch evidence: %s", (variant) => { + const fixture = nestedFetchEvidence(variant); + const evidence = reduceOpenClawToolEvidence( + fixture.session, + fixture.trajectory, + fixture.expectation, + ); + expect(assessPersonalPublicFetchToolEvidence(evidence).matches).toBe(variant === "valid"); + expect(JSON.stringify(evidence)).not.toContain("must-not-remain"); + expect(JSON.stringify(evidence)).not.toContain("United States"); + }); + it("accepts a progressive Tool Search wrapper only with projected web_fetch proof", () => { const evidence = reduceOpenClawToolEvidence( progressivePublicFetchSessionJsonLines(), @@ -1375,91 +1405,4 @@ describe("failed public-fetch evidence", () => { ); }, ); - - it("records HTTP status from tool results without trusting the assistant's explanation", () => { - const session = publicFetchSessionJsonLines({ - isError: true, - payload: publicFetchPayload({ status: 503, text: REDUCER_SECRET_SENTINEL }), - }); - const evidence = reduceOpenClawToolEvidence(session, "", PUBLIC_FETCH_EXPECTATION); - const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); - expect(artifact.webFetchDiagnostics).toEqual([ - { httpStatus: 503, toolError: true, paired: true }, - ]); - expect(artifact.matches).toBe(false); - expect(JSON.stringify(artifact)).not.toContain(REDUCER_SECRET_SENTINEL); - const assistantOnly = reduceOpenClawToolEvidence( - JSON.stringify({ - message: { - role: "assistant", - content: [{ type: "text", text: "Wikimedia returned HTTP 503" }], - }, - }), - "", - PUBLIC_FETCH_EXPECTATION, - ); - expect( - projectPersonalPublicFetchToolEvidenceArtifact(assistantOnly).webFetchDiagnostics, - ).toEqual([]); - }); - - it("reads the pinned producer's error prefix without retaining its response detail", () => { - const evidence = reduceOpenClawToolEvidence( - publicFetchSessionJsonLines({ - payload: { - status: "error", - tool: "web_fetch", - error: `Web fetch failed (503): ${REDUCER_SECRET_SENTINEL}`, - }, - }), - "", - PUBLIC_FETCH_EXPECTATION, - ); - const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); - expect(artifact.webFetchDiagnostics).toEqual([ - { httpStatus: 503, toolError: true, paired: true }, - ]); - expect(artifact.matches).toBe(false); - expect(evidence.webFetchResults[0]?.httpSuccess).toBe(false); - expect(JSON.stringify(evidence)).not.toContain(REDUCER_SECRET_SENTINEL); - }); - - it.each([ - { status: "error", tool: "web_fetch", error: "Agent said Web fetch failed (503): details" }, - { status: "error", tool: "web_fetch", error: "Web fetch failed (999): details" }, - { status: "error", tool: "web_fetch", error: "Web fetch failed (5030): details" }, - { status: "error", tool: "other", error: "Web fetch failed (503): details" }, - { status: "ok", tool: "web_fetch", error: "Web fetch failed (503): details" }, - ])("does not infer HTTP status from unsupported error shapes: %j", (payload) => { - const evidence = reduceOpenClawToolEvidence( - publicFetchSessionJsonLines({ payload }), - "", - PUBLIC_FETCH_EXPECTATION, - ); - expect( - projectPersonalPublicFetchToolEvidenceArtifact(evidence).webFetchDiagnostics[0]?.httpStatus, - ).toBeNull(); - }); - - it("keeps missing tool HTTP status unknown and bounds diagnostic records", () => { - const evidence = reduceOpenClawToolEvidence( - publicFetchSessionJsonLines({ - isError: true, - payload: { status: "error", error: "private details" }, - }), - "", - PUBLIC_FETCH_EXPECTATION, - ); - expect(projectPersonalPublicFetchToolEvidenceArtifact(evidence).webFetchDiagnostics).toEqual([ - { httpStatus: null, toolError: true, paired: true }, - ]); - evidence.webFetchResults = Array.from({ length: 100 }, () => ({ - ...evidence.webFetchResults[0]!, - httpStatus: 999, - })); - const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); - expect(artifact.webFetchDiagnostics).toHaveLength(64); - expect(artifact.webFetchDiagnostics.every((record) => record.httpStatus === null)).toBe(true); - expect(JSON.stringify(artifact)).not.toContain("private details"); - }); }); diff --git a/test/e2e/support/common-egress-web-fetch-diagnostics.test.ts b/test/e2e/support/common-egress-web-fetch-diagnostics.test.ts new file mode 100644 index 00000000000..b41762b6a18 --- /dev/null +++ b/test/e2e/support/common-egress-web-fetch-diagnostics.test.ts @@ -0,0 +1,215 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import { + reduceOpenClawToolEvidence, + projectPersonalPublicFetchToolEvidenceArtifact, +} from "../live/common-egress-agent-helpers.ts"; + +const PUBLIC_FETCH_URL = "https://www.wikidata.org/w/api.php?test=must-not-remain"; +const PUBLIC_FETCH_EXPECTATION = { content: "United States", url: PUBLIC_FETCH_URL }; +const REDUCER_SECRET_SENTINEL = "nvidia-api-key-must-not-remain"; +function publicFetchPayload(overrides: Record = {}) { + return { + url: PUBLIC_FETCH_URL, + finalUrl: PUBLIC_FETCH_URL, + status: 200, + text: "United States", + ...overrides, + }; +} +function publicFetchSessionJsonLines(options: { + payload: Record; + isError?: boolean; +}) { + return [ + { + message: { + role: "assistant", + content: [ + { + type: "toolCall", + id: "fetch-1", + name: "web_fetch", + arguments: { url: PUBLIC_FETCH_URL, maxChars: 8000 }, + }, + ], + }, + }, + { + message: { + role: "toolResult", + toolCallId: "fetch-1", + toolName: "web_fetch", + isError: options.isError ?? false, + details: options.payload, + content: [{ type: "text", text: JSON.stringify(options.payload) }], + }, + }, + ] + .map((message) => JSON.stringify(message)) + .join("\n"); +} + +describe("public-fetch failure diagnostics", () => { + it("records HTTP status from tool results without trusting the assistant's explanation", () => { + const session = publicFetchSessionJsonLines({ + isError: true, + payload: publicFetchPayload({ status: 503, text: REDUCER_SECRET_SENTINEL }), + }); + const evidence = reduceOpenClawToolEvidence(session, "", PUBLIC_FETCH_EXPECTATION); + const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); + expect(artifact.webFetchDiagnostics).toEqual([ + { httpStatus: 503, toolError: true, paired: true, errorCategory: "http" }, + ]); + expect(artifact.matches).toBe(false); + expect(JSON.stringify(artifact)).not.toContain(REDUCER_SECRET_SENTINEL); + const assistantOnly = reduceOpenClawToolEvidence( + JSON.stringify({ + message: { + role: "assistant", + content: [{ type: "text", text: "Wikimedia returned HTTP 503" }], + }, + }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + expect( + projectPersonalPublicFetchToolEvidenceArtifact(assistantOnly).webFetchDiagnostics, + ).toEqual([]); + }); + + it("reads the pinned producer's error prefix without retaining its response detail", () => { + const evidence = reduceOpenClawToolEvidence( + publicFetchSessionJsonLines({ + payload: { + status: "error", + tool: "web_fetch", + error: `Web fetch failed (503): ${REDUCER_SECRET_SENTINEL}`, + }, + }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); + expect(artifact.webFetchDiagnostics).toEqual([ + { httpStatus: 503, toolError: true, paired: true, errorCategory: "http" }, + ]); + expect(artifact.matches).toBe(false); + expect(evidence.webFetchResults[0]?.httpSuccess).toBe(false); + expect(JSON.stringify(evidence)).not.toContain(REDUCER_SECRET_SENTINEL); + }); + + it.each([ + { status: "error", tool: "web_fetch", error: "Agent said Web fetch failed (503): details" }, + { status: "error", tool: "web_fetch", error: "Web fetch failed (999): details" }, + { status: "error", tool: "web_fetch", error: "Web fetch failed (5030): details" }, + { status: "error", tool: "other", error: "Web fetch failed (503): details" }, + { status: "ok", tool: "web_fetch", error: "Web fetch failed (503): details" }, + ])("does not infer HTTP status from unsupported error shapes: %j", (payload) => { + const evidence = reduceOpenClawToolEvidence( + publicFetchSessionJsonLines({ payload }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + expect( + projectPersonalPublicFetchToolEvidenceArtifact(evidence).webFetchDiagnostics[0]?.httpStatus, + ).toBeNull(); + }); + + it("keeps missing tool HTTP status unknown and bounds diagnostic records", () => { + const evidence = reduceOpenClawToolEvidence( + publicFetchSessionJsonLines({ + isError: true, + payload: { status: "error", error: "private details" }, + }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + expect(projectPersonalPublicFetchToolEvidenceArtifact(evidence).webFetchDiagnostics).toEqual([ + { httpStatus: null, toolError: true, paired: true, errorCategory: "unknown" }, + ]); + evidence.webFetchResults = Array.from({ length: 100 }, () => ({ + ...evidence.webFetchResults[0]!, + httpStatus: 999, + })); + const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); + expect(artifact.webFetchDiagnostics).toHaveLength(64); + expect(artifact.webFetchDiagnostics.every((record) => record.httpStatus === null)).toBe(true); + expect(JSON.stringify(artifact)).not.toContain("private details"); + }); + it.each([ + ["Invalid URL: must be http or https", "invalid-url"], + ["maxChars must be a positive integer", "invalid-max-chars"], + ["web_fetch is disabled.", "disabled"], + ["fetch failed", "fetch-failed"], + [ + "Web fetch extraction failed: Readability disabled and no fetch provider is available.", + "extraction", + ], + [`Web fetch extraction failed: ${REDUCER_SECRET_SENTINEL}`, "extraction"], + [`unknown failure: ${REDUCER_SECRET_SENTINEL}`, "other"], + [`Invalid URL: must be http or https ${REDUCER_SECRET_SENTINEL}`, "other"], + ])("retains only the category for a tool error: %s", (error, category) => { + const evidence = reduceOpenClawToolEvidence( + publicFetchSessionJsonLines({ + payload: { status: "error", tool: "web_fetch", error }, + }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); + expect(artifact.webFetchDiagnostics).toEqual([ + { httpStatus: null, toolError: true, paired: true, errorCategory: category }, + ]); + expect(artifact.matches).toBe(false); + expect(evidence.webFetchResults[0]?.resultSuccess).toBe(false); + expect(JSON.stringify(evidence)).not.toContain(REDUCER_SECRET_SENTINEL); + expect(JSON.stringify(artifact)).not.toContain(REDUCER_SECRET_SENTINEL); + }); + + it.each([ + { status: "error", tool: "other", error: "fetch failed" }, + { status: "ok", tool: "web_fetch", error: "fetch failed" }, + { status: "error", tool: "web_fetch", error: { message: "fetch failed" } }, + { status: "error", error: "fetch failed" }, + ])("keeps unsupported error envelopes unknown: %j", (payload) => { + const evidence = reduceOpenClawToolEvidence( + publicFetchSessionJsonLines({ payload, isError: true }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + expect( + projectPersonalPublicFetchToolEvidenceArtifact(evidence).webFetchDiagnostics[0] + ?.errorCategory, + ).toBe("unknown"); + }); + + it("does not classify a successful result as an error", () => { + const evidence = reduceOpenClawToolEvidence( + publicFetchSessionJsonLines({ payload: publicFetchPayload({ error: "fetch failed" }) }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + expect( + projectPersonalPublicFetchToolEvidenceArtifact(evidence).webFetchDiagnostics[0] + ?.errorCategory, + ).toBeNull(); + expect(evidence.webFetchResults[0]?.resultSuccess).toBe(true); + }); + + it("discards unrecognized categories at the host artifact boundary", () => { + const evidence = reduceOpenClawToolEvidence( + publicFetchSessionJsonLines({ + payload: { status: "error", tool: "web_fetch", error: "fetch failed" }, + }), + "", + PUBLIC_FETCH_EXPECTATION, + ); + Object.assign(evidence.webFetchResults[0]!, { errorCategory: REDUCER_SECRET_SENTINEL }); + const artifact = projectPersonalPublicFetchToolEvidenceArtifact(evidence); + expect(artifact.webFetchDiagnostics[0]?.errorCategory).toBe("unknown"); + expect(JSON.stringify(artifact)).not.toContain(REDUCER_SECRET_SENTINEL); + }); +}); diff --git a/test/e2e/support/compatible-anthropic-switch.test.ts b/test/e2e/support/compatible-anthropic-switch.test.ts index 2afeb724955..eca5d1e9091 100644 --- a/test/e2e/support/compatible-anthropic-switch.test.ts +++ b/test/e2e/support/compatible-anthropic-switch.test.ts @@ -51,11 +51,13 @@ describe("compatible Anthropic inference switch setup", () => { ).toThrow("COMPATIBLE_ANTHROPIC_API_KEY is required"); }); - it("passes the mock bridge through endpoint validation without DNS rewriting (#9166)", async () => { + it("rejects the mock bridge before DNS rewriting (#9166)", async () => { const endpointUrl = compatibleAnthropicMockEndpointUrl(18_766); const rewrite = vi.fn(); - await expect(normalizeCustomEndpointUrl(endpointUrl, rewrite)).resolves.toBe(endpointUrl); + await expect(normalizeCustomEndpointUrl(endpointUrl, rewrite)).rejects.toThrow( + "cannot use an unverified OpenShell host bridge endpoint", + ); expect(rewrite).not.toHaveBeenCalled(); }); diff --git a/test/e2e/support/deepagents-observability-cleanup.test.ts b/test/e2e/support/deepagents-observability-cleanup.test.ts index bc6f92c87e4..c0ee52b717e 100644 --- a/test/e2e/support/deepagents-observability-cleanup.test.ts +++ b/test/e2e/support/deepagents-observability-cleanup.test.ts @@ -224,6 +224,14 @@ esac`, *) echo 'blocked by policy'; exit 7 ;; esac ;; *) exit 93 ;; +esac`, + ); + command( + "node", + `source="$(cat)" +case "$source" in + *currentRuntimeProviderBundles*) printf '127.0.0.1' ;; + *) printf '%s' "$source" | "$REAL_NODE" "$@" ;; esac`, ); command("ip", "echo '1: fixture inet 192.168.1.2/24'"); @@ -249,6 +257,7 @@ esac`, REPO: workspace.root, SANDBOX_NAME: "test-sandbox", NEMOCLAW_CLI_BIN: cli, + REAL_NODE: process.execPath, REAL_TSX: path.resolve("node_modules/.bin/tsx"), REAL_HELPER: path.resolve("test/e2e/live/deepagents-observability-contract.ts"), NATIVE_STATUS: String(nativeStatus), diff --git a/test/e2e/support/deepagents-observability-contract.test.ts b/test/e2e/support/deepagents-observability-contract.test.ts index 3e49e5d8d04..6906c7e065c 100644 --- a/test/e2e/support/deepagents-observability-contract.test.ts +++ b/test/e2e/support/deepagents-observability-contract.test.ts @@ -293,7 +293,7 @@ describe("Deep Agents observability policy proof", () => { encoding: "utf8", env: { PATH: process.env.PATH }, input: - 'FAILED:HTTPError:HTTP Error 403: Forbidden:{"error":"policy_denied","detail":"POST example.com:4318/v1/traces not permitted by policy"}\n', + 'FAILED:HTTPError:HTTP Error 403: Forbidden:{"error":"policy_denied","policy":"observability-otlp-local","rule":"POST /not-traces","detail":"POST /not-traces not permitted by policy","layer":"l7","protocol":"rest","method":"POST","path":"/not-traces","host":"host.openshell.internal","port":4318,"binary":"/opt/venv/bin/python3","rule_missing":{"type":"rest_allow","layer":"l7","method":"POST","path":"/not-traces","host":"host.openshell.internal","port":4318,"binary":"/opt/venv/bin/python3"},"next_steps":[]}\n', }); expect(proxyDenial.status, proxyDenial.stderr).toBe(0); expect(proxyDenial.stdout.trim()).toBe("policy-denied"); @@ -307,6 +307,44 @@ describe("Deep Agents observability policy proof", () => { expect(interleavedCurlDenial.status, interleavedCurlDenial.stderr).toBe(0); expect(interleavedCurlDenial.stdout.trim()).toBe("policy-denied"); }); + + it.each([ + ["7", "000", "* Immediate connect fail for 198.18.0.1: Permission denied", true], + ["7", "000", "curl: (7) Could not connect to server", false], + ["7", "000", "* Immediate connect fail for 198.18.0.1: Connection refused", false], + [ + "7", + "000", + "* Immediate connect fail for 198.18.0.1: Permission denied\n* Immediate connect fail for fd00::1: Connection refused", + false, + ], + ["28", "000", "* Immediate connect fail for 198.18.0.1: Permission denied", false], + ["7", "200", "* Immediate connect fail for 198.18.0.1: Permission denied", false], + [ + "7", + "000\nNEMOCLAW_OTLP_CURL_HTTP_CODE=000", + "* Immediate connect fail for 198.18.0.1: Permission denied", + false, + ], + ["7", "", "* Immediate connect fail for 198.18.0.1: Permission denied", false], + ])( + "requires native connect denial evidence for curl exit %s, HTTP %s, and %s", + (code, http, diagnostic, denied) => { + const result = spawnSync( + path.join(process.cwd(), "node_modules", ".bin", "tsx"), + ["test/e2e/live/deepagents-observability-contract.ts", "denial-state", code], + { + encoding: "utf8", + env: { PATH: process.env.PATH }, + input: `${http ? `NEMOCLAW_OTLP_CURL_HTTP_CODE=${http}\n` : ""}${diagnostic}\n`, + }, + ); + expect({ exit: result.status, state: result.stdout.trim() }, result.stderr).toEqual({ + exit: 0, + state: denied ? "policy-denied" : "other-failure", + }); + }, + ); }); describe("bounded private OTLP capture server", () => { diff --git a/test/e2e/support/deepagents-observability-dns-denial.test.ts b/test/e2e/support/deepagents-observability-dns-denial.test.ts new file mode 100644 index 00000000000..59d9e37ed25 --- /dev/null +++ b/test/e2e/support/deepagents-observability-dns-denial.test.ts @@ -0,0 +1,230 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { hasConfirmedOpenShellPolicyDenial } from "../live/deepagents-observability-contract.ts"; + +const hostname = "host.openshell.internal"; +const dnsFailure = "FAILED:URLError:"; +const dnsEvent = `[2.0] [sandbox] [OCSF ] NET:REFUSE [MED] DENIED ${hostname} [reason:policy_dns_no_valid_address]`; +const port = 4319; +const mappingEvent = `[2.0] [sandbox] [OCSF ] NET:OPEN [MED] DENIED /opt/venv/bin/python3(0) -> 198.18.0.11:${port} [reason:transparent_tcp_mapping_denied]`; +const mappingFailure = `NEMOCLAW_OTLP_PROBE_DNS=${JSON.stringify({ hostname, port, addresses: ["198.18.0.11"] })}\nFAILED:URLError:`; +const policyEvent = `[2.0] [sandbox] [OCSF ] NET:OPEN [MED] DENIED /opt/venv/bin/python3(0) -> ${hostname}:${port} [reason:transparent_tcp_policy_denied]`; + +describe("Deep Agents observability TCP policy denial evidence", () => { + it.each([ + ["fresh matching denial", "", policyEvent, true], + ["stale denial", policyEvent, policyEvent, false], + ["missing denial", "", "", false], + ["different host", "", policyEvent.replace(hostname, "other.example.test"), false], + ["different port", "", policyEvent.replace(":4319", ":4318"), false], + [ + "different executable", + "", + policyEvent.replace("/opt/venv/bin/python3", "/usr/bin/python3"), + false, + ], + [ + "unproven identity", + "", + policyEvent.replace("transparent_tcp_policy_denied", "identity_unavailable"), + false, + ], + ] as const)("classifies %s with Python permission denial", (_label, before, after, expected) => { + expect( + hasConfirmedOpenShellPolicyDenial(mappingFailure, 7, { hostname, port, before, after }), + ).toBe(expected); + }); +}); + +describe("Deep Agents observability native mapping denial evidence", () => { + it("accepts a fresh denial for the probe's resolved synthetic address and port", () => { + expect( + hasConfirmedOpenShellPolicyDenial(mappingFailure, 7, { + hostname, + port, + before: "", + after: mappingEvent, + }), + ).toBe(true); + }); + + it.each([ + ["missing audit", mappingFailure, "", "", 7], + ["stale event", mappingFailure, mappingEvent, mappingEvent, 7], + [ + "different address", + mappingFailure, + "", + mappingEvent.replace("198.18.0.11", "198.18.0.12"), + 7, + ], + ["different port", mappingFailure, "", mappingEvent.replace(":4319", ":4318"), 7], + [ + "different executable", + mappingFailure, + "", + mappingEvent.replace("/opt/venv/bin/python3", "/usr/bin/python3"), + 7, + ], + [ + "unproven identity", + mappingFailure, + "", + mappingEvent.replace("transparent_tcp_mapping_denied", "identity_unavailable"), + 7, + ], + ["different host", mappingFailure.replace(hostname, "other.example.test"), "", mappingEvent, 7], + [ + "unresolved address", + mappingFailure.replace('"198.18.0.11"', '"192.0.2.1"'), + "", + mappingEvent, + 7, + ], + [ + "non-synthetic address", + mappingFailure.replace("198.18.0.11", "192.0.2.1"), + "", + mappingEvent.replace("198.18.0.11", "192.0.2.1"), + 7, + ], + ["missing DNS record", mappingFailure.split("\n")[1], "", mappingEvent, 7], + [ + "duplicate DNS record", + mappingFailure + "\n" + mappingFailure.split("\n")[0], + "", + mappingEvent, + 7, + ], + [ + "malformed DNS record", + mappingFailure.replace('{"hostname"', '{broken"hostname"'), + "", + mappingEvent, + 7, + ], + [ + "null DNS record", + "NEMOCLAW_OTLP_PROBE_DNS=null\n" + mappingFailure.split("\n")[1], + "", + mappingEvent, + 7, + ], + [ + "transport failure", + mappingFailure.replace("[Errno 13] Permission denied", "timed out"), + "", + mappingEvent, + 7, + ], + ["successful process", mappingFailure, "", mappingEvent, 0], + ] as const)("rejects %s", (_label, output, before, after, status) => { + expect( + hasConfirmedOpenShellPolicyDenial(output, status, { hostname, port, before, after }), + ).toBe(false); + }); +}); + +describe("Deep Agents observability DNS denial evidence", () => { + it("preserves the Python URL error when the probe DNS lookup is denied", () => { + const shell = fs.readFileSync( + "test/e2e/e2e-cloud-experimental/checks/11-deepagents-code-observability.sh", + "utf8", + ); + const source = shell.match(/python_probe_source\(\) \{\n cat <<'PY'\n([\s\S]*?)\nPY/u)?.[1]; + expect(source).toBeDefined(); + const result = spawnSync( + "python3", + [ + "-c", + ` +import socket +def deny_dns(*args, **kwargs): + raise socket.gaierror(-5, "No address associated with hostname") +socket.getaddrinfo = deny_dns +${source} +`, + "POST", + `http://${hostname}:${port}/v1/traces`, + "probe", + ], + { encoding: "utf8" }, + ); + expect(result.status, result.stderr).toBe(7); + expect( + hasConfirmedOpenShellPolicyDenial(result.stdout, result.status ?? undefined, { + hostname, + port, + before: "", + after: dnsEvent, + }), + ).toBe(true); + }); + it.each(["policy_dns_ineligible", "policy_dns_no_valid_address"])( + "accepts a fresh matching %s event with the failed Python probe", + (reason) => { + expect( + hasConfirmedOpenShellPolicyDenial(dnsFailure, 7, { + hostname, + before: "", + after: dnsEvent.replace("policy_dns_no_valid_address", reason), + }), + ).toBe(true); + }, + ); + + it.each([ + ["stale event", dnsFailure, dnsEvent, dnsEvent, 7], + ["missing event", dnsFailure, "", "", 7], + ["different host", dnsFailure, "", dnsEvent.replace(hostname, "other.example.test"), 7], + [ + "unrelated reason", + dnsFailure, + "", + dnsEvent.replace("policy_dns_no_valid_address", "timeout"), + 7, + ], + ["transport failure", "FAILED:URLError:", "", dnsEvent, 7], + ["successful process", dnsFailure, "", dnsEvent, 0], + ] as const)("rejects %s", (_label, output, before, after, status) => { + expect(hasConfirmedOpenShellPolicyDenial(output, status, { hostname, before, after })).toBe( + false, + ); + }); + + it.each([ + ["DNS", dnsFailure, dnsEvent], + ["mapping", mappingFailure, mappingEvent], + ["TCP policy", mappingFailure, policyEvent], + ])("reads %s audit files through the live classifier command", (_label, input, event) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "dcode-dns-audit-")); + const before = path.join(directory, "before.log"); + const after = path.join(directory, "after.log"); + fs.writeFileSync(before, ""); + fs.writeFileSync(after, event); + try { + const result = spawnSync( + path.join(process.cwd(), "node_modules/.bin/tsx"), + [ + "test/e2e/live/deepagents-observability-contract.ts", + "denial-state", + "7", + before, + after, + `http://${hostname}:${port}/v1/traces`, + ], + { encoding: "utf8", env: { PATH: process.env.PATH }, input }, + ); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout.trim()).toBe("policy-denied"); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); +}); diff --git a/test/e2e/support/dgx-station-express.test.ts b/test/e2e/support/dgx-station-express.test.ts index e0f18a8ee49..93472c644a3 100644 --- a/test/e2e/support/dgx-station-express.test.ts +++ b/test/e2e/support/dgx-station-express.test.ts @@ -19,6 +19,10 @@ import { ShellProbe, trustedShellCommand } from "../fixtures/shell-probe.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; import { boundedTestTempRoot } from "../../helpers/bounded-temp-root"; +import { + nativeLocalIdentity, + type NativeLocalBinding, +} from "../../../src/lib/inference/native-local/contract.ts"; import { stationVllmCommands, assertStationWorkload, @@ -28,6 +32,7 @@ import { runStationExpressSmoke, stationModelCacheMetadata, stationSmokeEnvironment, + stationNativeInferenceCommand, STATION_SMOKE_MODEL, STATION_SMOKE_SANDBOX, } from "./dgx-station-express.ts"; @@ -40,6 +45,20 @@ const base = { E2E_MANAGED_IMAGE_REVISION: "b".repeat(40), }; const directories: string[] = []; +function stationAttachment(overrides: Partial = {}) { + const binding: NativeLocalBinding = { + agent: "openclaw", + provider: "vllm-local", + sandboxName: STATION_SMOKE_SANDBOX, + gatewayName: "nemoclaw", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated", + ...overrides, + }; + return { ...binding, ...nativeLocalIdentity(binding), schemaVersion: 1, providerId: "owned" }; +} + afterEach(() => { for (const directory of directories.splice(0)) fs.rmSync(directory, { recursive: true, force: true }); @@ -158,6 +177,61 @@ describe("Station Express smoke boundaries", () => { expect(() => assertStationReady(JSON.stringify([ready, ready]))).toThrow("ready sandbox"); }); + it.each([ + undefined, + { ...stationAttachment(), providerId: "" }, + { ...stationAttachment(), profileId: "another-profile" }, + stationAttachment({ sandboxName: "another-sandbox" }), + stationAttachment({ gatewayName: "another-gateway" }), + stationAttachment({ agent: "hermes" }), + stationAttachment({ provider: "ollama-local" }), + stationAttachment({ endpointUrl: "http://host.openshell.internal:18000/v1" }), + stationAttachment({ credentialEnv: "ANOTHER_TOKEN" }), + stationAttachment({ authMode: "sentinel" }), + ])("rejects missing or mismatched Station inference authority %#", (attachment) => { + expect(() => + stationNativeInferenceCommand({ + sandboxes: { [STATION_SMOKE_SANDBOX]: { nativeLocalProviderAttachment: attachment } }, + }), + ).toThrow("authenticated native vLLM attachment"); + }); + + it("passes the scoped native credential and exact model request to curl", () => { + const token = "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; + const probe = invokeStationInference(token); + expect(probe.status).toBe(0); + expect(probe.stdout.trim().split("\n")).toEqual([ + "-q", + "--fail", + "--silent", + "--show-error", + "--max-time", + "90", + "http://host.openshell.internal:8000/v1/chat/completions", + "-H", + `Authorization: Bearer ${token}`, + "-H", + "Content-Type: application/json", + "--data-raw", + JSON.stringify({ + model: STATION_SMOKE_MODEL, + messages: [{ role: "user", content: "Reply with one word: PONG" }], + max_tokens: 8192, + parallel_tool_calls: false, + }), + ]); + }); + + it.each([ + "host-secret", + "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN\nhost-secret", + ])("rejects an unscoped or raw credential before calling curl %#", (token) => { + const rejected = invokeStationInference(token); + expect(rejected.status).toBe(2); + expect(rejected.stdout).toBe(""); + }); + it.each([ { model: "other-model", choices: [{ message: { content: "PONG" } }] }, { model: STATION_SMOKE_MODEL, choices: [{ message: { content: "" } }] }, @@ -241,6 +315,7 @@ describe("Station Express smoke boundaries", () => { sandboxes: { [STATION_SMOKE_SANDBOX]: { agent: "openclaw", + nativeLocalProviderAttachment: stationAttachment(), workload: { kind: "managed-image", sourceRevision: base.E2E_MANAGED_IMAGE_REVISION, @@ -265,14 +340,14 @@ describe("Station Express smoke boundaries", () => { label === "station-express-install" && timeoutAt === "host" ? observed : normal; return { ...result, stdout: outputs[label] ?? "" }; }); - const exec = vi - .fn() + const execShell = vi + .fn() .mockResolvedValue(timeoutAt === "inference" ? observed : normal); const cleanup = new CleanupRegistry(); const writeEvidence = vi.fn(async () => {}); const result = runStationExpressSmoke({ host: { command }, - sandbox: { exec }, + sandbox: { execShell }, cleanup, environment: { ...base, HOME: home, NEMOCLAW_STATION_RUNNER_HOME: home }, repoRoot: "/candidate", @@ -292,7 +367,7 @@ describe("Station Express smoke boundaries", () => { (error: Error) => error.message, ), ).resolves.toBe(expectedFailure); - expect(exec).toHaveBeenCalledTimes(inferenceCalls); + expect(execShell).toHaveBeenCalledTimes(inferenceCalls); expect(command).toHaveBeenCalledWith( "bash", ["uninstall.sh", "--yes", "--destroy-user-data"], @@ -333,14 +408,14 @@ describe("Station Express smoke boundaries", () => { return empty; }) .mockResolvedValue(empty); - const exec = vi.fn(); + const execShell = vi.fn(); const writeEvidence = vi.fn(async () => {}); const phases = { inspect: vi.fn(), install: vi.fn(), assert: vi.fn(), cleanup: vi.fn() }; await expect( runStationExpressSmoke({ cleanup, host: { command }, - sandbox: { exec }, + sandbox: { execShell }, environment: { ...base, HOME: home, NEMOCLAW_STATION_RUNNER_HOME: home }, repoRoot: "/candidate", phases, @@ -363,7 +438,7 @@ describe("Station Express smoke boundaries", () => { ["uninstall.sh", "--yes", "--destroy-user-data"], expect.anything(), ); - expect(exec).not.toHaveBeenCalled(); + expect(execShell).not.toHaveBeenCalled(); expect(phases.inspect).toHaveBeenCalledOnce(); expect(phases.install).toHaveBeenCalledOnce(); expect(phases.assert).not.toHaveBeenCalled(); @@ -402,6 +477,22 @@ function cleanupHome() { directories.push(home); return home; } +function invokeStationInference(token: string) { + const home = cleanupHome(); + const bin = path.join(home, "bin"); + fs.mkdirSync(bin); + fs.writeFileSync(path.join(bin, "curl"), '#!/bin/sh\nprintf "%s\\n" "$@"\n', { mode: 0o755 }); + const script = stationNativeInferenceCommand({ + sandboxes: { [STATION_SMOKE_SANDBOX]: { nativeLocalProviderAttachment: stationAttachment() } }, + }); + return spawnSync("/bin/sh", ["-c", script], { + env: { PATH: `${bin}:/usr/bin:/bin`, NEMOCLAW_LOCAL_INFERENCE_TOKEN: token }, + encoding: "utf8", + timeout: 1000, + killSignal: "SIGKILL", + }); +} + async function stationProbeOutcome(home: string, timeOut: boolean) { const progress = startTestProgress( "Station timeout result", @@ -686,7 +777,7 @@ it("allows consecutive tsx IPC servers within the generated temporary path lengt ); directories.push(root); const tmp = path.join(root, "x".repeat(env.TMPDIR!.length - root.length - 1)); - fs.mkdirSync(tmp); + fs.mkdirSync(tmp, { recursive: true }); const program = path.join(root, "ipc.ts"); fs.writeFileSync(program, 'process.stdout.write("ready")'); const executable = path.join(process.cwd(), "node_modules/tsx/dist/cli.mjs"); diff --git a/test/e2e/support/dgx-station-express.ts b/test/e2e/support/dgx-station-express.ts index cb52c3c9eca..6fbfc1c89b4 100644 --- a/test/e2e/support/dgx-station-express.ts +++ b/test/e2e/support/dgx-station-express.ts @@ -18,10 +18,16 @@ import { } from "../../../src/lib/inference/vllm-models.ts"; import { detectVllmProfile, resolveVllmModelRuntime } from "../../../src/lib/inference/vllm.ts"; import { parseStrictOpenShellSandboxListJson } from "../../../src/lib/adapters/openshell/sandbox-identity.ts"; +import { normalizeNativeLocalProviderAttachment } from "../../../src/lib/inference/native-local/profile.ts"; +import { + NATIVE_LOCAL_AUTH_HEADER_SCRIPT, + NATIVE_LOCAL_CREDENTIAL_ENV, +} from "../../../src/lib/inference/native-local/agent-config.ts"; +import { shellQuote } from "../../../src/lib/core/shell-quote.ts"; import { buildAvailabilityProbeEnv } from "../fixtures/availability-env.ts"; import type { CleanupRegistry } from "../fixtures/cleanup.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; -import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; +import { trustedSandboxShellScript, type SandboxClient } from "../fixtures/clients/sandbox.ts"; export const STATION_SMOKE_SANDBOX = "e2e-station-express"; export const STATION_SMOKE_MODEL = "nvidia-nemotron-3.5-super-vl-120b-a12b-nvfp4"; @@ -240,10 +246,34 @@ export function assertStationInference(output: string): void { ); } +export function stationNativeInferenceCommand(value: unknown) { + const entry = record(record(record(value).sandboxes)[STATION_SMOKE_SANDBOX]); + const route = normalizeNativeLocalProviderAttachment(entry.nativeLocalProviderAttachment); + requireCondition( + route?.sandboxName === STATION_SMOKE_SANDBOX && + route.gatewayName === "nemoclaw" && + route.agent === "openclaw" && + route.provider === "vllm-local" && + route.authMode === "authenticated" && + route.credentialEnv === NATIVE_LOCAL_CREDENTIAL_ENV && + route.endpointUrl === "http://host.openshell.internal:8000/v1", + "Station inference requires its authenticated native vLLM attachment", + ); + const payload = JSON.stringify({ + model: STATION_SMOKE_MODEL, + messages: [{ role: "user", content: "Reply with one word: PONG" }], + max_tokens: 8192, + parallel_tool_calls: false, + }); + return trustedSandboxShellScript( + `${NATIVE_LOCAL_AUTH_HEADER_SCRIPT}; exec curl -q --fail --silent --show-error --max-time 90 ${shellQuote(`${route.endpointUrl}/chat/completions`)} -H "$AUTH_HEADER" -H 'Content-Type: application/json' --data-raw ${shellQuote(payload)}`, + ); +} + export async function runStationExpressSmoke(options: { host: Pick; cleanup: Pick; - sandbox: Pick; + sandbox: Pick; environment: NodeJS.ProcessEnv; repoRoot: string; phases: { inspect(): void; install(): void; assert(): void; cleanup(): void }; @@ -386,32 +416,14 @@ export async function runStationExpressSmoke(options: { initialImages.includes(vllmImage), "The selected Station vLLM image was not pre-pulled", ); - assertStationWorkload( - parseJson(fs.readFileSync(path.join(env.HOME!, ".nemoclaw", "sandboxes.json"), "utf8")), - env.E2E_MANAGED_IMAGE_REVISION!, + const registry = parseJson( + fs.readFileSync(path.join(env.HOME!, ".nemoclaw", "sandboxes.json"), "utf8"), ); + assertStationWorkload(registry, env.E2E_MANAGED_IMAGE_REVISION!); await options.verifyPublication(env); - const response = await options.sandbox.exec( + const response = await options.sandbox.execShell( STATION_SMOKE_SANDBOX, - [ - "curl", - "-q", - "--fail", - "--silent", - "--show-error", - "--max-time", - "90", - "https://inference.local/v1/chat/completions", - "-H", - "Content-Type: application/json", - "--data-raw", - JSON.stringify({ - model: STATION_SMOKE_MODEL, - messages: [{ role: "user", content: "Reply with one word: PONG" }], - max_tokens: 8192, - parallel_tool_calls: false, - }), - ], + stationNativeInferenceCommand(registry), { env, artifactName: "station-routed-inference", timeoutMs: 120_000 }, ); requireCondition( diff --git a/test/e2e/support/dns-rebinding-server.test.ts b/test/e2e/support/dns-rebinding-server.test.ts new file mode 100644 index 00000000000..31f94053fa5 --- /dev/null +++ b/test/e2e/support/dns-rebinding-server.test.ts @@ -0,0 +1,94 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import dgram from "node:dgram"; +import net from "node:net"; +import { Resolver } from "node:dns/promises"; +import { expect, it } from "vitest"; +import { + readDnsTcpFrame, + reboundDnsAnswer, + startRebindingDnsServer, +} from "../fixtures/dns-rebinding-server.ts"; + +const HOST = "fixture.example.test"; +function query(host: string, type = 1): Buffer { + const header = Buffer.from([0, 123, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0]); + return Buffer.concat([ + header, + ...host + .split(".") + .map((part) => Buffer.concat([Buffer.from([part.length]), Buffer.from(part)])), + Buffer.from([0, 0, type, 0, 1]), + ]); +} + +it("overrides A and suppresses AAAA only for the selected hostname", () => { + const answer = reboundDnsAnswer(query(HOST), HOST, "192.0.2.1")!; + expect(answer.subarray(-4)).toEqual(Buffer.from([192, 0, 2, 1])); + expect(answer.readUInt16BE(0)).toBe(123); + expect(answer.readUInt16BE(6)).toBe(1); + expect(reboundDnsAnswer(query(HOST, 28), HOST, "192.0.2.1")!.readUInt16BE(6)).toBe(0); + expect(reboundDnsAnswer(query("other.example.test"), HOST, "192.0.2.1")).toBeNull(); + expect(reboundDnsAnswer(query(HOST).subarray(0, 14), HOST, "192.0.2.1")).toBeNull(); +}); + +it("forwards UDP and TCP controls, changes real DNS answers, and closes twice", async () => { + const udp = dgram.createSocket("udp4"); + const answer = (bytes: Buffer) => + reboundDnsAnswer(bytes, HOST, "203.0.113.8") ?? + reboundDnsAnswer(bytes, "other.example.test", "203.0.113.8")!; + udp.on("message", (bytes, peer) => udp.send(answer(bytes), peer.port, peer.address)); + await new Promise((resolve) => udp.bind(0, "127.0.0.1", resolve)); + let tcpQueries = 0; + const tcp = net.createServer((client) => { + void readDnsTcpFrame(client) + .then((bytes) => { + tcpQueries += 1; + const response = answer(bytes); + const prefix = Buffer.alloc(2); + prefix.writeUInt16BE(response.length); + client.end(Buffer.concat([prefix, response])); + }) + .catch(() => client.destroy()); + }); + await new Promise((resolve) => tcp.listen(udp.address().port, "127.0.0.1", resolve)); + const server = await startRebindingDnsServer(HOST, "127.0.0.1", udp.address().port); + const resolver = new Resolver(); + resolver.setServers([`127.0.0.1:${server.port}`]); + try { + expect(await resolver.resolve4(HOST)).toEqual(["203.0.113.8"]); + const tcpResponse = await new Promise((resolve, reject) => { + const client = net.createConnection({ host: "127.0.0.1", port: server.port }); + const payload = query(HOST); + const prefix = Buffer.alloc(2); + prefix.writeUInt16BE(payload.length); + let bytes = Buffer.alloc(0); + client.on("error", reject); + client.setTimeout(3000, () => client.destroy(new Error("test DNS timeout"))); + client.on("connect", () => { + client.write(prefix); + client.write(payload); + }); + client.on("data", (chunk) => { + bytes = Buffer.concat([bytes, typeof chunk === "string" ? Buffer.from(chunk) : chunk]); + }); + client.on("end", () => resolve(bytes)); + }); + expect(tcpQueries).toBe(1); + expect(tcpResponse.subarray(-4)).toEqual(Buffer.from([203, 0, 113, 8])); + server.setAddress("192.0.2.1"); + expect(await resolver.resolve4(HOST)).toEqual(["192.0.2.1"]); + await expect(resolver.resolve6(HOST)).rejects.toMatchObject({ code: "ENODATA" }); + expect(await resolver.resolve4("other.example.test")).toEqual(["203.0.113.8"]); + expect(server.answers()).toBe(1); + server.setAddress("10.20.30.40"); + expect(server.answers()).toBe(0); + expect(await resolver.resolve4(HOST)).toEqual(["10.20.30.40"]); + } finally { + await server.close(); + await server.close(); + udp.close(); + await new Promise((resolve) => tcp.close(() => resolve())); + } +}); diff --git a/test/e2e/support/e2e-clients.test.ts b/test/e2e/support/e2e-clients.test.ts index a2f3a396e87..a1fde5cc08a 100644 --- a/test/e2e/support/e2e-clients.test.ts +++ b/test/e2e/support/e2e-clients.test.ts @@ -357,6 +357,7 @@ describe("E2E fixture clients", () => { await host.expectStatus("assistant"); await host.cleanupSandbox("assistant"); await host.cleanupSandbox("retained", { preserveGatewayRegistration: true }); + await host.cleanupSandbox("owned-last", { cleanupGateway: true }); expect(runner.calls.map((call) => ({ command: call.command, args: call.args }))).toEqual([ { command: "nemoclaw", args: ["list"] }, @@ -366,6 +367,10 @@ describe("E2E fixture clients", () => { command: "nemoclaw", args: ["retained", "destroy", "--yes", "--no-cleanup-gateway"], }, + { + command: "nemoclaw", + args: ["owned-last", "destroy", "--yes", "--cleanup-gateway"], + }, ]); }); @@ -745,70 +750,6 @@ describe("E2E fixture clients", () => { ).expectOpenshellStatusConnected(); }); - it("gateway client proves registration, listener, and host runtime are removed", async () => { - const runner = new FakeRunner(); - const privateRuntimeEnv = { ...process.env, HOME: "/private/export-home" }; - runner.enqueue({ exitCode: 1, stderr: "No active gateway" }); - runner.enqueue({ exitCode: 1 }); - runner.enqueue({ exitCode: 1 }); - runner.enqueue({ exitCode: 0 }); - const gateway = new GatewayClient( - new HostCliClient(runner, { cliPath: "nemoclaw" }), - new SandboxClient(runner), - ); - - await gateway.expectRemoved("nemoclaw", { - artifactName: "final-gateway", - env: privateRuntimeEnv, - gatewayPort: 18_080, - }); - - expect(runner.calls.map(({ command, args }) => [command, args])).toEqual([ - ["openshell", ["status"]], - ["lsof", ["-ti", ":18080", "-sTCP:LISTEN"]], - ["sh", expect.any(Array)], - ["docker", ["container", "ps", "--format", "{{.ID}}\t{{.Names}}"]], - ["true", []], - ]); - expect(runner.calls[0].options).toMatchObject({ - artifactName: "final-gateway-status", - env: { OPENSHELL_GATEWAY: "nemoclaw" }, - }); - expect(runner.calls[1].options).toMatchObject({ - artifactName: "final-gateway-listener", - }); - expect( - runner.calls.slice(2).every((call) => call.options?.env?.HOME === privateRuntimeEnv.HOME), - ).toBe(true); - }); - - it("gateway client rejects an inconclusive listener absence probe", async () => { - const runner = new FakeRunner(); - runner.enqueue({ exitCode: 1, stderr: "Status: Disconnected\nGateway: nemoclaw" }); - runner.enqueue({ exitCode: 1, stderr: "lsof: command unavailable" }); - const gateway = new GatewayClient( - new HostCliClient(runner, { cliPath: "nemoclaw" }), - new SandboxClient(runner), - ); - - await expect(gateway.expectRemoved("nemoclaw", { gatewayPort: 8_080 })).rejects.toThrow( - "gateway listener still exists or could not be disproved on port 8080", - ); - }); - - it("gateway client rejects a gateway that still accepts status connections", async () => { - const runner = new FakeRunner(); - runner.enqueue({ exitCode: 0, stdout: "Status: Connected\nGateway: nemoclaw" }); - const gateway = new GatewayClient( - new HostCliClient(runner, { cliPath: "nemoclaw" }), - new SandboxClient(runner), - ); - - await expect(gateway.expectRemoved("nemoclaw", { gatewayPort: 8_080 })).rejects.toThrow( - "openshell status did not prove gateway 'nemoclaw' disconnected", - ); - }); - it("initial cleanup verifies the requested gateway before deleting a sandbox", async () => { const runner = new FakeRunner(); runner.enqueue({ stdout: JSON.stringify({ gateway: "nemoclaw", status: "healthy" }) }); diff --git a/test/e2e/support/e2e-http-protocol.test.ts b/test/e2e/support/e2e-http-protocol.test.ts index 5f0afd466e8..79cc4918c56 100644 --- a/test/e2e/support/e2e-http-protocol.test.ts +++ b/test/e2e/support/e2e-http-protocol.test.ts @@ -2,6 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 import http from "node:http"; +import https from "node:https"; +import { createSecureContext, type ConnectionOptions } from "node:tls"; import { describe, expect, it } from "vitest"; import { closeServer, @@ -10,8 +12,152 @@ import { writeJsonResponse, writeSseEvents, } from "../fixtures/http-protocol.ts"; +import { startFakeHttpsCompatibleServer } from "../live/https-pin-compatible-server.ts"; +import { startRuntimeIdentityOAuthServer } from "../live/runtime-identity-oauth-server.ts"; describe("fake provider HTTP protocol", () => { + it.each([ + { + name: "compatible HTTPS", + start: () => startFakeHttpsCompatibleServer({ apiKey: "fixture-only", model: "test" }), + path: "/v1/models", + method: "GET", + headers: { authorization: "Bearer fixture-only" }, + status: 200, + }, + { + name: "OAuth HTTPS", + start: () => + startRuntimeIdentityOAuthServer({ + clientId: "fixture-client", + clientSecret: "fixture-secret", + initialRefreshToken: "fixture-refresh", + }), + path: "/resource", + method: "HEAD", + headers: {}, + status: 401, + }, + ])("returns the listening TCP port for the $name fixture", async (scenario) => { + const fixture = await scenario.start(); + try { + const status = await new Promise((resolve, reject) => { + const request = https.request( + { + hostname: "127.0.0.1", + port: fixture.port, + path: scenario.path, + method: scenario.method, + headers: scenario.headers, + ca: fixture.certificate, + }, + (response) => { + response.resume(); + response.on("end", () => resolve(response.statusCode)); + response.on("error", reject); + }, + ); + request.on("error", reject); + request.end(); + }); + expect(status).toBe(scenario.status); + } finally { + await fixture.close(); + } + }); + + it("records OAuth request timing without retaining credentials or changing rotation", async () => { + const options = { + clientId: "fixture-client", + clientSecret: "fixture-secret", + initialRefreshToken: "fixture-refresh", + }; + const fixture = await startRuntimeIdentityOAuthServer(options); + const tlsOptions: ConnectionOptions = { + secureContext: createSecureContext({ ca: fixture.certificate }), + }; + const exchange = (refreshToken: string) => + new Promise<{ status: number | undefined; body: Record }>( + (resolve, reject) => { + const request = https.request( + { + hostname: "127.0.0.1", + port: fixture.port, + path: "/oauth/token", + method: "POST", + ...tlsOptions, + headers: { "content-type": "application/x-www-form-urlencoded" }, + }, + (response) => { + let body = ""; + response.setEncoding("utf8"); + response.on("data", (chunk: string) => { + body += chunk; + }); + response.on("error", reject); + response.on("end", () => { + try { + resolve({ status: response.statusCode, body: JSON.parse(body) }); + } catch (error) { + reject(error); + } + }); + }, + ); + request.on("error", reject); + request.end( + new URLSearchParams({ + grant_type: "refresh_token", + client_id: options.clientId, + client_secret: options.clientSecret, + refresh_token: refreshToken, + }).toString(), + ); + }, + ); + try { + const startedAt = Date.now(); + const first = await exchange(options.initialRefreshToken); + expect(first.status).toBe(200); + expect(first.body.expires_in).toBe(600); + const second = await exchange(String(first.body.refresh_token)); + expect(second.status).toBe(200); + expect(second.body.access_token).not.toBe(first.body.access_token); + const stale = await exchange(options.initialRefreshToken); + expect(stale).toEqual({ status: 400, body: { error: "invalid_grant" } }); + const exhausted = await exchange(String(second.body.refresh_token)); + expect(exhausted).toEqual({ status: 400, body: { error: "invalid_grant" } }); + const completedAt = Date.now(); + const records = fixture.tokenRequests(); + expect(records).toHaveLength(4); + expect(records.map((record) => record.issuedVersion)).toEqual([1, 2, null, null]); + expect(records.map((record) => record.refreshTokenOk)).toEqual([true, true, false, true]); + records.forEach((record, index) => { + expect(record).toEqual({ + sequence: index + 1, + receivedAt: expect.any(String), + method: "POST", + path: "/oauth/token", + grantTypeOk: true, + clientIdOk: true, + refreshTokenOk: index !== 2, + clientSecretOk: true, + issuedVersion: index < 2 ? index + 1 : null, + }); + const receivedAt = Date.parse(record.receivedAt); + expect(receivedAt).toBeGreaterThanOrEqual(startedAt); + expect(receivedAt).toBeLessThanOrEqual(completedAt); + expect(new Date(receivedAt).toISOString()).toBe(record.receivedAt); + }); + const serializedRecords = JSON.stringify(records); + expect(fixture.secretValues().filter((secret) => serializedRecords.includes(secret))).toEqual( + [], + ); + } finally { + await fixture.close(); + } + }); + it("reads request bodies and writes JSON responses", async () => { let body = ""; const server = http.createServer(async (req, res) => { diff --git a/test/e2e/support/e2e-live-target-gating.test.ts b/test/e2e/support/e2e-live-target-gating.test.ts index db44e01dc22..ae3d9783c96 100644 --- a/test/e2e/support/e2e-live-target-gating.test.ts +++ b/test/e2e/support/e2e-live-target-gating.test.ts @@ -368,7 +368,7 @@ describe("live E2E target gating", () => { { file: "openshell-gateway-upgrade.test.ts", testName: - "openshell-gateway-upgrade: restores gateway registration and all sandbox state (#11898)", + "openshell-gateway-upgrade: refuses the 0.1.x cutover and preserves registered sandboxes (#12558)", }, ] as const)( "applies the Linux gate to $file at real Vitest collection", diff --git a/test/e2e/support/e2e-phase-lifecycle.test.ts b/test/e2e/support/e2e-phase-lifecycle.test.ts index 7754cdffede..26a21974e8e 100644 --- a/test/e2e/support/e2e-phase-lifecycle.test.ts +++ b/test/e2e/support/e2e-phase-lifecycle.test.ts @@ -9,6 +9,7 @@ import path from "node:path"; import { afterEach, beforeEach, describe, expect, expectTypeOf, it } from "vitest"; import { customNativeProvider } from "../../../src/lib/inference/native-provider/custom.ts"; + import { type CommandRunner, GatewayClient, @@ -555,34 +556,44 @@ describe("LifecyclePhaseFixture profile dispatch", () => { describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { const sandboxName = "e2e-dcode-cloud"; const validCredential = "valid-fixture-credential"; - const nativeProvider = customNativeProvider({ + const definition = customNativeProvider({ provider: "compatible-endpoint", endpointUrl: "https://inference-api.nvidia.com/v1", api: "openai-completions", credentialEnv: "COMPATIBLE_API_KEY", allowedIps: ["8.8.8.8"], }); - const attachment = { + const nativeHostedProviderAttachment = { schemaVersion: 1, - profileId: nativeProvider.profileId, - providerName: nativeProvider.providerName, - providerId: "provider-fixture-id", - endpointUrl: nativeProvider.endpoint, - allowedIps: nativeProvider.allowedIps, - custom: nativeProvider.custom, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "dcode-owned-provider", + endpointUrl: definition.endpointUrl, + allowedIps: definition.allowedIps, + custom: definition.custom, }; - const options = dcodeInvalidCredentialRebuildOptionsFromRegistryEntry( - { - agent: "langchain-deepagents-code", - gatewayName: "nemoclaw", - provider: "compatible-endpoint", - endpointUrl: nativeProvider.endpoint, - preferredInferenceApi: "openai-completions", - nativeHostedProviderAttachment: attachment, - model: "nvidia/nvidia/nemotron-3-ultra", - }, - validCredential, - ); + const entry = { + agent: "langchain-deepagents-code", + gatewayName: "nemoclaw", + provider: "compatible-endpoint", + model: "nvidia/nvidia/nemotron-3-ultra", + nativeHostedProviderAttachment, + }; + const options = dcodeInvalidCredentialRebuildOptionsFromRegistryEntry(entry, validCredential); + + function providerMetadata(id = "dcode-owned-provider"): ShellProbeResult { + return shellResult( + 0, + [ + `Name: ${definition.providerName}`, + `Id: ${id}`, + `Type: ${definition.profileId}`, + "Resource version: 1", + "Credential keys: COMPATIBLE_API_KEY", + "Config keys: ", + ].join("\n"), + ); + } function dcodeInstance(): NemoClawInstance { return instance({ @@ -631,6 +642,7 @@ describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { try { const runner = new FakeRunner(); enqueuePreamble(runner); + runner.enqueue(providerMetadata()); runner.enqueue(shellResult(0)); // install invalid provider credential runner.enqueue(shellResult(0, "401")); runner.enqueue(shellResult(0, `NAME PHASE\n${sandboxName} Ready\n`)); @@ -645,6 +657,7 @@ describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { runner.enqueue(shellResult(0, "container-b\ncontainer-a\n")); runner.enqueue(shellResult(0, "NEMOCLAW_DCODE_REBUILD_MARKER")); runner.enqueue(shellResult(0, `NAME PHASE\n${sandboxName} Ready\n`)); + runner.enqueue(providerMetadata()); runner.enqueue(shellResult(0)); // restore valid provider credential runner.enqueue(shellResult(0, "200")); runner.enqueue(shellResult(0, "rebuild complete")); @@ -703,17 +716,66 @@ describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { call.command === "openshell" && call.args.slice(0, 2).join(" ") === "provider update", ); expect(providerUpdates).toHaveLength(2); - expect(providerUpdates[0].args).toContain(nativeProvider.providerName); - const baselineProbe = runner.calls.find( - (call) => call.options?.artifactName === "lifecycle-dcode-route-baseline-1", - ); - expect(baselineProbe?.args).toContain(options.endpoint); - expect(baselineProbe?.args.join(" ")).not.toContain("inference.local"); + expect(providerUpdates.map((call) => call.args[4])).toEqual([ + definition.providerName, + definition.providerName, + ]); const invalidCredential = providerUpdates[0].options?.env?.COMPATIBLE_API_KEY; expect(invalidCredential).toMatch(/^nvapi-e2e-invalid-/); expect(providerUpdates[0].args).not.toContain(invalidCredential); expect(providerUpdates[0].options?.redactionValues).toContain(invalidCredential); expect(providerUpdates[1].options?.env?.COMPATIBLE_API_KEY).toBe(validCredential); + const probe = runner.calls.find( + (call) => call.options?.artifactName === "lifecycle-dcode-route-baseline-1", + )!; + expect(probe.command).toBe("openshell"); + expect(probe.args.slice(0, 6)).toEqual([ + "sandbox", + "exec", + "-n", + sandboxName, + "--", + "/usr/local/lib/nemoclaw/dcode-managed-exec", + ]); + expect(probe.args).toContain(options.endpoint); + expect(probe.args.join(" ")).not.toContain("inference.local"); + // Execute the shell payload after the sandbox wrapper and managed launcher. + const command = probe.args.slice(6); + const fakeCurl = path.join(home, "curl"); + fs.writeFileSync(fakeCurl, "#!/bin/sh\nprintf '%s\\0' \"$@\"\n", { mode: 0o700 }); + const handle = "openshell:resolve:env:s" + "a".repeat(64) + "_COMPATIBLE_API_KEY"; + const probeRun = spawnSync(command[0], command.slice(1), { + encoding: "utf8", + env: { PATH: `${home}:/usr/bin:/bin`, COMPATIBLE_API_KEY: handle }, + }); + expect(probeRun.status).toBe(0); + const probeArgs = probeRun.stdout.split("\0").filter(Boolean); + expect(probeArgs).toEqual([ + "-q", + "-sS", + "-o", + "/dev/null", + "-w", + "%{http_code}", + "--connect-timeout", + "5", + "--max-time", + "15", + "-H", + "Content-Type: application/json", + "-H", + `Authorization: Bearer ${handle}`, + "--data-binary", + JSON.stringify({ + model: options.model, + max_tokens: 8, + messages: [{ role: "user", content: "Reply with OK" }], + stream: false, + }), + options.endpoint, + ]); + expect(probe.options?.timeoutMs).toBe(25_000); + expect(probe.options?.env).not.toHaveProperty("COMPATIBLE_API_KEY"); const rebuild = runner.calls.find( (call) => call.command === "nemoclaw" && @@ -831,7 +893,12 @@ describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { options, ), ).rejects.toThrow(/gateway's only sandbox/); - expect(runner.calls).toHaveLength(1); + expect(runner.calls).toEqual([ + expect.objectContaining({ + command: "openshell", + args: ["sandbox", "list", "--names", "--page-size", "2", "-g", "nemoclaw"], + }), + ]); expect(cleanup.calls).toHaveLength(0); }); @@ -842,7 +909,9 @@ describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { try { const runner = new FakeRunner(); enqueuePreamble(runner); + runner.enqueue(providerMetadata()); runner.enqueue(shellResult(1, "invalid provider update failed")); + runner.enqueue(providerMetadata()); runner.enqueue(shellResult(1, "valid provider restoration failed")); const cleanup = new FakeCleanup(); @@ -868,9 +937,12 @@ describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { it("derives only the expected DCode compatible-endpoint binding", () => { expect(options).toEqual({ gatewayName: "nemoclaw", - providerName: nativeProvider.providerName, - credentialEnv: "COMPATIBLE_API_KEY", + providerName: definition.providerName, + providerId: "dcode-owned-provider", + profileId: definition.profileId, + endpointUrl: definition.endpoint, endpoint: "https://inference-api.nvidia.com/v1/chat/completions", + credentialEnv: "COMPATIBLE_API_KEY", model: "nvidia/nvidia/nemotron-3-ultra", validCredential, }); @@ -895,18 +967,109 @@ describe("LifecyclePhaseFixture DCode invalid-credential rebuild", () => { }, validCredential, ), - ).toThrow(/attached native completion provider/); + ).toThrow(/native hosted provider attachment/); expect(() => dcodeInvalidCredentialRebuildOptionsFromRegistryEntry( { - agent: "langchain-deepagents-code", - gatewayName: "nemoclaw", - provider: "compatible-endpoint", - model: "nvidia/model", - nativeHostedProviderAttachment: { ...attachment, providerName: "wrong-provider" }, + ...entry, + nativeHostedProviderAttachment: { + ...nativeHostedProviderAttachment, + providerName: "wrong-provider", + }, }, validCredential, ), ).toThrow(/native hosted provider attachment/); }); + + it.each([ + { nativeHostedProviderAttachment: undefined }, + { + nativeHostedProviderAttachment: { + ...nativeHostedProviderAttachment, + providerName: "compatible-endpoint", + }, + }, + { nativeHostedProviderAttachment: { ...nativeHostedProviderAttachment, providerId: "" } }, + { + nativeHostedProviderAttachment: { + ...nativeHostedProviderAttachment, + endpointUrl: "https://other.example/v1", + }, + }, + { nativeLocalProviderAttachment: {} }, + { credentialEnv: "OTHER_API_KEY" }, + ])("rejects a changed or missing native credential owner: %j", (change) => { + expect(() => + dcodeInvalidCredentialRebuildOptionsFromRegistryEntry( + { ...entry, ...change }, + validCredential, + ), + ).toThrow(/requires/); + }); + + it("refuses credential mutation and restoration when the provider identity changes", async () => { + const runner = new FakeRunner(); + enqueuePreamble(runner); + runner.enqueue(providerMetadata("replacement-id")); + runner.enqueue(providerMetadata("replacement-id")); + const cleanup = new FakeCleanup(); + await expect( + fixture(runner, cleanup).simulate( + "dcode-rebuild-invalid-credential", + dcodeInstance(), + options, + ), + ).rejects.toThrow(AggregateError); + expect( + runner.calls.filter((call) => call.args[0] === "provider" && call.args[1] === "update"), + ).toHaveLength(0); + expect(cleanup.calls).toHaveLength(1); + }); + + it.each([ + "", + "valid-fixture-credential", + "openshell:resolve:env:v1_COMPATIBLE_API_KEY\n", + "openshell:resolve:env:v1_COMPATIBLE_API_KEY\r\nInjected: x", + ])("rejects an empty, raw, or multiline sandbox credential: %j", async (credential) => { + const runner = new FakeRunner(); + enqueuePreamble(runner); + runner.enqueue(providerMetadata("replacement-id")); + runner.enqueue(providerMetadata("replacement-id")); + await expect( + fixture(runner, new FakeCleanup()).simulate( + "dcode-rebuild-invalid-credential", + dcodeInstance(), + options, + ), + ).rejects.toThrow(AggregateError); + const probe = runner.calls.find( + (call) => call.options?.artifactName === "lifecycle-dcode-route-baseline-1", + )!; + expect(probe.command).toBe("openshell"); + expect(probe.args.slice(0, 6)).toEqual([ + "sandbox", + "exec", + "-n", + sandboxName, + "--", + "/usr/local/lib/nemoclaw/dcode-managed-exec", + ]); + const command = probe.args.slice(6); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "dcode-probe-denial-")); + try { + fs.writeFileSync(path.join(directory, "curl"), "#!/bin/sh\necho unexpected-curl\n", { + mode: 0o700, + }); + const denied = spawnSync(command[0], command.slice(1), { + encoding: "utf8", + env: { PATH: `${directory}:/usr/bin:/bin`, COMPATIBLE_API_KEY: credential }, + }); + expect(denied.status).toBe(2); + expect(denied.stdout).toBe(""); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); }); diff --git a/test/e2e/support/e2e-phase-runtime.test.ts b/test/e2e/support/e2e-phase-runtime.test.ts index 175e354fd59..6326390acf4 100644 --- a/test/e2e/support/e2e-phase-runtime.test.ts +++ b/test/e2e/support/e2e-phase-runtime.test.ts @@ -198,6 +198,33 @@ describe("runtime phase fixture", () => { expect(call?.options?.artifactName).toBe("custom-chat"); }); + it("uses the scoped native router handle and retains completion failure checks", async () => { + const runner = new FakeRunner(); + runner.enqueue( + shellResult(0, JSON.stringify({ choices: [{ message: { content: "routed" } }] })), + ); + await fixture(runner).expectInferenceLocalChatCompletion(instance(), { + modelRouter: true, + model: "nvidia-routed", + }); + const script = runner.calls[0]?.args.join(" ") ?? ""; + expect(script).toContain("NEMOCLAW_LOCAL_INFERENCE_TOKEN"); + expect(script).toContain('"$AUTH_HEADER"'); + expect(script).toContain("http://host.openshell.internal:4000/v1/chat/completions"); + expect(script).not.toContain("https://inference.local"); + }); + + it.each([shellResult(1, "upstream failed"), shellResult(0, '{"choices":[]}')])( + "rejects a failed native router completion %j", + async (response) => { + const runner = new FakeRunner(); + runner.enqueue(response); + await expect( + fixture(runner).expectInferenceLocalChatCompletion(instance(), { modelRouter: true }), + ).rejects.toThrow(); + }, + ); + it("retries inference.local PONG chat completions and accepts reasoning content", async () => { const runner = new FakeRunner(); runner.enqueue( diff --git a/test/e2e/support/e2e-recovery-helpers.test.ts b/test/e2e/support/e2e-recovery-helpers.test.ts index ef9fbe87654..c57a1d7d5d5 100644 --- a/test/e2e/support/e2e-recovery-helpers.test.ts +++ b/test/e2e/support/e2e-recovery-helpers.test.ts @@ -157,6 +157,91 @@ function buildGateway(runner: CommandRunner): GatewayClient { } describe("GatewayClient recovery helpers (#2701)", () => { + it("gateway client proves registration, listener, and host runtime are removed", async () => { + const runner = new ScriptedRunner(); + const privateRuntimeEnv = { ...process.env, HOME: "/private/export-home" }; + runner.queue({ exitCode: 1, stderr: "No active gateway" }); + runner.queue({ exitCode: 1 }); + runner.queue({ exitCode: 1 }); + runner.queue({ exitCode: 0 }); + const gateway = new GatewayClient( + new HostCliClient(runner, { cliPath: "nemoclaw" }), + new SandboxClient(runner), + ); + + await gateway.expectRemoved("nemoclaw", { + artifactName: "final-gateway", + env: privateRuntimeEnv, + gatewayPort: 18_080, + }); + + expect(runner.calls.map(({ command, args }) => [command, args])).toEqual([ + ["openshell", ["status"]], + ["lsof", ["-ti", ":18080", "-sTCP:LISTEN"]], + ["sh", expect.any(Array)], + ["docker", ["container", "ps", "--format", "{{.ID}}\t{{.Names}}"]], + ["true", []], + ]); + expect(runner.calls[0].options).toMatchObject({ + artifactName: "final-gateway-status", + env: { OPENSHELL_GATEWAY: "nemoclaw" }, + }); + expect(runner.calls[1].options).toMatchObject({ + artifactName: "final-gateway-listener", + }); + expect( + runner.calls.slice(2).every((call) => call.options?.env?.HOME === privateRuntimeEnv.HOME), + ).toBe(true); + }); + + it("gateway client rejects an inconclusive listener absence probe", async () => { + const runner = new ScriptedRunner(); + runner.queue({ exitCode: 1, stderr: "Status: Disconnected\nGateway: nemoclaw" }); + runner.queue({ exitCode: 1, stderr: "lsof: command unavailable" }); + const gateway = new GatewayClient( + new HostCliClient(runner, { cliPath: "nemoclaw" }), + new SandboxClient(runner), + ); + + await expect(gateway.expectRemoved("nemoclaw", { gatewayPort: 8_080 })).rejects.toThrow( + "gateway listener still exists or could not be disproved on port 8080", + ); + }); + + it.each([ + { exitCode: 0, stdout: "Status: Connected\nGateway: nemoclaw" }, + { exitCode: 127, stderr: "openshell: command not found" }, + { exitCode: 2, stderr: "Gateway nemoclaw not found: unknown status" }, + { exitCode: 1, stderr: "configuration file does not exist" }, + ])("rejects status that does not prove gateway disconnection: %j", async (status) => { + const runner = new ScriptedRunner(); + runner.queue(status); + const gateway = buildGateway(runner); + await expect(gateway.expectRemoved("nemoclaw", { gatewayPort: 8_080 })).rejects.toThrow( + "openshell status did not prove gateway 'nemoclaw' disconnected", + ); + expect(runner.calls).toHaveLength(1); + }); + + it.each([ + { exitCode: 1, stderr: "daemon unavailable" }, + { exitCode: 0, stderr: "incomplete runtime observation" }, + { exitCode: null, signal: "SIGTERM" as const }, + { exitCode: 0, timedOut: true }, + { exitCode: 0, stdout: "unexpected inventory response" }, + ])("refuses an inconclusive runtime observation after listener absence: %j", async (failure) => { + const runner = new ScriptedRunner(); + runner.queue({ exitCode: 1, stderr: "No active gateway" }); + runner.queue({ exitCode: 1 }); + runner.queue({ exitCode: 1 }); + runner.queue(failure); + const gateway = new GatewayClient( + new HostCliClient(runner, { cliPath: "nemoclaw" }), + new SandboxClient(runner), + ); + await expect(gateway.expectRemoved()).rejects.toThrow("runtime observation failed"); + }); + describe("waitForMissingManagedSupervisor", () => { it("waits for the exact missing-supervisor proof and a quiet settle", async () => { const events: string[] = []; @@ -524,6 +609,23 @@ describe("GatewayClient recovery helpers (#2701)", () => { }); }); + it("uses the selected private home and a distinct supervisor identity artifact", async () => { + const runner = new ScriptedRunner(); + runner.queue({ stdout: "1234 987654 987654 S\n" }); + const gateway = buildGateway(runner); + await gateway.resolveGatewayIdentity( + { sandboxName: "switch-owner" }, + { env: { HOME: "/private/switch-home" }, artifactName: "after-switch-identity" }, + ); + expect(runner.calls[0]).toMatchObject({ + args: expect.arrayContaining(["switch-owner"]), + options: { + env: { HOME: "/private/switch-home", OPENSHELL_GATEWAY: "nemoclaw" }, + artifactName: "after-switch-identity", + }, + }); + }); + it("returns null when the PID probe fails despite valid-looking output", async () => { const runner = new ScriptedRunner(); runner.queue({ exitCode: 1, stdout: "1234 987654 987654 S\n" }); @@ -532,6 +634,16 @@ describe("GatewayClient recovery helpers (#2701)", () => { await expect(gateway.resolveGatewayPid(fakeInstance())).resolves.toBeNull(); }); + it.each([ + { timedOut: true }, + { signal: "SIGTERM" as const }, + { stderr: "incomplete process observation" }, + ])("rejects incomplete identity probes despite valid-looking output: %j", async (failure) => { + const runner = new ScriptedRunner(); + runner.queue({ stdout: "1234 987654 987654 S\n", ...failure }); + await expect(buildGateway(runner).resolveGatewayIdentity(fakeInstance())).resolves.toBeNull(); + }); + it("rejects a reused PID whose process start identity no longer matches", async () => { const runner = new ScriptedRunner(); runner.queue({ stdout: "1234 987654 123456 S\n" }); diff --git a/test/e2e/support/external-gateway-health-helpers.test.ts b/test/e2e/support/external-gateway-health-helpers.test.ts index 2a35f4a7ada..19767c0214a 100644 --- a/test/e2e/support/external-gateway-health-helpers.test.ts +++ b/test/e2e/support/external-gateway-health-helpers.test.ts @@ -4,9 +4,41 @@ import { describe, expect, it } from "vitest"; import { exactGatewayRelease } from "../fixtures/openshell-v0116-qualification.ts"; +import { externalGatewayHealthConfig } from "../fixtures/external-gateway-health-config.ts"; +import { getDockerDriverGatewayLocalTlsBundle } from "../../../src/lib/onboard/docker-driver-gateway-local-tls.ts"; describe("external gateway release qualification", () => { + it("renders v2 public-health TLS without sandbox user credentials (#12603)", () => { + const tls = getDockerDriverGatewayLocalTlsBundle('/private/health "fixture"'); + expect(externalGatewayHealthConfig("192.0.2.10", 4433, tls)).toBe( + String.raw`[openshell] +version = 2 + +[openshell.gateway] +bind_address = "192.0.2.10:4433" +compute_driver = "docker" +disable_tls = false +guest_tls_ca = "/private/health \"fixture\"/tls/ca.crt" + +[openshell.gateway.tls] +cert_path = "/private/health \"fixture\"/tls/server/tls.crt" +key_path = "/private/health \"fixture\"/tls/server/tls.key" + +[openshell.gateway.auth] +allow_unauthenticated_users = true +`, + ); + }); + it.each([ + ["openshell-gateway 0.1.5\n", "0.1.5"], + ["openshell-gateway v0.1.5\n", "0.1.5"], + ["openshell-gateway 0.1.5-dev.1\n", "0.1.5-dev.1"], + ["diagnostic 0.1.2\nopenshell-gateway 0.1.5\n", null], + ["openshell-gateway 0.1.2\n", "0.1.2"], + ["openshell-gateway v0.1.2\n", "0.1.2"], + ["openshell-gateway 0.1.2-dev.1\n", "0.1.2-dev.1"], + ["diagnostic 0.0.116\nopenshell-gateway 0.1.2\n", null], ["openshell-gateway 0.0.116\n", "0.0.116"], ["openshell-gateway v0.0.116\n", "0.0.116"], ["openshell-gateway 0.0.116-dev.1\n", "0.0.116-dev.1"], diff --git a/test/e2e/support/full-e2e-inference-probe.test.ts b/test/e2e/support/full-e2e-inference-probe.test.ts index 8bb79e3713f..a99c04e88d4 100644 --- a/test/e2e/support/full-e2e-inference-probe.test.ts +++ b/test/e2e/support/full-e2e-inference-probe.test.ts @@ -10,6 +10,10 @@ import path from "node:path"; import assert from "node:assert/strict"; import { runInNewContext } from "node:vm"; import { customNativeProvider } from "../../../src/lib/inference/native-provider/custom.ts"; +import { + nativeLocalIdentity, + NATIVE_LOCAL_CREDENTIAL_ENV, +} from "../../../src/lib/inference/native-local/contract.ts"; import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; import { @@ -38,22 +42,52 @@ function completion(content: string, finishReason = "stop"): string { } it.each([ - ["owned-id", "none", true], - ["different-id", "none", false], - ["owned-id", "endpoint", false], - ["owned-id", "api", false], - ["owned-id", "credential", false], - ["owned-id", "model", false], + ["hosted", "owned-id", "none", true], + ["hosted", "different-id", "none", false], + ["hosted", "owned-id", "endpoint", false], + ["hosted", "owned-id", "api", false], + ["hosted", "owned-id", "credential", false], + ["hosted", "owned-id", "model", false], + ["local", "owned-id", "none", true], + ["local", "different-id", "none", false], + ["local", "owned-id", "endpoint", false], + ["local", "owned-id", "api", false], + ["local", "owned-id", "credential", false], + ["local", "owned-id", "model", false], ] as const)( - "checks custom native provider identity %s and agent drift %s on the recorded gateway", - async (observedId, drift, expected) => { - const definition = customNativeProvider({ + "checks %s native provider identity %s and agent drift %s on the recorded gateway", + async (kind, observedId, drift, expected) => { + const binding = { + provider: "compatible-endpoint" as const, + agent: "openclaw" as const, + endpointUrl: "http://host.openshell.internal:42103/v1", + authMode: "authenticated" as const, + credentialEnv: NATIVE_LOCAL_CREDENTIAL_ENV, + gatewayName: "recorded-gateway", + sandboxName: "alpha", + }; + const local = { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1 as const, + providerId: "owned-id", + }; + const hosted = customNativeProvider({ provider: "compatible-endpoint", endpointUrl: "https://custom.example/v1", api: "openai-completions", credentialEnv: "SOURCE_KEY", allowedIps: ["8.8.8.8"], }); + const definition = + kind === "local" + ? { + ...local, + logicalProvider: local.provider, + endpoint: local.endpointUrl, + api: "openai-completions" as const, + } + : hosted; const openshell = vi .fn() .mockResolvedValueOnce({ @@ -63,12 +97,12 @@ it.each([ `Id: ${observedId}`, `Type: ${definition.profileId}`, "Resource version: 1", - "Credential keys: COMPATIBLE_API_KEY", + `Credential keys: ${definition.credentialEnv}`, "Config keys: ", ].join("\n"), }) .mockResolvedValueOnce({ exitCode: 0, stdout: definition.providerName }); - const handle = "openshell:resolve:env:v3_COMPATIBLE_API_KEY"; + const handle = `openshell:resolve:env:v3_${definition.credentialEnv}`; const config = { agents: { defaults: { @@ -94,7 +128,7 @@ it.each([ name === "node:fs" ? { readFileSync: () => JSON.stringify(config) } : assert, process: { argv: ["node", command[3]], - env: { COMPATIBLE_API_KEY: handle }, + env: { [definition.credentialEnv]: handle }, stdout: { write: (value: string) => { stdout += value; @@ -114,18 +148,23 @@ it.each([ model: "selected-model", env: {}, redactionValues: ["synthetic-key"], + timeoutMs: 30_000, entry: { name: "alpha", gatewayName: "recorded-gateway", - nativeHostedProviderAttachment: { - schemaVersion: 1, - profileId: definition.profileId, - providerName: definition.providerName, - providerId: "owned-id", - endpointUrl: definition.endpointUrl, - allowedIps: definition.allowedIps, - custom: definition.custom, - }, + ...(kind === "local" + ? { nativeLocalProviderAttachment: local } + : { + nativeHostedProviderAttachment: { + schemaVersion: 1, + profileId: hosted.profileId, + providerName: hosted.providerName, + providerId: "owned-id", + endpointUrl: hosted.endpointUrl, + allowedIps: hosted.allowedIps, + custom: hosted.custom, + }, + }), }, }); expect(result.ok).toBe(expected); @@ -141,34 +180,53 @@ it.each([ expect(openshell).toHaveBeenNthCalledWith( 1, expect.any(Array), - expect.objectContaining({ redactionValues: ["synthetic-key"], captureLimitBytes: 16 * 1024 }), + expect.objectContaining({ + redactionValues: ["synthetic-key"], + captureLimitBytes: 16 * 1024, + timeoutMs: 30_000, + }), ); }, ); describe("full E2E sandbox inference probe", () => { it.each([ - ["nvidia-prod", "NVIDIA_INFERENCE_API_KEY", "https://integrate.api.nvidia.com/v1"], - ["compatible-endpoint", "COMPATIBLE_API_KEY", "https://custom.example/team/v1"], - // Bootstrap and latency smoke use this hosted endpoint through its native provider. - ["compatible-endpoint", "COMPATIBLE_API_KEY", "https://inference-api.nvidia.com/v1"], - ])( + ["nvidia-prod", "NVIDIA_INFERENCE_API_KEY", "https://integrate.api.nvidia.com/v1", undefined], + ["compatible-endpoint", "COMPATIBLE_API_KEY", "https://custom.example/team/v1", undefined], + ["compatible-endpoint", "COMPATIBLE_API_KEY", "https://inference-api.nvidia.com/v1", undefined], + ["compatible-endpoint", "COMPATIBLE_API_KEY", "https://custom.example/team/v1", 60], + [ + "compatible-endpoint", + "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "http://host.openshell.internal:42103/v1", + 60, + ], + ] as const)( "uses the sandbox native %s handle and preserves the request as data", - (provider, credentialEnv, endpointUrl) => { + (provider, credentialEnv, endpointUrl, maxTimeSeconds) => { const directory = fs.mkdtempSync(path.join(os.tmpdir(), "full-e2e-native-probe-")); try { fs.writeFileSync(path.join(directory, "curl"), '#!/bin/sh\nprintf "%s\\n" "$@"\n', { mode: 0o700, }); const body = JSON.stringify({ prompt: 'literal $(exit 99) "quote"' }); - const [command, ...args] = buildSandboxInferenceChatCommand(body, provider, endpointUrl); + const [command, ...args] = buildSandboxInferenceChatCommand( + body, + provider, + endpointUrl, + maxTimeSeconds, + ); const handle = `openshell:resolve:env:v3_${credentialEnv}`; - const env = { PATH: `${directory}:/usr/bin:/bin`, [credentialEnv]: handle }; + const env = { + HOME: directory, + PATH: `${directory}:/usr/bin:/bin`, + [credentialEnv]: handle, + }; const result = execFileSync(command, args, { env, encoding: "utf8" }); expect(result.split("\n")).toEqual([ "-fsS", "--max-time", - "90", + String(maxTimeSeconds ?? 90), `${endpointUrl}/chat/completions`, "-H", "Content-Type: application/json", @@ -184,9 +242,20 @@ describe("full E2E sandbox inference probe", () => { }, ); - it.each(["", "nvapi-raw-key", "$(exit 99)"])( - "rejects the invalid native credential %j before invoking curl", - (invalid) => { + it.each([ + ["nvidia-prod", ""], + ["nvidia-prod", "nvapi-raw-key"], + ["nvidia-prod", "$(exit 99)"], + ["nvidia-prod", "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN"], + ["nvidia-prod", "openshell:resolve:env:v3_NEMOCLAW_LOCAL_INFERENCE_TOKEN\nraw"], + ["compatible-endpoint", ""], + ["compatible-endpoint", "nvapi-raw-key"], + ["compatible-endpoint", "$(exit 99)"], + ["compatible-endpoint", "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN"], + ["compatible-endpoint", "openshell:resolve:env:v3_NEMOCLAW_LOCAL_INFERENCE_TOKEN\nraw"], + ] as const)( + "rejects the invalid %s native credential %j before invoking curl", + (provider, invalid) => { const directory = fs.mkdtempSync(path.join(os.tmpdir(), "full-e2e-invalid-probe-")); try { const marker = path.join(directory, "curl-invoked"); @@ -197,12 +266,18 @@ describe("full E2E sandbox inference probe", () => { mode: 0o700, }, ); - const [, ...args] = buildSandboxInferenceChatCommand("{}", "nvidia-prod"); + const [, ...args] = buildSandboxInferenceChatCommand( + "{}", + provider, + "http://host.openshell.internal:42103/v1", + ); const denied = spawnSync("/bin/bash", args, { env: { HOME: directory, PATH: `${directory}:/usr/bin:/bin`, - NVIDIA_INFERENCE_API_KEY: invalid, + [provider === "nvidia-prod" + ? "NVIDIA_INFERENCE_API_KEY" + : "NEMOCLAW_LOCAL_INFERENCE_TOKEN"]: invalid, CURL_MARKER: marker, }, encoding: "utf8", diff --git a/test/e2e/support/full-e2e-native-model.test.ts b/test/e2e/support/full-e2e-native-model.test.ts index 252a9a12334..a805e0f5adf 100644 --- a/test/e2e/support/full-e2e-native-model.test.ts +++ b/test/e2e/support/full-e2e-native-model.test.ts @@ -112,3 +112,17 @@ it("retains a custom native endpoint and opaque credential reference through a m models: [{ id: "team/model", name: "team/model" }], }); }); + +it("retains the local endpoint and runtime credential through a model edit", () => { + const fixture = buildNativeModelRestartFixture( + "team/model", + "compatible-endpoint", + "http://host.openshell.internal:42103/v1", + ); + expect(JSON.parse(fixture.patch).models.providers[fixture.provider]).toMatchObject({ + baseUrl: "http://host.openshell.internal:42103/v1", + apiKey: "${NEMOCLAW_LOCAL_INFERENCE_TOKEN}", + api: "openai-completions", + models: [{ id: "team/model", name: "team/model" }], + }); +}); diff --git a/test/e2e/support/gpu-double-onboard-cleanup.test.ts b/test/e2e/support/gpu-double-onboard-cleanup.test.ts index 5850d2d5dfb..50739c8b303 100644 --- a/test/e2e/support/gpu-double-onboard-cleanup.test.ts +++ b/test/e2e/support/gpu-double-onboard-cleanup.test.ts @@ -22,11 +22,18 @@ async function runCleanupPaths() { const run = captured.test.mock.calls[0]![2] as (fixtures: object) => Promise; const stop = new Error("stop before onboarding"); const disposables = new Map Promise>(); - const host = { command: vi.fn().mockResolvedValue({ exitCode: 0, stdout: "", stderr: "" }) }; + const host = { + command: vi.fn().mockResolvedValue({ exitCode: 0, stdout: "", stderr: "" }), + openshellCommandPath: "/fixture/openshell", + isCommandAvailable: vi.fn().mockResolvedValue(true), + cleanupGatewayRegistration: vi.fn().mockResolvedValue(undefined), + }; + const lifecycle = { stopGatewayRuntime: vi.fn().mockResolvedValue(undefined) }; const execution = run({ artifacts: { target: { declare: vi.fn() } }, runtimeProvider: { requireAvailable: vi.fn() }, host, + lifecycle, sandbox: { openshell: vi.fn().mockResolvedValue({ exitCode: 0 }) }, cleanup: { trackDisposable: (name: string, dispose: () => Promise) => @@ -45,6 +52,16 @@ async function runCleanupPaths() { skip: vi.fn(), }); await expect(execution).rejects.toBe(stop); + expect(lifecycle.stopGatewayRuntime).toHaveBeenCalledWith( + expect.objectContaining({ userServiceMode: "permanent" }), + ); + expect(host.cleanupGatewayRegistration).toHaveBeenCalledWith( + "nemoclaw", + expect.objectContaining({ artifactName: "pre-cleanup-openshell-gateway" }), + ); + expect(lifecycle.stopGatewayRuntime.mock.invocationCallOrder[0]).toBeLessThan( + host.cleanupGatewayRegistration.mock.invocationCallOrder[0]!, + ); const dispose = disposables.get("stop gpu double-onboard Ollama processes")!; return { host, dispose }; } diff --git a/test/e2e/support/gpu-e2e-helpers.test.ts b/test/e2e/support/gpu-e2e-helpers.test.ts index 8ce1227225b..ec67b17808a 100644 --- a/test/e2e/support/gpu-e2e-helpers.test.ts +++ b/test/e2e/support/gpu-e2e-helpers.test.ts @@ -2,9 +2,21 @@ // SPDX-License-Identifier: Apache-2.0 import assert from "node:assert/strict"; +import { NEMOCLAW_CREATE_ATTEMPT_NONCE_HEX_LENGTH } from "../../../src/lib/adapters/openshell/sandbox-identity.ts"; +import { captureGpuInstallFailure } from "../fixtures/gpu-install-failure.ts"; import { ChildProcess, execFileSync } from "node:child_process"; import { once } from "node:events"; -import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import fs, { + appendFileSync, + renameSync, + chmodSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, + symlinkSync, +} from "node:fs"; import { createServer } from "node:net"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -26,6 +38,7 @@ import { ollamaProxyTokenFile, openClawModelConfigProjectionScript, REPO_ROOT, + SANDBOX_NAME, startAttachedOllama, trackGpuGatewayCleanup, waitForAttachedOllama, @@ -44,6 +57,268 @@ import { const GPU_MODEL = "qwen3.5:9b"; const SYNC_E2E_CHILD_OPTIONS = { killSignal: "SIGKILL" as const, timeout: 5_000 }; +describe("GPU install failure evidence", () => { + const probe = { + command: [], + signal: null, + timedOut: false, + artifacts: { stdout: "", stderr: "", result: "" }, + }; + const attempt = "a".repeat(NEMOCLAW_CREATE_ATTEMPT_NONCE_HEX_LENGTH); + const marker = `Create-attempt label: ai.nvidia.nemoclaw.create-attempt=${attempt}`; + const status = { + name: SANDBOX_NAME, + labels: { "ai.nvidia.nemoclaw.create-attempt": attempt }, + phase: "Error", + exit_code: 7, + restart_count: 0, + main_process_instance_id: "private-runtime-identity", + main_process_started_at_ms: 1234, + next_restart_at_ms: null, + conditions: [{ message: "private-error-token" }], + policy: { env: { TOKEN: "private-policy-token" } }, + }; + + it.each([SANDBOX_NAME, "e2e-hermes-gpu"])( + "projects only current-attempt process fields for %s", + async (sandboxName) => { + const openshell = vi.fn(async () => ({ + ...probe, + exitCode: 0, + stdout: JSON.stringify({ ...status, name: sandboxName }), + stderr: "", + })); + const artifacts = { writeText: vi.fn(), writeJson: vi.fn() }; + const install = { + exitCode: 1, + stdout: marker, + stderr: "", + startedAt: new Date().toISOString(), + finishedAt: new Date().toISOString(), + }; + await captureGpuInstallFailure(install, { openshell }, artifacts, {}, sandboxName); + expect(openshell).toHaveBeenCalledWith(["sandbox", "get", sandboxName, "--output", "json"], { + env: {}, + timeoutMs: 15_000, + captureLimitBytes: 64 * 1024, + persistArtifacts: false, + }); + expect(artifacts.writeJson).toHaveBeenCalledExactlyOnceWith( + "gpu-install-failure-process.json", + { + attempt, + phase: "Error", + mainProcessObserved: true, + exit_code: 7, + restart_count: 0, + main_process_started_at_ms: 1234, + }, + ); + expect(install.exitCode).toBe(1); + }, + ); + + it.each([ + ["success", 0, marker, status], + ["missing attempt", 1, "", status], + [ + "ambiguous attempt", + 1, + `${marker}\nai.nvidia.nemoclaw.create-attempt=${"b".repeat(NEMOCLAW_CREATE_ATTEMPT_NONCE_HEX_LENGTH)}`, + status, + ], + ["renamed sandbox", 1, marker, { ...status, name: "other" }], + ["stale attempt", 1, marker, { ...status, labels: {} }], + ["invalid response", 1, marker, null], + ])("retains no status for %s", async (_name, exitCode, stdout, observed) => { + const openshell = vi.fn(async () => ({ + ...probe, + exitCode: 0, + stdout: JSON.stringify(observed), + stderr: "private-command-error", + })); + const artifacts = { writeText: vi.fn(), writeJson: vi.fn() }; + await captureGpuInstallFailure( + { + exitCode, + stdout, + stderr: "", + startedAt: "", + finishedAt: "", + }, + { openshell }, + artifacts, + {}, + SANDBOX_NAME, + ); + expect(artifacts.writeJson).not.toHaveBeenCalled(); + expect(artifacts.writeText).not.toHaveBeenCalled(); + }); + + it.each([61, 63, 64])("refuses a %i-character attempt before querying status", async (length) => { + const invalidAttempt = "a".repeat(length); + const openshell = vi.fn(); + const artifacts = { writeText: vi.fn(), writeJson: vi.fn() }; + await captureGpuInstallFailure( + { exitCode: 1, stdout: `ai.nvidia.nemoclaw.create-attempt=${invalidAttempt}`, stderr: "" }, + { openshell }, + artifacts, + {}, + SANDBOX_NAME, + ); + expect(openshell).not.toHaveBeenCalled(); + expect(artifacts.writeText).not.toHaveBeenCalled(); + expect(artifacts.writeJson).not.toHaveBeenCalled(); + }); + + it.each([ + { timedOut: true, signal: null }, + { timedOut: false, signal: "SIGTERM" as const }, + ])("rejects incomplete status results %j", async (completion) => { + const openshell = vi.fn(async () => ({ + ...probe, + ...completion, + exitCode: 0, + stdout: JSON.stringify(status), + stderr: "", + })); + const artifacts = { writeText: vi.fn(), writeJson: vi.fn() }; + await captureGpuInstallFailure( + { exitCode: 1, stdout: marker, stderr: "" }, + { openshell }, + artifacts, + {}, + SANDBOX_NAME, + ); + expect(artifacts.writeJson).not.toHaveBeenCalled(); + }); + + it.each([ + [ + "private current bundle", + (file: string) => writeFileSync(file, "running=false exit_code=7", { mode: 0o600 }), + true, + false, + ], + ["symlink", (file: string) => symlinkSync("/etc/passwd", file), false, false], + [ + "public file", + (file: string) => writeFileSync(file, "private-canary", { mode: 0o644 }), + false, + false, + ], + [ + "oversized file", + (file: string) => writeFileSync(file, "x".repeat(8193), { mode: 0o600 }), + false, + false, + ], + [ + "stale bundle", + (file: string) => writeFileSync(file, "private-canary", { mode: 0o600 }), + false, + true, + ], + ] as const)( + "copies only the %s without changing the failure", + async (_name, prepare, copied, stale) => { + const home = mkdtempSync(path.join(tmpdir(), "gpu-failure-evidence-")); + const directory = path.join( + home, + ".nemoclaw", + "onboard-failures", + `${new Date().toISOString().replace(/[:.]/gu, "-")}-${SANDBOX_NAME}`, + ); + try { + mkdirSync(directory, { recursive: true, mode: 0o700 }); + prepare(path.join(directory, "sandbox-container.log")); + const openshell = vi.fn(async () => { + throw new Error("private-diagnostic-error"); + }); + const artifacts = { writeText: vi.fn(), writeJson: vi.fn() }; + const install = { + exitCode: 1, + stdout: `${marker}\nDiagnostics saved: ${directory}`, + stderr: "", + startedAt: new Date(stale ? Date.now() + 60_000 : Date.now() - 60_000).toISOString(), + finishedAt: new Date(Date.now() + 60_000).toISOString(), + }; + await captureGpuInstallFailure( + install, + { openshell }, + artifacts, + { HOME: home }, + SANDBOX_NAME, + ); + expect(artifacts.writeText.mock.calls).toEqual( + copied ? [["gpu-install-failure-container.log", "running=false exit_code=7"]] : [], + ); + expect(artifacts.writeJson).not.toHaveBeenCalled(); + expect(install.exitCode).toBe(1); + } finally { + rmSync(home, { recursive: true, force: true }); + } + }, + ); + it.each([ + ["file growth", (file: string) => appendFileSync(file, "private-canary".repeat(1000))], + [ + "parent replacement", + (file: string) => { + const directory = path.dirname(file); + renameSync(directory, `${directory}-moved`); + symlinkSync(`${directory}-moved`, directory); + }, + ], + ] as const)("rejects %s during the bounded read", async (_name, mutate) => { + const home = mkdtempSync(path.join(tmpdir(), "gpu-failure-race-")); + const directory = path.join( + home, + ".nemoclaw", + "onboard-failures", + `${new Date().toISOString().replace(/[:.]/gu, "-")}-${SANDBOX_NAME}`, + ); + try { + mkdirSync(directory, { recursive: true, mode: 0o700 }); + const file = path.join(directory, "sandbox-container.log"); + writeFileSync(file, "running=false", { mode: 0o600 }); + const actualRead = fs.readSync; + const read = vi.spyOn(fs, "readSync").mockImplementationOnce((fd, buffer, options) => { + mutate(file); + return actualRead(fd, buffer, options); + }); + const artifacts = { writeText: vi.fn(), writeJson: vi.fn() }; + await captureGpuInstallFailure( + { + exitCode: 1, + stdout: `${marker}\nDiagnostics saved: ${directory}`, + stderr: "", + startedAt: new Date(Date.now() - 60_000).toISOString(), + finishedAt: new Date(Date.now() + 60_000).toISOString(), + }, + { + openshell: async () => { + throw new Error("private-error"); + }, + }, + artifacts, + { HOME: home }, + SANDBOX_NAME, + ); + expect(read).toHaveBeenCalledWith(expect.any(Number), expect.any(Buffer), { + offset: 0, + length: 8193, + position: 0, + }); + expect(artifacts.writeText).not.toHaveBeenCalled(); + expect(artifacts.writeJson).not.toHaveBeenCalled(); + } finally { + vi.restoreAllMocks(); + rmSync(home, { recursive: true, force: true }); + } + }); +}); + function loopbackPort(server: ReturnType): number { const address = server.address(); assert(address && typeof address !== "string", "loopback server has no TCP port"); diff --git a/test/e2e/support/gpu-e2e-native-route.test.ts b/test/e2e/support/gpu-e2e-native-route.test.ts new file mode 100644 index 00000000000..bc410822d32 --- /dev/null +++ b/test/e2e/support/gpu-e2e-native-route.test.ts @@ -0,0 +1,223 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { + nativeLocalIdentity, + type NativeLocalBinding, +} from "../../../src/lib/inference/native-local/contract.ts"; +import type { SandboxEntry } from "../../../src/lib/state/registry.ts"; +import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; +import { gpuOllamaChatCommand, readGpuOllamaRoute, SANDBOX_NAME } from "../live/gpu-e2e-helpers.ts"; + +const binding = { + provider: "ollama-local" as const, + agent: "openclaw" as const, + endpointUrl: "http://host.openshell.internal:11435/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated" as const, + gatewayName: "gpu-gateway", + sandboxName: SANDBOX_NAME, +}; +const receipt = { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1 as const, + providerId: "gpu-provider-id", +}; +const environment = { OPENSHELL_GATEWAY: "gpu-gateway" }; + +function changedReceipt(changes: Partial) { + const selected = { ...binding, ...changes }; + return { ...receipt, ...selected, ...nativeLocalIdentity(selected) }; +} +const metadata = [ + `Name: ${receipt.providerName}`, + `Id: ${receipt.providerId}`, + "Resource version: 1", + `Type: ${receipt.profileId}`, + "Credential keys: NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "Config keys: ", +].join("\n"); + +function entry(overrides: Record = {}): SandboxEntry { + return { + name: SANDBOX_NAME, + agent: null, + provider: "ollama-local", + nativeLocalProviderAttachment: receipt, + ...overrides, + } as SandboxEntry; +} + +function client(providerText = metadata, attachedText = receipt.providerName, exitCode = 0) { + const openshell = vi + .fn() + .mockResolvedValueOnce({ exitCode, stdout: providerText, stderr: "" }) + .mockResolvedValueOnce({ exitCode: 0, stdout: attachedText, stderr: "" }); + return { openshell, sandbox: { openshell } as unknown as SandboxClient }; +} + +describe("GPU Ollama native route evidence", () => { + it("observes the scoped provider identity and sandbox attachment", async () => { + const { sandbox, openshell } = client(); + expect(await readGpuOllamaRoute(sandbox, entry(), environment)).toEqual(receipt); + expect(openshell.mock.calls).toEqual([ + [ + ["provider", "get", "-g", "gpu-gateway", receipt.providerName], + { + env: environment, + timeoutMs: 30_000, + captureLimitBytes: 16 * 1024, + artifactName: "gpu-ollama-native-provider", + }, + ], + [ + ["sandbox", "provider", "list", "-g", "gpu-gateway", SANDBOX_NAME], + { + env: environment, + timeoutMs: 30_000, + captureLimitBytes: 16 * 1024, + artifactName: "gpu-ollama-native-attachment", + }, + ], + ]); + }); + + it.each([ + ["missing receipt", { nativeLocalProviderAttachment: undefined }], + ["wrong sandbox", { name: "e2e-other" }], + ["wrong registry gateway", { gatewayName: "other" }], + ["wrong agent", { agent: "hermes" }], + ["wrong provider", { provider: "vllm-local" }], + ["wrong receipt agent", { nativeLocalProviderAttachment: changedReceipt({ agent: "hermes" }) }], + ["sentinel auth", { nativeLocalProviderAttachment: changedReceipt({ authMode: "sentinel" }) }], + [ + "different credential authority", + { nativeLocalProviderAttachment: changedReceipt({ credentialEnv: "OTHER_KEY" }) }, + ], + [ + "wrong receipt identity", + { nativeLocalProviderAttachment: { ...receipt, providerName: "other" } }, + ], + [ + "wrong receipt sandbox", + { nativeLocalProviderAttachment: { ...receipt, sandboxName: "e2e-other" } }, + ], + [ + "wrong credential binding", + { nativeLocalProviderAttachment: { ...receipt, credentialEnv: "OTHER_KEY" } }, + ], + ])("rejects %s before querying the runtime", async (_label, overrides) => { + const { sandbox, openshell } = client(); + expect( + await readGpuOllamaRoute(sandbox, entry(overrides as Record), environment), + ).toBeNull(); + expect(openshell).not.toHaveBeenCalled(); + }); + + it("rejects a different gateway", async () => { + const { sandbox, openshell } = client(); + expect(await readGpuOllamaRoute(sandbox, entry(), { OPENSHELL_GATEWAY: "other" })).toBeNull(); + expect(openshell).not.toHaveBeenCalled(); + }); + + it.each([ + ["provider name", `Name: ${receipt.providerName}`, "Name: other"], + ["immutable id", `Id: ${receipt.providerId}`, "Id: other-id"], + ["profile", `Type: ${receipt.profileId}`, "Type: openai"], + ["credential", "Credential keys: NEMOCLAW_LOCAL_INFERENCE_TOKEN", "Credential keys: OTHER_KEY"], + [ + "extra credential", + "Credential keys: NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "Credential keys: NEMOCLAW_LOCAL_INFERENCE_TOKEN, OTHER_KEY", + ], + ["legacy config", "Config keys: ", "Config keys: OPENAI_BASE_URL"], + ])("rejects mismatched runtime %s", async (_label, original, replacement) => { + const { sandbox } = client(metadata.replace(original, replacement)); + expect(await readGpuOllamaRoute(sandbox, entry(), environment)).toBeNull(); + }); + + it("rejects failed provider reads even when stdout looks valid", async () => { + const { sandbox } = client(metadata, receipt.providerName, 1); + expect(await readGpuOllamaRoute(sandbox, entry(), environment)).toBeNull(); + }); + + it("rejects a provider attached only under a different name", async () => { + const { sandbox } = client(metadata, `${receipt.providerName}-other`); + expect(await readGpuOllamaRoute(sandbox, entry(), environment)).toBeNull(); + }); + + it("rejects a failed attachment read", async () => { + const { sandbox, openshell } = client(); + openshell + .mockReset() + .mockResolvedValueOnce({ exitCode: 0, stdout: metadata }) + .mockResolvedValueOnce({ exitCode: 1, stdout: receipt.providerName }); + expect(await readGpuOllamaRoute(sandbox, entry(), environment)).toBeNull(); + }); +}); + +describe("GPU Ollama native request", () => { + let directory: string; + beforeEach(() => { + directory = mkdtempSync(path.join(os.tmpdir(), "gpu-native-request-")); + writeFileSync(path.join(directory, "curl"), '#!/bin/sh\nprintf "%s\\n" "$@"\n', { + mode: 0o755, + }); + }); + afterEach(() => rmSync(directory, { recursive: true, force: true })); + + function run(handle: string, model = "qwen3.5:9b") { + return execFileSync( + "/bin/bash", + ["--noprofile", "--norc", "-c", gpuOllamaChatCommand(receipt, model)], + { + encoding: "utf8", + env: { + HOME: directory, + PATH: `${directory}:/usr/bin:/bin`, + NEMOCLAW_LOCAL_INFERENCE_TOKEN: handle, + }, + }, + ); + } + + it("sends the scoped handle and quoted payload to the native endpoint", () => { + const handle = "openshell:resolve:env:v2_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; + const model = "model'with quote"; + const args = run(handle, model).trim().split("\n"); + expect(args).toEqual([ + "-sS", + "--max-time", + "120", + `${receipt.endpointUrl}/chat/completions`, + "-H", + "Content-Type: application/json", + "-H", + `Authorization: Bearer ${handle}`, + "--data", + JSON.stringify({ + model, + messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], + reasoning_effort: "none", + seed: 0, + temperature: 0, + max_tokens: 32, + }), + ]); + }); + + it.each([ + "", + "raw-proxy-secret", + "openshell:resolve:env:OTHER_KEY", + "openshell:resolve:env:v2_NEMOCLAW_LOCAL_INFERENCE_TOKEN\nraw", + ])("rejects an invalid credential handle %j before curl", (handle) => { + expect(() => run(handle)).toThrow(); + }); +}); diff --git a/test/e2e/support/hermes-acp-live.test.ts b/test/e2e/support/hermes-acp-live.test.ts index ab5f1d4e88a..099250e28e2 100644 --- a/test/e2e/support/hermes-acp-live.test.ts +++ b/test/e2e/support/hermes-acp-live.test.ts @@ -22,6 +22,7 @@ import { isAcpResponse, isProcessAbsent, runHermesAcpLiveScenario, + verifyNoRemoteHermesAcpProcess, } from "../fixtures/hermes-acp-live.ts"; describe("Hermes ACP live evidence boundary", () => { @@ -47,6 +48,24 @@ describe("Hermes ACP live evidence boundary", () => { artifacts: { stdout: "", stderr: "", result: "" }, }); + it.each([ + ["empty pgrep result", {}, true], + [ + "Provisioning transport refusal", + { stderr: "Error: sandbox is not ready (phase: Provisioning)" }, + false, + ], + ["timeout", { timedOut: true }, false], + ["signal", { signal: "SIGTERM" as const }, false], + ["unexpected process output", { stdout: "1234\n" }, false], + ])("requires actual remote process absence for %s", async (_label, overrides, expected) => { + const run = vi.fn().mockResolvedValue(shellResult({ exitCode: 1, ...overrides })); + expect( + await verifyNoRemoteHermesAcpProcess(new SandboxClient({ run }), "e2e-hermes", {}, "cleanup"), + ).toBe(expected); + expect(run).toHaveBeenCalledTimes(1); + }); + it("recognizes the OpenShell 0.0.116 stopped-gateway response (#10947)", () => { expect( hermesAcpGatewayStoppedPreconditionPassed( @@ -153,7 +172,7 @@ readline.createInterface({ input: process.stdin }).on("line", line => { fs.rmSync(artifactDir, { force: true, recursive: true }); }); const sandbox = new SandboxClient({ - run: vi.fn().mockResolvedValue({ exitCode: 1, stdout: "", stderr: "" }), + run: vi.fn().mockResolvedValue(shellResult({ exitCode: 1 })), }); await expect( @@ -294,7 +313,7 @@ setTimeout(() => { env: {}, progress, sandbox: new SandboxClient({ - run: vi.fn().mockResolvedValue({ exitCode: 1, stdout: "", stderr: "" }), + run: vi.fn().mockResolvedValue(shellResult({ exitCode: 1 })), }), sandboxName: "e2e-hermes", scenario: "initialize", @@ -329,7 +348,7 @@ setTimeout(() => { progress.stop(); fs.rmSync(artifactDir, { force: true, recursive: true }); }); - const run = vi.fn().mockResolvedValue({ exitCode: 1, stdout: "", stderr: "" }); + const run = vi.fn().mockResolvedValue(shellResult({ exitCode: 1 })); const sandbox = new SandboxClient({ run }); await expect( diff --git a/test/e2e/support/hermes-discord-policy-binding.test.ts b/test/e2e/support/hermes-discord-policy-binding.test.ts index 4094893b6db..9f8d433ab92 100644 --- a/test/e2e/support/hermes-discord-policy-binding.test.ts +++ b/test/e2e/support/hermes-discord-policy-binding.test.ts @@ -18,6 +18,10 @@ import { } from "../fixtures/gateway-providers.ts"; import { requireSuccessfulPolicyBoundaryBuild } from "../fixtures/hermes-discord-policy-boundary-build.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; +import { + applyRestRewritePolicy, + applyWebSocketRewritePolicy, +} from "../live/messaging-providers-helpers.ts"; const HELPER = path.resolve(import.meta.dirname, "../fixtures/gateway-provider-policy-binding.ts"); const TYPESCRIPT = path.resolve("node_modules/typescript/bin/tsc"); @@ -175,13 +179,69 @@ describe("Hermes Discord E2E policy binding", () => { expect(applied).toEqual([]); }); + const applySharedPolicy = ( + host: HostCliClient, + protocol: "rest" | "websocket", + restMethods?: readonly ("GET" | "POST")[], + ) => + applyFixtureProviderPolicyEndpoint(host, "e2e-messaging", { + artifactName: "apply-fake-messaging-policy", + endpoint: { port: 43119 }, + protocol, + rewrite: { + rest: "request-body-credential-rewrite", + websocket: "websocket-credential-rewrite", + }[protocol] as "request-body-credential-rewrite" | "websocket-credential-rewrite", + env: {}, + providerName: "messaging-provider", + redactionValues: [], + restMethods, + }); + const applyMessagingPolicy = (host: HostCliClient, apply: typeof applyRestRewritePolicy) => + apply( + host, + { kind: "discord-gateway", port: "43119", captureFile: "unused" }, + "messaging-provider", + "DISCORD_BOT_TOKEN", + { DISCORD_BOT_TOKEN: "fixture-token" }, + ["fixture-token"], + ); + it.each([ - { requested: undefined, expected: ["GET", "POST"] }, - { requested: ["POST"] as const, expected: ["POST"] }, - ])( - "renders only declared REST methods in a rebuild-valid fixture policy [case %#]", - async ({ requested, expected }) => { - const providerName = "telegram-provider"; + { + caller: "shared", + protocol: "rest", + expected: ["GET", "POST"], + apply: (host: HostCliClient) => applySharedPolicy(host, "rest"), + }, + { + caller: "shared", + protocol: "rest", + expected: ["POST"], + apply: (host: HostCliClient) => applySharedPolicy(host, "rest", ["POST"]), + }, + { + caller: "shared", + protocol: "websocket", + expected: ["GET", "WEBSOCKET_TEXT"], + apply: (host: HostCliClient) => applySharedPolicy(host, "websocket"), + }, + { + caller: "messaging", + protocol: "rest", + expected: ["GET", "POST"], + apply: (host: HostCliClient) => applyMessagingPolicy(host, applyRestRewritePolicy), + }, + { + caller: "messaging", + protocol: "websocket", + expected: ["GET", "WEBSOCKET_TEXT"], + apply: (host: HostCliClient) => applyMessagingPolicy(host, applyWebSocketRewritePolicy), + }, + ] as const)( + "applies $caller $protocol fixture rules without mixing endpoint and rule updates [case %#]", + async ({ protocol, expected, apply }) => { + const providerName = "messaging-provider"; const source = YAML.stringify({ version: 1, network_policies: { @@ -191,50 +251,61 @@ describe("Hermes Discord E2E policy binding", () => { { host: "host.openshell.internal", port: 43119, - protocol: "rest", - rules: ["GET", "HEAD", "OPTIONS", "POST", "PUT", "PATCH"].map((method) => ({ - allow: { method, path: "**" }, - })), + protocol, + access: "read-write", }, ], binaries: [{ path: "/usr/local/bin/node" }, { path: "/usr/bin/node" }], }, }, }); - expect(() => parseAndValidateSandboxPolicy(source)).toThrow("sandbox policy schema"); const applied: string[] = []; + const policyPaths: string[] = []; const command = vi .fn() - .mockResolvedValueOnce(successfulProbe(providerName)) - .mockResolvedValueOnce(successfulProbe()) - .mockResolvedValueOnce(successfulProbe(source)) - .mockImplementationOnce(async (_command, args = []) => { - applied.push(fs.readFileSync(args[args.indexOf("--policy") + 1], "utf8")); - return successfulProbe(); + .mockImplementation(async (_command, args = []) => { + // OpenShell 0.1.5 rejects mixed endpoint and rule updates before gateway access. + expect(args.includes("--add-endpoint") && args.includes("--add-allow")).toBe(false); + const responses: Record ShellProbeResult> = { + "sandbox provider": () => successfulProbe(providerName), + "policy update": () => successfulProbe(), + "policy get": () => successfulProbe(source), + "policy set": () => { + const policyPath = args[args.indexOf("--policy") + 1]; + policyPaths.push(policyPath); + applied.push(fs.readFileSync(policyPath, "utf8")); + return successfulProbe(); + }, + }; + return responses[args.slice(0, 2).join(" ")](); }); const host = { command, openshellCommandPath: "/usr/local/bin/openshell", } as unknown as HostCliClient; - await applyFixtureProviderPolicyEndpoint(host, "e2e-telegram", { - artifactName: "apply-fake-telegram-policy", - endpoint: { port: 43119 }, - protocol: "rest", - rewrite: "request-body-credential-rewrite", - env: {}, - providerName, - redactionValues: [], - restMethods: requested, - }); + await apply(host); expect(applied).toHaveLength(1); const policy = parseAndValidateSandboxPolicy(applied[0]); expect(policy).toMatchObject({ network_policies: { fixture: { - endpoints: [{ rules: expected.map((method) => ({ allow: { method, path: "/**" } })) }], + endpoints: [ + { + host: "host.openshell.internal", + port: 43119, + protocol, + credential_binding: { provider: providerName }, + rules: expected.map((method) => ({ allow: { method, path: "/**" } })), + }, + ], + binaries: [{ path: "/usr/local/bin/node" }, { path: "/usr/bin/node" }], }, }, }); + expect(YAML.parse(applied[0]).network_policies.fixture.endpoints[0]).not.toHaveProperty( + "access", + ); + expect(policyPaths.every((policyPath) => !fs.existsSync(policyPath))).toBe(true); }, ); diff --git a/test/e2e/support/hermes-gpu-startup-fallback.test.ts b/test/e2e/support/hermes-gpu-startup-fallback.test.ts index 97f116b7f00..ddfebf46d0c 100644 --- a/test/e2e/support/hermes-gpu-startup-fallback.test.ts +++ b/test/e2e/support/hermes-gpu-startup-fallback.test.ts @@ -111,6 +111,26 @@ describe("Hermes GPU startup failure diagnostics", () => { "Docker GPU diagnostics saved: /tmp/docker", "/tmp/docker", ], + [ + "recognizes generic startup diagnostics", + " Diagnostics saved: /tmp/onboard-failure\n", + "/tmp/onboard-failure", + ], + [ + "uses the later generic bundle", + "Native GPU diagnostics saved: /tmp/native\n Diagnostics saved: /tmp/latest", + "/tmp/latest", + ], + [ + "uses the latest repeated marker", + "Diagnostics saved: /tmp/old\nDiagnostics saved: /tmp/new", + "/tmp/new", + ], + [ + "follows chronology across marker categories", + "Pre-rollback diagnostics saved: /tmp/old\nNative GPU diagnostics saved: /tmp/new", + "/tmp/new", + ], ["returns empty without a bundle", "GPU setup failed before diagnostics\n", ""], ])("extracts %s", (_name, output, expected) => { expect(extractHermesGpuDiagnosticsDirectory(output)).toBe(expected); diff --git a/test/e2e/support/hermes-gpu-startup-integrity.test.ts b/test/e2e/support/hermes-gpu-startup-integrity.test.ts index d942ac79703..e4fdcac05ae 100644 --- a/test/e2e/support/hermes-gpu-startup-integrity.test.ts +++ b/test/e2e/support/hermes-gpu-startup-integrity.test.ts @@ -83,6 +83,7 @@ function runProof(fixture: IntegrityFixture, extraEnv: NodeJS.ProcessEnv = {}) { [ "-c", buildHermesManagedStartupIntegrityScript({ + pythonPath: "python3", configPath: fixture.configPath, envPath: fixture.envPath, strictHashPath: fixture.strictHashPath, @@ -96,6 +97,7 @@ function runProof(fixture: IntegrityFixture, extraEnv: NodeJS.ProcessEnv = {}) { encoding: "utf-8", timeout: 5000, env: { + PATH: process.env.PATH, LANG: "C", LC_ALL: "C", ...extraEnv, @@ -120,6 +122,71 @@ describe("Hermes managed startup integrity proof", () => { expect(proof.stdout).toBe("OK\n"); }); + it.each([ + { scenario: "valid", accepted: true }, + { scenario: "changed-model", model: "changed", accepted: false }, + { scenario: "changed-user-slot", userKey: "changed", accepted: false }, + { scenario: "invalid-handle", reference: "unscoped-secret", accepted: false }, + { scenario: "forged-witness", witnessSource: "current", accepted: false }, + { scenario: "writable-witness", witnessMode: 0o666, accepted: false }, + ])("checks native startup credential materialization with $scenario", (input) => { + const fixture = createFixture(); + const token = "${NEMOCLAW_LOCAL_INFERENCE_TOKEN}"; + const provider = "compatible-endpoint"; + const endpoint = "http://host.openshell.internal:8000/v1"; + const base = { + _nemoclaw_upstream: { provider, provider_key: provider }, + model: { provider: "custom", base_url: endpoint, api_key: token, default: "test-model" }, + providers: { [provider]: { api: endpoint, api_key: token }, user: { api_key: token } }, + custom_providers: [{ name: provider, base_url: endpoint, api_key: token }], + }; + const original = JSON.stringify(base); + const current = structuredClone(base); + const reference = input.reference ?? "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; + current.model.api_key = reference; + current.providers[provider]!.api_key = reference; + current.custom_providers[0]!.api_key = reference; + current.model.default = input.model ?? base.model.default; + current.providers.user.api_key = input.userKey ?? token; + const serialized = spawnSync( + "python3", + [ + "-I", + "-c", + "import sys,yaml; print(yaml.safe_dump(yaml.safe_load(sys.stdin.read()), sort_keys=False), end='')", + ], + { + input: JSON.stringify(current), + encoding: "utf8", + timeout: 5000, + }, + ); + expect(serialized.status, serialized.stderr).toBe(0); + fs.writeFileSync(fixture.configPath, serialized.stdout); + const witness = path.join(path.dirname(fixture.configPath), ".native-local-startup-config"); + const witnessSources: Record = { original, current: serialized.stdout }; + fs.writeFileSync(witness, witnessSources[input.witnessSource ?? "original"]!, { mode: 0o600 }); + fs.chmodSync(witness, input.witnessMode ?? 0o600); + fs.chmodSync(fixture.strictHashPath, 0o644); + writeHash( + fixture.strictHashPath, + fixture.configPath, + original, + fixture.envPath, + fixture.baseEnv, + ); + fs.chmodSync(fixture.strictHashPath, 0o444); + writeHash( + fixture.compatHashPath, + fixture.configPath, + serialized.stdout, + fixture.envPath, + fs.readFileSync(fixture.envPath, "utf8"), + ); + const proof = runProof(fixture); + expect(proof.status === 0, proof.stderr).toBe(input.accepted); + }); + it("rejects a missing Hermes environment record", () => { const fixture = createFixture(); const [configRecord] = readHashRecords(fixture.compatHashPath); diff --git a/test/e2e/support/hermes-gpu-startup-proof.test.ts b/test/e2e/support/hermes-gpu-startup-proof.test.ts index 373e608ced5..d289c1f09c6 100644 --- a/test/e2e/support/hermes-gpu-startup-proof.test.ts +++ b/test/e2e/support/hermes-gpu-startup-proof.test.ts @@ -2,7 +2,15 @@ // SPDX-License-Identifier: Apache-2.0 import { spawnSync } from "node:child_process"; -import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { + chmodSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + symlinkSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -12,10 +20,12 @@ import { openshellMainProcessSpecEnvValue } from "../../../src/lib/onboard/docke import type { ManagedWorkloadAuthority } from "../../../src/lib/onboard/workload/authority.ts"; import { assertHermesContainerImageAuthority, + buildHermesGpuInferenceCommand, assertHermesGpuStartupOutputContract, assertHermesManagedWorkloadAuthority, HERMES_GPU_FALLBACK_DISCLOSURE_FRAGMENTS, hermesRuntimeIntendedCommand, + hermesGpuRuntimePair, normalizeImmutableImageContentId, } from "../live/hermes-gpu-startup-proof.ts"; @@ -96,6 +106,7 @@ describe("Hermes GPU failed-container diagnostics", () => { "diagnostic", "label=openshell.ai/sandbox-name=owned", "", + root, runtime, ], { encoding: "utf8", timeout: 5_000 }, @@ -108,6 +119,44 @@ describe("Hermes GPU failed-container diagnostics", () => { rmSync(root, { recursive: true, force: true }); } }); + it("copies private generic diagnostics and refuses symlink or unrelated paths", () => { + const root = mkdtempSync(join(tmpdir(), "gpu-diagnostic-path-")); + try { + const directory = join(root, "failure"); + mkdirSync(directory, { mode: 0o700 }); + const log = join(directory, "sandbox-container.log"); + writeFileSync(log, "redacted supervisor evidence", { mode: 0o600 }); + const runtime = join(root, "runtime"); + writeFileSync(runtime, "#!/bin/sh\nexit 0\n", { mode: 0o700 }); + const run = (dir: string, expectedRoot = root) => + spawnSync( + "bash", + [ + "-lc", + liveFailedGpuDiagnosticScript(), + "diagnostic", + "label=openshell.ai/sandbox-name=owned", + dir, + expectedRoot, + runtime, + ], + { encoding: "utf8", timeout: 5_000 }, + ); + expect(run(directory).stdout).toContain("redacted supervisor evidence"); + expect(run(directory, join(root, "other")).stdout).not.toContain( + "redacted supervisor evidence", + ); + const alias = join(root, "alias"); + symlinkSync(directory, alias); + expect(run(alias).stdout).not.toContain("redacted supervisor evidence"); + rmSync(log); + writeFileSync(join(root, "private"), "must not copy", { mode: 0o600 }); + symlinkSync(join(root, "private"), log); + expect(run(directory).stdout).not.toContain("must not copy"); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); }); describe("Hermes GPU startup output contract", () => { @@ -317,3 +366,118 @@ describe("Hermes GPU managed-image authority proof", () => { ).toThrow(); }); }); + +function runHermesNativeInferenceProbe(handle: string) { + const directory = mkdtempSync(join(tmpdir(), "hermes-gpu-native-probe-")); + try { + writeFileSync(join(directory, "curl"), '#!/bin/sh\nprintf "%s\\n" "$@"\n', { mode: 0o700 }); + return spawnSync( + "/bin/bash", + [ + "-c", + buildHermesGpuInferenceCommand( + "http://host.openshell.internal:42103/v1", + 'model/$(exit 99)"quoted', + ), + ], + { + env: { PATH: `${directory}:/usr/bin:/bin`, NEMOCLAW_LOCAL_INFERENCE_TOKEN: handle }, + encoding: "utf8", + }, + ); + } finally { + rmSync(directory, { recursive: true, force: true }); + } +} + +it("sends the Hermes GPU request through its scoped native credential", () => { + const handle = "openshell:resolve:env:v3_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; + const result = runHermesNativeInferenceProbe(handle); + expect(result.status).toBe(0); + expect(result.stdout.trim().split("\n")).toEqual([ + "-fsS", + "--max-time", + "60", + "http://host.openshell.internal:42103/v1/chat/completions", + "-H", + "Content-Type: application/json", + "-H", + `Authorization: Bearer ${handle}`, + "--data", + JSON.stringify({ + model: 'model/$(exit 99)"quoted', + messages: [{ role: "user", content: "reply with OK" }], + max_tokens: 8, + }), + ]); +}); + +it.each([ + "", + "raw-host-key", + "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "openshell:resolve:env:v3_NEMOCLAW_LOCAL_INFERENCE_TOKEN\nraw-host-key", +])("rejects an unscoped Hermes GPU credential before curl (%j)", (handle) => { + const result = runHermesNativeInferenceProbe(handle); + expect(result.status).toBe(2); + expect(result.stdout).toBe(""); +}); + +describe("Hermes GPU supervisor identity", () => { + const sandboxId = "11111111-1111-4111-8111-111111111111"; + const pair = () => + ["sandbox", "supervisor"].map((role, index) => ({ + Id: String(index + 1).repeat(64), + Config: { + Env: + role === "supervisor" + ? [ + `OPENSHELL_MAIN_PROCESS_SPEC=${openshellMainProcessSpecEnvValue(["nemoclaw-start"], false)}`, + ] + : [], + Labels: { + "openshell.ai/managed-by": "openshell", + "openshell.ai/sandbox-name": "owned", + "openshell.ai/sandbox-id": sandboxId, + "openshell.ai/sandbox-namespace": "gateway-namespace", + ...(role === "sandbox" ? { "openshell.ai/sandbox-workspace": "default" } : {}), + "openshell.ai/isolation-role": role, + }, + }, + })); + it("reads command transport from the supervisor without attributing its image to the workload", () => { + const rows = pair(); + const result = hermesGpuRuntimePair(rows, "owned", sandboxId)!; + expect(result.workload).toBe(rows[0]); + expect(result.supervisor).toBe(rows[1]); + expect(result.workload.Config.Env).toEqual([]); + expect( + hermesRuntimeIntendedCommand( + Object.fromEntries( + result.supervisor.Config.Env.map((entry) => entry.split(/=(.*)/su).slice(0, 2)), + ), + ), + ).toEqual(["nemoclaw-start"]); + }); + it.each([ + ["openshell.ai/managed-by", "foreign"], + ["openshell.ai/sandbox-name", "other"], + ["openshell.ai/sandbox-id", "other"], + ["openshell.ai/sandbox-namespace", "other"], + ["openshell.ai/sandbox-workspace", "other"], + ["openshell.ai/isolation-role", "sandbox"], + ])("rejects a supervisor with mismatched %s", (key, value) => { + const rows = pair(); + Object.assign(rows[1].Config.Labels, { [key]: value }); + expect(hermesGpuRuntimePair(rows, "owned", sandboxId)).toBeNull(); + }); + it("rejects missing, duplicate and truncated container identities", () => { + const rows = pair(); + expect(hermesGpuRuntimePair(rows.slice(0, 1), "owned", sandboxId)).toBeNull(); + expect(hermesGpuRuntimePair(rows, "owned", null)).toBeNull(); + rows[1].Id = rows[0].Id; + expect(hermesGpuRuntimePair(rows, "owned", sandboxId)).toBeNull(); + rows[1].Id = "a".repeat(12); + expect(hermesGpuRuntimePair(rows, "owned", sandboxId)).toBeNull(); + }); +}); diff --git a/test/e2e/support/hermes-inference-switch-command-shape.test.ts b/test/e2e/support/hermes-inference-switch-command-shape.test.ts index 0714768dc34..1fae9bc6f2a 100644 --- a/test/e2e/support/hermes-inference-switch-command-shape.test.ts +++ b/test/e2e/support/hermes-inference-switch-command-shape.test.ts @@ -8,6 +8,7 @@ import { promisify } from "node:util"; import { afterEach, describe, expect, it, vi } from "vitest"; +import { customNativeProvider } from "../../../src/lib/inference/native-provider/custom.ts"; import { resolveAgentInferenceApi } from "../../../src/lib/inference/config.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; @@ -15,12 +16,11 @@ import { compatibleAnthropicSwitchBinding } from "../fixtures/compatible-anthrop import { DEFAULT_HOSTED_INFERENCE_MODEL } from "../fixtures/hosted-inference.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; import { - API_KEY_SHAPE_PATTERN, - NVIDIA_API_KEY_SHAPE_PATTERN, apiKeyShapeCommand, cleanupHermesSwitch, compatibleAnthropicMetadataArgs, expectAuthenticatedBaselineInventoryRequest, + expectOpenAiProvider, expectAuthenticatedProxyResolutionRequests, hasAuthenticatedProxyResolutionRequest, hermesApiCommand, @@ -36,6 +36,7 @@ import { parseInferenceRoute, prepareProxyResolutionRoute, PROXY_RESOLUTION_PROVIDER, + PROXY_RESOLUTION_MODEL, runHermesInferenceSetWithRetry, runHermesCliPongWithRetry, runHermesPongWithRetry, @@ -51,6 +52,38 @@ import { describe("Hermes inference switch command shape", () => { afterEach(() => vi.unstubAllEnvs()); + it.each([ + [PUBLIC_NVIDIA_SWITCH_PROVIDER, "openshell:resolve:env:v1_NVIDIA_INFERENCE_API_KEY", 0], + [PUBLIC_NVIDIA_SWITCH_PROVIDER, '"openshell:resolve:env:v12_NVIDIA_INFERENCE_API_KEY"', 0], + [ + PUBLIC_NVIDIA_SWITCH_PROVIDER, + `'openshell:resolve:env:s${"a".repeat(64)}_NVIDIA_INFERENCE_API_KEY'`, + 0, + ], + [PUBLIC_NVIDIA_SWITCH_PROVIDER, "nvapi-raw-key", 1], + [PUBLIC_NVIDIA_SWITCH_PROVIDER, "${NVIDIA_INFERENCE_API_KEY}", 1], + [PUBLIC_NVIDIA_SWITCH_PROVIDER, "openshell:resolve:env:NVIDIA_INFERENCE_API_KEY", 1], + [PUBLIC_NVIDIA_SWITCH_PROVIDER, "openshell:resolve:env:v1_COMPATIBLE_ANTHROPIC_API_KEY", 1], + ["compatible-anthropic-endpoint", "${COMPATIBLE_ANTHROPIC_API_KEY}", 0], + ["compatible-anthropic-endpoint", '"${COMPATIBLE_ANTHROPIC_API_KEY}"', 0], + ["compatible-anthropic-endpoint", "'${COMPATIBLE_ANTHROPIC_API_KEY}'", 0], + ["compatible-anthropic-endpoint", "sk-raw-key", 1], + ["compatible-anthropic-endpoint", "${NVIDIA_INFERENCE_API_KEY}", 1], + ["compatible-anthropic-endpoint", "${COMPATIBLE_ANTHROPIC_API_KEY} trailing", 1], + ["compatible-anthropic-endpoint", '"${COMPATIBLE_ANTHROPIC_API_KEY}', 1], + ])("checks the deployed %s credential reference %s", (provider, scalar, status) => { + const [command, ...args] = apiKeyShapeCommand(String(provider)); + const result = spawnSync(command!, args.slice(0, -1), { + input: `model:\n api_key: ${scalar}\n`, + encoding: "utf8", + }); + expect({ status: result.status, stdout: result.stdout, stderr: result.stderr }).toEqual({ + status, + stdout: "", + stderr: "", + }); + }); + it.each(["123", " 123\n"])("reads one PID from a successful gateway probe: %s", (stdout) => { expect(parseHermesGatewayPid({ exitCode: 0, stdout })).toBe("123"); }); @@ -116,15 +149,6 @@ describe("Hermes inference switch command shape", () => { return import("../live/hermes-inference-switch-helpers.ts"); } - function matchesApiKeyShape(line: string, pattern: string = API_KEY_SHAPE_PATTERN): boolean { - return ( - spawnSync("grep", ["-Eq", pattern], { - encoding: "utf8", - input: `${line}\n`, - }).status === 0 - ); - } - it("preserves the requested frontend for other Hermes upstreams (#6289)", () => { expect(resolveAgentInferenceApi("hermes", "nvidia-prod", "openai-completions")).toBe( "openai-completions", @@ -163,7 +187,10 @@ describe("Hermes inference switch command shape", () => { }); it("probes native NVIDIA through its sandbox-attached provider route (#12558)", () => { - const command = sandboxInferenceCommand('{"model":"nvidia/test"}'); + const command = sandboxInferenceCommand( + '{"model":"nvidia/test"}', + "https://integrate.api.nvidia.com/v1", + ); expect(command).toContain("https://integrate.api.nvidia.com/v1/chat/completions"); expect(command).toContain('AUTH_HEADER="Authorization: Bearer ${NVIDIA_INFERENCE_API_KEY}"'); @@ -172,6 +199,48 @@ describe("Hermes inference switch command shape", () => { expect(command).not.toContain("inference.local"); }); + it("probes a custom hosted Anthropic provider using its injected credential and recorded endpoint", async () => { + const definition = customNativeProvider({ + provider: "compatible-anthropic-endpoint", + api: "openai-completions", + endpointUrl: "https://anthropic.example.com/team", + credentialEnv: "COMPATIBLE_ANTHROPIC_API_KEY", + allowedIps: ["8.8.8.8"], + }); + const receipt = { + schemaVersion: 1, + profileId: definition.profileId, + providerName: definition.providerName, + providerId: "owned-id", + endpointUrl: definition.endpointUrl, + allowedIps: definition.allowedIps, + custom: definition.custom, + }; + const command = sandboxInferenceCommand( + '{"model":"mock-anthropic-model"}', + definition.endpoint, + "compatible-anthropic-endpoint", + ); + expect(command).toContain("https://anthropic.example.com/team/v1/chat/completions"); + expect(command).toContain( + 'AUTH_HEADER="Authorization: Bearer ${COMPATIBLE_ANTHROPIC_API_KEY:-}"', + ); + expect(command).not.toContain("inference.local"); + const hostCommand = vi.fn(async () => ({ + exitCode: 0, + stdout: `Name: ${receipt.providerName}\nType: ${receipt.profileId}\nId: owned-id\nResource version: 1\nCredential keys: COMPATIBLE_ANTHROPIC_API_KEY\nConfig keys: `, + stderr: "", + })); + await expectOpenAiProvider({ command: hostCommand } as unknown as HostCliClient, receipt); + expect(hostCommand.mock.calls[0]).toContainEqual([ + "provider", + "get", + "-g", + "nemoclaw", + receipt.providerName, + ]); + }); + it.each([ ["absent", undefined], [ @@ -228,58 +297,6 @@ describe("Hermes inference switch command shape", () => { ); }); - it("uses direct single-line argv for the in-sandbox API-key probe", () => { - const command = apiKeyShapeCommand("compatible-anthropic-endpoint"); - const nativeCommand = apiKeyShapeCommand(PUBLIC_NVIDIA_SWITCH_PROVIDER); - - expect(command).toEqual(["grep", "-Eq", API_KEY_SHAPE_PATTERN, "/sandbox/.hermes/config.yaml"]); - expect(nativeCommand).toEqual([ - "grep", - "-Eq", - NVIDIA_API_KEY_SHAPE_PATTERN, - "/sandbox/.hermes/config.yaml", - ]); - expect([...command, ...nativeCommand].every((argument) => !/[\r\n]/u.test(argument))).toBe( - true, - ); - }); - - it("accepts only complete sk-prefixed YAML scalars", () => { - expect( - [" api_key: sk-value", ' api_key: "sk-value"', " api_key: 'sk-value'"].every((line) => - matchesApiKeyShape(line), - ), - ).toBe(true); - expect( - [ - " api_key: not-sk-value", - " api_key: sk-value trailing", - ' api_key: "sk-value', - ' api_key: sk-value"', - ].some((line) => matchesApiKeyShape(line)), - ).toBe(false); - }); - - it("accepts only a provider-bound reference for a native NVIDIA switch", () => { - const reference = "openshell:resolve:env:v1_NVIDIA_INFERENCE_API_KEY"; - expect( - [` api_key: ${reference}`, ` api_key: "${reference}"`, ` api_key: '${reference}'`].every( - (line) => matchesApiKeyShape(line, NVIDIA_API_KEY_SHAPE_PATTERN), - ), - ).toBe(true); - expect( - [ - " api_key: sk-OPENSHELL-PROXY-REWRITE", - " api_key: nvapi-raw-key", - " api_key: ${NVIDIA_INFERENCE_API_KEY}", - " api_key: openshell:resolve:env:NVIDIA_INFERENCE_API_KEY", - ` api_key: ${reference} trailing`, - ` api_key: "${reference}`, - ` api_key: ${reference}"`, - ].some((line) => matchesApiKeyShape(line, NVIDIA_API_KEY_SHAPE_PATTERN)), - ).toBe(false); - }); - it("keeps initial hosted onboarding independent from the switch target", () => { expect(hostedInstallModel({ NEMOCLAW_SWITCH_MODEL: "mock-anthropic-model" })).toBe( DEFAULT_HOSTED_INFERENCE_MODEL, @@ -303,7 +320,7 @@ describe("Hermes inference switch command shape", () => { const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", - stdout: "Type: openai\nCredentials: NVIDIA_INFERENCE_API_KEY\nConfig: OPENAI_BASE_URL\n", + stdout: "route synced", }); await prepareProxyResolutionRoute({ apiKey: "nvapi-public-fixture-key", @@ -312,48 +329,69 @@ describe("Hermes inference switch command shape", () => { redactionValues: ["nvapi-public-fixture-key"], }); expect(command.mock.calls[0]).toEqual([ - "openshell", + "node", expect.arrayContaining([ - "provider", - "create", - "--credential", - "NVIDIA_INFERENCE_API_KEY", - "OPENAI_BASE_URL=https://integrate.api.nvidia.com/v1", + "--provider", + "compatible-endpoint", + "--endpoint-url", + "https://integrate.api.nvidia.com/v1", + "--credential-env", + "COMPATIBLE_API_KEY", + "--inference-api", + "openai-completions", ]), expect.objectContaining({ - env: expect.objectContaining({ NVIDIA_INFERENCE_API_KEY: "nvapi-public-fixture-key" }), + env: expect.objectContaining({ COMPATIBLE_API_KEY: "nvapi-public-fixture-key" }), redactionValues: ["nvapi-public-fixture-key"], }), ]); }); - it("keeps proxy-resolution evidence on a dedicated OpenAI provider", async () => { - const command = vi - .fn() - .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "created" }) - .mockResolvedValueOnce({ exitCode: 0, stderr: "", stdout: "route synced" }); - - await prepareProxyResolutionRoute({ - apiKey: "hosted-key", + it("selects the native local route before testing split provider and namespaced model flags", async () => { + const command = vi.fn().mockResolvedValue({ exitCode: 0, stderr: "", stdout: "route synced" }); + const mockBaseline = { + baseUrl: "http://host.openshell.internal:11434/v1", + logFile: "fixture-server.log", + requestsFile: "fixture-requests.jsonl", + environmentKeys: () => [], + requests: () => [{ method: "GET", path: "/v1/models", bodyBytes: 0 }], + close: async () => undefined, + } satisfies NonNullable[0]["mockBaseline"]>; + const selected = await prepareProxyResolutionRoute({ + apiKey: "fixture-key", host: { command } as unknown as HostCliClient, - mockBaseline: undefined, - redactionValues: ["hosted-key"], + mockBaseline, + redactionValues: ["fixture-key"], }); - expect(PROXY_RESOLUTION_PROVIDER).not.toBe("nvidia-prod"); - expect(command.mock.calls[0]?.[1]).toEqual( - expect.arrayContaining([ - "--name", + expect(selected).toEqual({ model: PROXY_RESOLUTION_MODEL, requestOffset: 1 }); + expect(command).toHaveBeenCalledExactlyOnceWith( + "node", + [ + expect.any(String), + "inference", + "set", + "--sandbox", + SANDBOX_NAME, + "--provider", PROXY_RESOLUTION_PROVIDER, - "--type", - "openai", - "--credential", - "NVIDIA_INFERENCE_API_KEY", - ]), - ); - expect(command.mock.calls[1]?.[1]).toEqual( - expect.arrayContaining(["--provider", PROXY_RESOLUTION_PROVIDER]), + "--model", + PROXY_RESOLUTION_MODEL, + "--endpoint-url", + mockBaseline.baseUrl, + "--credential-env", + "COMPATIBLE_API_KEY", + "--inference-api", + "openai-completions", + "--no-verify", + ], + expect.objectContaining({ + env: expect.objectContaining({ COMPATIBLE_API_KEY: "fixture-key" }), + redactionValues: ["fixture-key"], + timeoutMs: 180_000, + }), ); + expect(JSON.stringify(command.mock.calls[0][1])).not.toContain("fixture-key"); }); it("uses authenticated model inventory as baseline readiness evidence", () => { diff --git a/test/e2e/support/inference-adapter.test.ts b/test/e2e/support/inference-adapter.test.ts index 4decb437da9..ff742df90f8 100644 --- a/test/e2e/support/inference-adapter.test.ts +++ b/test/e2e/support/inference-adapter.test.ts @@ -256,6 +256,52 @@ describe("E2E inference adapter", () => { ).toMatchObject({ choices: [{ message: { content: "PONG" } }] }); }); + it.each([ + ["test-model", "test-model", false, false], + ["changed-model", "changed-model", false, false], + ["unadvertised-model", "test-model", false, false], + ["changed-model", "changed-model", true, false], + ["unadvertised-model", "test-model", true, false], + ["changed-model", "changed-model", true, true], + ["changed-model", "changed-model", false, true], + ])( + "returns the advertised response model for request %s: model=%s, stream=%s, tool=%s", + async (requested, expected, stream, tool) => { + const fake = await startFakeOpenAiCompatibleServer({ + progress: NOOP_PROGRESS, + additionalModels: ["changed-model"], + requestCanaryMarker: "MODEL_IDENTITY_CANARY", + toolCallOnCanary: { name: "identity_tool", arguments: "{}" }, + }); + try { + const response = await fetch(`${fake.baseUrl}/chat/completions`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + model: requested, + stream, + messages: [{ role: "user", content: tool ? "MODEL_IDENTITY_CANARY" : "hello" }], + tools: [{ type: "function", function: { name: "identity_tool" } }], + }), + }); + expect(response.status).toBe(200); + const text = await response.text(); + const chunks = stream + ? text + .split("\n") + .filter((line) => line.startsWith("data: {")) + .map((line) => JSON.parse(line.slice(6))) + : [JSON.parse(text)]; + expect(chunks.map((chunk) => chunk.model)).toEqual( + stream ? [expected, expected] : [expected], + ); + expect(chunks.at(-1).choices[0].finish_reason).toBe(tool ? "tool_calls" : "stop"); + } finally { + await fake.close(); + } + }, + ); + it("keeps unrelated ambient secrets out of adapter and fake-server child environments", async () => { const secretName = "UNRELATED_E2E_SENTINEL_SECRET"; const secretValue = "sentinel-value-that-must-not-propagate"; diff --git a/test/e2e/support/inference-redirect-dns-denial.test.ts b/test/e2e/support/inference-redirect-dns-denial.test.ts new file mode 100644 index 00000000000..7fcbda01651 --- /dev/null +++ b/test/e2e/support/inference-redirect-dns-denial.test.ts @@ -0,0 +1,66 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import { + INFERENCE_CURL_HTTP_CODE_MARKER as marker, + isExpectedMcpDnsPolicyDenial, + waitForMcpCurlPolicyDenial, +} from "../fixtures/curl-policy-denial.ts"; + +const hostname = "host.openshell.internal"; +const probe = { + exitCode: 6, + stdout: `HTTP/1.1 302 Found\r\nLocation: http://${hostname}:8000/v1/chat/completions\r\n\r\n${marker}302\n`, + stderr: `curl: (6) Could not resolve host: ${hostname}\n`, + timedOut: false, + signal: null, +}; +const before = { ...probe, exitCode: 0, stdout: "", stderr: "" }; +const event = `[2.0] [sandbox] NET:REFUSE [MED] DENIED ${hostname} [reason:policy_dns_ineligible]`; +const after = { ...before, stdout: event }; + +describe("Native inference redirect DNS denial", () => { + it.each(["policy_dns_ineligible", "policy_dns_no_valid_address"])( + "accepts a matching redirect and fresh %s audit event", + async (reason) => { + expect( + await waitForMcpCurlPolicyDenial( + probe, + before, + async () => ({ ...after, stdout: event.replace("policy_dns_ineligible", reason) }), + hostname, + marker, + "302", + ), + ).toBe(true); + }, + ); + + it.each([ + ["stale event", probe, after, after], + ["absent audit", probe, before, before], + ["wrong host", probe, before, { ...after, stdout: event.replace(hostname, "other.test") }], + [ + "resolver outage", + probe, + before, + { ...after, stdout: event.replace("policy_dns_ineligible", "policy_dns_upstream_nxdomain") }, + ], + [ + "different redirect", + { ...probe, stdout: probe.stdout.replace(hostname, "other.test") }, + before, + after, + ], + ["missing redirect response", { ...probe, stdout: `${marker}302\n` }, before, after], + ["duplicate marker", { ...probe, stdout: `${probe.stdout}${marker}302\n` }, before, after], + ["successful request", { ...probe, exitCode: 0 }, before, after], + ["timeout", { ...probe, timedOut: true }, before, after], + ["failed audit read", probe, before, { ...after, exitCode: 1 }], + ] as const)("rejects %s", (_label, result, previous, current) => { + expect(isExpectedMcpDnsPolicyDenial(result, previous, current, hostname, marker, "302")).toBe( + false, + ); + }); +}); diff --git a/test/e2e/support/inference-switch-retry.test.ts b/test/e2e/support/inference-switch-retry.test.ts index 272d9205da2..6cf41773a1c 100644 --- a/test/e2e/support/inference-switch-retry.test.ts +++ b/test/e2e/support/inference-switch-retry.test.ts @@ -10,6 +10,8 @@ import { runInferenceSetWithRetry, writeInferenceSwitchRetryEvidence, } from "../fixtures/inference-switch-retry.ts"; +import { completeInferencePostCommit } from "../../../src/lib/actions/inference-set-gateway-restart"; +import { OPENCLAW_PAIRING_OBSERVATION_FAILURE_CATEGORIES } from "../../../src/lib/actions/sandbox/launch-readiness/openclaw-pairing-qualification"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; function result(exitCode: number, stderr = ""): ShellProbeResult { @@ -164,3 +166,72 @@ describe("inference switch retry", () => { expect(() => inferenceSetAttemptCount("11")).toThrow(/between 1 and 10/u); }); }); + +describe("pairing observation diagnostics at the retry boundary", () => { + it.each(OPENCLAW_PAIRING_OBSERVATION_FAILURE_CATEGORIES)( + "keeps %s failures terminal and records the category", + async (failureCategory) => { + const appendAuditEntry = vi.fn(); + const log = vi.fn(); + let message = ""; + try { + await completeInferencePostCommit( + { + result: { + sandboxName: "alpha", + provider: "nvidia-prod", + model: "nvidia/model-b", + primaryModelRef: "inference/nvidia/model-b", + inSandboxConfigSynced: true, + }, + openClawGatewayRestartRequired: true, + openClawPairing: { + state: "required", + target: { + sandboxName: "alpha", + gatewayName: "gateway", + openclawVersion: "2026.9.2", + stateDirectory: "/sandbox/.openclaw", + }, + }, + }, + { + appendAuditEntry, + log, + restartSandboxGateway: async () => ({ + ok: true, + restarted: true, + healthPassed: true, + forwardRecovered: true, + }), + settleOpenClawPairing: () => ({ + ok: false, + failureLayer: "initial-state-unavailable", + failureCategory, + }), + }, + ); + } catch (error) { + message = (error as Error).message; + } + expect(message).toContain(`initial-state-unavailable; observation: ${failureCategory}`); + expect(message).toContain("The committed route was not rolled back."); + expect(message).toContain("doctor --fix"); + expect(appendAuditEntry).toHaveBeenCalledWith( + expect.objectContaining({ + reason: expect.stringContaining( + `pairing convergence failed: initial-state-unavailable; observation: ${failureCategory}`, + ), + }), + ); + expect(log.mock.calls.flat().join("\n")).not.toContain("Inference route synced"); + const run = vi.fn(async () => result(1, message)); + const delay = vi.fn(); + await expect(runInferenceSetWithRetry({ attempts: 3, run, delay })).resolves.toMatchObject({ + exitCode: 1, + }); + expect(run).toHaveBeenCalledOnce(); + expect(delay).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/test/e2e/support/launch-agent-turn-monitor-cleanup.test.ts b/test/e2e/support/launch-agent-turn-monitor-cleanup.test.ts new file mode 100644 index 00000000000..260e229b53c --- /dev/null +++ b/test/e2e/support/launch-agent-turn-monitor-cleanup.test.ts @@ -0,0 +1,219 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { chmodSync, existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { createServer } from "node:net"; +import { join } from "node:path"; + +import { expect, it } from "vitest"; +import { superviseChild } from "../../helpers/process-supervisor.ts"; +import { LAUNCH_TURN_SCRIPT, OPENCLAW_SESSION_EVIDENCE_SCRIPT } from "../live/launch-agent-turn.ts"; + +async function monitorFixture() { + const runId = randomUUID().replaceAll("-", ""); + const root = `/tmp/nemoclaw-launch-turn-${runId}`; + const socket = join(root, "pty-input-mode.sock"); + const server = createServer(); + mkdirSync(root, { mode: 0o700 }); + chmodSync(root, 0o700); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(socket, resolve); + }); + chmodSync(socket, 0o600); + let closed: Promise | undefined; + const close = () => + (closed ??= new Promise((resolve, reject) => { + server.close((error) => (error ? reject(error) : resolve())); + })); + return { + runId, + root, + socket, + close, + cleanup: async () => { + await close(); + rmSync(root, { recursive: true, force: true }); + }, + }; +} + +async function runWait( + fixture: Awaited>, + onProbe?: () => void, + expireWait = false, +) { + const waitFunction = LAUNCH_TURN_SCRIPT.match( + /wait_for_pty_monitor_exit\(\) \{[\s\S]*?\n\}/, + )?.[0]; + assert.ok(waitFunction, "Launch monitor wait function is missing"); + // The host path is absent while the sandbox path holds a real Unix socket. + const child = spawn( + "bash", + [ + "-c", + ` +session_evidence() { + printf 'sandbox-probe\n' + "$NODE_EXECUTABLE" -e "$EVIDENCE_SCRIPT" "$1" /sandbox/.openclaw/agents/main/sessions \ + "$BASELINE_PATH" '' "$SANDBOX_MONITOR_ROOT" "$RUN_ID" +} +${waitFunction} +pty_monitor_root="$SANDBOX_MONITOR_ROOT/absent-host-path" +wait_for_pty_monitor_exit +`, + ], + { + detached: true, + stdio: ["ignore", "pipe", "pipe"], + env: { + ...process.env, + NODE_EXECUTABLE: process.execPath, + EVIDENCE_SCRIPT: expireWait + ? `let clockReads = 0; +Object.defineProperty(globalThis, "performance", { + value: { now: () => clockReads++ === 0 ? 0 : 5_000 }, +}); +${OPENCLAW_SESSION_EVIDENCE_SCRIPT}` + : OPENCLAW_SESSION_EVIDENCE_SCRIPT, + BASELINE_PATH: `/tmp/nemoclaw-launch-session-${fixture.runId}.json`, + SANDBOX_MONITOR_ROOT: fixture.root, + RUN_ID: fixture.runId, + }, + }, + ); + let stderr = ""; + let sawProbe = false; + child.stdout.once("data", () => onProbe?.()); + const result = await superviseChild(child, { + timeoutMs: 8_000, + killGraceMs: 1_000, + onStdout: () => { + sawProbe = true; + }, + onStderr: (chunk) => { + stderr += chunk; + }, + }); + return { ...result, stderr, sawProbe }; +} + +it("waits for the sandbox monitor socket when the host path is absent", async () => { + const fixture = await monitorFixture(); + let closeTimer: ReturnType | undefined; + let closePromise: Promise | undefined; + try { + const result = await runWait(fixture, () => { + closeTimer = setTimeout(() => { + closePromise = fixture.close(); + }, 150); + }); + await closePromise; + expect(result.timedOut).toBe(false); + expect(result.exitCode).toBe(0); + expect(result.sawProbe).toBe(true); + expect(existsSync(fixture.socket)).toBe(false); + expect(existsSync(fixture.root)).toBe(true); + } finally { + clearTimeout(closeTimer); + await closePromise; + await fixture.cleanup(); + } +}); + +it("rejects a persistent sandbox monitor socket without removing it", async () => { + const fixture = await monitorFixture(); + try { + const result = await runWait(fixture, undefined, true); + expect(result.timedOut).toBe(false); + expect(result.exitCode).toBe(2); + expect(result.stderr).toContain('"reason":"pty_monitor_socket_still_present"'); + expect(existsSync(fixture.socket)).toBe(true); + } finally { + await fixture.cleanup(); + } +}); + +it("accepts an already absent sandbox monitor socket", async () => { + const fixture = await monitorFixture(); + try { + await fixture.close(); + const result = await runWait(fixture); + expect(result.timedOut).toBe(false); + expect(result.exitCode).toBe(0); + expect(result.sawProbe).toBe(true); + } finally { + await fixture.cleanup(); + } +}); + +it("rejects a non-socket entry at the sandbox monitor path", async () => { + const fixture = await monitorFixture(); + try { + await fixture.close(); + writeFileSync(fixture.socket, "unexpected", { mode: 0o600 }); + const result = await runWait(fixture); + expect(result.timedOut).toBe(false); + expect(result.exitCode).toBe(2); + expect(result.stderr).toContain('"reason":"pty_socket_invalid"'); + expect(existsSync(fixture.socket)).toBe(true); + } finally { + await fixture.cleanup(); + } +}); + +it("preserves a failed monitor wait when later cleanup succeeds", async () => { + const fixture = await monitorFixture(); + try { + await fixture.close(); + const cleanup = LAUNCH_TURN_SCRIPT.match(/\ncleanup\(\) \{[\s\S]*?\n\}/)?.[0]; + assert.ok(cleanup, "Launch cleanup function is missing"); + const sessionDir = join(fixture.root, "host-session"); + mkdirSync(sessionDir); + const child = spawn( + "bash", + [ + "-c", + ` +session_dir="$1" +evidence_error="$session_dir/evidence.err" +session_pid='' +last_evidence_status=3 +provider_unavailable_candidate=1 +NEMOCLAW_LAUNCH_RUN_ID=fixture +remove_session_baseline() { return 0; } +wait_for_pty_monitor_exit() { return 2; } +remove_pty_monitor() { return 0; } +${cleanup} +trap cleanup EXIT +exit 1 +`, + "monitor-cleanup", + sessionDir, + ], + { + detached: true, + stdio: ["ignore", "pipe", "pipe"], + }, + ); + let stderr = ""; + const result = await superviseChild(child, { + timeoutMs: 3_000, + killGraceMs: 1_000, + onStderr: (chunk) => { + stderr += chunk; + }, + }); + expect(result.timedOut).toBe(false); + expect(result.exitCode).toBe(1); + expect(stderr).toContain("launch PTY monitor wait failed"); + expect(stderr).toContain("launch-cleanup=completed status=1"); + expect(stderr).not.toContain("launch-failure=provider-unavailable"); + expect(existsSync(sessionDir)).toBe(false); + } finally { + await fixture.cleanup(); + } +}); diff --git a/test/e2e/support/lifecycle-user-service.test.ts b/test/e2e/support/lifecycle-user-service.test.ts index d6cf1ede869..c0f3b89765c 100644 --- a/test/e2e/support/lifecycle-user-service.test.ts +++ b/test/e2e/support/lifecycle-user-service.test.ts @@ -183,13 +183,13 @@ function writeMacServiceStubs( } describe("reboot lifecycle OpenShell gateway user-service fixture", () => { - it("stages, enables, and removes the repository service without installer cleanup", () => { + it("stages, enables, and removes the repository service with configured binary and config directories", () => { const root = fs.mkdtempSync( path.join(os.tmpdir(), "nemoclaw-installer-lifecycle-stage-service-"), ); const home = path.join(root, "home"); const configHome = path.join(root, "config"); - const bin = path.join(home, ".local", "bin"); + const bin = path.join(root, "installed-bin"); const log = path.join(root, "systemctl.log"); const unit = path.join(configHome, "systemd", "user", "nemoclaw-openshell-gateway.service"); const installerCleanupSentinel = path.join(root, "installer-cleanup-sentinel"); @@ -213,6 +213,7 @@ describe("reboot lifecycle OpenShell gateway user-service fixture", () => { const env = buildAvailabilityProbeEnv({ HOME: home, PATH: `${bin}:${path.dirname(process.execPath)}:/usr/bin:/bin`, + XDG_BIN_HOME: bin, XDG_CONFIG_HOME: configHome, }); env.NEMOCLAW_INSTALLER_STAGED = installerCleanupSentinel; diff --git a/test/e2e/support/managed-image-activation-cleanup.test.ts b/test/e2e/support/managed-image-activation-cleanup.test.ts new file mode 100644 index 00000000000..bf425b1fbfe --- /dev/null +++ b/test/e2e/support/managed-image-activation-cleanup.test.ts @@ -0,0 +1,225 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { + inspectActivationSandboxId, + verifyExactCleanup, +} from "../live/managed-image-activation-e2e-helpers.ts"; + +describe("managed image activation cleanup evidence", () => { + it.each([ + { + boundary: "OpenShell query", + listStatus: 1, + engineStatus: 0, + listed: false, + container: false, + }, + { boundary: "engine query", listStatus: 0, engineStatus: 1, listed: false, container: false }, + { + boundary: "OpenShell absence", + listStatus: 0, + engineStatus: 0, + listed: true, + container: false, + }, + { + boundary: "container absence", + listStatus: 0, + engineStatus: 0, + listed: false, + container: true, + }, + ])("cleanup polling rejects failed $boundary evidence", async (scenario) => { + vi.useFakeTimers(); + const list = vi.fn(async () => ({ + exitCode: scenario.listStatus, + stderr: "", + stdout: scenario.listed ? "mi-act-hermes 1m Deleting\n" : "NAME CREATED PHASE\n", + })); + const command = vi.fn(async (_engine: string, args: string[]) => ({ + exitCode: scenario.engineStatus, + stderr: "", + stdout: scenario.container && args[0] === "ps" ? "a".repeat(64) : "", + })); + try { + const assertion = expect( + verifyExactCleanup({ command } as never, { list } as never, "mi-act-hermes", {}, [ + "sandbox-1", + ]), + ).rejects.toThrow(scenario.listStatus || scenario.engineStatus ? /failed/ : /exhausted/); + await vi.advanceTimersByTimeAsync(60_000); + await assertion; + } finally { + vi.useRealTimers(); + } + }); + + it("cleanup polling succeeds only after sandbox, container and volume inventories show absence", async () => { + vi.useFakeTimers(); + const list = vi.fn(async () => ({ exitCode: 0, stderr: "", stdout: "NAME CREATED PHASE\n" })); + let volumeObservations = 0; + const command = vi.fn(async (_engine: string, args: string[]) => ({ + exitCode: 0, + stderr: "", + stdout: + args[0] === "volume" && volumeObservations++ === 0 ? "openshell-control-sandbox-1" : "", + })); + try { + const assertion = expect( + verifyExactCleanup({ command } as never, { list } as never, "mi-act-hermes", {}, [ + "sandbox-1", + ]), + ).resolves.toBeUndefined(); + await vi.advanceTimersByTimeAsync(1_000); + await assertion; + expect(list).toHaveBeenCalledTimes(2); + expect(command).toHaveBeenCalledTimes(4); + } finally { + vi.useRealTimers(); + } + }); + + it.each(["docker", "podman"] as const)( + "checks old and replacement runtime volumes through %s", + async (engine) => { + const list = vi.fn(async () => ({ exitCode: 0, stdout: "NAME CREATED PHASE\n", stderr: "" })); + const command = vi.fn(async () => ({ exitCode: 0, stdout: "", stderr: "" })); + await verifyExactCleanup( + { command } as never, + { list } as never, + "mi-act-hermes", + { NEMOCLAW_GATEWAY_RUNTIME: engine }, + ["old-id", "new-id", "new-id"], + ); + expect(command.mock.calls.map((call) => call)).toEqual([ + [ + engine, + ["ps", "-aq", "--filter", "label=openshell.ai/sandbox-name=mi-act-hermes"], + expect.any(Object), + ], + [ + engine, + ["volume", "ls", "-q", "--filter", "label=openshell.ai/sandbox-id=old-id"], + expect.any(Object), + ], + [ + engine, + ["volume", "ls", "-q", "--filter", "label=openshell.ai/sandbox-id=new-id"], + expect.any(Object), + ], + ]); + }, + ); + + it.each(["old-id", "new-id"])("rejects a retained control volume for %s", async (retainedId) => { + vi.useFakeTimers(); + const list = vi.fn(async () => ({ exitCode: 0, stdout: "NAME CREATED PHASE\n", stderr: "" })); + const command = vi.fn(async (_engine: string, args: string[]) => ({ + exitCode: 0, + stderr: "", + stdout: + args[0] === "volume" && args.at(-1) === `label=openshell.ai/sandbox-id=${retainedId}` + ? `openshell-control-${retainedId}` + : "", + })); + try { + const assertion = expect( + verifyExactCleanup({ command } as never, { list } as never, "mi-act-hermes", {}, [ + "old-id", + "new-id", + ]), + ).rejects.toThrow(/exhausted/); + await vi.advanceTimersByTimeAsync(60_000); + await assertion; + } finally { + vi.useRealTimers(); + } + }); + + it.each([ + { exitCode: 1, timedOut: false }, + { exitCode: 0, timedOut: true }, + ])("rejects an indeterminate runtime volume inventory %#", async (failure) => { + const list = vi.fn(async () => ({ exitCode: 0, stdout: "NAME CREATED PHASE\n", stderr: "" })); + const command = vi.fn(async (_engine: string, args: string[]) => ({ + exitCode: 0, + timedOut: false, + stdout: "", + stderr: "", + ...(args[0] === "volume" ? failure : {}), + })); + await expect( + verifyExactCleanup({ command } as never, { list } as never, "mi-act-hermes", {}, [ + "sandbox-1", + ]), + ).rejects.toThrow(/runtime volumes.*failed/); + }); + + it.each([{ ids: [] }, { ids: [""] }, { ids: ["one\ntwo"] }])( + "rejects missing or ambiguous cleanup identities %#", + async ({ ids }) => { + const command = vi.fn(); + await expect( + verifyExactCleanup({ command } as never, {} as never, "mi-act-hermes", {}, ids), + ).rejects.toThrow(/observed runtime sandbox identities/); + expect(command).not.toHaveBeenCalled(); + }, + ); + + it.each(["docker", "podman"] as const)( + "captures the runtime identity before cleanup through %s", + async (engine) => { + const command = vi.fn(async () => ({ exitCode: 0, stdout: "sandbox-1\n", stderr: "" })); + await expect( + inspectActivationSandboxId( + { command } as never, + "mi-act-hermes", + { + NEMOCLAW_GATEWAY_RUNTIME: engine, + }, + "before-cleanup", + ), + ).resolves.toBe("sandbox-1"); + expect(command).toHaveBeenCalledWith( + engine, + [ + "ps", + "-a", + "--filter", + "label=openshell.ai/sandbox-name=mi-act-hermes", + "--filter", + "label=openshell.ai/isolation-role=sandbox", + "--format", + '{{.Label "openshell.ai/sandbox-id"}}', + ], + expect.any(Object), + ); + }, + ); + + it("preserves runtime identity evidence before and after rebuild", async () => { + const command = vi.fn(async () => ({ exitCode: 0, stdout: "sandbox-1\n", stderr: "" })); + await inspectActivationSandboxId({ command } as never, "mi-act-hermes", {}, "before-rebuild"); + await inspectActivationSandboxId({ command } as never, "mi-act-hermes", {}, "after-rebuild"); + expect( + command.mock.calls.map( + (call) => (call as unknown as [string, string[], { artifactName: string }])[2].artifactName, + ), + ).toEqual([ + "activation-runtime-sandbox-id-mi-act-hermes-before-rebuild", + "activation-runtime-sandbox-id-mi-act-hermes-after-rebuild", + ]); + }); + + it.each(["", "one\ntwo", ""])( + "rejects unusable runtime identity %j", + async (stdout) => { + const command = vi.fn(async () => ({ exitCode: 0, stdout, stderr: "" })); + await expect( + inspectActivationSandboxId({ command } as never, "mi-act-hermes", {}, "before-cleanup"), + ).rejects.toThrow(/must be unambiguous/); + }, + ); +}); diff --git a/test/e2e/support/managed-image-activation-diagnostics.test.ts b/test/e2e/support/managed-image-activation-diagnostics.test.ts index a37cf6ebf33..de3810106f4 100644 --- a/test/e2e/support/managed-image-activation-diagnostics.test.ts +++ b/test/e2e/support/managed-image-activation-diagnostics.test.ts @@ -2,11 +2,16 @@ // SPDX-License-Identifier: Apache-2.0 import fs, { existsSync, readFileSync, writeFileSync } from "node:fs"; +import assert from "node:assert/strict"; import { createHash } from "node:crypto"; import os from "node:os"; import path, { join } from "node:path"; import { describe, expect, it, vi } from "vitest"; +import * as dockerRuntime from "../../../src/lib/onboard/runtime-provider/docker.ts"; +import * as podmanRuntime from "../../../src/lib/onboard/runtime-provider/podman.ts"; +import { isValidName } from "../../../src/lib/name-validation.ts"; +import { CleanupRegistry } from "../fixtures/cleanup.ts"; import { approveOpenClawAdminScope, @@ -24,10 +29,12 @@ import { import { ArtifactSink } from "../fixtures/artifacts.ts"; import { captureManagedImageOnboardPairingDiagnostics, + createLegacyManagedVolumeFixture, collectOnboardFailureRuntimeDiagnostics, externalImageActivationAgents, externalImageActivationMatches, externalImageActivationOnboardArgs, + inspectRuntimeSandboxContainerId, managedActivationPostRestartAgentTurnScript, managedActivationOpenClawPluginScript, managedHermesBoundaryPoisonCommand, @@ -42,6 +49,69 @@ import { } from "../live/managed-image-activation-e2e-helpers.ts"; import { pendingAdminRequestId } from "../fixtures/issue-4462-admin-approval-evidence.ts"; +describe("legacy managed volume fixture", () => { + it.each(["docker", "podman"] as const)( + "seeds owned legacy state and removes only recorded fixtures through %s", + async (runtime) => { + vi.stubEnv("NEMOCLAW_GATEWAY_RUNTIME", runtime); + const provider = + runtime === "docker" + ? dockerRuntime.createDockerRuntimeProviderBundle() + : podmanRuntime.createCurrentPodmanRuntimeProviderBundle(); + assert(provider.containerEngine.supported, "built-in engine required"); + const helperId = "a".repeat(64); + const capture = vi + .spyOn(provider.containerEngine, "capture") + .mockImplementation((_operation, args) => ({ + status: 0, + stdout: args[0] === "create" ? helperId : "", + stderr: "", + })); + vi.spyOn(dockerRuntime, "createDockerRuntimeProviderBundle").mockReturnValue(provider); + vi.spyOn(podmanRuntime, "createCurrentPodmanRuntimeProviderBundle").mockReturnValue(provider); + const cleanup = new CleanupRegistry(); + const fixture = createLegacyManagedVolumeFixture(cleanup); + expect(isValidName(fixture.sandboxName)).toBe(true); + expect(capture).not.toHaveBeenCalled(); + fixture.seed(); + const commands = () => capture.mock.calls.map((call) => call[1]); + expect(commands().find((args) => args[0] === "volume" && args[1] === "create")).toEqual([ + "volume", + "create", + ...Object.entries(fixture.root.ownershipLabels).flatMap(([key, value]) => [ + "--label", + `${key}=${value}`, + ]), + fixture.root.resourceIdentity, + ]); + const probe = commands().find((args) => args[0] === "create")!; + expect(probe.at(-1)).toContain(JSON.stringify(fixture.marker)); + expect(probe.join(" ")).toContain( + runtime === "docker" + ? `src=${fixture.root.resourceIdentity},dst=/state,volume-nocopy` + : `${fixture.root.resourceIdentity}:/state:rw,nocopy`, + ); + expect(await cleanup.runAll()).toEqual({ + passed: ["remove only the volume qualification fixtures"], + failures: [], + }); + expect(commands().filter((args) => args[0] === "volume" && args[1] === "rm")).toEqual([ + ["volume", "rm", fixture.root.resourceIdentity], + ]); + const failingCleanup = new CleanupRegistry(); + createLegacyManagedVolumeFixture(failingCleanup).seed(); + capture.mockImplementation((_operation, args) => ({ + status: args[0] === "volume" && args[1] === "rm" ? 1 : 0, + stdout: "", + stderr: "volume is in use", + })); + const failedCleanup = await failingCleanup.runAll(); + expect(failedCleanup.failures).toHaveLength(1); + expect(failedCleanup.failures[0]?.message).toContain("volume is in use"); + }, + ); +}); + const MANAGED_ADMIN_PUBLIC_KEY_BYTES = Buffer.from( Array.from({ length: 32 }, (_value, index) => index), ); @@ -94,7 +164,13 @@ function prepareManagedAdminState(root: string, requestId: string): NodeJS.Proce helperPath, `import json\nfrom pathlib import Path\ndef read_openclaw_pairing_state(state_dir, timeout=1):\n records=json.loads((Path(state_dir) / "pairing-state.json").read_text(encoding="utf-8"))\n return records, {"timeout": timeout}\n`, ); + fs.writeFileSync( + join(root, "bin", "mktemp"), + '#!/bin/sh\nexec /usr/bin/mktemp "$TMPDIR/tmp.XXXXXXXXXX"\n', + { mode: 0o755 }, + ); return { + TMPDIR: root, FAKE_DEVICES_STATE: devicesPath, NEMOCLAW_OPENCLAW_PAIRING_STATE_HELPER: helperPath, OPENCLAW_STATE_DIR: stateRoot, @@ -190,6 +266,27 @@ printf '%s\n' "$@" >"$MANAGED_ACTIVATION_FIXTURE/openclaw-args" } describe("managed image activation failure diagnostics", () => { + it.each(["docker", "podman"] as const)("selects full sandbox IDs through %s", async (engine) => { + const result = { exitCode: 0, stdout: `${"a".repeat(64)}\n`, stderr: "" }; + const command = vi.fn(async () => result); + const env = { NEMOCLAW_GATEWAY_RUNTIME: engine }; + await expect( + inspectRuntimeSandboxContainerId({ command } as never, engine, "candidate", "lookup", env), + ).resolves.toBe(result); + expect(command).toHaveBeenCalledExactlyOnceWith( + engine, + [ + "ps", + "-aq", + "--no-trunc", + "--filter", + "label=openshell.ai/sandbox-name=candidate", + "--filter", + "label=openshell.ai/isolation-role=sandbox", + ], + { artifactName: "lookup", env, timeoutMs: 30_000 }, + ); + }); it.each<{ diagnostic: string; expectedRequestId?: string; @@ -203,7 +300,7 @@ describe("managed image activation failure diagnostics", () => { ) => void; }>([ { - diagnostic: "selection-state-read", + diagnostic: "selection-state-permission", prepare: (env, _devices, secret) => { writeFileSync( env.NEMOCLAW_OPENCLAW_PAIRING_STATE_HELPER!, @@ -258,11 +355,6 @@ describe("managed image activation failure diagnostics", () => { const requestId = "4edc8df0-20d0-4308-b0e8-850843ae0cf4"; const secret = "selector-private-diagnostic-value"; const approveLog = fixture.path("approve.log"); - // BSD mktemp without a template selects the system temp directory instead of TMPDIR. - fixture.writeExecutable( - "mktemp", - '#!/bin/sh\nexec /usr/bin/mktemp "$TMPDIR/tmp.XXXXXXXXXX"\n', - ); fixture.writeExecutable( "openclaw", `#!/bin/sh @@ -937,48 +1029,81 @@ ${adminApprovalConnectScript("nemoclaw", "fixture-sandbox", "managed-cron", outp expect(ONBOARD_FAILURE_LOG_ARTIFACT_OPTIONS).toEqual({ persistArtifacts: true }); }); - it("redacts a copied failed-startup log before artifact publication", async () => { - const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-startup-diagnostics-")); - const secret = "supplied-startup-diagnostic-secret"; - const containerId = "a".repeat(64); - const artifacts = new ArtifactSink(directory); - const command = vi.fn(async (executable: string, args: readonly string[]) => { - switch (`${executable}:${String(args[0])}`) { - case "docker:ps": - return { - exitCode: 0, - stdout: `${containerId}\tmanaged-container\timage\tExited\n`, - stderr: "", - }; - case "docker:cp": - fs.writeFileSync(String(args[2]), `startup log contains ${secret}\n`); - break; - } - return { exitCode: 0, stdout: "", stderr: "" }; - }); - - try { - await collectOnboardFailureRuntimeDiagnostics( - artifacts, - { command } as never, - "openclaw", - "managed-openclaw", - {}, - [secret], - ); + it.each([ + { + kind: "regular", + create: (destination: string, secret: string) => + fs.writeFileSync(destination, `startup log contains ${secret}\n`), + expected: { data: "startup log contains [REDACTED]\n" }, + }, + { + kind: "symlink", + create: (destination: string, secret: string, directory: string) => { + const target = path.join(directory, "outside-copy.txt"); + fs.writeFileSync(target, `startup log contains ${secret}\n`); + fs.symlinkSync(target, destination); + }, + expected: { error: "ENOENT" }, + }, + { + kind: "directory", + create: (destination: string) => fs.mkdirSync(destination), + expected: { error: "ENOENT" }, + }, + { + kind: "oversized", + create: (destination: string) => + fs.writeFileSync(destination, "x".repeat(2 * 1024 * 1024 + 1)), + expected: { error: "ENOENT" }, + }, + ])( + "publishes only bounded regular startup logs with redaction: $kind", + async ({ create, expected }) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-startup-diagnostics-")); + const secret = "supplied-startup-diagnostic-secret"; + const containerId = "a".repeat(64); + const artifacts = new ArtifactSink(directory); + const command = vi.fn(async (executable: string, args: readonly string[]) => { + switch (`${executable}:${String(args[0])}`) { + case "docker:ps": + return { + exitCode: 0, + stdout: `${containerId}\tmanaged-container\timage\tExited\n`, + stderr: "", + }; + case "docker:cp": + create(String(args[2]), secret, directory); + break; + } + return { exitCode: 0, stdout: "", stderr: "" }; + }); - const published = fs.readFileSync( - artifacts.pathFor( - "managed-activation-onboard-failure-openclaw-container-1-nemoclaw-start.log", - ), - "utf8", - ); - expect(published).toContain("[REDACTED]"); - expect(published).not.toContain(secret); - } finally { - fs.rmSync(directory, { force: true, recursive: true }); - } - }); + try { + await collectOnboardFailureRuntimeDiagnostics( + artifacts, + { command } as never, + "openclaw", + "managed-openclaw", + {}, + [secret], + ); + const published = await fs.promises + .readFile( + artifacts.pathFor( + "managed-activation-onboard-failure-openclaw-container-1-nemoclaw-start.log", + ), + "utf8", + ) + .then( + (data) => ({ data }), + (error: NodeJS.ErrnoException) => ({ error: error.code }), + ); + expect(published).toEqual(expected); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }, + ); const diagnosticEvents = [ "tail:-c", @@ -1097,11 +1222,26 @@ ${adminApprovalConnectScript("nemoclaw", "fixture-sandbox", "managed-cron", outp } as never, { env: {} } as never, "/unused-catalog.json", - "http://inference.invalid/v1", + "http://10.20.30.40:43123/v1", "langchain-deepagents-code", {} as never, + { + agents: { + "langchain-deepagents-code": { + onboard: vi.fn(), + publicLifecycle: vi.fn(), + cleanup: vi.fn(), + }, + }, + hermesSecretBoundary: vi.fn(), + externalImages: vi.fn(), + } as never, ), ).rejects.toThrow(error); + expect(nemoclaw.mock.calls[0]?.[1]?.env).toMatchObject({ + NEMOCLAW_ENDPOINT_URL: "http://10.20.30.40:43123/v1", + NEMOCLAW_TRUSTED_PRIVATE_INFERENCE_HOSTS: "10.20.30.40", + }); expect(events).toEqual(expectedEvents); expect( copy.mock.calls.map(() => diff --git a/test/e2e/support/managed-image-external-rebuild-diagnostics.test.ts b/test/e2e/support/managed-image-external-rebuild-diagnostics.test.ts new file mode 100644 index 00000000000..96db1e04c5c --- /dev/null +++ b/test/e2e/support/managed-image-external-rebuild-diagnostics.test.ts @@ -0,0 +1,118 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { expect, it, vi } from "vitest"; +import { ArtifactSink } from "../fixtures/artifacts.ts"; +import * as failureDiagnostics from "../fixtures/sandbox-failure-diagnostics.ts"; +import { qualifyExternalImage } from "../live/managed-image-activation-e2e-helpers.ts"; + +it.each([ + { exitCode: 1, timedOut: false }, + { exitCode: null, timedOut: true }, +])("retains external rebuild diagnostics before cleanup for %j (#12603)", async (failure) => { + const directory = fs.mkdtempSync( + path.join(os.tmpdir(), "nemoclaw-external-rebuild-diagnostics-"), + ); + vi.spyOn(os, "homedir").mockReturnValue(directory); + vi.stubEnv("NEMOCLAW_GATEWAY_RUNTIME", "docker"); + const events: string[] = []; + const ok = { exitCode: 0, stdout: "", stderr: "", timedOut: false }; + const rebuild = { ...ok, ...failure, stderr: "Hermes post-restore verification failed" }; + const imageId = `sha256:${"a".repeat(64)}`; + const reference = `ghcr.io/nvidia/nemoclaw/hermes-sandbox@${imageId}`; + const receipt = { + schemaVersion: 1, + kind: "external-image", + reference, + platform: "linux/amd64", + runtimeImageContentId: imageId, + shared: true, + }; + fs.mkdirSync(path.join(directory, ".nemoclaw")); + fs.writeFileSync( + path.join(directory, ".nemoclaw/sandboxes.json"), + JSON.stringify({ + sandboxes: { "ext-img-hermes": { toolDisclosure: "progressive", workload: receipt } }, + }), + ); + const capture = vi + .spyOn(failureDiagnostics, "captureSandboxFailureDiagnostics") + .mockImplementation(async () => { + await new Promise((resolve) => setImmediate(resolve)); + events.push("diagnostics"); + }); + const ready = vi.fn(async () => undefined); + const host = { + nemoclaw: vi.fn(async (args: string[]) => { + const operation = args[0] === "onboard" ? "onboard" : args[1]; + events.push(operation!); + return operation === "rebuild" ? rebuild : ok; + }), + command: vi.fn(async (_command: string, args: string[], options: { artifactName: string }) => ({ + ...ok, + stdout: + options.artifactName === "activation-runtime-sandbox-id-ext-img-hermes-before-rebuild" + ? "hermes-before-rebuild" + : options.artifactName === "activation-runtime-sandbox-id-ext-img-hermes-after-rebuild" + ? "hermes-after-rebuild" + : args[0] === "image" + ? imageId + : "", + })), + bestEffortCleanupSandbox: vi.fn(async () => ok), + cleanupGatewayRegistration: vi.fn(async () => ok), + destroySandbox: vi.fn(async () => { + events.push("destroy"); + return ok; + }), + }; + try { + const result = await qualifyExternalImage( + { + artifacts: new ArtifactSink(directory), + cleanup: { trackDisposable: vi.fn(), trackSandbox: vi.fn() }, + host, + lifecycle: { stopGatewayRuntime: vi.fn(), waitForSandboxReadyAfterGatewayRestart: ready }, + sandbox: { + cleanupSandbox: vi.fn(async () => ok), + exec: vi.fn(async () => ({ ...ok, stdout: "PONG" })), + list: vi.fn(async () => ok), + }, + } as never, + { env: {} } as never, + "docker", + "/unused-catalog.json", + "http://10.20.30.40:43123/v1", + "hermes", + { reference, platform: "linux/amd64" } as never, + ); + expect({ + events, + rebuilt: result.rebuilt, + verified: result.verified, + readinessCalls: ready.mock.calls.length, + }).toEqual({ + events: ["onboard", "rebuild", "diagnostics", "destroy"], + rebuilt: false, + verified: false, + readinessCalls: 1, + }); + expect(capture).toHaveBeenCalledExactlyOnceWith(host, rebuild, { + sandboxName: "ext-img-hermes", + artifactPrefix: "external-image-hermes-rebuild-failure", + redactionValues: ["nemoclaw-managed-activation-e2e-key"], + captureGatewayLog: true, + }); + expect( + host.command.mock.calls.filter((call) => call[1][0] === "volume").map((call) => call[1]), + ).toEqual([ + ["volume", "ls", "-q", "--filter", "label=openshell.ai/sandbox-id=hermes-before-rebuild"], + ["volume", "ls", "-q", "--filter", "label=openshell.ai/sandbox-id=hermes-after-rebuild"], + ]); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); diff --git a/test/e2e/support/managed-image-legacy-container-contract.test.ts b/test/e2e/support/managed-image-legacy-container-contract.test.ts index e7502954198..e1acbd96228 100644 --- a/test/e2e/support/managed-image-legacy-container-contract.test.ts +++ b/test/e2e/support/managed-image-legacy-container-contract.test.ts @@ -59,7 +59,7 @@ it("rejects a runtime without a reviewed container identity contract", () => { ); }); -it.each(["0.1.3", "0.1.5"] as const)( +it.each(["0.1.3", "0.1.5", "0.1.7"] as const)( "requires the native isolation identity for a %s workload", (runtimeVersion) => { expect(requireManagedImageHarnessOpenShellVersion(runtimeVersion)).toBe(runtimeVersion); diff --git a/test/e2e/support/managed-image-multiarch-startup-helpers.test.ts b/test/e2e/support/managed-image-multiarch-startup-helpers.test.ts index 1234f8f584f..791c31f3dc5 100644 --- a/test/e2e/support/managed-image-multiarch-startup-helpers.test.ts +++ b/test/e2e/support/managed-image-multiarch-startup-helpers.test.ts @@ -11,6 +11,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { DOCKER_ENGINE_27_MINIMUM_CLEANUP_PROCESS_TIMEOUT_MS, DOCKER_ENGINE_27_MINIMUM_PROBE_TIMEOUT_MS, + cleanupDockerEngine27ReceiptDaemon, dockerEngine27ReceiptDaemonName, dockerEngine27ReceiptIdentityArguments, finalizeDockerEngine27ReceiptProbe, @@ -24,9 +25,12 @@ import { } from "../live/managed-image-multiarch-startup-helpers.ts"; const sha = "a".repeat(40); +const docker = vi.hoisted(() => vi.fn()); +vi.mock("node:child_process", () => ({ spawnSync: docker })); let temporaryRoot = ""; beforeEach(() => { + docker.mockReset(); temporaryRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-multiarch-helper-")); const artifacts = path.join(temporaryRoot, "artifacts"); fs.mkdirSync(artifacts); @@ -49,6 +53,37 @@ afterEach(() => { }); describe("protected managed-image startup helpers", () => { + it("removes the owned daemon with its anonymous data volumes and verifies absence", () => { + const name = dockerEngine27ReceiptDaemonName(123, 1, "linux/amd64"); + docker + .mockReturnValueOnce({ status: 0, stdout: name, stderr: "" }) + .mockReturnValueOnce({ status: 0, stdout: name, stderr: "" }) + .mockReturnValueOnce({ status: 1, stdout: "", stderr: `No such container: ${name}` }); + + cleanupDockerEngine27ReceiptDaemon(name); + + expect(docker.mock.calls.map((call) => call[1])).toEqual([ + [ + "container", + "inspect", + "--format", + '{{ index .Config.Labels "io.nvidia.nemoclaw.e2e.docker27-receipt" }}', + name, + ], + ["rm", "-f", "--volumes", name], + ["container", "inspect", name], + ]); + }); + + it("preserves a daemon and its volumes when the ownership label differs", () => { + docker.mockReturnValueOnce({ status: 0, stdout: "another-run", stderr: "" }); + + expect(() => + cleanupDockerEngine27ReceiptDaemon(dockerEngine27ReceiptDaemonName(123, 1, "linux/amd64")), + ).toThrow("mismatched ownership"); + expect(docker).toHaveBeenCalledOnce(); + }); + it("preserves the probe failure and attempts fixture cleanup after daemon cleanup fails", () => { const cleanupFixture = vi.fn(); diff --git a/test/e2e/support/managed-image-protected-runtime-budget.test.ts b/test/e2e/support/managed-image-protected-runtime-budget.test.ts index c6927e38356..0396aa69f9e 100644 --- a/test/e2e/support/managed-image-protected-runtime-budget.test.ts +++ b/test/e2e/support/managed-image-protected-runtime-budget.test.ts @@ -11,7 +11,7 @@ import { PROTECTED_RUNTIME_JOB_MINUTES, PROTECTED_RUNTIME_ROLLBACK_MS, PROTECTED_RUNTIME_RUNNER_HEADROOM_MS, -} from "../live/managed-image-protected-runtime-budget.ts"; +} from "../live/managed-image-protected-runtime-helpers.ts"; const MINUTE = 60_000; const START = 1_700_000_000_000; diff --git a/test/e2e/support/managed-image-protected-runtime-readiness.test.ts b/test/e2e/support/managed-image-protected-runtime-readiness.test.ts index 16b26229ae3..c9b453fa675 100644 --- a/test/e2e/support/managed-image-protected-runtime-readiness.test.ts +++ b/test/e2e/support/managed-image-protected-runtime-readiness.test.ts @@ -346,7 +346,7 @@ describe.concurrent("protected managed-image runtime commands", () => { { case: "short", stdout: "abc123" }, { case: "ambiguous", stdout: `${PROVIDER_CONTAINER_ID}\n${"d".repeat(64)}` }, ])("rejects a $case provider run ID before authority capture", ({ stdout }) => { - expect(() => protectedProviderReportedContainerId(VLLM_PROVIDER_NAME, stdout)).toThrow( + expect(() => protectedProviderReportedContainerId(stdout)).toThrow( "did not report one full container ID", ); }); @@ -526,7 +526,7 @@ describe.concurrent("protected managed-image runtime commands", () => { expect(ollama.command.slice(0, 2)).toEqual(["bash", "-c"]); expect(ollama.exitCode).toBe(0); - expect(ollama.stdout).toBe("restart_mode=manual\nmanaged-image-ollama-ready\n"); + expect(ollama.stdout).toBe("restart_mode=manual\n"); expect(vllm.command.slice(0, 4)).toEqual(["bash", "--noprofile", "--norc", "-c"]); expect(vllm.exitCode).toBe(0); expect(vllm.stdout).toBe("managed-image-vllm-ready attempts=1\n"); @@ -565,7 +565,6 @@ describe.concurrent("protected managed-image runtime commands", () => { expect(result.command.slice(0, 2)).toEqual(["bash", "-c"]); expect(result.exitCode).toBe(1); expect(result.timedOut).toBe(false); - expect(result.stderr).toContain("managed-image-ollama-not-ready status=1"); expect(diagnosticLines).toHaveLength(200); expect(diagnosticLines[0]).toContain("runtime-log-line-061"); expect(diagnosticLines.at(-1)).toContain("runtime-log-line-260"); @@ -607,7 +606,6 @@ describe.concurrent("protected managed-image runtime commands", () => { expect(result.exitCode).toBe(1); expect(result.stderr).toContain("[shell-probe omitted "); - expect(result.stderr).toContain("managed-image-ollama-not-ready status=1"); expect(result.stderr).toContain("[REDACTED]"); expect(result.stderr).not.toContain(sensitiveValue); expect(Buffer.byteLength(stderrArtifact)).toBeLessThanOrEqual(command.captureLimitBytes + 256); diff --git a/test/e2e/support/mcp-bridge-cleanup.test.ts b/test/e2e/support/mcp-bridge-cleanup.test.ts new file mode 100644 index 00000000000..1dc3421a021 --- /dev/null +++ b/test/e2e/support/mcp-bridge-cleanup.test.ts @@ -0,0 +1,23 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import { shouldRetryMcpMutationAfterConcurrencyConflict } from "../live/mcp-bridge-cleanup.ts"; + +describe("MCP mutation concurrency retry", () => { + it("retries the explicit OpenShell optimistic-concurrency response", () => { + expect( + shouldRetryMcpMutationAfterConcurrencyConflict( + "Failed to detach provider: sandbox was modified by another operation.\nPlease retry the command.", + ), + ).toBe(true); + }); + + it.each([ + "Failed to detach provider: permission denied", + "sandbox was modified by another operation.", + "Please retry the command.", + ])("does not retry another failure: %s", (output) => { + expect(shouldRetryMcpMutationAfterConcurrencyConflict(output)).toBe(false); + }); +}); diff --git a/test/e2e/support/mcp-bridge-hermes-http.test.ts b/test/e2e/support/mcp-bridge-hermes-http.test.ts index 67cd5181df4..478a826d11a 100644 --- a/test/e2e/support/mcp-bridge-hermes-http.test.ts +++ b/test/e2e/support/mcp-bridge-hermes-http.test.ts @@ -38,6 +38,7 @@ function httpResult(status: number, body = "", result = "") { return { exitCode: 0, signal: null, + timedOut: false, stdout: body, stderr: `${HERMES_MCP_HTTP_STATUS_MARKER}${status}\n${result}`, }; @@ -348,7 +349,7 @@ describe("Hermes MCP HTTP failure diagnostics", () => { "--filter", "label=openshell.ai/sandbox-name=owned-hermes", "--format", - "{{.ID}}", + '{{.ID}}\t{{.Label "openshell.ai/isolation-role"}}', ]); }, ); @@ -441,6 +442,19 @@ describe("Hermes MCP HTTP failure diagnostics", () => { ).not.toThrow(); }); + it.each([ + ["times out", { timedOut: true, signal: null }, "timeout"], + ["receives SIGTERM", { timedOut: false, signal: "SIGTERM" as const }, "signal=SIGTERM"], + ])("rejects a complete HTTP response when the command %s", (_label, interruption, detail) => { + const result = { + ...httpResult(200, "", `${HERMES_MCP_RESULT_TOKEN_MARKER}present\n`), + ...interruption, + }; + expect(() => assertHermesMcpHttpResponse(result, [])).toThrow( + `Hermes real MCP tool call transport failed (${detail})`, + ); + }); + it("classifies only the exact Hermes gateway draining response", () => { const draining = JSON.stringify({ error: { code: "gateway_draining" } }); expect(isHermesGatewayDrainingResponse(httpResult(503, draining))).toBe(true); diff --git a/test/e2e/support/mcp-bridge-https-diagnostics.test.ts b/test/e2e/support/mcp-bridge-https-diagnostics.test.ts index 8f8bb4bffe0..3349fbc5a3d 100644 --- a/test/e2e/support/mcp-bridge-https-diagnostics.test.ts +++ b/test/e2e/support/mcp-bridge-https-diagnostics.test.ts @@ -8,7 +8,10 @@ import path from "node:path"; import https from "node:https"; import net from "node:net"; import { afterAll, describe, expect, it, vi } from "vitest"; -import { startFakeMcpHttpsServer } from "../live/mcp-bridge-servers.ts"; +import { + startFakeMcpHttpsServer, + withCompatibleMockFailureDiagnostics, +} from "../live/mcp-bridge-servers.ts"; import { shouldRetryMcpDiscoveryAfterRestart } from "../live/mcp-bridge-tool-discovery.ts"; import { createMcpFixtureTls } from "../fixtures/mcp-fixture-tls.ts"; import { CleanupRegistry } from "../fixtures/cleanup.ts"; @@ -268,3 +271,54 @@ it.each([ fs.rmSync(parent, { recursive: true, force: true }); } }); + +describe("compatible endpoint diagnostic failures", () => { + it("preserves the approval error when the diagnostic snapshot throws", async () => { + const failure = new Error("approval failed"); + const diagnostics = vi.fn(() => { + throw new Error("snapshot failed"); + }); + const writeJson = vi.fn(); + const operation = vi.fn().mockRejectedValue(failure); + await expect( + withCompatibleMockFailureDiagnostics({ diagnostics }, { writeJson }, operation), + ).rejects.toBe(failure); + expect(operation).toHaveBeenCalledOnce(); + expect(diagnostics).toHaveBeenCalledOnce(); + expect(writeJson).not.toHaveBeenCalled(); + }); + + it("bounds stalled persistence and preserves the approval error", async () => { + vi.useFakeTimers(); + const failure = new Error("approval failed"); + const diagnostics = vi + .fn() + .mockReturnValue({ receivedCount: 0, retainedLimit: 64, requests: [] }); + let release!: (value: string) => void; + const pending = new Promise((resolve) => { + release = resolve; + }); + const writeJson = vi.fn().mockReturnValue(pending); + const operation = vi.fn().mockRejectedValue(failure); + const outcome = withCompatibleMockFailureDiagnostics( + { diagnostics }, + { writeJson }, + operation, + ).catch((error: unknown) => error); + const settled = vi.fn(); + void outcome.then(settled); + try { + await vi.advanceTimersByTimeAsync(9_999); + expect(writeJson).toHaveBeenCalledOnce(); + expect(settled).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + expect(await outcome).toBe(failure); + expect(operation).toHaveBeenCalledOnce(); + expect(diagnostics).toHaveBeenCalledOnce(); + expect(vi.getTimerCount()).toBe(0); + } finally { + release("evidence.json"); + vi.useRealTimers(); + } + }); +}); diff --git a/test/e2e/support/mcp-bridge-reliability.test.ts b/test/e2e/support/mcp-bridge-reliability.test.ts index e0a0011c7ec..9c2ca4f1292 100644 --- a/test/e2e/support/mcp-bridge-reliability.test.ts +++ b/test/e2e/support/mcp-bridge-reliability.test.ts @@ -194,6 +194,7 @@ function gatewayResult(status: number, code: string) { return { exitCode: 0, signal: null, + timedOut: false, stdout: JSON.stringify({ error: { code } }), stderr: `${HTTP_STATUS_MARKER}${status}\n`, }; @@ -544,6 +545,58 @@ describe("MCP bridge transient classification", () => { expect(sleep).not.toHaveBeenCalled(); }); + it.each([ + ["openclaw-config", { exitCode: 1 }], + ["openclaw-config", { exitCode: null }], + ["openclaw-config", { signal: "SIGTERM" as const }], + ["openclaw-config", { timedOut: true }], + ["hermes-config", { exitCode: 1 }], + ["hermes-config", { exitCode: null }], + ["hermes-config", { signal: "SIGTERM" as const }], + ["hermes-config", { timedOut: true }], + ["deepagents-config", { exitCode: 1 }], + ["deepagents-config", { exitCode: null }], + ["deepagents-config", { signal: "SIGTERM" as const }], + ["deepagents-config", { timedOut: true }], + ])( + "requires successful status before accepting %s registration with %j", + async (expectedAdapter, failure) => { + const status = vi.fn(async () => ({ + ...HERMES_RESTART_SETTLING_STATUS, + stdout: JSON.stringify({ + ...HERMES_RESTART_SETTLING_PAYLOAD, + adapter: { registered: true }, + }), + ...failure, + })); + const execShell = vi.fn(); + const sleep = vi.fn(async () => undefined); + const outcome = await readConcurrentMcpStatusAndConfirmHermesRegistration({ + clients: { + artifacts: { writeJson: vi.fn(async (relativePath: string) => relativePath) }, + host: { nemoclaw: status } as unknown as Pick, + sandbox: { execShell }, + }, + committedAddResult: HERMES_ADD_POST_PROBE_NOT_READY, + credentialEnvName: "FAKE_MCP_SECRET", + env: {}, + redactionValues: [], + scenario: { + artifactPrefix: "registration", + expectedAdapter, + mcpUrl: HERMES_RESTART_SETTLING_PAYLOAD.url, + sandboxName: "test-sandbox", + }, + server: "concurrent", + sleep, + }); + expect(outcome.registered).toBe(false); + expect(status).toHaveBeenCalledOnce(); + expect(execShell).not.toHaveBeenCalled(); + expect(sleep).not.toHaveBeenCalled(); + }, + ); + it("does not repeat full status when Hermes registration needs a read-only confirmation (#9485)", async () => { const status = vi.fn(async () => HERMES_RESTART_SETTLING_STATUS); const execShell = vi @@ -851,6 +904,21 @@ describe("MCP bridge transient classification", () => { expect(wait).toHaveBeenCalledTimes(3); }); + it.each([ + ["times out", { timedOut: true, signal: null }], + ["receives SIGTERM", { timedOut: false, signal: "SIGTERM" as const }], + ])("does not retry gateway draining when the command %s", async (_label, interruption) => { + const failed = { ...gatewayResult(503, "gateway_draining"), ...interruption }; + const retry = vi.fn(async () => gatewayResult(200, "none")); + const wait = vi.fn(async () => undefined); + + await expect(retryHermesGatewayDraining({ initialResult: failed, retry, wait })).resolves.toBe( + failed, + ); + expect(retry).not.toHaveBeenCalled(); + expect(wait).not.toHaveBeenCalled(); + }); + it("does not retry a different Hermes HTTP failure", async () => { const failed = gatewayResult(503, "other"); const retry = vi.fn(async () => gatewayResult(200, "none")); diff --git a/test/e2e/support/mcp-bridge-runtime-compatibility.test.ts b/test/e2e/support/mcp-bridge-runtime-compatibility.test.ts index 6ffcd4cc336..a68881cec1a 100644 --- a/test/e2e/support/mcp-bridge-runtime-compatibility.test.ts +++ b/test/e2e/support/mcp-bridge-runtime-compatibility.test.ts @@ -48,7 +48,7 @@ function assertRuntimeVersion(version: string): () => void { describe("MCP bridge dev runtime compatibility", () => { it("selects the full lifecycle for the reviewed OpenShell runtime (#6426)", () => { - expect(MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION).toBe("0.0.116"); + expect(MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION).toBe("0.1.7"); expect( classifyMcpBridgeRuntimeCompatibility( assertRuntimeVersion(MCP_CREDENTIAL_BOUNDARY_OPENSHELL_VERSION), @@ -60,7 +60,7 @@ describe("MCP bridge dev runtime compatibility", () => { }); }); - it.each(["", "0.0.117"])( + it.each(["", "0.1.2"])( "does not accept an injected unvalidated runtime version %j", (actualVersion) => { expect(classifyMcpBridgeRuntimeCompatibility(() => actualVersion)).toEqual({ diff --git a/test/e2e/support/mcp-bridge-sandbox.test.ts b/test/e2e/support/mcp-bridge-sandbox.test.ts index 65bc08b2efb..e2af4582de6 100644 --- a/test/e2e/support/mcp-bridge-sandbox.test.ts +++ b/test/e2e/support/mcp-bridge-sandbox.test.ts @@ -1,25 +1,38 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { spawnSync } from "node:child_process"; +import { execFile, spawnSync } from "node:child_process"; +import { promisify } from "node:util"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import YAML from "yaml"; +import * as dnsServer from "../fixtures/dns-rebinding-server.ts"; import * as policy from "../../../src/lib/policy"; import { testTimeout } from "../../helpers/timeouts"; +import { + isExpectedMcpDnsPolicyDenial, + waitForMcpCurlPolicyDenial, +} from "../fixtures/curl-policy-denial.ts"; +import { createMcpFixtureTls } from "../fixtures/mcp-fixture-tls.ts"; +import { startFakeMcpHttpsServer } from "../live/mcp-bridge-servers.ts"; import { ArtifactSink } from "../fixtures/artifacts.ts"; import type { HostCliClient } from "../fixtures/clients/host.ts"; +import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; import { assertManagedMcpPolicySurvivedRemoval, buildMcpDnsRebindingProbeScript, + buildMcpSecretAbsenceScript, + captureManagedMcpPolicy, hostAddressForSandbox, hostPrivateAddressForSandbox, isExpectedMcpCurlPolicyDenial, restoreDnsRebindingHostsFixture, + setupDnsRebindingHostsFixture, + remapDnsRebindingHostname, } from "../live/mcp-bridge-sandbox.ts"; import { assertRawOpenShellAllowedIpsRebindingDenied, @@ -34,8 +47,115 @@ import { const SUITE_OPTIONS = { timeout: testTimeout(15_000) }; const tempDirs: string[] = []; +const WORKLOAD_ID = "a".repeat(64); +const SUPERVISOR_ID = "b".repeat(64); + +describe.each(["/bin/sh", "/bin/bash"])("MCP secret absence scan with %s", (shell) => { + function fixture() { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-mcp-secret-scan-")); + tempDirs.push(root); + const state = path.join(root, "agent state"); + fs.mkdirSync(state); + return { root, state, file: path.join(state, "config") }; + } + + function scan(paths: string[], secrets: string[], env: NodeJS.ProcessEnv = process.env) { + const result = spawnSync(shell, ["-c", buildMcpSecretAbsenceScript(paths, secrets)], { + encoding: "utf8", + env, + timeout: 5_000, + }); + expect(result.error).toBeUndefined(); + expect(result.stdout).toBe(""); + expect(result.stderr).toBe(""); + return result.status; + } + + it.each([0, 1])("rejects credential %s without printing its contents", (index) => { + const { state, file } = fixture(); + const secrets = ["fixture-first-secret", "fixture-rotated-secret"]; + fs.writeFileSync(file, secrets[index]); + expect(scan([state], secrets)).toBe(1); + }); + + it("rejects a match in a later path", () => { + const { root, state, file } = fixture(); + fs.writeFileSync(file, "clean"); + const log = path.join(root, "log"); + fs.writeFileSync(log, "fixture-secret"); + expect(scan([state, log], ["fixture-secret"])).toBe(1); + }); + + it("accepts a clean tree and a confirmed absent optional file", () => { + const { root, state, file } = fixture(); + fs.writeFileSync(file, "clean"); + expect(scan([state, path.join(root, "optional")], ["fixture-secret"])).toBe(0); + }); + + it("rejects a scan error without printing the diagnostic", () => { + const { root, state } = fixture(); + fs.writeFileSync(path.join(root, "grep"), "#!/bin/sh\necho sensitive >&2\nexit 2\n", { + mode: 0o700, + }); + expect( + scan([state], ["fixture-secret"], { ...process.env, PATH: `${root}:${process.env.PATH}` }), + ).toBe(2); + }); + + it("rejects a dangling path supplied to the scan", () => { + const { root } = fixture(); + const link = path.join(root, "dangling"); + fs.symlinkSync(path.join(root, "missing"), link); + expect(scan([link], ["fixture-secret"])).toBe(2); + }); + + it("rejects absence when the parent cannot be inspected", () => { + const { root } = fixture(); + expect(scan([path.join(root, "missing-parent", "state")], ["fixture-secret"])).toBe(2); + }); + + it("treats credential metacharacters as literal text", () => { + const { state, file } = fixture(); + const secret = "fixture.*'$(printf injected)"; + fs.writeFileSync(file, secret); + expect(scan([state], [secret])).toBe(1); + fs.writeFileSync(file, "fixtureXXX"); + expect(scan([state], [secret])).toBe(0); + }); + + it("rejects a scan without paths or credentials", () => { + expect(() => buildMcpSecretAbsenceScript([], ["fixture-secret"])).toThrow(); + expect(() => buildMcpSecretAbsenceScript(["/sandbox"], [])).toThrow(); + expect(() => buildMcpSecretAbsenceScript(["/sandbox"], [""])).toThrow(); + }); +}); + +function resolverRecord(id: string, role: string | null, name = "test-sandbox") { + return { + id, + role, + name, + sandboxId: "sandbox-id", + namespace: "default", + running: true, + pid: 321, + networkMode: "host", + resolvPath: `/var/lib/docker/containers/${id}/resolv.conf`, + }; +} + +beforeEach(() => { + vi.spyOn(dnsServer, "startRebindingDnsServer").mockResolvedValue({ + port: 1053, + family: 4, + answers: () => 1, + setAddress: vi.fn(), + close: vi.fn(async () => {}), + }); +}); afterEach(() => { + vi.useRealTimers(); for (const tempDir of tempDirs.splice(0)) { fs.rmSync(tempDir, { force: true, recursive: true }); } @@ -62,6 +182,7 @@ function denialResult( } = {}, ) { return { + signal: null, exitCode: overrides.exitCode ?? 0, stderr: overrides.stderr ?? "", stdout: overrides.stdout ?? "", @@ -69,26 +190,115 @@ function denialResult( }; } -async function captureRestoreCommand(hostBackupPath: string, sandboxBackupPath: string) { - let restoreArgs: string[] = []; - let restoreScript = ""; - const host = { - command: async (_command: string, args: string[]) => { - restoreScript = args[1] ?? ""; - restoreArgs = args.slice(2); - return denialResult(); - }, - } as unknown as HostCliClient; +describe("MCP curl policy denial classification", SUITE_OPTIONS, () => { + it.each([ + {}, + { managed: {} }, + { managed: { endpoints: [{ host: "other.example.test", allowed_ips: ["10.0.0.2"] }] } }, + { managed: { endpoints: [{ host: "mcp.example.test", allowed_ips: [] }] } }, + ])("rejects missing or mismatched managed endpoint evidence %j", async (networkPolicies) => { + const sandbox = { + openshell: async () => ({ + ...denialResult(), + stdout: YAML.stringify({ version: 1, network_policies: networkPolicies }), + }), + } as unknown as SandboxClient; + await expect( + captureManagedMcpPolicy(sandbox, { + artifactName: "managed-policy", + label: "capture managed policy", + policyKey: "managed", + sandboxName: "test-sandbox", + url: "https://mcp.example.test/mcp", + }), + ).rejects.toThrow(/expected managed MCP host|expected at least one managed MCP address pin/); + }); - await restoreDnsRebindingHostsFixture(host, "test-sandbox", { - hostname: "mcp-rebind.example.test", - hostBackupPath, - sandboxBackupPath, + it("binds DNS redirection to the selected supervisor cgroup", async () => { + const role = "sandbox"; + const id = SUPERVISOR_ID; + const command = vi.fn( + async (_command: string, args: string[], options?: { artifactName?: string }) => { + const stdout = + { + "dns-rebinding-workload-resource": `${WORKLOAD_ID}\t${role ?? ""}\n`, + "dns-rebinding-supervisor-resource": `${SUPERVISOR_ID}\n`, + "dns-rebinding-resolver-cgroup": JSON.stringify({ + cgroup: `system.slice/docker-${SUPERVISOR_ID}.scope`, + upstream: "127.0.0.53", + }), + "dns-rebinding-resolver-identity": JSON.stringify( + resolverRecord(args.at(-1)!, args.at(-1) === WORKLOAD_ID ? role : "supervisor"), + ), + }[options?.artifactName ?? ""] ?? ""; + return { ...denialResult(), stdout }; + }, + ); + const host = { command } as unknown as HostCliClient; + const fixture = await setupDnsRebindingHostsFixture( + host, + "test-sandbox", + "rebind.example.test", + ); + expect(fixture).toMatchObject({ + containerId: id, + cgroup: `system.slice/docker-${id}.scope`, + upstream: "127.0.0.53", + }); + await remapDnsRebindingHostname(host, "test-sandbox", fixture, "10.20.30.40", "remap"); + const script = command.mock.calls.at(-1)![1][1]; + expect(script).toContain("sudo -n tee /etc/hosts"); + const rule = command.mock.calls.find(([, args]) => args.includes("-A"))?.[1]; + expect(rule).toEqual( + expect.arrayContaining([ + "--path", + `system.slice/docker-${id}.scope`, + "-d", + "127.0.0.53", + "--dport", + "53", + ]), + ); + await restoreDnsRebindingHostsFixture(host, "test-sandbox", fixture); }); - return { restoreArgs, restoreScript }; -} -describe("MCP curl policy denial classification", SUITE_OPTIONS, () => { + it.each(["duplicate", "wrong-namespace", "wrong-path", "stopped"])( + "rejects %s resolver evidence before modifying DNS", + async (failure) => { + const command = vi.fn( + async (_command: string, args: string[], options?: { artifactName?: string }) => { + const id = args.at(-1)!; + const supervisorOverrides = { + duplicate: {}, + "wrong-namespace": { namespace: "other" }, + "wrong-path": { resolvPath: "/etc/resolv.conf" }, + stopped: { running: false }, + }[failure]; + const record = { + ...resolverRecord(id, id === WORKLOAD_ID ? "sandbox" : "supervisor"), + ...(id === SUPERVISOR_ID ? supervisorOverrides : {}), + }; + const stdout = + { + "dns-rebinding-workload-resource": `${WORKLOAD_ID}\tsandbox\n`, + "dns-rebinding-supervisor-resource": + failure === "duplicate" ? `${SUPERVISOR_ID}\n${SUPERVISOR_ID}` : SUPERVISOR_ID, + "dns-rebinding-resolver-identity": JSON.stringify(record), + }[options?.artifactName ?? ""] ?? `${WORKLOAD_ID}\tsandbox\n`; + return { ...denialResult(), stdout }; + }, + ); + await expect( + setupDnsRebindingHostsFixture( + { command } as unknown as HostCliClient, + "test-sandbox", + "rebind.example.test", + ), + ).rejects.toThrow(); + expect(command.mock.calls.some(([name]) => name === "bash")).toBe(false); + }, + ); + it("separates the managed endpoint alias from the concrete rebinding address", async () => { let probeScript = ""; const host = { @@ -105,7 +315,9 @@ describe("MCP curl policy denial classification", SUITE_OPTIONS, () => { it("accepts an L7 HTTP 403 denial", () => { expect( - isExpectedMcpCurlPolicyDenial(denialResult({ stdout: "NEMOCLAW_MCP_CURL_HTTP_CODE=403\n" })), + isExpectedMcpCurlPolicyDenial( + denialResult({ stdout: '{"error":"policy_denied"}\nNEMOCLAW_MCP_CURL_HTTP_CODE=403\n' }), + ), ).toBe(true); }); @@ -147,6 +359,163 @@ describe("MCP curl policy denial classification", SUITE_OPTIONS, () => { ).toBe(false); }); + it.each([ + ["* Immediate connect fail for 198.18.0.1: Permission denied", true], + ["* Immediate connect fail for fd00::1: Permission denied", true], + ["* Immediate connect fail for 198.18.0.1: Connection refused", false], + [ + "* Immediate connect fail for 198.18.0.1: Permission denied\n* Immediate connect fail for fd00::1: Connection refused", + false, + ], + ["curl: (7) Permission denied", false], + ["* Failed to open file: Permission denied", false], + ["curl: (6) Could not resolve host", false], + [ + "* Immediate connect fail for 198.18.0.1: Permission denied\n* connect to 10.0.0.1 port 443 from 10.0.0.2 port 5000 failed: Connection refused", + false, + ], + ])("classifies curl connect diagnostic %s", (stderr, expected) => { + expect( + isExpectedMcpCurlPolicyDenial( + denialResult({ exitCode: 7, stdout: "NEMOCLAW_MCP_CURL_HTTP_CODE=000\n", stderr }), + ), + ).toBe(expected); + }); + + it.each([ + ["000", 7, "* Immediate connect fail for 198.18.0.1: Permission denied", false], + ["302", 7, "* Immediate connect fail for 198.18.0.1: Permission denied", true], + ["302", 7, "* Immediate connect fail for 198.18.0.1: Connection refused", false], + ["302", 7, "curl: (7) Permission denied", false], + ["302", 0, "", false], + ["302", 28, "* Immediate connect fail for 198.18.0.1: Permission denied", false], + ] as const)( + "requires explicit redirect evidence %s/%s/%s", + (expectedCode, exitCode, stderr, expected) => { + expect( + isExpectedMcpCurlPolicyDenial( + denialResult({ + exitCode, + stdout: + "HTTP/1.1 302 Found\r\nlocation: http://host.openshell.internal:8000/v1/chat/completions\r\n\r\nNEMOCLAW_MCP_CURL_HTTP_CODE=302\n", + stderr, + }), + "NEMOCLAW_MCP_CURL_HTTP_CODE=", + expectedCode, + ), + ).toBe(expected); + }, + ); + + it.each([ + "NEMOCLAW_MCP_CURL_HTTP_CODE=403\nNEMOCLAW_MCP_CURL_HTTP_CODE=200\npolicy_denied", + "NEMOCLAW_MCP_CURL_HTTP_CODE=403\nNEMOCLAW_MCP_CURL_HTTP_CODE=403\npolicy_denied", + "NEMOCLAW_MCP_CURL_HTTP_CODE=403\ncredential_endpoint_mismatch", + ])("rejects ambiguous or nonpolicy HTTP evidence [case %#]", (stdout) => { + expect(isExpectedMcpCurlPolicyDenial(denialResult({ stdout }))).toBe(false); + }); + + it.each(["openclaw-config", "hermes-config", "deepagents-config", null] as const)( + "executes curl with adapter %s and removes outgoing header diagnostics", + (adapter) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "mcp-curl-command-")); + tempDirs.push(directory); + const capture = path.join(directory, "argv"); + fs.writeFileSync(path.join(directory, "nemoclaw-start"), '#!/bin/sh\nexec "$@"\n', { + mode: 0o755, + }); + fs.writeFileSync( + path.join(directory, "curl"), + '#!/bin/sh\nprintf "%s\\n" "$@" > "$CURL_ARGV_CAPTURE"\nprintf "NEMOCLAW_MCP_CURL_HTTP_CODE=200\\n"\nprintf "> authorization: Bearer runtime-handle\\n* Immediate connect fail for 198.18.0.1: Permission denied\\n" >&2\n', + { mode: 0o755 }, + ); + const script = buildMcpDnsRebindingProbeScript( + adapter, + "https://mcp.example.test/mcp", + "REBIND_MCP_SECRET", + ) + .replaceAll("/tmp/nemoclaw-mcp-rebinding", path.join(directory, "probe")) + .replace("/opt/hermes/.venv/bin/python", "python3") + .replace("/opt/venv/bin/python3", "python3"); + const result = spawnSync("/bin/bash", ["-c", script], { + encoding: "utf8", + env: { + ...process.env, + PATH: `${directory}:${process.env.PATH}`, + CURL_ARGV_CAPTURE: capture, + REBIND_MCP_SECRET: "openshell:resolve:env:v1_REBIND_MCP_SECRET", + }, + }); + expect(result.status, result.stderr).toBe(0); + expect(fs.readFileSync(capture, "utf8")).toContain( + adapter === null + ? "authorization: Bearer openshell:resolve:env:REBIND_MCP_SECRET" + : "authorization: Bearer openshell:resolve:env:v1_REBIND_MCP_SECRET", + ); + const argumentsCaptured = fs.readFileSync(capture, "utf8").trim().split("\n"); + expect(JSON.parse(argumentsCaptured.at(-1)!)).toMatchObject({ + id: 1, + method: "initialize", + params: { protocolVersion: "2025-11-25" }, + }); + expect(argumentsCaptured).not.toContain("mcp-protocol-version: 2025-11-25"); + expect(result.stderr).not.toContain("authorization:"); + expect(result.stderr).toContain("Immediate connect fail"); + }, + ); + + it.each([true, false])( + "sends one initialization control with accepted credential=%s", + async (accepted) => { + const tls = createMcpFixtureTls(); + const server = await startFakeMcpHttpsServer({ + tls: tls.tls, + secret: accepted ? "openshell:resolve:env:REBIND_MCP_SECRET" : "different-credential", + }); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "mcp-rebind-initialize-")); + tempDirs.push(directory); + const caPath = path.join(directory, "ca.pem"); + fs.writeFileSync(caPath, tls.tls.cert); + try { + const script = buildMcpDnsRebindingProbeScript( + null, + `https://127.0.0.1:${server.port}/mcp`, + "REBIND_MCP_SECRET", + ).replaceAll("/tmp/nemoclaw-mcp-rebinding", path.join(directory, "probe")); + const result = await promisify(execFile)("/bin/bash", ["-c", script], { + env: { ...process.env, CURL_CA_BUNDLE: caPath }, + timeout: 10_000, + }); + expect(result.stdout).toContain(`NEMOCLAW_MCP_CURL_HTTP_CODE=${accepted ? 200 : 401}`); + expect( + server.requests.map((request) => ({ + method: request.method, + path: request.path, + rpcMethod: request.rpcMethod, + id: JSON.parse(request.body).id, + auth: request.auth, + sessionId: request.sessionId, + protocolVersion: request.protocolVersion, + })), + ).toEqual([ + { + method: "POST", + path: "/mcp", + rpcMethod: "initialize", + id: 1, + auth: "Bearer openshell:resolve:env:REBIND_MCP_SECRET", + sessionId: "", + protocolVersion: "", + }, + ]); + expect(result.stderr).not.toContain("authorization:"); + } finally { + await server.close(); + tls.close(); + } + }, + ); + it("runs the rebinding request beneath each adapter runtime identity", () => { const runtimes = { "openclaw-config": "nemoclaw-start node -e", @@ -164,24 +533,23 @@ describe("MCP curl policy denial classification", SUITE_OPTIONS, () => { expect(script, adapter).toMatch(/spawnSync|subprocess\.run/); expect(script, adapter).toContain("'curl'"); expect(script, adapter).toContain("NEMOCLAW_MCP_CURL_HTTP_CODE=%{http_code}"); - expect(script, adapter).toContain( - "authorization: Bearer openshell:resolve:env:REBIND_MCP_SECRET", - ); + expect(script, adapter).toContain("NEMOCLAW_MCP_AUTHORIZATION"); expect(script, adapter).not.toContain("fake-rebind-mcp-secret-value"); const syntax = spawnSync("/bin/bash", ["-n"], { input: script, encoding: "utf8" }); expect(syntax.status, `${adapter}: ${syntax.stderr}`).toBe(0); }); }); - it("pins the resolve-validate-connect source contract to OpenShell v0.0.116", () => { - const commit = "d1155aa70042d3e2ee49dbfa15346b108b7c1d92"; + it("pins the resolve-validate-connect source contract to OpenShell v0.1.2", () => { + const commit = "6648bd0c290efbc41ba131ee9831ee45cd431f94"; const sourcePath = "crates/openshell-supervisor-network/src/proxy.rs"; + const sourceUrl = `https://github.com/NVIDIA/OpenShell/blob/${commit}/${sourcePath}`; const citations = [ - `${sourcePath}:3857-3886`, - `${sourcePath}:3911-3951`, - `${sourcePath}:3983-4043`, - `${sourcePath}:3998-4031`, - `${sourcePath}:5613-5655`, + `${sourceUrl}#L4201-L4230`, + `${sourceUrl}#L4255-L4295`, + `${sourceUrl}#L4327-L4388`, + `${sourceUrl}#L4342-L4375`, + `${sourceUrl}#L5967-L6037`, ]; const docsPath = "docs/deployment/set-up-mcp-bridge.mdx"; @@ -225,6 +593,13 @@ network_policies: path: "/mcp", port: 31337, protocol: "mcp", + enforcement: "enforce", + mcp: { + versions: ["2025-11-25"], + max_body_bytes: 4096, + strict_tool_names: true, + allow_all_known_mcp_methods: false, + }, rules: [{ allow: { method: "tools/list" } }], }), ]); @@ -252,10 +627,11 @@ network_policies: }); }); - it("passes only an exact HTTP 403 and rejects an allowed response", () => { + it("preserves raw HTTP and transport outcomes for the strict caller", () => { const binDir = fakeCurlPath(); const script = buildRawOpenShellAllowedIpsRebindingProbeScript( `http://${RAW_OPENSHELL_REBIND_HOSTNAME}:31337/mcp`, + binDir, ); const run = (status: string, curlRc = "0") => spawnSync("/bin/bash", ["-c", script], { @@ -273,70 +649,273 @@ network_policies: expect(denied.stdout).toContain(`${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}403`); const allowed = run("200"); - expect(allowed.status).toBe(1); + expect(allowed.status).toBe(0); expect(allowed.stdout).toContain(`${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}200`); const transportFailure = run("000", "7"); expect(transportFailure.status).toBe(7); }); - it("applies and restores the raw proof through live OpenShell policy authority", async () => { - const rootDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-raw-policy-proof-")); - tempDirs.push(rootDir); - const artifacts = new ArtifactSink(path.join(rootDir, "artifacts")); - await artifacts.ensureRoot(); - const basePolicy = "version: 1\nnetwork_policies: {}\n"; - let currentPolicy = basePolicy; - const policyMutations: Array<{ document: string; operation: string | undefined }> = []; - const setPolicy = vi - .spyOn(policy, "setPolicyDocument") - .mockImplementation(async (_sandboxName, document, options) => { - currentPolicy = document; - policyMutations.push({ document, operation: options?.operation }); - return true; - }); - const host = { - command: async (_command: string, _args: string[], options?: { artifactName?: string }) => ({ - ...denialResult(), - stdout: options?.artifactName === "host-address-for-sandbox" ? "route 10.20.30.40\n" : "", - }), - } as unknown as HostCliClient; - const openshellCalls: string[][] = []; - const sandbox = { - openshell: async (args: string[]) => { - openshellCalls.push(args); - return { + it.each(["http", "dns"])( + "applies and restores the raw proof after %s policy denial", + async (transport) => { + const rootDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-raw-policy-proof-")); + tempDirs.push(rootDir); + const artifacts = new ArtifactSink(path.join(rootDir, "artifacts")); + await artifacts.ensureRoot(); + const basePolicy = "version: 1\nnetwork_policies: {}\n"; + let currentPolicy = basePolicy; + const policyMutations: Array<{ document: string; operation: string | undefined }> = []; + const setPolicy = vi + .spyOn(policy, "setPolicyDocument") + .mockImplementation(async (_sandboxName, document, options) => { + currentPolicy = document; + policyMutations.push({ document, operation: options?.operation }); + return true; + }); + const host = { + command: async ( + _command: string, + _args: string[], + options?: { artifactName?: string }, + ) => ({ ...denialResult(), - stdout: args.includes("--full") ? currentPolicy : basePolicy, - }; - }, - execShell: async () => ({ - ...denialResult(), - stdout: `${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}403\n`, - }), - } as never; - - try { - await assertRawOpenShellAllowedIpsRebindingDenied({ - artifacts, - host, - policySettleMs: 0, - sandbox, - sandboxName: "raw-proof", - timeoutMs: 1_000, + stdout: + { + "host-address-for-sandbox": "route 10.20.30.40\n", + "dns-rebinding-workload-resource": `${WORKLOAD_ID}\tsandbox\n`, + "dns-rebinding-supervisor-resource": SUPERVISOR_ID, + "dns-rebinding-resolver-cgroup": JSON.stringify({ + cgroup: `system.slice/docker-${SUPERVISOR_ID}.scope`, + upstream: "127.0.0.53", + }), + "dns-rebinding-resolver-identity": JSON.stringify( + resolverRecord( + _args.at(-1)!, + _args.at(-1) === WORKLOAD_ID ? "sandbox" : "supervisor", + "raw-proof", + ), + ), + }[options?.artifactName ?? ""] ?? "", + }), + } as unknown as HostCliClient; + const openshellCalls: string[][] = []; + let denialExecuted = false; + const sandbox = { + openshell: async (args: string[]) => { + openshellCalls.push(args); + return { + ...denialResult(), + stdout: + args[0] === "logs" + ? denialExecuted + ? `2026-10-20T20:41:43.600Z OCSF NET:REFUSE [MED] DENIED ${RAW_OPENSHELL_REBIND_HOSTNAME} [reason:policy_dns_no_valid_address]\n` + : "" + : args.includes("--full") + ? currentPolicy + : basePolicy, + }; + }, + execShell: async ( + _name: string, + _script: unknown, + options: { + artifactName: + | "raw-openshell-rebinding-allowed-control" + | "raw-openshell-rebinding-policy-denial"; + }, + ) => + ({ + "raw-openshell-rebinding-allowed-control": async () => { + const port = + YAML.parse(currentPolicy).network_policies[RAW_OPENSHELL_REBIND_POLICY_KEY] + .endpoints[0].port; + const response = await fetch(`http://127.0.0.1:${port}/mcp`); + await response.text(); + return { + ...denialResult(), + stdout: `${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}${response.status}\n`, + }; + }, + "raw-openshell-rebinding-policy-denial": async () => { + denialExecuted = true; + return transport === "dns" + ? denialResult({ + exitCode: 6, + stdout: `${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}000\n`, + stderr: `curl: (6) Could not resolve host: ${RAW_OPENSHELL_REBIND_HOSTNAME}\n`, + }) + : denialResult({ + stdout: `{"error":"policy_denied"}\n${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}403\n`, + }); + }, + })[options.artifactName]!(), + } as never; + + try { + await assertRawOpenShellAllowedIpsRebindingDenied({ + artifacts, + host, + policySettleMs: 0, + sandbox, + sandboxName: "raw-proof", + timeoutMs: 1_000, + }); + } finally { + setPolicy.mockRestore(); + } + + expect(policyMutations).toHaveLength(3); + expect(policyMutations[0]?.document).toContain(RAW_OPENSHELL_REBIND_POLICY_KEY); + expect(policyMutations[0]?.operation).toBe("allow the raw OpenShell rebinding control"); + expect(policyMutations[1]?.operation).toBe( + "run the raw OpenShell allowed_ips rebinding proof", + ); + expect(policyMutations[2]?.document).toBe(basePolicy.trim()); + expect(policyMutations[2]?.operation).toBe( + "restore the raw OpenShell allowed_ips rebinding proof policy", + ); + expect(openshellCalls.some((args) => args[0] === "policy" && args[1] === "set")).toBe(false); + }, + ); + + it.each([ + [RAW_OPENSHELL_REBIND_HOSTNAME, RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER], + ["mcp-rebind.example.test", "NEMOCLAW_MCP_CURL_HTTP_CODE="], + ["inference-rebind.trycloudflare.com", "NEMOCLAW_INFERENCE_CURL_HTTP_CODE="], + ])("accepts only a fresh exact policy DNS refusal for %s", (hostname, marker) => { + const result = denialResult({ + exitCode: 6, + stdout: `${marker}000\n`, + stderr: `curl: (6) Could not resolve host: ${hostname}\n`, + }); + const event = `2026-10-20T20:41:43.600Z OCSF NET:REFUSE [MED] DENIED ${hostname} [reason:policy_dns_no_valid_address]\n`; + expect( + isExpectedMcpDnsPolicyDenial( + result, + denialResult(), + denialResult({ stdout: event }), + hostname, + marker, + ), + ).toBe(true); + expect( + isExpectedMcpDnsPolicyDenial( + result, + denialResult({ exitCode: 1 }), + denialResult({ stdout: event }), + hostname, + marker, + ), + ).toBe(false); + expect( + isExpectedMcpDnsPolicyDenial( + result, + denialResult(), + denialResult({ stdout: event }), + "other.example.test", + marker, + ), + ).toBe(false); + }); + + it.each([ + ["first read", "after-1", [0], true], + ["delayed third read", "after-3", [0, 250, 500], true], + ["exhausted reads", null, [0, 250, 500, 750, 1000], false], + ] as const)( + "confirms DNS denial with bounded audit polling: %s", + async (_name, freshAt, times, expected) => { + vi.useFakeTimers(); + vi.setSystemTime(0); + const hostname = "inference-rebind.trycloudflare.com"; + const marker = "NEMOCLAW_INFERENCE_CURL_HTTP_CODE="; + const result = denialResult({ + exitCode: 6, + stdout: `${marker}000\n`, + stderr: `curl: (6) Could not resolve host: ${hostname}\n`, }); - } finally { - setPolicy.mockRestore(); - } - - expect(policyMutations).toHaveLength(2); - expect(policyMutations[0]?.document).toContain(RAW_OPENSHELL_REBIND_POLICY_KEY); - expect(policyMutations[0]?.operation).toBe("run the raw OpenShell allowed_ips rebinding proof"); - expect(policyMutations[1]?.document).toBe(basePolicy.trim()); - expect(policyMutations[1]?.operation).toBe( - "restore the raw OpenShell allowed_ips rebinding proof policy", - ); - expect(openshellCalls.some((args) => args[0] === "policy" && args[1] === "set")).toBe(false); + const before = denialResult(); + const after = denialResult({ + stdout: `fresh OCSF NET:REFUSE [MED] DENIED ${hostname} [reason:policy_dns_no_valid_address]\n`, + }); + const observedTimes: number[] = []; + const readAudit = vi.fn(async (suffix: string) => { + observedTimes.push(Date.now()); + return suffix === freshAt ? after : before; + }); + const confirmation = waitForMcpCurlPolicyDenial(result, before, readAudit, hostname, marker); + await vi.advanceTimersByTimeAsync(1000); + expect(await confirmation).toBe(expected); + expect(observedTimes).toEqual(times); + expect(readAudit.mock.calls.at(-1)).toEqual([`after-${times.length}`]); + expect(vi.getTimerCount()).toBe(0); + }, + ); + + it.each([ + [ + 7, + "NEMOCLAW_MCP_CURL_HTTP_CODE=000\n", + "* Immediate connect fail for 192.0.2.1: Permission denied\n", + true, + ], + [ + 7, + "NEMOCLAW_MCP_CURL_HTTP_CODE=000\n", + "* connect to 192.0.2.1 port 443 failed: Connection refused\n", + false, + ], + [0, "policy_denied\nNEMOCLAW_MCP_CURL_HTTP_CODE=403\n", "", true], + ] as const)( + "does not read DNS logs for a completed non-DNS result [case %#]", + async (exitCode, stdout, stderr, expected) => { + const readAudit = vi.fn(async () => denialResult()); + const result = denialResult({ exitCode, stdout, stderr }); + expect( + await waitForMcpCurlPolicyDenial(result, denialResult(), readAudit, "mcp.example.test"), + ).toBe(expected); + expect(readAudit).not.toHaveBeenCalled(); + }, + ); + + it.each([ + "missing", + "stale", + "other-host", + "upstream-nxdomain", + "audit-failed", + "timeout", + "http-success", + ])("rejects a raw DNS denial with %s evidence", (condition) => { + const event = `2026-10-20T20:41:43.600Z OCSF NET:REFUSE [MED] DENIED ${RAW_OPENSHELL_REBIND_HOSTNAME} [reason:policy_dns_no_valid_address]\n`; + const result = denialResult({ + exitCode: 6, + stdout: `${RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER}${condition === "http-success" ? "200" : "000"}\n`, + stderr: `curl: (6) Could not resolve host: ${RAW_OPENSHELL_REBIND_HOSTNAME}\n`, + timedOut: condition === "timeout", + }); + const before = denialResult({ stdout: condition === "stale" ? event : "" }); + const after = denialResult({ + exitCode: condition === "audit-failed" ? 1 : 0, + stdout: + condition === "missing" + ? "" + : condition === "other-host" + ? event.replace(RAW_OPENSHELL_REBIND_HOSTNAME, "other.example.test") + : condition === "upstream-nxdomain" + ? event.replace("policy_dns_no_valid_address", "policy_dns_upstream_nxdomain") + : event, + }); + expect( + isExpectedMcpDnsPolicyDenial( + result, + before, + after, + RAW_OPENSHELL_REBIND_HOSTNAME, + RAW_OPENSHELL_REBIND_HTTP_CODE_MARKER, + ), + ).toBe(false); }); it("accepts an unchanged surviving policy only after the unrelated policy is absent", () => { @@ -387,98 +966,74 @@ network_policies: ).toThrow(); }); - it("restores host DNS strictly while treating the ephemeral sandbox as best effort", async () => { - const { restoreScript } = await captureRestoreCommand( - "/tmp/host-backup", - "/tmp/sandbox-backup", + function cleanupFixture(failure: string) { + const close = vi.fn(async () => {}); + const removeBackup = vi.fn(); + const command = vi.fn( + async (_command: string, _args: string[], options?: { artifactName?: string }) => ({ + ...denialResult(), + exitCode: + (failure === "rule-removal" && options?.artifactName?.includes("-D-")) || + (failure === "rule-inventory" && + options?.artifactName === "dns-rebinding-rules-absent") || + (failure === "host-restore" && options?.artifactName === "dns-rebinding-restore-hosts") + ? 1 + : 0, + }), ); + const fixture = { + hostname: "mcp-rebind.example.test", + hostBackupPath: "/tmp/fixture/hosts", + backupReady: true, + removeBackup, + containerId: SUPERVISOR_ID, + pid: 321, + cgroup: `system.slice/docker-${SUPERVISOR_ID}.scope`, + upstream: "127.0.0.53", + comment: "owned-fixture", + restored: false, + server: { port: 1053, family: 4, answers: () => 1, setAddress: vi.fn(), close }, + }; + return { command, fixture, close, removeBackup }; + } - expect(restoreScript).toContain("set -uo pipefail"); - expect(restoreScript).not.toContain("set -euo pipefail"); - expect(restoreScript).toContain('if ! sudo -n tee /etc/hosts < "$host_backup"'); - expect(restoreScript).toContain('if ! cmp -s "$host_backup" /etc/hosts'); - expect(restoreScript).toContain("host_restore_failed=1"); - expect(restoreScript).toContain('if [ "$host_restore_failed" -ne 0 ]; then exit 1; fi'); - expect(restoreScript).toContain("for attempt in 1 2 3; do"); - expect(restoreScript).toContain('runtime_command=("$@")'); - expect(restoreScript).toContain( - '"${runtime_command[@]}" container exec --user 0 --interactive "$container_id"', + it("restores hosts, removes DNS rules, and closes the forwarder once", async () => { + const { command, fixture, close, removeBackup } = cleanupFixture("success"); + await restoreDnsRebindingHostsFixture( + { command } as unknown as HostCliClient, + "test-sandbox", + fixture, ); - expect(restoreScript).toContain( - "::warning::could not restore ephemeral sandbox /etc/hosts; cleanup will destroy the sandbox", + expect(close).toHaveBeenCalledOnce(); + expect(fixture.restored).toBe(true); + expect(removeBackup).toHaveBeenCalledOnce(); + const calls = command.mock.calls.length; + await restoreDnsRebindingHostsFixture( + { command } as unknown as HostCliClient, + "test-sandbox", + fixture, ); - expect(restoreScript).toContain("failed to remove DNS rebinding hosts backups"); + expect(command).toHaveBeenCalledTimes(calls); }); - it("executes every restore outcome without an unlabeled errexit", async () => { - const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-mcp-restore-")); - const binDir = path.join(tempDir, "bin"); - const hostBackupPath = path.join(tempDir, "host-backup"); - const sandboxBackupPath = path.join(tempDir, "sandbox-backup"); - const fakeHostsPath = path.join(tempDir, "hosts"); - fs.mkdirSync(binDir); - const writeExecutable = (name: string, source: string) => { - const target = path.join(binDir, name); - fs.writeFileSync(target, source, { mode: 0o755 }); - }; - writeExecutable( - "sudo", - '#!/bin/sh\n[ "${FAKE_SUDO_STATUS:-0}" -eq 0 ] || exit "$FAKE_SUDO_STATUS"\ncat > "$FAKE_HOSTS_PATH"\n', - ); - writeExecutable("cmp", '#!/bin/sh\nexit "${FAKE_CMP_STATUS:-0}"\n'); - writeExecutable("sleep", "#!/bin/sh\nexit 0\n"); - - try { - const { restoreArgs, restoreScript } = await captureRestoreCommand( - hostBackupPath, - sandboxBackupPath, - ); - const runtimeCommand = restoreArgs[1] ?? "missing-runtime-command"; - writeExecutable( - runtimeCommand, - '#!/bin/sh\nif [ "$1" = container ] && [ "$2" = ps ]; then echo fake-container; exit 0; fi\nif [ "$1" = container ] && [ "$2" = exec ]; then cat >/dev/null; exit "${FAKE_RUNTIME_EXEC_STATUS:-0}"; fi\nexit 64\n', - ); - const runRestore = (extraEnv: Record = {}) => - spawnSync("/bin/bash", ["-c", restoreScript, ...restoreArgs], { - encoding: "utf8", - env: { - ...process.env, - PATH: `${binDir}:${process.env.PATH ?? ""}`, - FAKE_HOSTS_PATH: fakeHostsPath, - ...extraEnv, - }, - }); - const resetBackups = () => { - fs.writeFileSync(hostBackupPath, "original host entries\n"); - fs.writeFileSync(sandboxBackupPath, "original sandbox entries\n"); - }; - - resetBackups(); - const success = runRestore(); - expect(success.status, success.stderr).toBe(0); - expect(success.stdout).toContain("restored host /etc/hosts"); - expect(success.stdout).toContain("restored sandbox /etc/hosts"); - expect(success.stdout).toContain("removed DNS rebinding hosts backups"); - expect(fs.existsSync(hostBackupPath)).toBe(false); - expect(fs.existsSync(sandboxBackupPath)).toBe(false); - - resetBackups(); - const hostFailure = runRestore({ FAKE_SUDO_STATUS: "1" }); - expect(hostFailure.status).toBe(1); - expect(hostFailure.stderr).toContain("failed to restore host /etc/hosts"); - expect(fs.existsSync(hostBackupPath)).toBe(true); - expect(fs.existsSync(sandboxBackupPath)).toBe(true); - - resetBackups(); - const sandboxFailure = runRestore({ FAKE_RUNTIME_EXEC_STATUS: "1" }); - expect(sandboxFailure.status, sandboxFailure.stderr).toBe(0); - expect(sandboxFailure.stderr).toContain( - "::warning::could not restore ephemeral sandbox /etc/hosts; cleanup will destroy the sandbox", - ); - expect(fs.existsSync(hostBackupPath)).toBe(false); - expect(fs.existsSync(sandboxBackupPath)).toBe(false); - } finally { - fs.rmSync(tempDir, { recursive: true, force: true }); - } - }); + it.each([ + { failure: "rule-removal", backupRemovals: 1 }, + { failure: "rule-inventory", backupRemovals: 1 }, + { failure: "host-restore", backupRemovals: 0 }, + ])( + "closes the DNS forwarder and reports $failure cleanup failure", + async ({ failure, backupRemovals }) => { + const { command, fixture, close, removeBackup } = cleanupFixture(failure); + await expect( + restoreDnsRebindingHostsFixture( + { command } as unknown as HostCliClient, + "test-sandbox", + fixture, + ), + ).rejects.toThrow("DNS fixture cleanup failed"); + expect(close).toHaveBeenCalledOnce(); + expect(fixture.restored).toBe(false); + expect(removeBackup).toHaveBeenCalledTimes(backupRemovals); + }, + ); }); diff --git a/test/e2e/support/mcp-bridge-tool-discovery.test.ts b/test/e2e/support/mcp-bridge-tool-discovery.test.ts index 24a77f6994d..120331bc8ac 100644 --- a/test/e2e/support/mcp-bridge-tool-discovery.test.ts +++ b/test/e2e/support/mcp-bridge-tool-discovery.test.ts @@ -7,13 +7,13 @@ import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; import { ArtifactSink } from "../fixtures/artifacts.ts"; -import { shouldRetryMcpMutationAfterConcurrencyConflict } from "../live/mcp-bridge-cleanup.ts"; import { type FakeMcpHttpsServer, type FakeMcpRequest, HERMES_DEFERRED_TOOL_SEARCH_MISS, type StartedHttpServer, startCompatibleMock, + withCompatibleMockFailureDiagnostics, } from "../live/mcp-bridge-servers.ts"; import { assertAuthenticatedMcpDiscovery, @@ -314,6 +314,130 @@ afterEach(async () => { ); }); +describe("compatible endpoint failure diagnostics", () => { + it("does not collect diagnostics after successful approval", async () => { + const diagnostics = vi.fn(); + const writeJson = vi.fn(); + const operation = vi.fn().mockResolvedValue(undefined); + await withCompatibleMockFailureDiagnostics({ diagnostics }, { writeJson }, operation); + expect(operation).toHaveBeenCalledOnce(); + expect(diagnostics).not.toHaveBeenCalled(); + expect(writeJson).not.toHaveBeenCalled(); + }); + + it.each(["success", "failure"] as const)( + "preserves the original failed approval after diagnostic publication %s", + async (publicationResult) => { + const failure = new Error("original approval failure"); + const evidence = { receivedCount: 0, retainedLimit: 64, requests: [] }; + const diagnostics = vi.fn().mockReturnValue(evidence); + const publishers = { + success: () => Promise.resolve("evidence.json"), + failure: () => Promise.reject(new Error("diagnostic publication failed")), + }; + const writeJson = vi.fn(publishers[publicationResult]); + const operation = vi.fn().mockRejectedValue(failure); + await expect( + withCompatibleMockFailureDiagnostics({ diagnostics }, { writeJson }, operation), + ).rejects.toBe(failure); + expect(operation).toHaveBeenCalledOnce(); + expect(diagnostics).toHaveBeenCalledOnce(); + expect(writeJson).toHaveBeenCalledWith("admin-approval-compatible-requests.json", { + adminApprovalStartedAtMs: expect.any(Number), + capturedAtMs: expect.any(Number), + ...evidence, + }); + }, + ); + + it.each([ + ["/v1/chat/completions", "chat-completions", true, true, 200], + ["/chat/completions", "chat-completions", true, false, 200], + ["/v1/responses", "responses", true, null, 200], + ["/private-path?token=PRIVATE_QUERY", "other", false, null, 401], + ] as const)( + "records only transport facts for %s", + async (route, category, authenticated, stream, status) => { + const server = await startCompatibleMock({ + apiKey: COMPATIBLE_API_KEY, + model: COMPATIBLE_MODEL, + }); + compatibleMock = server; + const response = await fetch(`http://127.0.0.1:${server.port}${route}`, { + method: "POST", + headers: { + Authorization: `Bearer ${authenticated ? COMPATIBLE_API_KEY : "PRIVATE_BAD_TOKEN"}`, + "Content-Type": "application/json", + "X-Private-Header": "PRIVATE_HEADER", + }, + body: JSON.stringify({ + stream, + messages: [{ role: "user", content: "PRIVATE_PROMPT" }], + }), + }); + await response.text(); + expect(response.status).toBe(status); + const evidence = server.diagnostics(); + expect(evidence).toEqual({ + receivedCount: 1, + retainedLimit: 64, + requests: [ + { + receivedAtMs: expect.any(Number), + method: "POST", + route: category, + authenticated, + bodyComplete: expect.any(Boolean), + stream, + responseStatus: status, + responseFinished: true, + responseClosed: true, + }, + ], + }); + expect(JSON.stringify(evidence)).not.toMatch( + new RegExp(`PRIVATE_|${COMPATIBLE_API_KEY}|${COMPATIBLE_MODEL}|private-path|token=`), + ); + }, + ); + + it("bounds retained requests and returns independent snapshots", async () => { + const server = await startCompatibleMock({ + apiKey: COMPATIBLE_API_KEY, + model: COMPATIBLE_MODEL, + }); + compatibleMock = server; + await Promise.all( + Array.from({ length: 70 }, async () => { + const response = await fetch(`http://127.0.0.1:${server.port}/v1/models`, { + headers: { Authorization: `Bearer ${COMPATIBLE_API_KEY}` }, + }); + await response.text(); + }), + ); + const evidence = server.diagnostics(); + expect(evidence.receivedCount).toBe(70); + expect(evidence.requests).toHaveLength(64); + expect(evidence.requests).toEqual( + Array.from({ length: 64 }, () => ({ + receivedAtMs: expect.any(Number), + method: "GET", + route: "models", + authenticated: true, + bodyComplete: true, + stream: null, + responseStatus: 200, + responseFinished: true, + responseClosed: true, + })), + ); + evidence.requests[0]!.responseStatus = 599; + evidence.requests.length = 0; + expect(server.diagnostics().requests).toHaveLength(64); + expect(server.diagnostics().requests[0]!.responseStatus).toBe(200); + }); +}); + async function startDeferredCompatibleMock(): Promise { return startCompatibleMock({ apiKey: COMPATIBLE_API_KEY, @@ -1368,21 +1492,3 @@ describe("Hermes deferred MCP tool discovery", () => { expect(terminalMessage.tool_calls).toBeUndefined(); }); }); - -describe("MCP mutation concurrency retry", () => { - it("retries the explicit OpenShell optimistic-concurrency response", () => { - expect( - shouldRetryMcpMutationAfterConcurrencyConflict( - "Failed to detach provider: sandbox was modified by another operation.\nPlease retry the command.", - ), - ).toBe(true); - }); - - it.each([ - "Failed to detach provider: permission denied", - "sandbox was modified by another operation.", - "Please retry the command.", - ])("does not retry another failure: %s", (output) => { - expect(shouldRetryMcpMutationAfterConcurrencyConflict(output)).toBe(false); - }); -}); diff --git a/test/e2e/support/mcp-provider-rewrite-probe.test.ts b/test/e2e/support/mcp-provider-rewrite-probe.test.ts index 7f093f3883e..b967d4038b3 100644 --- a/test/e2e/support/mcp-provider-rewrite-probe.test.ts +++ b/test/e2e/support/mcp-provider-rewrite-probe.test.ts @@ -2,8 +2,9 @@ // SPDX-License-Identifier: Apache-2.0 import vm from "node:vm"; +import { EventEmitter } from "node:events"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { buildMcpProviderRewriteAuthorization, @@ -34,6 +35,7 @@ describe("managed MCP provider rewrite probe", () => { it.each([ undefined, "raw-secret", + "openshell:resolve:env:v1_FAKE_MCP_SECRET\n", "openshell:resolve:env:FAKE_MCP_SECRET", "openshell:resolve:env:v_FAKE_MCP_SECRET", "openshell:resolve:env:v144298782728593258901_FAKE_MCP_SECRET", @@ -88,3 +90,63 @@ describe("managed MCP provider rewrite probe", () => { ); }); }); + +function executeRewriteProbe(expectation: string) { + const request = Object.assign(new EventEmitter(), { end: vi.fn() }); + const send = vi.fn((_options: object, _callback: (response: EventEmitter) => void) => request); + const exit = vi.fn(); + const output: string[] = []; + vm.runInNewContext(MCP_PROVIDER_REWRITE_PROBE_SOURCE, { + require: () => ({ request: send }), + URL, + Buffer, + process: { + argv: ["node", "-", "https://mcp.example.test/mcp", "tools/list", expectation], + env: { FAKE_MCP_SECRET: "openshell:resolve:env:v1_FAKE_MCP_SECRET" }, + exit, + }, + console: { log: (line: string) => output.push(line), error: vi.fn() }, + }); + return { request, send, exit, output }; +} + +describe("MCP request denial evidence", () => { + it.each([ + ["deny", "EACCES", "connect", 0], + ["deny-strict", "EACCES", "connect", 0], + ["allow", "EACCES", "connect", 1], + ["deny", "EACCES", "open", 1], + ["deny", "ENOTFOUND", "getaddrinfo", 1], + ["deny", "ECONNREFUSED", "connect", 1], + ["deny", "ECONNRESET", "read", 1], + ["deny-strict", "ECONNRESET", "connect", 1], + ])("classifies %s with %s/%s as exit %s", (mode, code, syscall, status) => { + const probe = executeRewriteProbe(String(mode)); + probe.request.emit("error", { code, syscall, message: "transport error" }); + expect(probe.exit).toHaveBeenCalledWith(status); + expect(probe.output.join("\n")).toContain(JSON.stringify({ errorCode: code, syscall })); + }); + it.each([ + ["allow", 200, '{"result":{}}', 0], + ["deny", 200, '{"result":{}}', 1], + ["deny", 403, '{"error":"policy_denied"}', 0], + ["deny-strict", 403, '{"error":"policy_denied"}', 0], + ["deny", 403, '{"error":"credential_endpoint_mismatch"}', 1], + ["deny", 500, '{"error":"credential_unavailable"}', 1], + ])("classifies %s HTTP %s %s as exit %s", (mode, status, body, expected) => { + const probe = executeRewriteProbe(String(mode)); + const response = Object.assign(new EventEmitter(), { + statusCode: status, + setEncoding: vi.fn(), + }); + probe.send.mock.calls[0]![1](response); + response.emit("data", body); + response.emit("end"); + expect(probe.exit).toHaveBeenCalledWith(expected); + expect(probe.send.mock.calls[0]![0]).toMatchObject({ + autoSelectFamily: false, + hostname: "mcp.example.test", + path: "/mcp", + }); + }); +}); diff --git a/test/e2e/support/messaging-compatible-native.test.ts b/test/e2e/support/messaging-compatible-native.test.ts new file mode 100644 index 00000000000..6ce6d19f265 --- /dev/null +++ b/test/e2e/support/messaging-compatible-native.test.ts @@ -0,0 +1,75 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import vm from "node:vm"; +import { expect, it, vi } from "vitest"; +import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; +import { + assertOpenClawConfigShape, + onboardEnv, +} from "../live/messaging-compatible-endpoint.test.ts"; + +// Register no live target while exercising its configuration probe in isolation. +vi.mock("../fixtures/e2e-test.ts", async () => ({ + expect: (await import("vitest")).expect, + test: vi.fn(), +})); + +const endpointUrl = "http://10.20.30.40:18089/v1"; +const reference = "openshell:resolve:env:v7_COMPATIBLE_API_KEY"; + +it("admits only the mock endpoint host for native onboarding", () => { + expect(onboardEnv(endpointUrl)).toMatchObject({ + NEMOCLAW_ENDPOINT_URL: endpointUrl, + NEMOCLAW_TRUSTED_PRIVATE_INFERENCE_HOSTS: "10.20.30.40", + NEMOCLAW_PROVIDER: "custom", + NEMOCLAW_PREFERRED_API: "openai-completions", + }); +}); + +it.each([ + { label: "scoped reference", baseUrl: endpointUrl, apiKey: reference, status: 0 }, + { label: "raw credential", baseUrl: endpointUrl, apiKey: "raw-test-credential", status: 1 }, + { + label: "unscoped reference", + baseUrl: endpointUrl, + apiKey: "openshell:resolve:env:COMPATIBLE_API_KEY", + status: 1, + }, + { + label: "different revision", + baseUrl: endpointUrl, + apiKey: "openshell:resolve:env:v8_COMPATIBLE_API_KEY", + status: 1, + }, + { label: "retired route", baseUrl: "https://inference.local/v1", apiKey: reference, status: 1 }, +])("checks the native config with $label", async ({ baseUrl, apiKey, status }) => { + const exitStatuses: number[] = []; + const sandbox = { + exec: async (_name: string, args: string[]) => { + const config = { + models: { providers: { inference: { baseUrl, apiKey } } }, + agents: { defaults: { model: { primary: "inference/" + args[3] } } }, + channels: { telegram: {} }, + }; + vm.runInNewContext(args[2]!, { + require: () => ({ readFileSync: () => JSON.stringify(config) }), + process: { + argv: ["node", ...args.slice(3)], + env: { COMPATIBLE_API_KEY: reference }, + exit: (code: number) => exitStatuses.push(code), + }, + console: { log: () => undefined }, + }); + return { exitCode: exitStatuses.at(-1), stdout: "", stderr: "" }; + }, + } as unknown as SandboxClient; + const result = await assertOpenClawConfigShape(sandbox, endpointUrl).then( + () => "accepted", + () => "rejected", + ); + expect({ exitStatuses, result }).toEqual({ + exitStatuses: [status], + result: status === 0 ? "accepted" : "rejected", + }); +}); diff --git a/test/e2e/support/messaging-providers-runtime-proofs.test.ts b/test/e2e/support/messaging-providers-runtime-proofs.test.ts index 752234d78b0..d2d308f8377 100644 --- a/test/e2e/support/messaging-providers-runtime-proofs.test.ts +++ b/test/e2e/support/messaging-providers-runtime-proofs.test.ts @@ -22,6 +22,7 @@ import { FAKE_API_PROXY_READINESS_SOURCE, FAKE_API_PROXY_SOURCE, isNvidiaEndpointRateLimitFailure, + isUnresolvedPlaceholderRejection, messagingEnv, OPENSHELL_EXEC_ARGUMENT_LIMIT_BYTES, parseRuntimeProofPort, @@ -36,15 +37,24 @@ import { parseInstalledWechatProof } from "../live/messaging-providers-wechat-ru const FAKE_TELEGRAM_API = path.resolve(import.meta.dirname, "../lib/fake-telegram-api.cjs"); const FAKE_SLACK_API = path.resolve(import.meta.dirname, "../lib/fake-slack-api.cjs"); const FAKE_WECHAT_API = path.resolve(import.meta.dirname, "../lib/fake-wechat-api.mts"); -const OPENSHELL_BRIDGE_ADDRESS = "172.18.0.1"; -const OPENSHELL_NETWORK_INSPECT = JSON.stringify([ - { - Driver: "bridge", - IPAM: { Config: [{ Subnet: "172.18.0.0/16", Gateway: OPENSHELL_BRIDGE_ADDRESS }] }, - }, -]); const execFileAsync = promisify(execFile); +it.each([ + [ + '500 {"error":"credential_unavailable","message":"Credential placeholder could not be resolved"}', + true, + ], + ['500 {"error":"credential_injection_failed"}', true], + ["unresolved credential placeholder", true], + ['401 {"error":"bad_auth"}', false], + ['200 {"ok":false,"error":"invalid_auth"}', false], + ['500 {"error":"internal_server_error"}', false], + ['403 {"error":"credential_endpoint_mismatch"}', false], + ["TIMEOUT", false], +])("classifies missing-credential rejection in %s", (response, expected) => { + expect(isUnresolvedPlaceholderRejection(response)).toBe(expected); +}); + it("pins the fake messaging API to the reviewed Node runtime", () => { expect(FAKE_API_IMAGE).toBe( "node:24.22.1-trixie-slim@sha256:43b9ba1142b8dcf33fa8f922ca1be85714adc444e0586c7cf7726daaeabc1bea", @@ -235,7 +245,6 @@ function fakeDockerHost( missingProxyControl?: MissingProxyControl; proxyEnvironment?: readonly string[]; publishedAddress?: string; - networkInspect?: string; proxyRunning?: boolean; proxyReady?: boolean; } = {}, @@ -252,7 +261,6 @@ function fakeDockerHost( const commands: Array<{ command: string; args: string[] }> = []; const containers = new Set(); const networks = new Set(); - const networkInspect = options.networkInspect ?? OPENSHELL_NETWORK_INSPECT; let runtimeProviderId: "docker" | "podman" = "docker"; let proxyRunning = options.proxyRunning !== false; const dockerCommand = (args: string[]) => { @@ -268,7 +276,7 @@ function fakeDockerHost( ); return successfulCommand( executedArgs[2] === "openshell-docker" - ? networkInspect + ? "[]" : JSON.stringify([ runtimeProviderId === "podman" ? { @@ -497,10 +505,6 @@ describe("messaging provider installed-runtime proofs", () => { "e2e-hermes-discord", "--add-endpoint", "host.openshell.internal:43117:read-write:websocket:enforce:websocket-credential-rewrite,allowed-ip=10.0.0.0/8,allowed-ip=172.16.0.0/12,allowed-ip=192.168.0.0/16", - "--add-allow", - "host.openshell.internal:43117:GET:/**", - "--add-allow", - "host.openshell.internal:43117:WEBSOCKET_TEXT:/**", "--binary", allowedBinaries[0], "--binary", @@ -559,10 +563,6 @@ describe("messaging provider installed-runtime proofs", () => { "e2e-hermes-discord", "--add-endpoint", "host.openshell.internal:43118:read-write:rest:enforce:request-body-credential-rewrite,allowed-ip=10.0.0.0/8,allowed-ip=172.16.0.0/12,allowed-ip=192.168.0.0/16", - "--add-allow", - "host.openshell.internal:43118:GET:/**", - "--add-allow", - "host.openshell.internal:43118:POST:/**", "--binary", allowedBinaries[0], "--wait", @@ -613,8 +613,6 @@ describe("messaging provider installed-runtime proofs", () => { "e2e-telegram", "--add-endpoint", "host.openshell.internal:43119:read-write:rest:enforce:request-body-credential-rewrite,allowed-ip=10.0.0.0/8,allowed-ip=172.16.0.0/12,allowed-ip=192.168.0.0/16", - "--add-allow", - "host.openshell.internal:43119:POST:/**", "--binary", "/usr/local/bin/node", "--binary", @@ -796,7 +794,7 @@ describe("messaging provider installed-runtime proofs", () => { } }); - it("rejects a fake API proxy port that Docker publishes beyond the OpenShell bridge", async () => { + it("rejects a fake API proxy port that Docker publishes beyond loopback", async () => { const { host } = fakeDockerHost({ publishedAddress: "0.0.0.0" }); const cleanup: CleanupAction[] = []; @@ -810,7 +808,7 @@ describe("messaging provider installed-runtime proofs", () => { }); it("publishes the isolated proxy through rootless Podman without binding its bridge gateway", async () => { - const { calls, host } = fakeDockerHost({ networkInspect: "[]" }); + const { calls, host } = fakeDockerHost(); const cleanup: CleanupAction[] = []; try { @@ -823,9 +821,6 @@ describe("messaging provider installed-runtime proofs", () => { expect(publications).toContain("0.0.0.0::8080"); expect(publications).toContain(`0.0.0.0::${String(FAKE_API_PROXY_READINESS_PORT)}`); - expect(publications.some((entry) => entry.startsWith(`${OPENSHELL_BRIDGE_ADDRESS}::`))).toBe( - false, - ); expect(calls).not.toContainEqual(["network", "inspect", "openshell-docker"]); expect(api.port).toBe("32100"); } finally { @@ -836,7 +831,6 @@ describe("messaging provider installed-runtime proofs", () => { it("rejects effective proxy capabilities reported by rootless Podman", async () => { const { host } = fakeDockerHost({ missingProxyControl: "--cap-drop", - networkInspect: "[]", }); const cleanup: CleanupAction[] = []; @@ -852,35 +846,28 @@ describe("messaging provider installed-runtime proofs", () => { } }); - it("rejects ambiguous OpenShell IPv4 gateways and cleans its temporary directory", async () => { - const networkInspect = JSON.stringify([ - { - Driver: "bridge", - IPAM: { - Config: [ - { Subnet: "172.18.0.0/16", Gateway: OPENSHELL_BRIDGE_ADDRESS }, - { Subnet: "172.19.0.0/16", Gateway: "172.19.0.1" }, - ], - }, - }, - ]); - const { calls, host } = fakeDockerHost({ networkInspect }); + it("publishes the Docker proxy on loopback without an OpenShell bridge and cleans its resources", async () => { + const { calls, host, resources } = fakeDockerHost(); const cleanup: CleanupAction[] = []; - let fixtureDir: string | undefined; + let captureFile: string | undefined; try { - await expect(startFakeDiscordApi(host, cleanup)).rejects.toThrow( - /exactly one IPv4 bridge gateway/u, - ); - expect(calls).toEqual([["network", "inspect", "openshell-docker"]]); - expect(cleanup).toHaveLength(1); - fixtureDir = cleanup[0]!.name.replace(/^remove /u, ""); - expect(fs.existsSync(fixtureDir)).toBe(true); + const api = await startFakeDiscordApi(host, cleanup); + captureFile = api.captureFile; + const proxyRun = calls.filter((args) => args[0] === "run").at(-1)!; + expect(optionValues(proxyRun, "-p")).toEqual([ + `127.0.0.1::${String(FAKE_API_PROXY_READINESS_PORT)}`, + "127.0.0.1::8080", + ]); + expect(calls).not.toContainEqual(["network", "inspect", "openshell-docker"]); + expect(api.port).toBe("32100"); + expect(fs.existsSync(captureFile)).toBe(true); } finally { await runCleanup(cleanup); } - expect(fixtureDir).toBeDefined(); - expect(fs.existsSync(fixtureDir!)).toBe(false); + expect(resources()).toEqual({ containers: [], networks: [] }); + expect(captureFile).toBeDefined(); + expect(fs.existsSync(captureFile!)).toBe(false); }); it("fails with proxy diagnostics when the detached proxy stops before readiness", async () => { diff --git a/test/e2e/support/native-lifecycle-migration-contract.test.ts b/test/e2e/support/native-lifecycle-migration-contract.test.ts index ced4fb51f29..93c5de472c5 100644 --- a/test/e2e/support/native-lifecycle-migration-contract.test.ts +++ b/test/e2e/support/native-lifecycle-migration-contract.test.ts @@ -6,26 +6,351 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { restorationScript } from "../live/openclaw-restoration.ts"; +import * as fakeInference from "../fixtures/fake-openai-compatible.ts"; + +const captured = vi.hoisted(() => ({ test: vi.fn() })); +vi.mock("../fixtures/e2e-test.ts", async () => ({ + expect: (await import("vitest")).expect, + test: captured.test, +})); + +async function runCronTarget( + endpoint: string, + runResult = { exitCode: 0, stdout: '{"ok":true,"ran":true}', stderr: "" }, +) { + vi.stubEnv("NEMOCLAW_ENDPOINT_URL", endpoint); + vi.stubEnv("NEMOCLAW_SANDBOX_NAME", "cron-test"); + vi.spyOn(Date, "now").mockReturnValue(42); + vi.resetModules(); + captured.test.mockClear(); + await import("../live/cron-preflight-inference-local.test.ts"); + const run = captured.test.mock.calls[0]![2] as (fixtures: object) => Promise; + const success = { exitCode: 0, stdout: "", stderr: "" }; + const exec = vi + .fn() + .mockResolvedValueOnce(success) + .mockResolvedValueOnce({ + ...success, + stdout: '{"name":"nemoclaw-native-cron-42","id":"cron-42"}', + }) + .mockResolvedValueOnce(runResult) + .mockResolvedValueOnce(success); + const command = vi.fn( + async (_command: string, _args: string[], _options: { env: NodeJS.ProcessEnv }) => success, + ); + const complete = vi.fn(); + const cleanupSandbox = vi.fn(async () => success); + const trackDisposable = vi.fn(); + const finished = run({ + artifacts: { target: { declare: vi.fn(), complete } }, + cleanup: { trackDisposable }, + host: { command }, + progress: { phase: vi.fn() }, + runtimeProvider: { requireAvailable: vi.fn() }, + sandbox: { + exec, + cleanupSandbox, + openshell: vi.fn(async () => ({ + ...success, + stdout: JSON.stringify({ + paired: [{ scopes: ["operator.admin"], tokens: [{ scopes: ["operator.admin"] }] }], + }), + })), + }, + secrets: { required: () => "test-hosted-key" }, + }); + return { finished, exec, command, complete, cleanupSandbox, trackDisposable }; +} + +describe("native scheduled inference target", () => { + it.each(["https://inference-api.nvidia.com/v1", "https://selected.example/team/v1"])( + "reports the selected %s endpoint after running and removing scheduled work", + async (endpoint) => { + const fixture = await runCronTarget(endpoint); + await fixture.finished; + expect(fixture.command.mock.calls[0]![2].env).toMatchObject({ + NEMOCLAW_PROVIDER: "custom", + NEMOCLAW_ENDPOINT_URL: endpoint, + COMPATIBLE_API_KEY: "test-hosted-key", + }); + expect(fixture.exec.mock.calls.map((call) => call[1].slice(0, 3))).toEqual([ + ["openclaw", "cron", "add"], + ["openclaw", "cron", "add"], + ["openclaw", "cron", "run"], + ["openclaw", "cron", "remove"], + ]); + expect(fixture.exec.mock.calls.slice(2).map((call) => call[1][3])).toEqual([ + "cron-42", + "cron-42", + ]); + expect(fixture.complete).toHaveBeenCalledExactlyOnceWith({ + id: "cron-preflight-inference-local", + status: "passed", + nativeCronId: "cron-42", + nativeRunAccepted: true, + inferenceRoute: endpoint, + }); + await fixture.trackDisposable.mock.calls[0]![1](); + expect(fixture.cleanupSandbox).toHaveBeenLastCalledWith( + "cron-test", + expect.objectContaining({ + artifactName: "cron-preflight-cleanup-openshell-delete", + }), + ); + }, + ); + + it.each([ + { exitCode: 42, stdout: "", stderr: "command failed" }, + { exitCode: 0, stdout: '{"ok":true,"ran":false}', stderr: "" }, + { exitCode: 0, stdout: '{"ok":true,"ran":true}', stderr: "EAI_AGAIN" }, + ])("does not report completion for a rejected cron run %j", async (runResult) => { + const fixture = await runCronTarget("https://selected.example/v1", runResult); + await expect(fixture.finished).rejects.toThrow(); + expect(fixture.complete).not.toHaveBeenCalled(); + await fixture.trackDisposable.mock.calls[0]![1](); + expect(fixture.cleanupSandbox).toHaveBeenCalledTimes(2); + }); +}); const LIVE_ROOT = path.join(import.meta.dirname, "..", "live"); +describe("OpenShell stop/start sandbox cleanup", () => { + let directory: string; + + beforeEach(() => { + directory = fs.mkdtempSync(path.join(os.tmpdir(), "survival-cleanup-")); + vi.stubEnv("HOME", directory); + vi.stubEnv("XDG_CONFIG_HOME", path.join(directory, "config")); + vi.stubEnv("OPENSHELL_GATEWAY", "unrelated-gateway"); + vi.stubEnv("NEMOCLAW_SANDBOX_NAME", "e2e-survival-cleanup"); + vi.stubEnv("NVIDIA_INFERENCE_API_KEY", "must-not-reach-cleanup"); + vi.stubEnv("COMPATIBLE_API_KEY", "also-must-not-reach-cleanup"); + }); + + afterEach(() => { + fs.rmSync(directory, { recursive: true, force: true }); + }); + + async function runSurvival( + options: { + install?: () => Promise; + cleanup?: () => Promise; + cliAvailable?: boolean; + listFails?: boolean; + approvalFails?: boolean; + pluginFails?: boolean; + } = {}, + ) { + vi.resetModules(); + captured.test.mockClear(); + await import("../live/sandbox-survival.test.ts"); + const run = captured.test.mock.calls[0]![2] as (fixtures: object) => Promise; + const success = { exitCode: 0, stdout: "", stderr: "", signal: null, timedOut: false }; + const cleanupSandbox = vi.fn(async (_name: string, _options: { env?: NodeJS.ProcessEnv }) => + options.cleanup?.(), + ); + const list = vi.fn(async (_options: { env?: NodeJS.ProcessEnv }) => ({ + ...success, + exitCode: options.listFails ? 1 : 0, + })); + const trackDisposable = vi.fn((_name: string, _cleanup: () => Promise) => undefined); + const complete = vi.fn(); + const isCommandAvailable = vi.fn(async () => options.cliAvailable ?? true); + const waitForSandboxReadyAfterGatewayRestart = vi.fn(); + const execResults: Record = { + "openclaw cron": { + ...success, + exitCode: 1, + stderr: "scope upgrade pending approval (requestId: 12345678-1234-4123-8123-123456789abc)", + }, + "openclaw agent": { + ...success, + stdout: JSON.stringify({ + runId: "survival-run", + status: "ok", + summary: "completed", + result: { payloads: [], meta: {} }, + }), + }, + "openclaw audit": { + ...success, + stdout: JSON.stringify({ + events: [ + ["tool.action.started", "started"], + ["tool.action.finished", options.pluginFails ? "failed" : "succeeded"], + ].map(([action, status]) => ({ + kind: "tool_action", + runId: "survival-run", + toolName: "nemoclaw_e2e_check_v1", + toolCallId: "call-1", + action, + status, + })), + }), + }, + }; + const exec = vi.fn( + async (_name: string, args: string[], _options: { env?: NodeJS.ProcessEnv }) => + execResults[args.slice(0, 2).join(" ")] ?? { + ...success, + stdout: args[2] === "echo alive" ? "alive" : args[2].includes("/health") ? "ready" : "", + }, + ); + const commandResults: Record Promise> = { + "install-sh-sandbox-survival": async () => (await options.install?.()) ?? success, + "openclaw-explicit-admin-approval": async () => ({ + ...success, + exitCode: options.approvalFails ? 1 : 0, + stdout: options.approvalFails ? "" : "ISSUE_5324_ADMIN_APPROVAL_OK", + }), + }; + const command = vi.fn( + async ( + _command: string, + _args: string[], + commandOptions: { artifactName: string; env?: NodeJS.ProcessEnv }, + ) => commandResults[commandOptions.artifactName]?.() ?? success, + ); + const finished = run({ + artifacts: { target: { declare: vi.fn(), complete } }, + cleanup: { trackGateway: vi.fn(), trackDisposable }, + host: { + commandPath: "nemoclaw", + openshellCommandPath: "openshell", + isCommandAvailable, + command, + }, + lifecycle: { + trackInstallerGatewayUserService: vi.fn(), + stopGatewayRuntime: vi.fn(), + waitForSandboxReadyAfterGatewayRestart, + }, + progress: { phase: vi.fn() }, + runtimeProvider: { requireAvailable: vi.fn() }, + sandbox: { + cleanupSandbox, + list, + openshell: vi.fn(async () => success), + exec, + }, + secrets: { required: () => "test-hosted-key" }, + stateValidation: { + expectLocalRegistryContains: vi.fn(), + writeSandboxMarkers: vi.fn(), + expectSandboxMarkers: vi.fn(), + }, + }); + return { + finished, + cleanupSandbox, + list, + trackDisposable, + complete, + isCommandAvailable, + exec, + command, + waitForSandboxReadyAfterGatewayRestart, + }; + } + + function expectGatewayContext(env: NodeJS.ProcessEnv | undefined) { + expect(env).toMatchObject({ + HOME: directory, + XDG_CONFIG_HOME: path.join(directory, "config"), + OPENSHELL_GATEWAY: "nemoclaw", + }); + expect(env).not.toHaveProperty("NVIDIA_INFERENCE_API_KEY"); + expect(env).not.toHaveProperty("COMPATIBLE_API_KEY"); + } + + it("deletes only from the owned gateway after installation fails despite an ambient override", async () => { + const fixture = await runSurvival({ + install: vi.fn().mockRejectedValue(new Error("install failed")), + }); + await expect(fixture.finished).rejects.toThrow("install failed"); + await fixture.trackDisposable.mock.calls[0]![1](); + expect(fixture.cleanupSandbox).toHaveBeenCalledTimes(1); + expect(fixture.cleanupSandbox.mock.calls[0]![0]).toBe("e2e-survival-cleanup"); + expectGatewayContext(fixture.cleanupSandbox.mock.calls[0]![1].env); + expect(fixture.complete).not.toHaveBeenCalled(); + }); + + it("propagates deferred deletion failure", async () => { + const fixture = await runSurvival({ + install: vi.fn().mockRejectedValue(new Error("install failed")), + cleanup: vi.fn().mockRejectedValue(new Error("sandbox deletion failed")), + }); + await expect(fixture.finished).rejects.toThrow("install failed"); + await expect(fixture.trackDisposable.mock.calls[0]![1]()).rejects.toThrow( + "sandbox deletion failed", + ); + expect(fixture.complete).not.toHaveBeenCalled(); + }); + + it("does not delete when the OpenShell executable is unavailable", async () => { + const fixture = await runSurvival({ + install: vi.fn().mockRejectedValue(new Error("install failed")), + cliAvailable: false, + }); + await expect(fixture.finished).rejects.toThrow("install failed"); + await fixture.trackDisposable.mock.calls[0]![1](); + expect(fixture.cleanupSandbox).not.toHaveBeenCalled(); + expect(fixture.complete).not.toHaveBeenCalled(); + }); + + it("uses gateway context for final deletion and absence verification without duplicate deletion", async () => { + const fixture = await runSurvival(); + await fixture.finished; + expectGatewayContext(fixture.cleanupSandbox.mock.calls[0]![1].env); + expectGatewayContext(fixture.list.mock.calls[0]![0].env); + expectGatewayContext(fixture.waitForSandboxReadyAfterGatewayRestart.mock.calls[0]![1].env); + expect(fixture.exec.mock.calls.map((call) => call[2].env)).toEqual( + Array(fixture.exec.mock.calls.length).fill(fixture.cleanupSandbox.mock.calls[0]![1].env), + ); + expectGatewayContext( + fixture.command.mock.calls.find( + (call) => call[2].artifactName === "openclaw-explicit-admin-approval", + )![2].env, + ); + await fixture.trackDisposable.mock.calls[0]![1](); + expect(fixture.cleanupSandbox).toHaveBeenCalledTimes(1); + expect(fixture.isCommandAvailable).not.toHaveBeenCalled(); + expect(fixture.complete).toHaveBeenCalledTimes(1); + }); + + it("rejects an unsuccessful absence check even when deletion succeeds", async () => { + const fixture = await runSurvival({ listFails: true }); + await expect(fixture.finished).rejects.toThrow("sandbox destroy verification failed"); + expect(fixture.complete).not.toHaveBeenCalled(); + }); + + it.each([ + { + approvalFails: true, + pluginFails: false, + message: "explicit admin approval did not complete", + }, + { approvalFails: false, pluginFails: true, message: "the plugin tool must start and succeed" }, + ])("retains deferred cleanup when post-start verification fails: %j", async (options) => { + const fixture = await runSurvival(options); + await expect(fixture.finished).rejects.toThrow(options.message); + expect(fixture.complete).not.toHaveBeenCalled(); + expect(fixture.cleanupSandbox).not.toHaveBeenCalled(); + await fixture.trackDisposable.mock.calls[0]![1](); + expect(fixture.cleanupSandbox).toHaveBeenCalledTimes(1); + expectGatewayContext(fixture.cleanupSandbox.mock.calls[0]![1].env); + }); +}); + function liveSource(name: string): string { return fs.readFileSync(path.join(LIVE_ROOT, name), "utf8"); } describe("native lifecycle E2E migration contracts", () => { - it("runs scheduled inference through native OpenClaw cron", () => { - const source = liveSource("cron-preflight-inference-local.test.ts"); - - expect(source).toMatch(/"openclaw",\s*"cron",\s*"add"/u); - expect(source).toContain('["openclaw", "cron", "run", cronId]'); - expect(source).toContain('["openclaw", "cron", "remove", cronId]'); - expect(source).not.toContain("preflightCronModelProvider"); - }); - it.each([ [ "rebuild-openclaw.test.ts", @@ -188,3 +513,257 @@ exit 2 expect(runRestoration({ PLUGIN_JSON: pluginJson })).toEqual({ status: 1, stdout: "" }); }); }); + +describe("GPU double-onboard gateway cleanup", () => { + it.each([ + { + stopFails: false, + cliAvailable: true, + expected: "install boundary", + order: ["stop", "remove", "install"], + }, + { stopFails: true, cliAvailable: true, expected: "owned stop failed", order: ["stop"] }, + { + stopFails: false, + cliAvailable: false, + expected: "install boundary", + order: ["stop", "install"], + }, + ])( + "preserves cleanup order and stop failure (failure=$stopFails)", + async ({ stopFails, cliAvailable, expected, order }) => { + vi.stubEnv("NEMOCLAW_SANDBOX_NAME", "e2e-gpu-cleanup"); + vi.stubEnv("NEMOCLAW_CLI_BIN", "fixture-cli"); + vi.resetModules(); + captured.test.mockClear(); + await import("../live/gpu-double-onboard.test.ts"); + const run = captured.test.mock.calls[0]![2] as (fixtures: object) => Promise; + const observed: string[] = []; + const success = { exitCode: 0, stdout: "", stderr: "", signal: null, timedOut: false }; + const stopGatewayRuntime = vi.fn(async () => { + observed.push("stop"); + await (stopFails ? Promise.reject(new Error("owned stop failed")) : Promise.resolve()); + }); + const cleanupGatewayRegistration = vi.fn(async () => { + observed.push("remove"); + }); + const command = vi.fn( + async (_command: string, _args: string[], options: { artifactName: string }) => { + const actions: Record never> = { + "phase-1-install-ollama": () => { + observed.push("install"); + throw new Error("install boundary"); + }, + }; + return actions[options.artifactName]?.() ?? success; + }, + ); + const openshell = vi.fn(async () => success); + const trackDisposable = vi.fn((_name: string, _cleanup: () => Promise) => undefined); + await expect( + run({ + artifacts: { target: { declare: vi.fn() } }, + cleanup: { trackDisposable, trackSandbox: vi.fn() }, + host: { + command, + cleanupGatewayRegistration, + openshellCommandPath: "openshell", + isCommandAvailable: vi.fn(async () => cliAvailable), + }, + lifecycle: { stopGatewayRuntime }, + progress: { phase: vi.fn() }, + runtimeProvider: { requireAvailable: vi.fn() }, + sandbox: { openshell }, + skip: vi.fn(), + }), + ).rejects.toThrow(expected); + expect(observed).toEqual(order); + expect(stopGatewayRuntime).toHaveBeenCalledExactlyOnceWith({ + env: expect.objectContaining({ + OPENSHELL_GATEWAY: "nemoclaw", + NEMOCLAW_SANDBOX_NAME: "e2e-gpu-cleanup", + }), + userServiceMode: "permanent", + }); + expect(cleanupGatewayRegistration.mock.calls).toHaveLength( + Number(!stopFails && cliAvailable), + ); + expect(openshell.mock.calls).toEqual([ + [["sandbox", "delete", "e2e-gpu-cleanup"], expect.any(Object)], + ]); + observed.length = 0; + const terminalCleanup = trackDisposable.mock.calls.find( + ([name]) => name === "stop GPU gateway runtime before removing its state", + )![1]; + const outcomes = await Promise.allSettled([terminalCleanup()]); + expect(outcomes[0]!.status).toBe(stopFails ? "rejected" : "fulfilled"); + expect(observed).toEqual(order.filter((step) => step !== "install")); + }, + ); +}); + +describe("Messaging token rotation gateway cleanup", () => { + it.each([ + { + stopFails: false, + cliAvailable: true, + expected: "install boundary", + order: ["stop", "remove", "install"], + }, + { stopFails: true, cliAvailable: true, expected: "owned stop failed", order: ["stop"] }, + { + stopFails: false, + cliAvailable: false, + expected: "install boundary", + order: ["stop", "install"], + }, + ])( + "preserves runtime cleanup before registration (failure=$stopFails, CLI=$cliAvailable)", + async ({ stopFails, cliAvailable, expected, order }) => { + vi.stubEnv("NEMOCLAW_SANDBOX_NAME", "e2e-token-cleanup"); + vi.resetModules(); + captured.test.mockClear(); + const server = await import("../fixtures/fake-openai-compatible.ts"); + vi.spyOn(server, "startFakeOpenAiCompatibleServer").mockResolvedValue({ + baseUrl: "http://host.openshell.internal:19000/v1", + logFile: "/fixture/fake-server.log", + requestsFile: "/fixture/fake-requests.json", + requests: () => [], + environmentKeys: () => [], + close: async () => undefined, + } satisfies Awaited>); + await import("../live/token-rotation.test.ts"); + const run = captured.test.mock.calls[0]![2] as (fixtures: object) => Promise; + const observed: string[] = []; + const success = { exitCode: 0, stdout: "", stderr: "", signal: null, timedOut: false }; + const stopGatewayRuntime = vi.fn(async () => { + observed.push("stop"); + await (stopFails ? Promise.reject(new Error("owned stop failed")) : Promise.resolve()); + }); + const cleanupGatewayRegistration = vi.fn(async () => { + observed.push("remove"); + }); + const command = vi.fn( + async (_command: string, _args: string[], options: { artifactName: string }) => { + const actions: Record never> = { + "phase-0-install-token-a": () => { + observed.push("install"); + throw new Error("install boundary"); + }, + }; + return actions[options.artifactName]?.() ?? success; + }, + ); + const trackGateway = vi.fn( + ( + _owner: { cleanupGatewayRegistration: () => Promise }, + _name: string, + _options: object, + ) => undefined, + ); + await expect( + run({ + artifacts: { target: { declare: vi.fn() } }, + cleanup: { trackGateway, trackDisposable: vi.fn(), trackSandbox: vi.fn() }, + host: { + command, + cleanupGatewayRegistration, + openshellCommandPath: "openshell", + isCommandAvailable: vi.fn(async () => cliAvailable), + }, + lifecycle: { stopGatewayRuntime }, + progress: { phase: vi.fn() }, + runtimeProvider: { requireAvailable: vi.fn() }, + sandbox: {}, + }), + ).rejects.toThrow(expected); + expect(observed).toEqual(order); + expect(stopGatewayRuntime).toHaveBeenCalledExactlyOnceWith({ + env: expect.any(Object), + userServiceMode: "permanent", + }); + expect(cleanupGatewayRegistration.mock.calls).toHaveLength( + Number(!stopFails && cliAvailable), + ); + observed.length = 0; + const terminal = trackGateway.mock.calls[0]![0].cleanupGatewayRegistration; + const outcomes = await Promise.allSettled([terminal()]); + expect(outcomes[0]!.status).toBe(stopFails ? "rejected" : "fulfilled"); + expect(observed).toEqual(order.filter((step) => step !== "install")); + }, + ); +}); + +describe("Hermes fresh runner gateway cleanup", () => { + it.each([ + { + stopFails: false, + cliAvailable: true, + expected: "prerequisite boundary", + order: ["stop", "remove", "prerequisite"], + }, + { stopFails: true, cliAvailable: true, expected: "owned stop failed", order: ["stop"] }, + { + stopFails: false, + cliAvailable: false, + expected: "prerequisite boundary", + order: ["stop", "prerequisite"], + }, + ])( + "preserves owned cleanup before prerequisites ($stopFails/$cliAvailable)", + async ({ stopFails, cliAvailable, expected, order }) => { + vi.stubEnv("NEMOCLAW_SANDBOX_NAME", "e2e-hermes-cleanup"); + vi.resetModules(); + captured.test.mockClear(); + await import("../live/hermes-e2e.test.ts"); + const run = captured.test.mock.calls[0]![2] as (fixtures: object) => Promise; + const observed: string[] = []; + const success = { exitCode: 0, stdout: "", stderr: "", signal: null, timedOut: false }; + const openshell = vi.fn(async () => success); + const stopGatewayRuntime = vi.fn(async () => { + observed.push("stop"); + await (stopFails ? Promise.reject(new Error("owned stop failed")) : Promise.resolve()); + }); + const cleanupGatewayRegistration = vi.fn(async () => { + observed.push("remove"); + }); + await expect( + run({ + artifacts: { target: { declare: vi.fn() } }, + cleanup: { trackGateway: vi.fn(), trackDisposable: vi.fn(), trackSandbox: vi.fn() }, + host: { + command: vi.fn(async () => success), + openshellCommandPath: "openshell", + isCommandAvailable: vi.fn(async () => cliAvailable), + cleanupGatewayRegistration, + }, + lifecycle: { stopGatewayRuntime, trackInstallerGatewayUserService: vi.fn() }, + inference: { env: () => ({}), redactionValues: () => [], mode: "mock" }, + progress: { phase: vi.fn() }, + runtimeProvider: { + requireAvailable: vi.fn(async () => { + observed.push("prerequisite"); + throw new Error("prerequisite boundary"); + }), + }, + sandbox: { openshell }, + }), + ).rejects.toThrow(expected); + expect(observed).toEqual(order); + expect(openshell).toHaveBeenCalledExactlyOnceWith( + ["sandbox", "delete", "e2e-hermes-cleanup"], + expect.any(Object), + ); + expect(stopGatewayRuntime).toHaveBeenCalledExactlyOnceWith({ + env: expect.objectContaining({ + NEMOCLAW_AGENT: "hermes", + NEMOCLAW_SANDBOX_NAME: "e2e-hermes-cleanup", + }), + userServiceMode: "permanent", + }); + expect(cleanupGatewayRegistration.mock.calls).toHaveLength( + Number(!stopFails && cliAvailable), + ); + }, + ); +}); diff --git a/test/e2e/support/network-policy-probe.test.ts b/test/e2e/support/network-policy-probe.test.ts index 71b1b5df99f..7a7ec82d407 100644 --- a/test/e2e/support/network-policy-probe.test.ts +++ b/test/e2e/support/network-policy-probe.test.ts @@ -1,45 +1,165 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { describe, expect, it } from "vitest"; +import { EventEmitter } from "node:events"; +import fs from "node:fs"; +import vm from "node:vm"; +import { describe, expect, it, vi } from "vitest"; import { - buildNetworkPolicyCurlProbe, - parseNetworkPolicyCurlOutput, + buildNetworkPolicyRequest, + isNetworkPolicyDenial, + NETWORK_POLICY_RESULT_PREFIX, } from "./network-policy-probe.ts"; -describe("network-policy curl probe", () => { - it("keeps the status format with its parser", () => { - expect(buildNetworkPolicyCurlProbe("http://host.openshell.internal:1234/")).toContain( - String.raw`-w '\nSTATUS_%{http_code}\n'`, - ); +function executeProbe(url = "https://example.com/") { + const request = new EventEmitter() as EventEmitter & { destroy: ReturnType }; + request.destroy = vi.fn((error: Error) => request.emit("error", error)); + const output: string[] = []; + const get = vi.fn( + ( + _url: URL, + _options: object, + _callback: ( + response: EventEmitter & { statusCode: number; setEncoding: (encoding: string) => void }, + ) => void, + ) => request, + ); + const deadline = vi.fn((_callback: () => void, _delay: number) => undefined); + vm.runInNewContext(buildNetworkPolicyRequest(url)[2]!, { + URL, + require: vi.fn(() => ({ get })), + console: { log: (line: string) => output.push(line) }, + setTimeout: deadline, + clearTimeout: vi.fn(), }); + return { request, output, get, deadline }; +} - it("quotes a URL as one shell argument", () => { - expect(buildNetworkPolicyCurlProbe("http://host.openshell.internal:1234/'; echo UNSAFE")).toBe( - String.raw`curl -sS --connect-timeout 10 --max-time 20 -w '\nSTATUS_%{http_code}\n' 'http://host.openshell.internal:1234/'\''; echo UNSAFE' 2>&1`, - ); +const denialResult = (record: unknown) => ({ + exitCode: 0, + stdout: NETWORK_POLICY_RESULT_PREFIX + JSON.stringify(record), +}); + +describe("native network policy probe", () => { + it("executes one URL argument without shell interpretation and preserves connect EACCES", () => { + const url = "http://host.openshell.internal:1234/'; echo UNSAFE"; + const probe = executeProbe(url); + probe.request.emit("error", { code: "EACCES", syscall: "connect" }); + expect(buildNetworkPolicyRequest(url).slice(0, 2)).toEqual(["/usr/local/bin/node", "-e"]); + expect(probe.get.mock.calls[0]?.[0].href).toBe(new URL(url).href); + expect(probe.get.mock.calls[0]?.[1]).toEqual({ autoSelectFamily: false }); + expect(isNetworkPolicyDenial({ exitCode: 0, stdout: probe.output.join("\n") })).toBe(true); }); it.each([ + [200, "allowed marker", false], + [403, "policy_denied", true], + ])("retains HTTP status %s and response body", (status, body, denied) => { + const probe = executeProbe(); + const response = Object.assign(new EventEmitter(), { + statusCode: status, + setEncoding: vi.fn(), + }); + probe.get.mock.calls[0]![2](response); + response.emit("data", body); + response.emit("end"); + expect(probe.output.join("\n")).toContain(body); + expect(isNetworkPolicyDenial({ exitCode: 0, stdout: probe.output.join("\n") })).toBe(denied); + }); + + it("rejects an oversized response without treating truncation as policy denial", () => { + const probe = executeProbe(); + const response = Object.assign(new EventEmitter(), { statusCode: 403, setEncoding: vi.fn() }); + probe.get.mock.calls[0]![2](response); + response.emit("data", "x".repeat(65537)); + expect(probe.request.destroy).toHaveBeenCalledOnce(); + expect(isNetworkPolicyDenial({ exitCode: 0, stdout: probe.output.join("\n") })).toBe(false); + }); + + it.each(["ECONNREFUSED", "ECONNRESET", "ETIMEDOUT", "ENOTFOUND", "EAGAIN", "ECANCELED"])( + "rejects %s as denial evidence", + (code) => { + const probe = executeProbe(); + probe.request.emit("error", { code, syscall: "connect" }); + expect(isNetworkPolicyDenial({ exitCode: 0, stdout: probe.output.join("\n") })).toBe(false); + }, + ); + + it("fails a bounded deadline without manufacturing policy evidence", () => { + const probe = executeProbe(); + probe.deadline.mock.calls[0]![0](); + expect(probe.deadline.mock.calls[0]?.[1]).toBe(20_000); + expect(isNetworkPolicyDenial({ exitCode: 0, stdout: probe.output.join("\n") })).toBe(false); + }); + + it.each([ + [{ errorCode: "EACCES", syscall: "connect" }, true], + [{ errorCode: "EACCES", syscall: "open" }, false], + [{ status: 403, response: '{"detail":"policy_denied"}' }, true], + [{ status: 403, response: "origin authentication refused" }, false], + [{ status: 200, response: "policy_denied" }, false], + [{ status: 0, response: "curl: (7) Failed to connect" }, false], + [null, false], + ])("classifies structured network result %j", (record, expected) => { + expect(isNetworkPolicyDenial(denialResult(record))).toBe(expected); + }); + + it.each([ + { exitCode: 1 }, + { timedOut: true }, + { signal: "SIGTERM" }, + { stdout: NETWORK_POLICY_RESULT_PREFIX + "invalid JSON" }, + { stdout: "error connect EACCES" }, + { + stdout: `${denialResult({ errorCode: "EACCES", syscall: "connect" }).stdout}\n${denialResult({ errorCode: "EACCES", syscall: "connect" }).stdout}`, + }, + ])("rejects incomplete or ambiguous command evidence %j", (override) => { + expect( + isNetworkPolicyDenial({ + ...denialResult({ errorCode: "EACCES", syscall: "connect" }), + ...override, + }), + ).toBe(false); + }); +}); + +describe("installed web_fetch error evidence", () => { + const source = fs.readFileSync( + new URL("../live/network-policy.test.ts", import.meta.url), + "utf8", + ); + const classifier = source.slice( + source.indexOf("function errorChainDetail(error) {"), + source.indexOf("\nconst approved = await webFetch.execute"), + ); + + it.each([ + [{ code: "EACCES", syscall: "connect" }, true], + [{ cause: { code: "EACCES", syscall: "connect" } }, true], [ - "a terminal LF record", - '{"detail":"policy_denied"}\nSTATUS_403\n', - { response: '{"detail":"policy_denied"}', status: 403 }, - ], - ["a terminal CRLF record", "denied\r\nSTATUS_403\r\n", { response: "denied", status: 403 }], - [ - "response whitespace", - "\n denied \n\nSTATUS_403\n", - { response: "\n denied \n", status: 403 }, + { + errors: [ + { code: "EACCES", syscall: "connect" }, + { code: "EACCES", syscall: "connect" }, + ], + }, + true, ], [ - "status-like response text", - '{"detail":"STATUS_403"}\nSTATUS_000\n', - { response: '{"detail":"STATUS_403"}', status: 0 }, + { + errors: [ + { code: "EACCES", syscall: "connect" }, + { code: "ETIMEDOUT", syscall: "connect" }, + ], + }, + false, ], - ["no terminal record", '{"detail":"STATUS_403"}', null], - ])("separates response and status from %s", (_label, output, expected) => { - expect(parseNetworkPolicyCurlOutput(output)).toEqual(expected); + [{ code: "EACCES", syscall: "open" }, false], + [{ message: "E2E_NATIVE_CONNECT_EACCES" }, false], + [{ name: "SsrFBlockedError", message: "Blocked hostname" }, false], + ])("classifies actual web_fetch probe error chain %j", (error, expected) => { + const result = vm.runInNewContext(`${classifier}\nerrorChainDetail(input);`, { input: error }); + expect(result.nativeDenied).toBe(expected); }); }); diff --git a/test/e2e/support/network-policy-probe.ts b/test/e2e/support/network-policy-probe.ts index ac8c3cd406e..543db388598 100644 --- a/test/e2e/support/network-policy-probe.ts +++ b/test/e2e/support/network-policy-probe.ts @@ -1,22 +1,54 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { shellQuote } from "../fixtures/clients/command.ts"; +export const NETWORK_POLICY_RESULT_PREFIX = "NEMOCLAW_NETWORK_RESULT "; -const CURL_STATUS_FORMAT = String.raw`\nSTATUS_%{http_code}\n`; -const CURL_STATUS_RECORD = /(?:^|\r?\n)STATUS_(\d{3})\r?\n?$/u; +/** Preserve the syscall error; curl's exit 7 also represents unrelated failures. */ +export const NETWORK_POLICY_ERROR_REPORT = `console.log(${JSON.stringify(NETWORK_POLICY_RESULT_PREFIX)} + JSON.stringify({ errorCode: error.code ?? null, syscall: error.syscall ?? null }));`; -export function buildNetworkPolicyCurlProbe(url: string): string { - return `curl -sS --connect-timeout 10 --max-time 20 -w ${shellQuote(CURL_STATUS_FORMAT)} ${shellQuote(url)} 2>&1`; +export function buildNetworkPolicyRequest(url: string): string[] { + const script = ` +const url = new URL(${JSON.stringify(url)}); +const client = require(url.protocol === "https:" ? "node:https" : "node:http"); +const request = client.get(url, { autoSelectFamily: false }, (response) => { + let body = ""; + response.setEncoding("utf8"); + response.on("data", (chunk) => { + body += chunk; + if (body.length > 65536) request.destroy(new Error("response exceeded probe limit")); + }); + response.on("end", () => console.log(${JSON.stringify(NETWORK_POLICY_RESULT_PREFIX)} + JSON.stringify({ status: response.statusCode, response: body }))); +}); +const deadline = setTimeout(() => request.destroy(new Error("probe deadline exceeded")), 20000); +request.on("close", () => clearTimeout(deadline)); +request.on("error", (error) => { ${NETWORK_POLICY_ERROR_REPORT} }); +`; + return ["/usr/local/bin/node", "-e", script]; } -export function parseNetworkPolicyCurlOutput( - output: string, -): { response: string; status: number } | null { - const match = CURL_STATUS_RECORD.exec(output); - if (!match) return null; - return { - response: output.slice(0, match.index), - status: Number.parseInt(match[1]!, 10), - }; +export function isNetworkPolicyDenial(result: { + stdout: string; + exitCode: number | null; + timedOut?: boolean; + signal?: string | null; +}): boolean { + if (result.exitCode !== 0 || result.timedOut || result.signal) return false; + const records = result.stdout + .split(/\r?\n/u) + .filter((line) => line.startsWith(NETWORK_POLICY_RESULT_PREFIX)); + if (records.length !== 1) return false; + try { + const record = JSON.parse(records[0]!.slice(NETWORK_POLICY_RESULT_PREFIX.length)); + return ( + record !== null && + typeof record === "object" && + !Array.isArray(record) && + ((record.errorCode === "EACCES" && record.syscall === "connect") || + (record.status === 403 && + typeof record.response === "string" && + /policy[_ ]denied|not allowed by any policy/iu.test(record.response))) + ); + } catch { + return false; + } } diff --git a/test/e2e/support/openclaw-agent-assertion.test.ts b/test/e2e/support/openclaw-agent-assertion.test.ts index dd42786ad12..18650be2506 100644 --- a/test/e2e/support/openclaw-agent-assertion.test.ts +++ b/test/e2e/support/openclaw-agent-assertion.test.ts @@ -79,70 +79,93 @@ function publicFetchSessionJsonLines(options: { isError?: boolean; payload?: obj const SECRET_SENTINEL = "nvidia-api-key-must-not-remain"; describe("OpenClaw agent assertion", () => { - it("keeps OpenShell SSH configuration outside artifacts and removes its temporary file", async () => { - const fixtureRoot = mkdtempSync(join(tmpdir(), "nemoclaw-openclaw-assertion-")); - const artifactRoot = join(fixtureRoot, "artifacts"); - const temporaryRoot = join(fixtureRoot, "tmp"); - mkdirSync(temporaryRoot); - vi.stubEnv("TMPDIR", temporaryRoot); - const sshConfigSentinel = `ProxyCommand connect -H bearer=${SECRET_SENTINEL}`; - const shellResult = { - artifacts: { result: "", stderr: "", stdout: "" }, - command: [], - durationMs: 1, - exitCode: 0, - signal: null, - stderr: "", - stdout: sshConfigSentinel, - timedOut: false, - }; - const sandbox = { - exec: vi.fn(), - openshell: vi.fn().mockResolvedValue(shellResult), - } as unknown as SandboxClient; - const host = { - command: vi.fn().mockImplementation(async (_command: string, args: string[]) => { - const configPath = args[1]; - expect(readFileSync(configPath!, "utf8")).toBe(sshConfigSentinel); - expect(statSync(configPath!).mode & 0o777).toBe(0o600); - return { + it.each([ + { collectToolEvidenceOnFailure: false, expectedReducedArtifacts: [] }, + { + collectToolEvidenceOnFailure: true, + expectedReducedArtifacts: [ + { toolCallCount: 1, toolResultCount: 1, execCalls: 0, execResults: 0 }, + ], + }, + ])( + "keeps failed-turn evidence and SSH cleanup safe with tool collection $collectToolEvidenceOnFailure", + async ({ collectToolEvidenceOnFailure, expectedReducedArtifacts }) => { + const fixtureRoot = mkdtempSync(join(tmpdir(), "nemoclaw-openclaw-assertion-")); + const artifactRoot = join(fixtureRoot, "artifacts"); + const temporaryRoot = join(fixtureRoot, "tmp"); + mkdirSync(temporaryRoot); + vi.stubEnv("TMPDIR", temporaryRoot); + const sshConfigSentinel = `ProxyCommand connect -H bearer=${SECRET_SENTINEL}`; + const shellResult = { + artifacts: { result: "", stderr: "", stdout: "" }, + command: [], + durationMs: 1, + exitCode: 0, + signal: null, + stderr: "", + stdout: sshConfigSentinel, + timedOut: false, + }; + const sandbox = { + exec: vi.fn().mockResolvedValue({ ...shellResult, - exitCode: 1, - stderr: `credential=${SECRET_SENTINEL}`, - stdout: "wrong reply", - }; - }), - } as unknown as HostCliClient; - - try { - await expect( - runOpenClawAgentAssertion(host, sandbox, new ArtifactSink(artifactRoot), { - apiKey: SECRET_SENTINEL, - expected: "PERSONAL_PUBLIC_FETCH_OK", - label: "personal-public-fetch", - persistCommandArtifacts: false, - prompt: "fetch the fixed public reference", - redactOutputInFailure: true, - sandboxName: "personal-sandbox", + stdout: + "__NEMOCLAW_TOOL_EVIDENCE__=" + + JSON.stringify(reduceOpenClawToolEvidence(publicFetchSessionJsonLines(), "")), + }), + openshell: vi.fn().mockResolvedValue(shellResult), + } as unknown as SandboxClient; + const host = { + command: vi.fn().mockImplementation(async (_command: string, args: string[]) => { + const configPath = args[1]; + expect(readFileSync(configPath!, "utf8")).toBe(sshConfigSentinel); + expect(statSync(configPath!).mode & 0o777).toBe(0o600); + return { + ...shellResult, + exitCode: 1, + stderr: `credential=${SECRET_SENTINEL}`, + stdout: "wrong reply", + }; }), - ).rejects.toThrow(/agent output omitted; exit=1/u); + } as unknown as HostCliClient; - expect(sandbox.openshell).toHaveBeenCalledWith( - ["sandbox", "ssh-config", "personal-sandbox"], - expect.objectContaining({ persistArtifacts: false }), - ); - expect(readdirSync(temporaryRoot)).toEqual([]); - expect(existsSync(join(artifactRoot, "ssh"))).toBe(false); - const persistedActions = readdirSync(join(artifactRoot, "actions")) - .map((name) => readFileSync(join(artifactRoot, "actions", name), "utf8")) - .join("\n"); - expect(persistedActions).not.toContain(SECRET_SENTINEL); - expect(sandbox.exec).not.toHaveBeenCalled(); - } finally { - vi.unstubAllEnvs(); - rmSync(fixtureRoot, { force: true, recursive: true }); - } - }); + try { + await expect( + runOpenClawAgentAssertion(host, sandbox, new ArtifactSink(artifactRoot), { + apiKey: SECRET_SENTINEL, + collectToolEvidenceOnFailure, + expected: "PERSONAL_PUBLIC_FETCH_OK", + label: "personal-public-fetch", + persistCommandArtifacts: false, + prompt: "fetch the fixed public reference", + redactOutputInFailure: true, + sandboxName: "personal-sandbox", + }), + ).rejects.toThrow(/agent output omitted; exit=1/u); + + expect(sandbox.openshell).toHaveBeenCalledWith( + ["sandbox", "ssh-config", "personal-sandbox"], + expect.objectContaining({ persistArtifacts: false }), + ); + expect(readdirSync(temporaryRoot)).toEqual([]); + expect(existsSync(join(artifactRoot, "ssh"))).toBe(false); + const persistedActions = readdirSync(join(artifactRoot, "actions")) + .map((name) => readFileSync(join(artifactRoot, "actions", name), "utf8")) + .join("\n"); + expect(persistedActions).not.toContain(SECRET_SENTINEL); + expect(sandbox.exec).toHaveBeenCalledTimes(expectedReducedArtifacts.length); + const reducedArtifacts = readdirSync(join(artifactRoot, "actions")) + .filter((name) => name.endsWith("-reduced.json")) + .map((name) => JSON.parse(readFileSync(join(artifactRoot, "actions", name), "utf8"))); + expect(reducedArtifacts).toHaveLength(expectedReducedArtifacts.length); + expect(reducedArtifacts).toMatchObject(expectedReducedArtifacts); + expect(persistedActions).not.toContain(PUBLIC_FETCH_EXPECTATION.url); + } finally { + vi.unstubAllEnvs(); + rmSync(fixtureRoot, { force: true, recursive: true }); + } + }, + ); it("collects failed-turn records and a separate HTTP probe before preserving the reply failure", async () => { const fixtureRoot = mkdtempSync(join(tmpdir(), "nemoclaw-fetch-diagnostic-")); @@ -483,6 +506,83 @@ function replyFixture(reply: string, meta: Record = {}) { }; } +describe("weather reply assertion", () => { + it.each(["WEATHER_AGENT_OK", "\nWEATHER_AGENT_OK\n"])( + "accepts the exact weather reply with surrounding whitespace: %j", + async (reply) => { + const f = replyFixture(reply); + try { + await expect( + runOpenClawAgentAssertion( + f.host as unknown as HostCliClient, + f.sandbox as unknown as SandboxClient, + f.artifacts, + { + ...f.options, + expected: "WEATHER_AGENT_OK", + label: "weather", + finalReplyContract: "exact-token", + collectToolEvidenceOnFailure: true, + }, + ), + ).resolves.toMatchObject({ reply: reply.trim() }); + expect(f.host.command).toHaveBeenCalledTimes(1); + expect(f.sandbox.exec).not.toHaveBeenCalled(); + } finally { + rmSync(f.root, { force: true, recursive: true }); + } + }, + ); + + it.each([ + { + caseName: "tool-call markup instead of execution", + reply: '<<>>', + visible: '<<>>', + }, + { + caseName: "a printed command with a fabricated final token", + reply: "printf WEATHER_AGENT_OK", + visible: "WEATHER_AGENT_OK", + }, + { + caseName: "a missing final visible reply", + reply: "WEATHER_AGENT_OK", + visible: undefined, + }, + ])("rejects $caseName and retains tool counts without retrying", async ({ reply, visible }) => { + const f = replyFixture(reply, { finalAssistantVisibleText: visible }); + try { + await expect( + runOpenClawAgentAssertion( + f.host as unknown as HostCliClient, + f.sandbox as unknown as SandboxClient, + f.artifacts, + { + ...f.options, + expected: "WEATHER_AGENT_OK", + label: "weather", + finalReplyContract: "exact-token", + collectToolEvidenceOnFailure: true, + }, + ), + ).rejects.toThrow("payload and final visible reply must contain only the expected token"); + expect(f.host.command).toHaveBeenCalledTimes(1); + expect(f.sandbox.exec).toHaveBeenCalledTimes(1); + const reduced = JSON.parse( + readFileSync( + join(f.root, "artifacts", "actions", "weather-attempt-1-reduced.json"), + "utf8", + ), + ); + expect(reduced).toMatchObject({ execCalls: 0, execResults: 0, toolCallCount: 0 }); + expect(JSON.stringify(reduced)).not.toContain(SECRET_SENTINEL); + } finally { + rmSync(f.root, { force: true, recursive: true }); + } + }); +}); + describe("unavailable-search assertion wiring", () => { it.each([ { diff --git a/test/e2e/support/openclaw-agent-output.test.ts b/test/e2e/support/openclaw-agent-output.test.ts index 344a40406e3..bb4ca09946a 100644 --- a/test/e2e/support/openclaw-agent-output.test.ts +++ b/test/e2e/support/openclaw-agent-output.test.ts @@ -433,7 +433,7 @@ describe("OpenClaw native state doctor evidence", () => { describe("OpenClaw native state process identities", () => { const identities = ` EUID EGID PID PPID COMMAND - 0 0 1 0 openshell-sandb + 998 998 1 0 openshell-sandb 998 998 892 1 bash 998 998 1315 892 openclaw-gatewa 998 998 2196 1 ps`; @@ -456,8 +456,10 @@ describe("OpenClaw native state process identities", () => { ["duplicate PID", 0, false, `${identities}\n998 998 892 1 bash`, false], ["non-Bash parent", 0, false, identities.replace("1 bash", "1 sh"), false], ["indirect parent", 0, false, identities.replace("892 1 bash", "892 900 bash"), false], - ["non-root PID1 user", 0, false, identities.replace("0 0 1 0", "998 0 1 0"), false], - ["non-root PID1 group", 0, false, identities.replace("0 0 1 0", "0 998 1 0"), false], + ["root PID1 user", 0, false, identities.replace("998 998 1 0", "0 998 1 0"), false], + ["root PID1 group", 0, false, identities.replace("998 998 1 0", "998 0 1 0"), false], + ["different PID1 user", 0, false, identities.replace("998 998 1 0", "999 998 1 0"), false], + ["different PID1 group", 0, false, identities.replace("998 998 1 0", "998 999 1 0"), false], ["malformed row", 0, false, `${identities}\ntruncated`, false], ["empty stdout", 0, false, "", false], ["failed command", 1, false, identities, false], diff --git a/test/e2e/support/openclaw-inference-switch-baseline.test.ts b/test/e2e/support/openclaw-inference-switch-baseline.test.ts index 78145655459..9a846d3788f 100644 --- a/test/e2e/support/openclaw-inference-switch-baseline.test.ts +++ b/test/e2e/support/openclaw-inference-switch-baseline.test.ts @@ -37,9 +37,12 @@ vi.mock("../fixtures/managed-image-receipt.ts", () => ({ const disposables: Array<() => unknown> = []; const platform = Object.getOwnPropertyDescriptor(process, "platform")!; +async function disposeResources(): Promise { + for (const dispose of disposables.splice(0).reverse()) await dispose(); +} afterEach(async () => { Object.defineProperty(process, "platform", platform); - for (const dispose of disposables.splice(0).reverse()) await dispose(); + await disposeResources(); vi.doUnmock("../live/openclaw-inference-switch-helpers.ts"); vi.restoreAllMocks(); vi.unstubAllEnvs(); @@ -173,6 +176,8 @@ it.each( expect(name).toBe("NVIDIA_API_KEY"); return "nvapi-public-fixture-key"; }); + const cleanupSandbox = vi.fn(async () => undefined); + const expectRemoved = vi.fn(async () => undefined); await expect( run({ artifacts: { target: { declare: vi.fn() } }, @@ -183,9 +188,15 @@ it.each( }, host: { nemoclaw, - command: vi.fn(async () => result), + cleanupSandbox, + command: vi.fn(async (_cmd, _args, options) => + options?.artifactName === "native-baseline-endpoint-address" + ? { ...result, stdout: "route 10.20.30.40" } + : result, + ), }, progress, + gateway: { expectRemoved }, runtimeProvider: { id: runtime, requireAvailable: vi.fn() }, sandbox: { openshell: vi.fn(async () => result), cleanupSandbox: vi.fn() }, secrets: { required }, @@ -223,6 +234,15 @@ it.each( expect(environment.NVIDIA_API_KEY).toBeUndefined(); expect(options.redactionValues.includes("nvapi-public-fixture-key")).toBe(!nativeCustom); expect(options.redactionValues).toContain(environment.COMPATIBLE_API_KEY); + await disposeResources(); + expect(cleanupSandbox).toHaveBeenCalledExactlyOnceWith( + expect.any(String), + expect.objectContaining({ cleanupGateway: true }), + ); + expect(expectRemoved).toHaveBeenCalledExactlyOnceWith( + "nemoclaw", + expect.objectContaining({ artifactName: "cleanup-inference-switch-gateway-absent" }), + ); }, ); diff --git a/test/e2e/support/openclaw-inference-switch-helpers.test.ts b/test/e2e/support/openclaw-inference-switch-helpers.test.ts index f7290c99a51..270ba4ddcaf 100644 --- a/test/e2e/support/openclaw-inference-switch-helpers.test.ts +++ b/test/e2e/support/openclaw-inference-switch-helpers.test.ts @@ -11,11 +11,16 @@ import { patchOpenClawInferenceConfig } from "../../../src/lib/actions/inference import type { HostCliClient } from "../fixtures/clients/host.ts"; import { PUBLIC_NVIDIA_SWITCH_MODEL } from "../live/public-nvidia-switch-provider.ts"; +import { OPERATOR_SWITCH_API_KEY } from "../fixtures/registered-inference.ts"; import { startTestProgress } from "../fixtures/progress.ts"; import { + createOpenClawInferenceSwitchCleanup, + readOpenClawBootId, getRouteOutput, readOpenClawConfig, + httpStatusFromResponse, + httpBodyFromResponse, agentReplyContainsToken, anthropicToolCount, classifyExhaustedPostSwitchEvidence, @@ -28,6 +33,84 @@ import { startMockOpenClawBaselineProvider, } from "../live/openclaw-inference-switch-helpers.ts"; +describe("OpenClaw authenticated restart identity", () => { + const bootId = "0ad13a83-485c-427c-a6ba-a64e7d86621a"; + const valid = { + exitCode: 0, + timedOut: false, + signal: null, + stderr: "", + stdout: JSON.stringify({ rpc: { ok: true, kind: "read", server: { bootId } } }), + }; + + it.each([ + [valid, bootId], + [{ ...valid, exitCode: 1 }, null], + [{ ...valid, timedOut: true }, null], + [{ ...valid, signal: "SIGTERM" }, null], + [{ ...valid, stderr: "[sandbox-safety-net] loaded (launcher)\n" }, bootId], + [{ ...valid, exitCode: 1, stderr: "observation failed" }, null], + [{ ...valid, stdout: "{}" }, null], + [{ ...valid, stdout: "null" }, null], + [ + { + ...valid, + stdout: JSON.stringify({ rpc: { ok: false, kind: "read", server: { bootId } } }), + }, + null, + ], + [ + { + ...valid, + stdout: JSON.stringify({ rpc: { ok: true, kind: "connect", server: { bootId } } }), + }, + null, + ], + [ + { + ...valid, + stdout: JSON.stringify({ rpc: { ok: true, kind: "read", server: { bootId: 42 } } }), + }, + null, + ], + [ + { + ...valid, + stdout: '{"rpc":{"ok":true,"kind":"read","server":{"bootId":"untrusted text"}}}', + }, + null, + ], + [{ ...valid, stdout: "not json" }, null], + ])("accepts only a successful, complete boot observation %#", async (result, expected) => { + const exec = vi.fn().mockResolvedValue(result); + await expect( + readOpenClawBootId({ exec } as never, "alpha", { + env: { HOME: "/private-home" }, + artifactName: "boot-before", + }), + ).resolves.toBe(expected); + expect(exec).toHaveBeenCalledWith( + "alpha", + [ + "openclaw", + "gateway", + "status", + "--port", + "18789", + "--require-rpc", + "--json", + "--timeout", + "10000", + ], + { + env: { HOME: "/private-home" }, + artifactName: "boot-before", + timeoutMs: 20_000, + }, + ); + }); +}); + describe("openclaw-inference-switch post-switch retry classification", () => { const attempt = { exitCode: 1, @@ -237,54 +320,70 @@ describe("openclaw-inference-switch gateway model-run output", () => { }); describe("openclaw-inference-switch mock-Anthropic baseline", () => { - it("serves authenticated PONG replies for the lifecycle gateway checks", async () => { - const progress = startTestProgress("OpenClaw baseline", ["serve baseline", "verify baseline"], { - logLine: () => undefined, - }); - const baseline = await startMockOpenClawBaselineProvider(progress); - try { - const endpoint = new URL(`${baseline.baseUrl}/chat/completions`); - expect(endpoint.hostname).toBe("host.openshell.internal"); - endpoint.hostname = "127.0.0.1"; - const payload = { - model: MOCK_BASELINE_MODEL, - messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], - stream: true, - }; - const denied = await fetch(endpoint, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify(payload), - }); - expect(denied.status).toBe(401); - const response = await fetch(endpoint, { - method: "POST", - headers: { - Authorization: `Bearer ${MOCK_BASELINE_API_KEY}`, - "Content-Type": "application/json", + it.each([undefined, OPERATOR_SWITCH_API_KEY])( + "serves authenticated PONG replies using the selected fixture credential: %s", + async (apiKey) => { + const progress = startTestProgress( + "OpenClaw baseline", + ["serve baseline", "verify baseline"], + { + logLine: () => undefined, }, - body: JSON.stringify(payload), - }); - expect(response.status).toBe(200); - const chunks = (await response.text()) - .split("\n\n") - .filter((chunk) => chunk.startsWith("data: {")) - .map((chunk) => JSON.parse(chunk.slice("data: ".length))); - expect(chunks.map((chunk) => chunk.choices[0].delta.content ?? "").join("")).toBe("PONG"); - expect(baseline.requests()).toContainEqual( - expect.objectContaining({ - auth: "ok", - method: "POST", - path: "/v1/chat/completions", + ); + const baseline = await startMockOpenClawBaselineProvider(progress, undefined, apiKey); + try { + const endpoint = new URL(`${baseline.baseUrl}/chat/completions`); + expect(endpoint.hostname).toBe("host.openshell.internal"); + endpoint.hostname = "127.0.0.1"; + const payload = { model: MOCK_BASELINE_MODEL, + messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], stream: true, - }), - ); - } finally { - await baseline.close(); - progress.stop(); - } - }); + }; + const denied = await fetch(endpoint, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(payload), + }); + expect(denied.status).toBe(401); + const wrongCredential = await fetch(endpoint, { + method: "POST", + headers: { + "Content-Type": "application/json", + Authorization: `Bearer ${apiKey ? MOCK_BASELINE_API_KEY : OPERATOR_SWITCH_API_KEY}`, + }, + body: JSON.stringify(payload), + }); + expect(wrongCredential.status).toBe(401); + const response = await fetch(endpoint, { + method: "POST", + headers: { + Authorization: `Bearer ${apiKey ?? MOCK_BASELINE_API_KEY}`, + "Content-Type": "application/json", + }, + body: JSON.stringify(payload), + }); + expect(response.status).toBe(200); + const chunks = (await response.text()) + .split("\n\n") + .filter((chunk) => chunk.startsWith("data: {")) + .map((chunk) => JSON.parse(chunk.slice("data: ".length))); + expect(chunks.map((chunk) => chunk.choices[0].delta.content ?? "").join("")).toBe("PONG"); + expect(baseline.requests()).toContainEqual( + expect.objectContaining({ + auth: "ok", + method: "POST", + path: "/v1/chat/completions", + model: MOCK_BASELINE_MODEL, + stream: true, + }), + ); + } finally { + await baseline.close(); + progress.stop(); + } + }, + ); it("uses an authenticated local baseline with the compatible env wiring", () => { expect(mockBaselineInference("http://127.0.0.1:34567/v1")).toEqual({ @@ -390,6 +489,7 @@ function sandboxForOpenClawConfig(config: object, handle = scopedNativeHandle) { env: { ...process.env, NVIDIA_INFERENCE_API_KEY: handle, + NEMOCLAW_E2E_OPERATOR_TOKEN: handle, COMPATIBLE_ANTHROPIC_API_KEY: handle, COMPATIBLE_API_KEY: handle, }, @@ -402,18 +502,25 @@ function sandboxForOpenClawConfig(config: object, handle = scopedNativeHandle) { }; } -it("checks the actual switched native model budget with a materialized handle", async () => { - const model = PUBLIC_NVIDIA_SWITCH_MODEL; - const config: any = {}; - patchOpenClawInferenceConfig(config, "nvidia-prod", model, "openai-completions"); - config.models.providers.inference.apiKey = scopedNativeHandle; - await probeOpenClawConfig(sandboxForOpenClawConfig(config) as never, "/tmp/switch-home", { - model, - inferenceApi: "openai-completions", - artifactName: "switched-native-config", - nativeNvidia: true, - }); -}); +it.each([ + scopedNativeHandle, + "openshell:resolve:env:v2_NVIDIA_INFERENCE_API_KEY", + `openshell:resolve:env:s${"a".repeat(64)}_NVIDIA_INFERENCE_API_KEY`, +])( + "checks the native model budget with a scoped handle from any retained revision: %s", + async (handle) => { + const model = PUBLIC_NVIDIA_SWITCH_MODEL; + const config: any = {}; + patchOpenClawInferenceConfig(config, "nvidia-prod", model, "openai-completions"); + config.models.providers.inference.apiKey = handle; + await probeOpenClawConfig(sandboxForOpenClawConfig(config) as never, "/tmp/switch-home", { + model, + inferenceApi: "openai-completions", + artifactName: "switched-native-config", + nativeNvidia: true, + }); + }, +); it("checks the actual switched Anthropic model budget", async () => { const model = "mock-anthropic-model"; @@ -486,7 +593,7 @@ it.each([ "native", "${OTHER_API_KEY}", "${NVIDIA_INFERENCE_API_KEY}", - "openshell:resolve:env:v2_NVIDIA_INFERENCE_API_KEY", + "openshell:resolve:env:v2_OTHER_API_KEY", "openshell:resolve:env:NVIDIA_INFERENCE_API_KEY", null, ])("rejects an incorrect native OpenClaw credential value after redaction: %s", async (value) => { @@ -521,3 +628,119 @@ it.each(["", "nvapi-fixture-secret", "openshell:resolve:env:NVIDIA_INFERENCE_API ).rejects.toThrow(); }, ); + +it.each([ + "openshell:resolve:env:v7_NEMOCLAW_E2E_OPERATOR_TOKEN", + "raw-key", + "openshell:resolve:env:v7_OTHER_TOKEN", +])( + "reads registered OpenClaw configuration only with the matching scoped credential: %s", + async (handle) => { + const sandbox = sandboxForOpenClawConfig( + { models: { providers: { "operator-prod": { apiKey: handle } } } }, + handle, + ); + const operation = readOpenClawConfig(sandbox as never, "switch-test", { + artifactName: "registered-config", + env: {}, + timeoutMs: 1000, + registeredProvider: "operator-prod", + credentialEnv: "NEMOCLAW_E2E_OPERATOR_TOKEN", + }); + const outcome = await operation.then( + (result) => JSON.parse(result.stdout), + () => "rejected", + ); + expect(outcome).toEqual( + handle.startsWith("openshell:resolve:env:v7_NEMOCLAW_") + ? { models: { providers: { "operator-prod": {} } } } + : "rejected", + ); + }, +); + +it("reads the final curl status while preserving only the response body", () => { + const response = + '{"model":"selected"}\n__NEMOCLAW_HTTP_STATUS__=100\n__NEMOCLAW_HTTP_STATUS__=200'; + expect(httpStatusFromResponse(response)).toBe("200"); + expect(httpBodyFromResponse(response)).toBe('{"model":"selected"}'); + expect(httpStatusFromResponse("no status")).toBe(""); +}); + +function switchCleanupFixture() { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "switch-cleanup-")); + disposables.push(() => fs.rmSync(home, { recursive: true, force: true })); + fs.writeFileSync(path.join(home, "ownership-record"), "retained fixture state"); + const cleanupSandbox = vi.fn().mockResolvedValue(undefined); + const verifyGatewayRemoved = vi.fn().mockResolvedValue(undefined); + return { + home, + cleanupSandbox, + verifyGatewayRemoved, + input: { + host: { cleanupSandbox } as unknown as HostCliClient, + verifyGatewayRemoved, + home, + sandboxName: "switch-owner", + env: { HOME: home }, + retain: false, + }, + }; +} + +it("removes private state only after owned destroy and gateway absence, once", async () => { + const fixture = switchCleanupFixture(); + fixture.verifyGatewayRemoved.mockImplementation(async () => { + expect(fs.existsSync(path.join(fixture.home, "ownership-record"))).toBe(true); + }); + const cleanup = createOpenClawInferenceSwitchCleanup(fixture.input); + await cleanup(); + await cleanup(); + expect(fixture.cleanupSandbox).toHaveBeenCalledExactlyOnceWith("switch-owner", { + cleanupGateway: true, + artifactName: "cleanup-inference-switch-sandbox-and-gateway", + env: { HOME: fixture.home }, + timeoutMs: 120_000, + }); + expect(fixture.verifyGatewayRemoved).toHaveBeenCalledOnce(); + expect(fs.existsSync(fixture.home)).toBe(false); +}); + +it.each([ + "failed before registry publication", + "ownership record is incomplete", + "owned destroy failed after onboarding", +])("preserves partial-onboard state after %s without raw-name fallback", async (message) => { + const fixture = switchCleanupFixture(); + fixture.cleanupSandbox.mockRejectedValue(new Error(message)); + const cleanup = createOpenClawInferenceSwitchCleanup(fixture.input); + await expect(cleanup()).rejects.toThrow(message); + await expect(cleanup()).rejects.toThrow(message); + expect(fixture.cleanupSandbox).toHaveBeenCalledOnce(); + expect(fixture.verifyGatewayRemoved).not.toHaveBeenCalled(); + expect(fs.readFileSync(path.join(fixture.home, "ownership-record"), "utf8")).toBe( + "retained fixture state", + ); +}); + +it.each(["status remains connected", "listener still exists", "runtime observation failed"])( + "preserves private state when the gateway proof rejects: %s", + async (message) => { + const fixture = switchCleanupFixture(); + fixture.verifyGatewayRemoved.mockRejectedValue(new Error(message)); + const cleanup = createOpenClawInferenceSwitchCleanup(fixture.input); + await expect(cleanup()).rejects.toThrow(message); + await expect(cleanup()).rejects.toThrow(message); + expect(fixture.cleanupSandbox).toHaveBeenCalledOnce(); + expect(fixture.verifyGatewayRemoved).toHaveBeenCalledOnce(); + expect(fs.existsSync(path.join(fixture.home, "ownership-record"))).toBe(true); + }, +); + +it("retains the owned runtime and private state when sandbox retention is requested", async () => { + const fixture = switchCleanupFixture(); + await createOpenClawInferenceSwitchCleanup({ ...fixture.input, retain: true })(); + expect(fixture.cleanupSandbox).not.toHaveBeenCalled(); + expect(fixture.verifyGatewayRemoved).not.toHaveBeenCalled(); + expect(fs.existsSync(fixture.home)).toBe(true); +}); diff --git a/test/e2e/support/openclaw-stopped-recovery.test.ts b/test/e2e/support/openclaw-stopped-recovery.test.ts new file mode 100644 index 00000000000..3e7a84da6a9 --- /dev/null +++ b/test/e2e/support/openclaw-stopped-recovery.test.ts @@ -0,0 +1,188 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import type { ArtifactSink } from "../fixtures/artifacts.ts"; +import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; +import type { RuntimeProviderPrerequisite } from "../fixtures/runtime-provider.ts"; +import { proveStoppedDockerAgentRecovery } from "../live/openclaw-stopped-recovery.ts"; + +const NAME = "e2e-recovery"; +const ID = "source-id"; +const CONTAINER = "a".repeat(64); +const ROW = { + id: ID, + name: NAME, + labels: {}, + resource_version: 1, + created_at: "2026-10-19T00:00:00Z", + phase: "Ready", + current_policy_version: 1, +}; +const page = (rows = [ROW], token = "") => + JSON.stringify({ sandboxes: rows, next_page_token: token }); +const result = (stdout = "") => ({ exitCode: 0, stdout, stderr: "" }); + +function fixture( + initial = page(), + terminal = page([{ ...ROW, phase: "Error" }]), + agent: "openclaw" | "langchain-deepagents-code" = "openclaw", +) { + const openshell = vi + .fn() + .mockResolvedValueOnce(result(initial)) + .mockResolvedValue(result(terminal)); + const exec = vi + .fn() + .mockResolvedValueOnce(result()) + .mockResolvedValue(result("stopped-source-1000\nstopped-source-1000-unknown-root")); + const command = vi + .fn() + .mockResolvedValueOnce( + result( + JSON.stringify([ + CONTAINER, + { + "openshell.ai/managed-by": "openshell", + "openshell.ai/sandbox-name": NAME, + "openshell.ai/sandbox-id": ID, + }, + true, + ]), + ), + ) + .mockResolvedValue(result()); + const runtime = { + id: "docker", + command, + resolveSandboxResourceHandle: vi + .fn() + .mockResolvedValueOnce(CONTAINER) + .mockResolvedValue("b".repeat(64)), + }; + const rebuild = vi.fn().mockResolvedValue(undefined); + const writeJson = vi.fn(); + return { + exec, + command, + rebuild, + writeJson, + openshell, + run: () => + proveStoppedDockerAgentRecovery( + { openshell, exec } as unknown as SandboxClient, + runtime as unknown as RuntimeProviderPrerequisite, + { writeJson } as unknown as ArtifactSink, + NAME, + rebuild, + "native-readiness", + agent, + ), + }; +} + +describe("stopped Docker recovery inventory", () => { + beforeEach(() => { + vi.stubEnv("NEMOCLAW_EXPERIMENTAL_PROFILE", ""); + vi.stubEnv("E2E_TARGET_ID", "rebuild-openclaw"); + vi.spyOn(Date, "now").mockReturnValue(1000); + }); + + it.each([ + ["0.1.5 page", page()], + ["legacy array", JSON.stringify([ROW])], + ])("recovers the identified source from a complete %s", async (_name, inventory) => { + const f = fixture(inventory); + await f.run(); + expect(f.command.mock.calls.map(([args]) => args[0])).toEqual(["inspect", "kill"]); + expect(f.rebuild).toHaveBeenCalledOnce(); + expect(f.writeJson).toHaveBeenCalledWith( + "openclaw-stopped-source-recovery.json", + expect.objectContaining({ + sourceContainerId: CONTAINER, + replacementContainerId: "b".repeat(64), + sourcePhase: "Error", + workspacePreserved: true, + unknownNativeRootStatePreserved: true, + }), + ); + }); + + it.each([ + ["partial page", page([ROW], "next")], + ["missing page token", JSON.stringify({ sandboxes: [ROW] })], + ["invalid JSON", "{"], + ["missing sandbox", page([])], + ["ambiguous identity", page([ROW, ROW])], + ["invalid ID", page([{ ...ROW, id: "unsafe/id" }])], + ["not ready", page([{ ...ROW, phase: "Error" }])], + ])("rejects %s before container inspection or mutation", async (_name, inventory) => { + const f = fixture(inventory); + await expect(f.run()).rejects.toThrow(); + expect(f.command).not.toHaveBeenCalled(); + expect(f.rebuild).not.toHaveBeenCalled(); + }); + + it("rejects a changed source ID before rebuild", async () => { + const f = fixture(page(), page([{ ...ROW, id: "replacement-id", phase: "Error" }])); + await expect(f.run()).rejects.toThrow("The source identity changed"); + expect(f.rebuild).not.toHaveBeenCalled(); + }); + + it("prepares Deep Agents recovery state as its owner without chown", async () => { + vi.stubEnv("OPENSHELL_GATEWAY", "recovery-gateway"); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "stopped-dcode-")); + const home = path.join(directory, "sandbox"); + const bin = path.join(directory, "bin"); + try { + fs.mkdirSync(path.join(home, ".deepagents", ".state"), { recursive: true, mode: 0o700 }); + fs.mkdirSync(bin); + fs.writeFileSync(path.join(home, ".bash_profile"), "existing user profile"); + fs.writeFileSync(path.join(bin, "chown"), "#!/bin/sh\nexit 99\n", { mode: 0o700 }); + fs.writeFileSync(path.join(bin, "sync"), "#!/bin/sh\nexit 0\n", { mode: 0o700 }); + const f = fixture(undefined, undefined, "langchain-deepagents-code"); + f.exec.mockReset().mockImplementation(async (_name, args) => { + const prepared = spawnSync(args[0], [args[1], args[2].replaceAll("/sandbox", home)], { + env: { PATH: `${bin}:/usr/bin:/bin` }, + encoding: "utf8", + }); + return { exitCode: prepared.status, stdout: prepared.stdout, stderr: prepared.stderr }; + }); + await f.run(); + const marker = path.join(home, ".deepagents", ".state", ".stopped-recovery-marker"); + expect(fs.readFileSync(marker, "utf8")).toBe("stopped-source-1000"); + expect(fs.statSync(marker).uid).toBe(process.getuid!()); + expect( + fs.readFileSync(path.join(home, ".stopped-recovery-unknown-root-marker"), "utf8"), + ).toBe("stopped-source-1000-unknown-root"); + expect(fs.readlinkSync(path.join(home, ".bash_profile"))).toBe( + path.join(home, "hostile-env.sh"), + ); + expect(f.command.mock.calls.map(([args]) => args[0])).toEqual(["inspect", "kill"]); + expect(f.exec).toHaveBeenNthCalledWith( + 1, + NAME, + expect.any(Array), + expect.objectContaining({ + timeoutMs: 10_000, + env: expect.objectContaining({ OPENSHELL_GATEWAY: "recovery-gateway" }), + }), + ); + expect(f.rebuild).toHaveBeenCalledOnce(); + expect(f.writeJson).toHaveBeenCalledWith( + "deepagents-stopped-source-recovery.json", + expect.objectContaining({ + sourceProfile: "broken-symlink", + agentStatePreserved: true, + unknownNativeRootStatePreserved: true, + }), + ); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); +}); diff --git a/test/e2e/support/openshell-exact-main-child-contracts.test.ts b/test/e2e/support/openshell-exact-main-child-contracts.test.ts index 303b3ff94d7..2116d4ccd3c 100644 --- a/test/e2e/support/openshell-exact-main-child-contracts.test.ts +++ b/test/e2e/support/openshell-exact-main-child-contracts.test.ts @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { spawnSync } from "node:child_process"; -import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -115,6 +115,52 @@ describe("OpenShell exact-main child contracts", () => { } }); + it.each([ + { capBnd: "0000000000000000", tls: false, named: true, success: true }, + { capBnd: "0000000000000001", tls: false, named: true, success: false }, + { capBnd: "0000000000000000", tls: true, named: true, success: false }, + { capBnd: "0000000000000000", tls: false, named: false, success: false }, + ])("reads only the selected container's entrypoint: %j", ({ capBnd, tls, named, success }) => { + const directory = mkdtempSync(path.join(tmpdir(), "nemoclaw-host-proc-")); + try { + const proc = path.join(directory, "42"); + mkdirSync(proc); + writeFileSync( + path.join(proc, "cmdline"), + `${named ? "nemoclaw-dcode-entrypoint" : "unrelated"}\0-f\0/dev/null\0`, + ); + writeFileSync(path.join(proc, "status"), `CapBnd:\t${capBnd}\n`); + writeFileSync( + path.join(proc, "environ"), + tls ? "OPENSHELL_TLS_KEY=fixture-secret\0" : "PATH=/usr/bin\0", + ); + const source = ENTRYPOINT_CHILD_PROBE.replace( + 'Path("/proc")', + `Path(${JSON.stringify(directory)})`, + ); + const harness = [ + "import sys", + "from types import SimpleNamespace", + "from unittest.mock import patch", + "source, container = sys.argv[1:]", + "sys.argv = ['entrypoint-proof', container]", + "with patch('subprocess.run', return_value=SimpleNamespace(stdout='PID\\n42\\n')) as top:", + " exec(source)", + " top.assert_called_once_with(['docker', 'top', container, '-eo', 'pid'], check=True, capture_output=True, text=True)", + ].join("\n"); + const result = spawnSync("python3", ["-c", harness, source, CONTAINER_ID], { + encoding: "utf8", + }); + expect(result.status === 0, result.stderr).toBe(success); + expect(result.stdout + result.stderr).not.toContain("fixture-secret"); + expect(JSON.parse(result.stdout || "null")).toEqual( + success ? { capBnd, surface: "entrypoint", supervisorTlsEnvNames: [] } : null, + ); + } finally { + rmSync(directory, { recursive: true, force: true }); + } + }); + it("proves entrypoint, exec, and NemoClaw connect children", async () => { const { command, host, nemoclaw } = fakeHost(); command @@ -136,12 +182,19 @@ describe("OpenShell exact-main child contracts", () => { "label=openshell.ai/managed-by=openshell", "--filter", `label=openshell.ai/sandbox-name=${SANDBOX_NAME}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", "--format", '{{.ID}}\t{{.Names}}\t{{.Label "openshell.ai/sandbox-id"}}\t{{.Label "openshell.ai/sandbox-workspace"}}', ]); - expect(command.mock.calls[1]?.[1]).toEqual( - expect.arrayContaining(["exec", "--user", "0", CONTAINER_ID, "python3", "-c"]), - ); + expect(command.mock.calls[1]?.[1]).toEqual([ + "-n", + "python3", + "-c", + ENTRYPOINT_CHILD_PROBE, + CONTAINER_ID, + ]); + expect(command.mock.calls[1]?.[0]).toBe("sudo"); expect(nemoclaw.mock.calls[0]?.[0]).toEqual( expect.arrayContaining([SANDBOX_NAME, "exec", "--", "python3", "-c"]), ); diff --git a/test/e2e/support/openshell-exact-main-driver-config.test.ts b/test/e2e/support/openshell-exact-main-driver-config.test.ts index 648eb28d4e5..1c794d59e6c 100644 --- a/test/e2e/support/openshell-exact-main-driver-config.test.ts +++ b/test/e2e/support/openshell-exact-main-driver-config.test.ts @@ -7,14 +7,19 @@ import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it, vi } from "vitest"; +import * as gatewayService from "../../../src/lib/onboard/docker-driver-gateway-service.ts"; import { resolveOpenShellSiblingComponents } from "../../helpers/openshell-components.ts"; import { createOpenShellDriverConfigTestWrapper } from "../live/openshell-driver-config-test-wrapper.ts"; import { + assertGatewayListeners, + assertRuntimeMounts, + requireRunningSandboxContainer, EXACT_MAIN_DRIVER_CONFIG_JSON, EXACT_MAIN_DRIVER_CONFIG_PROOF_ENV, EXACT_MAIN_TMPFS_MOUNT, prepareExactMainDriverConfigProof, + readGatewayPid, } from "../live/openshell-exact-main-driver-config.ts"; const originalProofEnv = process.env[EXACT_MAIN_DRIVER_CONFIG_PROOF_ENV]; @@ -28,6 +33,209 @@ afterEach(() => { }); describe("OpenShell driver configuration for main-branch E2E", () => { + it.each(["service", "standalone"])("verifies the active %s gateway PID", (manager) => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "gateway-pid-")); + vi.stubEnv("NEMOCLAW_OPENSHELL_GATEWAY_STATE_DIR", directory); + vi.spyOn(gatewayService, "getTrustedActiveOpenShellGatewayUserServicePid").mockReturnValue( + manager === "service" ? 12345 : null, + ); + const kill = vi.spyOn(process, "kill").mockReturnValue(true); + fs.writeFileSync(path.join(directory, "openshell-gateway.pid"), "23456\n"); + try { + const expected = manager === "service" ? 12345 : 23456; + expect(readGatewayPid()).toBe(expected); + expect(kill).toHaveBeenCalledWith(expected, 0); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); + + it("rejects a missing PID file when no trusted gateway service exists", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "gateway-pid-")); + vi.stubEnv("NEMOCLAW_OPENSHELL_GATEWAY_STATE_DIR", directory); + vi.spyOn(gatewayService, "getTrustedActiveOpenShellGatewayUserServicePid").mockReturnValue( + null, + ); + const kill = vi.spyOn(process, "kill").mockReturnValue(true); + try { + expect(() => readGatewayPid()).toThrow(/ENOENT/); + expect(kill).not.toHaveBeenCalled(); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); + + it.each(["sandbox", "supervisor"] as const)("selects exactly one scoped %s", async (role) => { + const command = vi.fn(async (_command: string, _args: string[]) => ({ + exitCode: 0, + stdout: "container-id\n", + stderr: "", + })); + await expect( + requireRunningSandboxContainer({ command } as never, "candidate", "gateway-id", role, "test"), + ).resolves.toBe("container-id"); + expect(command.mock.calls[0]?.[1]).toEqual([ + "ps", + "--no-trunc", + "--filter", + "label=openshell.ai/sandbox-name=candidate", + "--filter", + "label=openshell.ai/managed-by=openshell", + "--filter", + "label=openshell.ai/sandbox-namespace=gateway-id", + "--filter", + "label=openshell.ai/isolation-role=" + role, + "--format", + "{{.ID}}", + ]); + }); + + it.each(["", "one\ntwo\n"])("rejects a missing or ambiguous role: %j", async (stdout) => { + const command = vi.fn(async () => ({ exitCode: 0, stdout, stderr: "" })); + await expect( + requireRunningSandboxContainer( + { command } as never, + "candidate", + "gateway-id", + "sandbox", + "test", + ), + ).rejects.toThrow(); + }); + + function splitRuntimeFixture(fault: string) { + const hash = "a".repeat(64); + const tmpfs = { Destination: "/run/nemoclaw-dcode-mcp", Type: "tmpfs", RW: true }; + const channel = { + Destination: "/.openshell/channel", + Type: "volume", + RW: true, + Source: "/volumes/channel", + }; + const state = { + Destination: "/.openshell/supervisor", + Type: "volume", + RW: false, + Source: "/volumes/auth", + }; + const command = vi.fn(async (_command: string, args: string[]) => { + const respond: Record { exitCode: number; stdout: string; stderr: string }> = { + exec: () => { + expect(args).toEqual([ + "exec", + "--user", + "0", + "workload", + "sha256sum", + "/.openshell/runtime/openshell-sandbox", + ]); + return { + exitCode: 0, + stdout: + (fault === "binary" ? "b".repeat(64) : hash) + + " /.openshell/runtime/openshell-sandbox\n", + stderr: "", + }; + }, + inspect: () => { + const supervisor = args[3] === "supervisor"; + let value: unknown; + switch (args[2]) { + case "{{json .HostConfig.Mounts}}": + value = [ + { + Type: "tmpfs", + Target: tmpfs.Destination, + TmpfsOptions: { Mode: 0o1777, SizeBytes: 1_048_576 }, + }, + ]; + break; + case "{{json .HostConfig.Tmpfs}}": + value = fault === "legacy tmpfs" ? { [tmpfs.Destination]: "rw" } : null; + break; + case "{{json .Mounts}}": + value = supervisor + ? [ + { + ...channel, + RW: fault === "writable supervisor", + Source: fault === "channel" ? "/volumes/other" : channel.Source, + }, + state, + ] + : [tmpfs, channel, ...(fault === "credential exposure" ? [state] : [])]; + break; + case "{{json .HostConfig.Binds}}": + value = []; + break; + case "{{json .NetworkSettings.Networks}}": + value = fault === "network attachment" ? { bridge: {} } : { none: {} }; + break; + case "{{json .HostConfig.NetworkMode}}": + value = supervisor ? "host" : fault === "network mode" ? "bridge" : "none"; + break; + case "{{json .Config.Labels}}": + value = { + "openshell.ai/sandbox-id": supervisor && fault === "identity" ? "other" : "same-id", + }; + break; + default: + throw new Error("Unexpected inspection " + args[2]); + } + return { exitCode: 0, stdout: JSON.stringify(value), stderr: "" }; + }, + }; + return respond[args[0]!]!(); + }); + return { + host: { command }, + proof: { provenance: { artifacts: { standaloneSandbox: { binarySha256: hash } } } }, + }; + } + + it("accepts the expected split-runtime mount and network layout", async () => { + const { host, proof } = splitRuntimeFixture(""); + await expect( + assertRuntimeMounts(host as never, proof as never, "workload", "supervisor", "test"), + ).resolves.toBeUndefined(); + }); + + it.each([ + "binary", + "channel", + "writable supervisor", + "credential exposure", + "network attachment", + "network mode", + "identity", + "legacy tmpfs", + ])("rejects split-runtime %s mismatch", async (fault) => { + const { host, proof } = splitRuntimeFixture(fault); + await expect( + assertRuntimeMounts(host as never, proof as never, "workload", "supervisor", "test"), + ).rejects.toThrow(); + }); + + function observeGatewayListener(address: string, pid: number) { + const command = vi.fn(async () => ({ + exitCode: 0, + stdout: " LISTEN 0 4096 " + address + " 0.0.0.0:* users:((gateway,pid=" + pid + ",fd=3))\n", + stderr: "", + })); + return assertGatewayListeners({ command } as never, 42, { port: 8080 } as never, "test"); + } + + it("accepts the gateway's loopback listener", async () => { + await expect(observeGatewayListener("127.0.0.1:8080", 42)).resolves.toBeUndefined(); + }); + + it.each([ + ["0.0.0.0:8080", 42], + ["[::]:8080", 42], + ["127.0.0.1:8080", 43], + ] as const)("rejects gateway listener %s owned by PID %i", async (address, pid) => { + await expect(observeGatewayListener(address, pid)).rejects.toThrow(); + }); it("resolves one canonical executable set for CLI, gateway, and sandbox (#11547)", () => { const rootDirectory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-components-")); const installDirectory = path.join(rootDirectory, "install"); diff --git a/test/e2e/support/openshell-exact-main-runtime-contracts.test.ts b/test/e2e/support/openshell-exact-main-runtime-contracts.test.ts index 0857965c28f..bb15eeb62f4 100644 --- a/test/e2e/support/openshell-exact-main-runtime-contracts.test.ts +++ b/test/e2e/support/openshell-exact-main-runtime-contracts.test.ts @@ -1,108 +1,99 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { spawnSync } from "node:child_process"; +import { execFile, spawnSync } from "node:child_process"; + +import { promisify } from "node:util"; +import { createConnection } from "node:net"; import { describe, expect, it, vi } from "vitest"; import * as policy from "../../../src/lib/policy"; import { - assertExactMainPolicyNftAndIdentityContracts, + assertExactMainPolicyAndIdentityContracts, buildExactMainLiveExeIdentityScript, - buildExactMainNftInspectionScript, DIRECT_BYPASS_PROBE_CODE, - inspectExactMainNftRuleset, + inspectExactMainIsolation, parseExactMainPolicyStatus, + waitForRestartedPolicy, } from "../live/openshell-exact-main-runtime-contracts.ts"; -function match(left: Record, right: string | number): Record { - return { match: { left, op: "==", right } }; -} - -function rejectRule(family: "ipv4" | "ipv6", protocol: "tcp" | "udp") { +function completeIsolation(): Record { return { - rule: { - chain: "output", - expr: [ - match({ meta: { key: "l4proto" } }, protocol), - { - reject: { - expr: "port-unreachable", - type: family === "ipv4" ? "icmp" : "icmpv6", - }, - }, - ], - family: "inet", - table: "openshell_bypass", - }, + Id: "a".repeat(64), + HostConfig: { NetworkMode: "none", CapDrop: ["ALL"], CapAdd: null, Privileged: false }, }; } -function completeRuleset(): Record { - return { - nftables: [ - { metainfo: { json_schema_version: 1 } }, - { table: { family: "inet", name: "openshell_bypass" } }, - { - chain: { - family: "inet", - hook: "output", - name: "output", - policy: "accept", - prio: 0, - table: "openshell_bypass", - type: "filter", - }, - }, - { - rule: { - chain: "output", - expr: [ - match({ payload: { field: "daddr", protocol: "ip" } }, "10.200.0.1"), - match({ payload: { field: "dport", protocol: "tcp" } }, 3128), - { accept: null }, - ], - family: "inet", - table: "openshell_bypass", - }, - }, - { - rule: { - chain: "output", - expr: [match({ meta: { key: "oifname" } }, "lo"), { accept: null }], - family: "inet", - table: "openshell_bypass", - }, - }, - rejectRule("ipv4", "tcp"), - rejectRule("ipv6", "tcp"), - rejectRule("ipv4", "udp"), - rejectRule("ipv6", "udp"), - ], - }; -} +describe("OpenShell exact-main policy, native isolation, and process-identity proof helpers", () => { + it.each([ + { label: "composed effective policy", revisionHash: "base", effectiveHash: "changed" }, + { label: "stored base revision", revisionHash: "changed", effectiveHash: "composed" }, + ])("waits for the same $label hash after restart", async ({ revisionHash, effectiveHash }) => { + vi.useFakeTimers(); + const status = (hash: string, state: string) => ({ + activeVersion: 4, + hash, + sandbox: "exact-main-proof", + status: state, + version: 4, + }); + const effective = status("composed", "effective"); + const revision = status("base", "loaded"); + const result = (value: ReturnType) => ({ + exitCode: 0, + stdout: JSON.stringify({ ...value, active_version: value.activeVersion }), + stderr: "", + timedOut: false, + signal: null, + }); + const openshell = vi + .fn() + .mockResolvedValueOnce(result(status(revisionHash, "loaded"))) + .mockResolvedValueOnce(result(status(effectiveHash, "effective"))) + .mockResolvedValueOnce(result(revision)) + .mockResolvedValueOnce(result(effective)); + try { + const pending = waitForRestartedPolicy( + { openshell } as never, + "exact-main-proof", + effective, + revision, + ); + await vi.advanceTimersByTimeAsync(1_000); + await expect(pending).resolves.toEqual({ effective, revision }); + expect(openshell).toHaveBeenCalledTimes(4); + expect(openshell.mock.calls.map(([args]) => args)).toEqual([ + ["policy", "get", "--rev", "4", "exact-main-proof", "--output", "json"], + ["policy", "get", "exact-main-proof", "--output", "json"], + ["policy", "get", "--rev", "4", "exact-main-proof", "--output", "json"], + ["policy", "get", "exact-main-proof", "--output", "json"], + ]); + } finally { + vi.useRealTimers(); + } + }); -describe("OpenShell exact-main policy, nft, and process-identity proof helpers", () => { it.each([ { label: "stop fails", failure: "stop", operations: ["stop"], - dockerOperations: ["ps", "exec"], + dockerOperations: ["ps", "inspect"], message: "stop exact-main OpenShell sandbox container failed", }, { label: "start fails", failure: "start", operations: ["stop", "start"], - dockerOperations: ["ps", "exec"], + dockerOperations: ["ps", "inspect"], message: "start exact-main OpenShell sandbox container failed", }, { label: "restart completes", failure: null, operations: ["stop", "start"], - dockerOperations: ["ps", "exec", "ps"], + dockerOperations: ["ps", "inspect", "ps"], message: "inspect restarted container", }, ])( @@ -125,7 +116,7 @@ describe("OpenShell exact-main policy, nft, and process-identity proof helpers", JSON.stringify({ active_version: version, config_revision: version, - hash: `sha256:policy-${version}`, + hash: `sha256:${state}-policy-${version}`, policy_source: "sandbox", sandbox: "exact-main-proof", status: state, @@ -147,13 +138,17 @@ describe("OpenShell exact-main policy, nft, and process-identity proof helpers", ].forEach((result) => openshell.mockResolvedValueOnce(result)); const command = vi .fn() - .mockResolvedValueOnce(success("a".repeat(64))) - .mockResolvedValueOnce(success(JSON.stringify(completeRuleset()))) + .mockResolvedValueOnce( + success( + `${"a".repeat(64)}\topenshell-default--exact-main-proof-proofid\tproofid\tdefault`, + ), + ) + .mockResolvedValueOnce(success(JSON.stringify(completeIsolation()))) .mockRejectedValueOnce(new Error("inspect restarted container")); try { await expect( - assertExactMainPolicyNftAndIdentityContracts({ + assertExactMainPolicyAndIdentityContracts({ artifacts: {} as never, cleanup: { add: vi.fn() } as never, host: { command } as never, @@ -225,7 +220,7 @@ describe("OpenShell exact-main policy, nft, and process-identity proof helpers", try { await expect( - assertExactMainPolicyNftAndIdentityContracts({ + assertExactMainPolicyAndIdentityContracts({ artifacts: {} as never, cleanup: { add: vi.fn() } as never, host: {} as never, @@ -248,6 +243,138 @@ describe("OpenShell exact-main policy, nft, and process-identity proof helpers", expect(openshellCalls.some((args) => args[0] === "policy" && args[1] === "set")).toBe(false); }); + it("closes owned TCP/UDP listeners and restores policy after the complete native proof", async () => { + vi.stubEnv("NEMOCLAW_OPENSHELL_EXACT_MAIN_PROOF", "1"); + const base = "version: 1\nnetwork_policies: {}"; + const result = (stdout: string) => ({ + artifacts: { result: "", stderr: "", stdout: "" }, + command: [], + exitCode: 0, + signal: null, + stderr: "", + stdout, + timedOut: false, + }); + const status = (version: number, state: string) => + result( + JSON.stringify({ + active_version: version, + hash: `policy-${version}`, + policy_source: "sandbox", + sandbox: "exact-main-proof", + status: state, + version, + }), + ); + let port = 0; + const mutations = vi + .spyOn(policy, "setPolicyDocument") + .mockImplementation(async (_name, document) => { + port = Number(document.match(/port: (\d+)/u)?.[1] ?? port); + return true; + }); + const openshell = vi.fn(); + [ + result(base), + status(1, "effective"), + status(2, "effective"), + status(2, "loaded"), + result(""), + result(""), + status(2, "loaded"), + status(2, "effective"), + result("Acknowledged initial policy revision as loaded [version:2]"), + result(base), + ].forEach((value) => openshell.mockResolvedValueOnce(value)); + const command = vi.fn(async (name: string, args: string[]) => { + const handlers: Record Promise>> = { + python3: async () => ({ + ...result(""), + ...(await promisify(execFile)(name, args, { timeout: 5_000 })), + }), + inspect: async () => result(JSON.stringify(completeIsolation())), + ps: async () => + result( + `${"a".repeat(64)}\topenshell-default--exact-main-proof-proofid\tproofid\tdefault`, + ), + }; + return await handlers[name === "python3" ? name : args[0]](); + }); + const identityConnection = () => + new Promise((resolve, reject) => { + const socket = createConnection({ host: "127.0.0.1", port }, () => + socket.write("identity-control"), + ); + socket.on("error", reject); + socket.on("data", () => socket.end()); + socket.on("close", () => resolve()); + }); + const execShell = vi.fn(async () => { + await identityConnection(); + await identityConnection(); + return result( + "NEMOCLAW_LIVE_EXE_OLD_FIRST=allowed\nNEMOCLAW_LIVE_EXE_OLD_SECOND=allowed\nNEMOCLAW_LIVE_EXE_NEW=EACCES\nNEMOCLAW_LIVE_EXE_OLD_LINK=/tmp/live-python (deleted)\n", + ); + }); + const cleanup: Array<() => Promise> = []; + const artifacts = { writeJson: vi.fn() }; + try { + await assertExactMainPolicyAndIdentityContracts({ + artifacts: artifacts as never, + cleanup: { + add: (_label: string, action: () => Promise) => cleanup.push(action), + } as never, + host: { command } as never, + sandbox: { + openshell, + execShell, + exec: vi.fn(async () => + result( + JSON.stringify({ + tcp: { connected: false, error: { errno: 13 } }, + udp: { echoed: false, error: { errno: 13 } }, + }), + ), + ), + } as never, + sandboxName: "exact-main-proof", + mcpUrl: "https://mcp.example.test/mcp", + }); + expect(command.mock.calls.filter(([name]) => name === "python3")).toHaveLength(2); + expect(mutations).toHaveBeenLastCalledWith( + "exact-main-proof", + base, + expect.objectContaining({ operation: "restore the exact-main policy proof" }), + ); + expect(artifacts.writeJson).toHaveBeenCalledWith( + "exact-main-policy-native-identity-summary.json", + expect.objectContaining({ + isolationAfterRestart: expect.objectContaining({ networkMode: "none" }), + }), + ); + await expect( + new Promise((resolve, reject) => { + const socket = createConnection({ host: "127.0.0.1", port }, () => { + socket.destroy(); + resolve(); + }); + socket.on("error", reject); + }), + ).rejects.toMatchObject({ code: "ECONNREFUSED" }); + await cleanup.reduce(async (previous, action) => { + await previous; + await action(); + }, Promise.resolve()); + } finally { + await cleanup.reduce(async (previous, action) => { + await previous; + await action(); + }, Promise.resolve()); + mutations.mockRestore(); + vi.unstubAllEnvs(); + } + }); + it("parses effective and stored revision identities without discarding version or hash", () => { expect( parseExactMainPolicyStatus( @@ -291,44 +418,62 @@ describe("OpenShell exact-main policy, nft, and process-identity proof helpers", expect(status.configRevision).toBe("7692118364955054884"); }); - it("requires the policy-accept chain, proxy/loopback accepts, and all four L4 rejects", () => { - expect(inspectExactMainNftRuleset(JSON.stringify(completeRuleset()))).toEqual({ - chainPolicy: "accept", - loopbackAcceptIndex: 1, - proxyAcceptIndex: 0, - rejectIndexes: { ipv4Tcp: 2, ipv4Udp: 4, ipv6Tcp: 3, ipv6Udp: 5 }, - ruleCount: 6, + it("requires the native workload network and capability boundary", () => { + expect( + inspectExactMainIsolation(JSON.stringify([completeIsolation()]), "a".repeat(64)), + ).toEqual({ + containerId: "a".repeat(64), + networkMode: "none", + capDrop: ["ALL"], + privileged: false, }); }); - it("inspects nft rules in either supported OpenShell network topology", () => { - const script = buildExactMainNftInspectionScript(); - const syntax = spawnSync("sh", ["-n"], { encoding: "utf8", input: script }); - - expect(syntax.status, syntax.stderr).toBe(0); - expect(script).toContain("/usr/sbin/nft /sbin/nft /usr/bin/nft"); - expect(script).toContain("nft executable not found"); - expect(script).toContain("/var/run/netns/sandbox-*"); - expect(script).toContain('ip netns pids "$namespace" 2>/dev/null'); - expect(script).toContain('nsenter --net="$namespace_path" -- "$nft_path"'); - expect(script).toContain('exec "$nft_path" -j list table inet openshell_bypass'); - expect(script).toContain('if [ "$active_namespace_count" -gt 1 ]'); + it.each([ + { NetworkMode: "host" }, + { CapDrop: [] }, + { CapAdd: ["NET_ADMIN"] }, + { Privileged: true }, + ])("rejects a weakened native isolation boundary [%j]", (changed) => { + const value = completeIsolation(); + value.HostConfig = { ...(value.HostConfig as object), ...changed }; + expect(() => inspectExactMainIsolation(JSON.stringify([value]), "a".repeat(64))).toThrow( + "native network/capability isolation", + ); }); - it("accepts immediate refused and permission-denied bypass outcomes", () => { - expect(DIRECT_BYPASS_PROBE_CODE).toContain("denial_errnos = {errno.ECONNREFUSED, errno.EPERM}"); - expect(DIRECT_BYPASS_PROBE_CODE).toContain( - 'tcp_error.get("errno") not in denial_errnos or udp_error.get("errno") not in denial_errnos', - ); + it.each([ + [], + [completeIsolation(), completeIsolation()], + [{ ...completeIsolation(), Id: "b".repeat(64) }], + ])("rejects missing, ambiguous or replaced workload identity [%j]", (...value) => { + expect(() => inspectExactMainIsolation(JSON.stringify(value), "a".repeat(64))).toThrow(); }); - it("rejects the feasible sequential-install partial state instead of calling it green", () => { - const partial = completeRuleset(); - const entries = partial.nftables as Array>; - entries.pop(); - expect(() => inspectExactMainNftRuleset(JSON.stringify(partial))).toThrow( - "expected exactly one ipv6 udp reject rule, got 0", - ); + it.each([ + { name: "broker EACCES", error: "EACCES", status: 0 }, + { name: "unreachable listener", error: "ECONNREFUSED", status: 1 }, + { name: "generic permission failure", error: "EPERM", status: 1 }, + { name: "network timeout", error: "ETIMEDOUT", status: 1 }, + ])("executes the denial probe and rejects unrelated failures [$name]", ({ error, status }) => { + const shim = `import socket, errno, sys +class DeniedSocket: + def settimeout(self, value): pass + def connect(self, address): raise OSError(errno.${error}, "controlled failure") + def close(self): pass +def denied(*args, **kwargs): raise OSError(errno.${error}, "controlled failure") +socket.create_connection = denied +socket.socket = lambda *args, **kwargs: DeniedSocket() +sys.argv = ["probe", "host.openshell.internal", "12345"] +`; + const result = spawnSync("python3", ["-c", shim + DIRECT_BYPASS_PROBE_CODE], { + encoding: "utf8", + }); + expect(result.status, result.stderr).toBe(status); + expect(JSON.parse(result.stdout)).toMatchObject({ + tcp: { connected: false }, + udp: { echoed: false }, + }); }); it("keeps the old live inode and new altered-path denial in one process-level probe", () => { @@ -340,11 +485,11 @@ describe("OpenShell exact-main policy, nft, and process-identity proof helpers", expect(script).toContain("NEMOCLAW_LIVE_EXE_OLD_FIRST=%s"); expect(script).toContain("NEMOCLAW_LIVE_EXE_OLD_SECOND=%s"); expect(script).toContain("NEMOCLAW_LIVE_EXE_NEW=%s"); - expect(script).toContain('while [ "$allowed_attempt" -lt 50 ]'); - expect(script).toContain('[ "$allowed_status" = 200 ] && break'); - expect(script).toContain('connect_until_allowed "$first_result"'); - expect(script).toContain('connect_until_allowed "$second_result"'); - expect(script).toContain("expected initial live-exe CONNECT 200"); - expect(script).toContain("expected replacement live-exe CONNECT 403"); + expect(script).toContain("for _ in range(50):"); + expect(script).toContain('if status == "allowed":'); + expect(script).toContain("publish(sys.argv[3])"); + expect(script).toContain("publish(sys.argv[4])"); + expect(script).toContain("expected initial live-exe broker connection allowed"); + expect(script).toContain("expected replacement live-exe broker EACCES"); }); }); diff --git a/test/e2e/support/openshell-exact-main-snapshot-residue.test.ts b/test/e2e/support/openshell-exact-main-snapshot-residue.test.ts new file mode 100644 index 00000000000..d0c88ecc675 --- /dev/null +++ b/test/e2e/support/openshell-exact-main-snapshot-residue.test.ts @@ -0,0 +1,71 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { DCODE_MCP_SNAPSHOT_RESIDUE_PROBE } from "../live/openshell-exact-main-mcp-proof.ts"; + +describe("Deep Agents snapshot residue verification", () => { + let root: string; + let directory: string; + + beforeEach(() => { + root = mkdtempSync(path.join(tmpdir(), "nemoclaw-snapshot-residue-")); + directory = path.join(root, "snapshot"); + }); + + afterEach(() => { + rmSync(root, { recursive: true, force: true }); + }); + + function runProbe() { + const source = DCODE_MCP_SNAPSHOT_RESIDUE_PROBE.replace( + 'Path("/run/nemoclaw-dcode-mcp")', + `Path(${JSON.stringify(directory)})`, + ); + return spawnSync("python3", ["-I", "-c", source], { encoding: "utf8" }); + } + + it("accepts an empty directory", () => { + mkdirSync(directory); + const result = runProbe(); + expect(result.status, result.stderr).toBe(0); + }); + + it("rejects a directory with a leftover file", () => { + mkdirSync(directory); + writeFileSync(path.join(directory, "leftover"), "proof"); + const result = runProbe(); + expect(result.status, result.stderr).toBe(1); + }); + + it("rejects a missing directory", () => { + const result = runProbe(); + expect(result.status, result.stderr).toBe(1); + }); + + it("rejects a regular file in place of the directory", () => { + writeFileSync(directory, "proof"); + const result = runProbe(); + expect(result.status, result.stderr).toBe(1); + }); + + it("rejects a directory read error", () => { + const harness = [ + "import sys", + "from unittest.mock import patch", + "with patch('pathlib.Path.iterdir', side_effect=PermissionError('directory read denied')):", + " exec(sys.argv[1])", + ].join("\n"); + const result = spawnSync("python3", ["-I", "-c", harness, DCODE_MCP_SNAPSHOT_RESIDUE_PROBE], { + encoding: "utf8", + }); + expect(result.status, result.stderr).toBe(1); + expect(result.stderr).toContain("PermissionError: directory read denied"); + }); +}); diff --git a/test/e2e/support/openshell-gateway-auth-sandbox.test.ts b/test/e2e/support/openshell-gateway-auth-sandbox.test.ts new file mode 100644 index 00000000000..c74adc459d9 --- /dev/null +++ b/test/e2e/support/openshell-gateway-auth-sandbox.test.ts @@ -0,0 +1,68 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { expect, it, vi } from "vitest"; +import { createGatewayAuthSandbox } from "../live/openshell-gateway-auth-source-contract-helpers.ts"; + +const { spawn } = vi.hoisted(() => ({ spawn: vi.fn() })); +vi.mock("node:child_process", async (importOriginal) => ({ + ...(await importOriginal()), + spawnSync: spawn, +})); + +it("uses the gateway-issued launch token and deletes its exact fixture sandbox", () => { + const stateDir = fs.mkdtempSync(path.join(os.tmpdir(), "auth-sandbox-test-")); + const id = "11111111-2222-3333-4444-555555555555"; + const cleanup = { add: vi.fn() }; + try { + const configPath = path.join(stateDir, "gateway.toml"); + fs.writeFileSync(configPath, 'sandbox_label = "fixture"\n'); + const bundleDir = path.join( + stateDir, + "runtime-state", + "openshell", + "docker-sandbox-tokens", + "fixture", + id, + ); + fs.mkdirSync(bundleDir, { recursive: true }); + fs.writeFileSync( + path.join(bundleDir, "supervisor-auth.json"), + JSON.stringify({ gateway_token: "fixture-issued-session" }), + ); + spawn.mockReturnValue({ + status: 0, + stdout: JSON.stringify({ id, name: "auth-contract", phase: "Ready" }), + }); + expect(createGatewayAuthSandbox({ cleanup, configPath, port: 12345, stateDir })).toEqual({ + name: "auth-contract", + token: "fixture-issued-session", + }); + const invocation = spawn.mock.calls[1]!; + expect(invocation[2].env.XDG_STATE_HOME).toBe(path.join(stateDir, "runtime-state")); + expect(invocation[1]).toContain("--no-auto-providers"); + cleanup.add.mock.calls[0]![1](); + expect(spawn.mock.lastCall?.[1]).toEqual(["sandbox", "delete", "auth-contract"]); + } finally { + fs.rmSync(stateDir, { recursive: true, force: true }); + } +}); + +it("does not include sensitive command output in a fixture failure", () => { + spawn.mockReturnValue({ + status: 1, + stdout: "secret-output", + stderr: "invalid token nvapi-testSecret000000", + }); + expect(() => + createGatewayAuthSandbox({ + cleanup: { add: vi.fn() }, + configPath: "/unused", + port: 12345, + stateDir: "/unused", + }), + ).toThrow("OpenShell auth fixture gateway add failed (1): invalid token "); +}); diff --git a/test/e2e/support/openshell-gateway-auth-source-contract-helpers.test.ts b/test/e2e/support/openshell-gateway-auth-source-contract-helpers.test.ts index 982d839eb35..51900eea648 100644 --- a/test/e2e/support/openshell-gateway-auth-source-contract-helpers.test.ts +++ b/test/e2e/support/openshell-gateway-auth-source-contract-helpers.test.ts @@ -78,9 +78,10 @@ describe("OpenShell gateway auth source contract helpers", () => { expect(serializedArgs).not.toContain("openshell-gateway.toml"); }); - it("uses empty stdin for the mTLS-only Docker probe", () => { + it("uses empty stdin and the health RPC for the portable mTLS route probe", () => { const { args, input } = buildSandboxTokenContainerProbeInvocation({ dockerBin: "docker", + grpcPath: "/openshell.v1.OpenShell/Health", networkName: "nemoclaw-auth-source-net", payload: Buffer.from("sandbox request"), port: 47321, @@ -91,6 +92,9 @@ describe("OpenShell gateway auth source contract helpers", () => { valuesAfterFlag(args, "--env").some((value) => value.startsWith("PROBE_AUTHORIZATION=")), ).toBe(false); expect(input).toBe(""); + expect(valuesAfterFlag(args, "--env")).toContain( + "PROBE_GRPC_PATH=/openshell.v1.OpenShell/Health", + ); }); it("uses host networking to reach a loopback-only Linux gateway", () => { diff --git a/test/e2e/support/openshell-gateway-upgrade-workflow-boundary.test.ts b/test/e2e/support/openshell-gateway-upgrade-workflow-boundary.test.ts index 94b2c5b8097..c208a59607a 100644 --- a/test/e2e/support/openshell-gateway-upgrade-workflow-boundary.test.ts +++ b/test/e2e/support/openshell-gateway-upgrade-workflow-boundary.test.ts @@ -20,10 +20,13 @@ import { validateE2eWorkflow } from "../../../tools/e2e/workflow-boundary.mts"; import { readWorkflow } from "../../helpers/e2e-workflow-contract"; import { captureGatewayUpgradeProbeEvidence, + captureLegacyGatewayForward, + captureLegacyGatewayServiceForward, currentGatewayUpgradeInstallerArgs, currentNemoclawUpgradeRef, gatewayCredentialNonExposureScript, gatewayUpgradeRecoverySucceeded, + gatewayServiceWasPreserved, gatewayUpgradeAgentResponseIsSuccessful, gatewayUpgradeInstallerCommand, prepareGatewayUpgradeUserManager, @@ -31,6 +34,8 @@ import { isolateGatewayUpgradeFixtureEnv, legacyGatewayUpgradeBaseImageOverrideEnabled, legacyGatewayUpgradeHostFirewallOptions, + legacyGatewayForwardIdentity, + legacyGatewayServiceForwardIdentity, oldGatewayUpgradeInstallerArgs, throwGatewayUpgradeSetupFailures, upgradeGatewayCleanupScript, @@ -38,6 +43,206 @@ import { } from "../live/openshell-gateway-upgrade-helpers.ts"; describe("OpenShell gateway upgrade boundary", () => { + const legacyForwardListing = { + exitCode: 0, + signal: null, + timedOut: false, + stdout: "SANDBOX BIND PORT PID STATUS\nalpha 127.0.0.1 18789 4321 running\n", + }; + const legacyForwardListener = { ...legacyForwardListing, stdout: "4321\n" }; + + it("retains the legacy SSH forward identity when its validated PID owns the port (#12558)", () => { + expect( + legacyGatewayForwardIdentity(legacyForwardListing, legacyForwardListener, "alpha", "18789"), + ).toBe("alpha\t127.0.0.1\t18789\t4321\trunning"); + expect( + legacyGatewayForwardIdentity( + { + ...legacyForwardListing, + stdout: legacyForwardListing.stdout.replace("running", "\u001b[32mrunning\u001b[0m"), + }, + { ...legacyForwardListener, stdout: "4321\n4321\n" }, + "alpha", + "18789", + ), + ).toBe("alpha\t127.0.0.1\t18789\t4321\trunning"); + }); + + it.each([ + { label: "failed listing", listing: { ...legacyForwardListing, exitCode: 1 } }, + { label: "listing timeout", listing: { ...legacyForwardListing, timedOut: true } }, + { label: "listing signal", listing: { ...legacyForwardListing, signal: "SIGTERM" as const } }, + { + label: "missing sandbox", + listing: { ...legacyForwardListing, stdout: "beta 127.0.0.1 18789 4321 running" }, + }, + { + label: "different port", + listing: { ...legacyForwardListing, stdout: "alpha 127.0.0.1 18790 4321 running" }, + }, + { + label: "dead process", + listing: { ...legacyForwardListing, stdout: "alpha 127.0.0.1 18789 4321 dead" }, + }, + { + label: "invalid PID", + listing: { ...legacyForwardListing, stdout: "alpha 127.0.0.1 18789 0 running" }, + }, + { + label: "ambiguous listing", + listing: { ...legacyForwardListing, stdout: legacyForwardListing.stdout.repeat(2) }, + }, + { + label: "extra fields", + listing: { ...legacyForwardListing, stdout: "alpha 127.0.0.1 18789 4321 running unexpected" }, + }, + { label: "failed listener probe", listener: { ...legacyForwardListener, exitCode: 1 } }, + { label: "listener timeout", listener: { ...legacyForwardListener, timedOut: true } }, + { + label: "listener signal", + listener: { ...legacyForwardListener, signal: "SIGTERM" as const }, + }, + { label: "missing listener", listener: { ...legacyForwardListener, stdout: "" } }, + { label: "foreign listener", listener: { ...legacyForwardListener, stdout: "5678" } }, + { label: "ambiguous listener", listener: { ...legacyForwardListener, stdout: "4321\n5678" } }, + ])("rejects a legacy forward with $label (#12558)", ({ listing, listener }) => { + expect( + legacyGatewayForwardIdentity( + listing ?? legacyForwardListing, + listener ?? legacyForwardListener, + "alpha", + "18789", + ), + ).toBeNull(); + }); + + it("uses the retained runtime and process listener to capture the legacy forward (#12558)", async () => { + const command = vi + .fn() + .mockResolvedValueOnce(legacyForwardListing) + .mockResolvedValueOnce(legacyForwardListener); + const host = { command, openshellCommandPath: "/legacy/bin/openshell" }; + const env = { HOME: "/legacy/home" }; + const result = await captureLegacyGatewayForward(host, "alpha", "18789", env, "legacy"); + expect(result.identity).toBe("alpha\t127.0.0.1\t18789\t4321\trunning"); + expect(command.mock.calls).toEqual([ + [ + "/legacy/bin/openshell", + ["forward", "list"], + { artifactName: "legacy-dashboard-forward-list", env, timeoutMs: 15_000 }, + ], + [ + "lsof", + ["-ti", ":18789", "-sTCP:LISTEN"], + { artifactName: "legacy-dashboard-forward-listener", env, timeoutMs: 15_000 }, + ], + ]); + }); + + const serviceArgv = [ + "/legacy/bin/openshell", + "--gateway", + "nemoclaw", + "--workspace", + "default", + "forward", + "service", + "alpha", + "--target-port", + "18789", + "--target-host", + "127.0.0.1", + "--local", + "127.0.0.1:18789", + ]; + const processStat = (start = "12345", state = "S") => + `4321 (openshell) ${state} ${Array(18).fill("0").join(" ")} ${start} 0\n`; + const serviceProbes = () => + [ + "4321\n", + "/legacy/bin/openshell\n", + processStat(), + "/legacy/bin/openshell\n", + "/legacy/bin/openshell\n", + `${serviceArgv.join("\0")}\0`, + processStat(), + "4321\n", + ].map((stdout) => ({ exitCode: 0, stdout, signal: null, timedOut: false })); + + it("retains the v0.0.123 ForwardTcp process identity without an SSH registry entry (#12558)", () => { + const probes = serviceProbes(); + const identity = legacyGatewayServiceForwardIdentity(probes, "alpha", "18789"); + expect(identity).toBe(`4321\t12345\t/legacy/bin/openshell\t${JSON.stringify(serviceArgv)}`); + probes[2]!.stdout = probes[6]!.stdout = processStat("12346"); + expect(legacyGatewayServiceForwardIdentity(probes, "alpha", "18789")).not.toBe(identity); + }); + + it.each([ + { label: "missing listener", index: 0, change: { stdout: "" } }, + { label: "ambiguous listener", index: 0, change: { stdout: "4321\n5678\n" } }, + { label: "replaced listener", index: 7, change: { stdout: "5678\n" } }, + { label: "missing command", index: 1, change: { stdout: "" } }, + { label: "foreign executable", index: 3, change: { stdout: "/foreign/openshell\n" } }, + { label: "failed read", index: 4, change: { exitCode: 1 } }, + { label: "timeout", index: 3, change: { timedOut: true } }, + { label: "signal", index: 3, change: { signal: "SIGTERM" as const } }, + { + label: "different sandbox", + index: 5, + change: { stdout: `${serviceArgv.join("\0").replace("alpha", "beta")}\0` }, + }, + { + label: "different gateway", + index: 5, + change: { stdout: `${serviceArgv.join("\0").replace("nemoclaw", "foreign")}\0` }, + }, + { + label: "different port", + index: 5, + change: { stdout: `${serviceArgv.join("\0").replace("18789", "18790")}\0` }, + }, + { + label: "extra argument", + index: 5, + change: { stdout: `${serviceArgv.join("\0")}\0--unexpected\0` }, + }, + { label: "reused PID", index: 6, change: { stdout: processStat("99999") } }, + { label: "zombie process", index: 6, change: { stdout: processStat("12345", "Z") } }, + { + label: "different PID in stat", + index: 6, + change: { stdout: processStat().replace("4321", "5678") }, + }, + ])("rejects a ForwardTcp observation with $label (#12558)", ({ index, change }) => { + const probes = serviceProbes(); + const changed = probes.map((probe, i) => (i === index ? { ...probe, ...change } : probe)); + expect(legacyGatewayServiceForwardIdentity(changed, "alpha", "18789")).toBeNull(); + }); + + it("captures the retained ForwardTcp executable and process around its listener (#12558)", async () => { + const probes = serviceProbes(); + const pendingProbes = [...probes]; + const command = vi.fn().mockImplementation(async () => pendingProbes.shift()!); + const result = await captureLegacyGatewayServiceForward( + { command, openshellCommandPath: "/legacy/bin/openshell" }, + "alpha", + "18789", + {}, + "legacy", + ); + expect(result.identity).toBe(legacyGatewayServiceForwardIdentity(probes, "alpha", "18789")); + expect(command.mock.calls.map(([executable, args]) => [executable, args])).toEqual([ + ["lsof", ["-ti", ":18789", "-sTCP:LISTEN"]], + ["which", ["/legacy/bin/openshell"]], + ["cat", ["/proc/4321/stat"]], + ["readlink", ["-f", "/proc/4321/exe"]], + ["readlink", ["-f", "/legacy/bin/openshell"]], + ["cat", ["/proc/4321/cmdline"]], + ["cat", ["/proc/4321/stat"]], + ["lsof", ["-ti", ":18789", "-sTCP:LISTEN"]], + ]); + }); + it.each([ { statuses: ["ok"], expected: true }, { statuses: ["error"], expected: false }, @@ -66,6 +271,22 @@ describe("OpenShell gateway upgrade boundary", () => { const beforeService = service("a".repeat(32)); const afterService = service("b".repeat(32)); + it("keeps the same active systemd invocation when a 0.1.x cutover is denied (#12558)", () => { + const evidence = { + before: beforeService, + after: beforeService, + expectedFragmentPath: fragmentPath, + }; + expect(gatewayServiceWasPreserved(evidence)).toBe(true); + expect(gatewayServiceWasPreserved({ ...evidence, after: afterService })).toBe(false); + expect( + gatewayServiceWasPreserved({ + ...evidence, + after: { ...beforeService, stdout: beforeService.stdout.replace("=active", "=inactive") }, + }), + ).toBe(false); + }); + it("requires a successful query and a new active service invocation for systemd recovery", () => { const evidence = { before: beforeService, diff --git a/test/e2e/support/openshell-v0116-tls-server-name-source.test.ts b/test/e2e/support/openshell-v0116-tls-server-name-source.test.ts index f4f59b709d1..f1c883a3395 100644 --- a/test/e2e/support/openshell-v0116-tls-server-name-source.test.ts +++ b/test/e2e/support/openshell-v0116-tls-server-name-source.test.ts @@ -10,11 +10,21 @@ import { describe, expect, it, vi } from "vitest"; import * as dockerDriverGatewayEnv from "../../../src/lib/onboard/docker-driver-gateway-env.ts"; import { createDockerDriverGatewayRuntimeHelpers } from "../../../src/lib/onboard/docker-driver-gateway-runtime.ts"; -import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; +import { + OPENSHELL_V0116_QUALIFICATION, + OPENSHELL_V012_QUALIFICATION, + OPENSHELL_V013_QUALIFICATION, + OPENSHELL_V015_QUALIFICATION, + OPENSHELL_V017_QUALIFICATION, +} from "../fixtures/openshell-v0116-qualification.ts"; import { assertOpenShellTlsServerNameSource, type OpenShellTlsServerNameSource, OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES, + OPENSHELL_V012_TLS_SERVER_NAME_SOURCES, + OPENSHELL_V013_TLS_SERVER_NAME_SOURCES, + OPENSHELL_V015_TLS_SERVER_NAME_SOURCES, + OPENSHELL_V017_TLS_SERVER_NAME_SOURCES, verifyOpenShellTlsServerNameSourceBoundary, } from "../live/openshell-v0116-tls-server-name-source.ts"; @@ -36,7 +46,7 @@ function checkFor(reviewedSource: OpenShellTlsServerNameSource, category: "drive return check!; } -describe("OpenShell 0.0.116 TLS server-name boundary", () => { +describe("OpenShell TLS server-name boundary", () => { it("covers every local supervisor driver changed by the upstream security fix", () => { const checks = OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES.flatMap(({ checks }) => checks); @@ -67,19 +77,19 @@ describe("OpenShell 0.0.116 TLS server-name boundary", () => { try { const helpers = createDockerDriverGatewayRuntimeHelpers({ gatewayPort: 18_080, - getBlueprintMaxOpenshellVersion: () => OPENSHELL_V0116_QUALIFICATION.version, + getBlueprintMaxOpenshellVersion: () => OPENSHELL_V017_QUALIFICATION.version, getCachedOpenshellBinary: () => null, getInstalledOpenshellVersion: () => null, isOpenshellDevVersion: () => false, loadDockerDriverGatewayEnv: () => dockerDriverGatewayEnv, runCapture: () => "", shouldUseOpenshellDevChannel: () => false, - supportedOpenshellFallbackVersion: OPENSHELL_V0116_QUALIFICATION.version, + supportedOpenshellFallbackVersion: OPENSHELL_V017_QUALIFICATION.version, }); expect( helpers.getDockerDriverGatewayEnv(null, "linux").OPENSHELL_DOCKER_SUPERVISOR_IMAGE, - ).toBe(OPENSHELL_V0116_QUALIFICATION.supervisorImage); + ).toBe(OPENSHELL_V017_QUALIFICATION.supervisorImage); } finally { vi.unstubAllEnvs(); fs.rmSync(stateDir, { recursive: true, force: true }); @@ -97,8 +107,14 @@ describe("OpenShell 0.0.116 TLS server-name boundary", () => { ).toThrow(/docker driver does not preserve/u); }); - it("fetches each exact source once before projecting driver and regression results", async () => { - const fixtures = OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES.map((reviewedSource) => { + it.each([ + [OPENSHELL_V0116_QUALIFICATION, OPENSHELL_V0116_TLS_SERVER_NAME_SOURCES], + [OPENSHELL_V012_QUALIFICATION, OPENSHELL_V012_TLS_SERVER_NAME_SOURCES], + [OPENSHELL_V013_QUALIFICATION, OPENSHELL_V013_TLS_SERVER_NAME_SOURCES], + [OPENSHELL_V015_QUALIFICATION, OPENSHELL_V015_TLS_SERVER_NAME_SOURCES], + [OPENSHELL_V017_QUALIFICATION, OPENSHELL_V017_TLS_SERVER_NAME_SOURCES], + ] as const)("fetches each exact source once for %j", async (qualification, sources) => { + const fixtures = sources.map((reviewedSource) => { const source = reviewedSource.checks.flatMap(({ orderedTokens }) => orderedTokens).join("\n"); return { reviewedSource: withBlobSha(reviewedSource, source), @@ -106,8 +122,7 @@ describe("OpenShell 0.0.116 TLS server-name boundary", () => { }; }); const fetchSource = vi.fn(async (input) => { - const sourcePath = - String(input).split(`${OPENSHELL_V0116_QUALIFICATION.sourceRevision}/`)[1] ?? ""; + const sourcePath = String(input).split(`${qualification.sourceRevision}/`)[1] ?? ""; const fixture = fixtures.find(({ reviewedSource }) => reviewedSource.path === sourcePath); return fixture ? new Response(fixture.source, { status: 200 }) @@ -118,12 +133,13 @@ describe("OpenShell 0.0.116 TLS server-name boundary", () => { verifyOpenShellTlsServerNameSourceBoundary( fetchSource, fixtures.map(({ reviewedSource }) => reviewedSource), + qualification, ), ).resolves.toMatchObject({ drivers: [{ driver: "docker" }, { driver: "podman" }, { driver: "vm" }], regressions: [{ driver: "docker" }, { driver: "podman" }, { driver: "vm" }], - sourceRevision: OPENSHELL_V0116_QUALIFICATION.sourceRevision, - version: "0.0.116", + sourceRevision: qualification.sourceRevision, + version: qualification.version, }); expect(fetchSource).toHaveBeenCalledTimes(4); }); diff --git a/test/e2e/support/owned-sandbox-cleanup.test.ts b/test/e2e/support/owned-sandbox-cleanup.test.ts index 37084ee4a2e..4c4eaefd264 100644 --- a/test/e2e/support/owned-sandbox-cleanup.test.ts +++ b/test/e2e/support/owned-sandbox-cleanup.test.ts @@ -3,18 +3,26 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; import { CleanupRegistry } from "../fixtures/cleanup.ts"; +import { HostCliClient } from "../fixtures/clients/host.ts"; +import { RuntimeProviderPrerequisite } from "../fixtures/runtime-provider.ts"; import { SandboxClient } from "../fixtures/clients/sandbox.ts"; import type { ShellProbeResult, ShellProbeRunOptions } from "../fixtures/shell-probe.ts"; -import { prepareOwnedSandboxForOnboard } from "../fixtures/owned-sandbox-cleanup.ts"; +import { + cleanupOwnedSandboxAfterOnboard, + cleanupRegisteredSandbox, + prepareOwnedSandboxForOnboard, +} from "../fixtures/owned-sandbox-cleanup.ts"; const state = vi.hoisted(() => ({ registered: false, gatewayPort: undefined as number | undefined, })); vi.mock("../../../src/lib/state/registry.ts", () => ({ - getSandbox: () => + getSandbox: vi.fn(() => state.registered ? { name: "e2e-mcp-bridge", gatewayPort: state.gatewayPort } : null, + ), })); function fixture() { @@ -69,7 +77,7 @@ function fixture() { const cleanup = new CleanupRegistry(); const prepare = (orphanGatewayName?: string) => prepareOwnedSandboxForOnboard(host, sandbox, cleanup, "e2e-mcp-bridge", orphanGatewayName); - return { calls, host, sandbox, openshell, gateway, presentGateway, cleanup, prepare }; + return { calls, host, sandbox, openshell, gateway, presentGateway, cleanup, prepare, response }; } describe("owned-sandbox cleanup", () => { @@ -80,6 +88,248 @@ describe("owned-sandbox cleanup", () => { state.gatewayPort = undefined; }); + it.each([false, true])( + "preserves Pi cleanup ordering with a registered sandbox: %s", + async (registered) => { + state.registered = registered; + const events: string[] = []; + const cleanup = new CleanupRegistry(); + const host = { + cleanupGatewayRegistration: vi.fn(async () => { + events.push("registration"); + }), + cleanupSandbox: vi.fn(async (_name: string, _options: ShellProbeRunOptions) => { + events.push("CLI destroy"); + }), + }; + const env = { HOME: "/owned-job-home", OPENSHELL_GATEWAY: "nemoclaw" }; + cleanup.trackDisposable("guard", () => { + events.push("guard"); + }); + cleanup.trackGateway(host, "nemoclaw", { env }); + cleanup.trackDisposable("OpenShell delete", () => { + events.push("OpenShell delete"); + }); + cleanup.trackDisposable("CLI destroy", () => + cleanupRegisteredSandbox("e2e-pi-qual", () => + host.cleanupSandbox("e2e-pi-qual", { env, timeoutMs: 300_000 }), + ), + ); + expect((await cleanup.runAll()).failures).toEqual([]); + expect(events).toEqual( + registered + ? ["CLI destroy", "OpenShell delete", "registration", "guard"] + : ["OpenShell delete", "registration", "guard"], + ); + expect(host.cleanupSandbox.mock.calls).toEqual( + registered ? [["e2e-pi-qual", { env, timeoutMs: 300_000 }]] : [], + ); + expect(getSandbox).toHaveBeenCalledWith("e2e-pi-qual"); + }, + ); + + it("retains a registry failure while later Pi cleanup callbacks still run", async () => { + const cleanup = new CleanupRegistry(); + const later = vi.fn(); + const destroy = vi.fn(async () => {}); + vi.mocked(getSandbox).mockImplementationOnce(() => { + throw new Error("invalid registry"); + }); + cleanup.trackDisposable("remaining cleanup", later); + cleanup.trackDisposable("CLI destroy", () => cleanupRegisteredSandbox("e2e-pi-qual", destroy)); + expect((await cleanup.runAll()).failures).toEqual([ + { name: "CLI destroy", message: "invalid registry" }, + ]); + expect(destroy).not.toHaveBeenCalled(); + expect(later).toHaveBeenCalledOnce(); + }); + + function finalCleanupFixture(environment: NodeJS.ProcessEnv = {}) { + const f = fixture(); + const run = vi.fn().mockResolvedValue(f.response()); + const runtime = new RuntimeProviderPrerequisite( + new HostCliClient({ run }), + undefined, + environment, + ); + const options = { env: { OPENSHELL_GATEWAY: "nemoclaw" }, timeoutMs: 60_000 }; + const finish = () => + cleanupOwnedSandboxAfterOnboard(f.sandbox, runtime, "e2e-mcp-bridge", options); + return { ...f, run, runtime, options, finish }; + } + + it.each([ + { environment: {}, command: "docker", prefix: [] }, + { + environment: { + NEMOCLAW_GATEWAY_RUNTIME: "podman", + OPENSHELL_PODMAN_SOCKET: "/run/user/1000/podman/podman.sock", + }, + command: "podman", + prefix: ["--url", "unix:///run/user/1000/podman/podman.sock"], + }, + ])( + "accepts released resources only after empty $command inventory", + async ({ environment, command, prefix }) => { + const f = finalCleanupFixture(environment); + await f.finish(); + expect(f.openshell.mock.calls.map(([args]) => args)).toEqual([ + ["gateway", "info", "-g", "nemoclaw", "-o", "json"], + ]); + const expectedOptions = expect.objectContaining({ + env: expect.objectContaining({ OPENSHELL_GATEWAY: "nemoclaw" }), + captureLimitBytes: 4096, + timeoutMs: 30_000, + }); + expect(f.run.mock.calls).toEqual([ + [ + expect.objectContaining({ + command, + args: [ + ...prefix, + "container", + "ps", + "--all", + "--no-trunc", + "--filter", + "label=openshell.ai/sandbox-name=e2e-mcp-bridge", + "--format", + "{{.ID}}", + ], + }), + expectedOptions, + ], + [ + expect.objectContaining({ + command, + args: [ + ...prefix, + "volume", + "ls", + "--filter", + "name=nemoclaw-openclaw-state-v1-e2e-mcp-bridge", + "--format", + "{{.Name}}", + ], + }), + expectedOptions, + ], + [ + expect.objectContaining({ + command, + args: [ + ...prefix, + "volume", + "ls", + "--filter", + "name=nemoclaw-hermes-state-v1-e2e-mcp-bridge", + "--format", + "{{.Name}}", + ], + }), + expectedOptions, + ], + ]); + expect(f.host.command).not.toHaveBeenCalled(); + }, + ); + + it("keeps raw deletion for an orphan when the selected gateway exists", async () => { + const f = finalCleanupFixture(); + f.gateway.response = f.presentGateway; + await f.finish(); + expect(f.calls).toEqual(["gateway info", "sandbox delete"]); + expect(f.openshell).toHaveBeenLastCalledWith(["sandbox", "delete", "e2e-mcp-bridge"], { + ...f.options, + artifactName: "cleanup-openshell-sandbox-e2e-mcp-bridge", + }); + expect(f.run).not.toHaveBeenCalled(); + }); + + it("preserves raw deletion failure when the gateway exists", async () => { + const f = finalCleanupFixture(); + f.gateway.response = f.presentGateway; + vi.spyOn(f.sandbox, "cleanupSandbox").mockRejectedValue(new Error("delete refused")); + await expect(f.finish()).rejects.toThrow("delete refused"); + expect(f.run).not.toHaveBeenCalled(); + }); + + it("refuses absent-gateway cleanup for a retained registration", async () => { + const f = finalCleanupFixture(); + state.registered = true; + await expect(f.finish()).rejects.toThrow("registered sandbox"); + expect(f.run).not.toHaveBeenCalled(); + expect(f.calls).toEqual(["gateway info"]); + }); + + it.each([ + { stdout: "a".repeat(64) + "\n" }, + { stdout: "malformed inventory" }, + { stdout: "\n" }, + { exitCode: 1 }, + { signal: "SIGTERM" as const }, + { timedOut: true }, + { stderr: "runtime unavailable" }, + ])("preserves uncertain runtime cleanup failure: %j", async (failure) => { + const f = finalCleanupFixture(); + f.run.mockResolvedValue(f.response(failure)); + const later = vi.fn(); + f.cleanup.trackDisposable("later cleanup", later); + f.cleanup.trackDisposable("raw cleanup", f.finish); + expect((await f.cleanup.runAll()).failures).toEqual([ + { + name: "raw cleanup", + message: + "Cannot verify runtime absence for sandbox 'e2e-mcp-bridge' after its gateway was removed.", + }, + ]); + expect(later).toHaveBeenCalledOnce(); + expect(f.calls).toEqual(["gateway info"]); + }); + + it.each([ + { openclaw: { stdout: "nemoclaw-openclaw-state-v1-e2e-mcp-bridge\n" }, hermes: {} }, + { openclaw: {}, hermes: { stdout: "nemoclaw-hermes-state-v1-e2e-mcp-bridge\n" } }, + { openclaw: { exitCode: 1 }, hermes: {} }, + { openclaw: {}, hermes: { signal: "SIGTERM" as const } }, + { openclaw: { timedOut: true }, hermes: {} }, + { openclaw: {}, hermes: { stderr: "volume inventory denied" } }, + ])( + "refuses remaining or uncertain state volumes with no containers: %j", + async ({ openclaw, hermes }) => { + const f = finalCleanupFixture(); + f.run + .mockResolvedValueOnce(f.response()) + .mockResolvedValueOnce(f.response(openclaw)) + .mockResolvedValueOnce(f.response(hermes)); + await expect(f.finish()).rejects.toThrow("Cannot verify runtime absence"); + expect(f.calls).toEqual(["gateway info"]); + expect(f.run).toHaveBeenCalledTimes(3); + }, + ); + + it.each([ + { exitCode: 1, stderr: "permission denied" }, + { stdout: JSON.stringify({ gateway: "foreign" }) }, + { exitCode: 1, signal: "SIGTERM" as const, stderr: "No gateway configured." }, + ])("does not interpret uncertain gateway results as absence: %j", async (failure) => { + const f = finalCleanupFixture(); + f.gateway.response = failure; + await expect(f.finish()).rejects.toThrow(); + expect(f.run).not.toHaveBeenCalled(); + expect(f.calls).toEqual(["gateway info"]); + }); + + it("rejects a registration acquired during the runtime inventory", async () => { + const f = finalCleanupFixture(); + f.run.mockImplementation(async () => { + state.registered = true; + return f.response(); + }); + await expect(f.finish()).rejects.toThrow("Cannot verify runtime absence"); + expect(f.calls).toEqual(["gateway info"]); + }); + it("does not start a gateway or run CLI recovery when fresh resources are absent", async () => { const f = fixture(); await f.prepare(); diff --git a/test/e2e/support/package-database-read-only.test.ts b/test/e2e/support/package-database-read-only.test.ts new file mode 100644 index 00000000000..34545d9a030 --- /dev/null +++ b/test/e2e/support/package-database-read-only.test.ts @@ -0,0 +1,52 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { expect, it, vi } from "vitest"; +import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; +import type { RuntimeProviderPrerequisite } from "../fixtures/runtime-provider.ts"; +import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; +import { expectPackageDatabaseReadOnly } from "../live/package-database-read-only.ts"; + +it("prepares a writable sentinel without chown and removes it after a failed sandbox probe", async () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dpkg-probe-")); + const sentinel = path.join(directory, "sentinel"); + const execSandboxAsRoot = vi.fn(async (_name: string, args: readonly string[]) => { + // Execute the fixture's real commands against a test-owned host path. + const command = [...args.slice(0, -1), sentinel]; + const result = spawnSync(command[0]!, command.slice(1), { encoding: "utf8" }); + return { + exitCode: result.status, + stdout: result.stdout, + stderr: result.stderr, + timedOut: false, + } as ShellProbeResult; + }); + const execShell = vi.fn(async () => { + const metadata = fs.statSync(sentinel); + expect({ mode: metadata.mode & 0o777, uid: metadata.uid }).toEqual({ + mode: 0o666, + uid: process.getuid!(), + }); + throw new Error("sandbox probe failed"); + }); + try { + await expect( + expectPackageDatabaseReadOnly({ + artifactPrefix: "package-database", + env: {}, + runtimeProvider: { execSandboxAsRoot } as unknown as RuntimeProviderPrerequisite, + sandbox: { execShell } as unknown as SandboxClient, + sandboxName: "alpha", + timeoutMs: 30_000, + }), + ).rejects.toThrow("sandbox probe failed"); + expect(execShell).toHaveBeenCalledOnce(); + expect(fs.existsSync(sentinel)).toBe(false); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); diff --git a/test/e2e/support/platform-parity-cloud-experimental.test.ts b/test/e2e/support/platform-parity-cloud-experimental.test.ts index bc9e26b1b29..4dd3fe5f895 100644 --- a/test/e2e/support/platform-parity-cloud-experimental.test.ts +++ b/test/e2e/support/platform-parity-cloud-experimental.test.ts @@ -5,6 +5,7 @@ import { spawn, spawnSync } from "node:child_process"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; +import vm from "node:vm"; import { describe, expect, it, vi } from "vitest"; @@ -19,7 +20,6 @@ import { HostCliClient } from "../fixtures/clients/host.ts"; import { CleanupRegistry } from "../fixtures/cleanup.ts"; import { DCODE_BASE_IMAGE, DCODE_BASE_IMAGE_ENV } from "../fixtures/dcode-base-image.ts"; import { SecretStore } from "../fixtures/secrets.ts"; -import { canonicalTargets } from "../registry/definitions/baseline.ts"; import type { ShellProbeResult, ShellProbeRunOptions, @@ -42,7 +42,6 @@ const dcodeTuiSessionGuard = path.join( process.cwd(), "test/e2e/e2e-cloud-experimental/dcode-tui-session-guard.sh", ); -const dcodeTavilyCheck = path.join(cloudChecksDir, "09-deepagents-code-tavily-opt-in.sh"); const dcodeApprovalCheck = path.join(cloudChecksDir, "12-deepagents-code-thread-auto-approval.sh"); const dcodeApprovalMainEntrypoint = `if [[ "\${BASH_SOURCE[0]}" == "$0" ]]; then main "$@" @@ -52,14 +51,6 @@ const dcodeFreshReonboardCheck = path.join(cloudChecksDir, "04-deepagents-code-f const dcodeLandlockCheck = path.join(cloudChecksDir, "05-deepagents-code-landlock-readonly.sh"); const dcodeObservabilityCheck = path.join(cloudChecksDir, "11-deepagents-code-observability.sh"); const DEFAULT_TEST_PATH = process.env.PATH ?? "/usr/bin:/bin"; -const tavilyBlocked = "BLOCKED:policy denied"; -const observabilityEnabled = "enabled"; -const denialRestored = /returns to the default Tavily denial/; -const denialNotRestored = /did not restore the default Tavily denial/; -const policyRemoveFailed = /policy-remove tavily failed/; -const observabilityPreserved = /preserve enabled observability after policy-remove/; -const observabilityDriftedAfter = /observability state drifted after policy-remove/; -const observabilityDriftedBefore = /observability state drifted before Tavily policy mutation/; function shellResult(exitCode: number, stdout: string, stderr = ""): ShellProbeResult { return { @@ -143,11 +134,129 @@ describe("P0-E cloud-experimental parity guardrails", () => { expect(script).not.toContain('spawnSync(\n "docker"'); }); - it("accepts only the provider-owned Podman host address outside RFC1918", () => { - const script = fs.readFileSync(dcodeObservabilityCheck, "utf8"); + it.each([ + ["docker", "linux", "nemoclaw", "docker", "host", "127.0.0.1", "127.0.0.1"], + ["podman", "linux", "nemoclaw", "podman", "", "", "podman"], + ["docker", "darwin", "nemoclaw", "docker", "host", "127.0.0.1", null], + ["docker", "linux", "other", "docker", "host", "127.0.0.1", null], + ["docker", "linux", "nemoclaw", "podman", "host", "127.0.0.1", null], + ["docker", "linux", "nemoclaw", "docker", "bridge", "127.0.0.1", null], + ["docker", "linux", "nemoclaw", "docker", "host", "0.0.0.0", null], + ])( + "binds OTLP to its selected runtime (%s/%s/%s/%s/%s/%s)", + (providerId, platform, gatewayName, targetProvider, networkName, address, expected) => { + const script = fs.readFileSync(dcodeObservabilityCheck, "utf8"); + const source = script.split("node <<'NODE'\n")[1].split("\nNODE")[0]; + let output = ""; + const provider = { + identity: { id: providerId }, + gateway: { + observeHostRuntime: () => ({ + network: { supervisorNetwork: { name: networkName, hostAddress: address } }, + }), + }, + }; + const run = () => + vm.runInNewContext(source, { + process: { + platform, + env: { SANDBOX_NAME: "fixture", OPENSHELL_GATEWAY: "nemoclaw" }, + stdout: { + write: (value: string) => { + output += value; + }, + }, + exit: () => { + throw new Error("refused"); + }, + }, + require: () => ({ + getSandbox: () => ({ gatewayName }), + currentRuntimeProviderBundles: () => [], + requireRuntimeProviderBundleForSandbox: () => provider, + resolvePrivilegedSandboxTarget: () => ({ + providerId: targetProvider, + resourceHandle: "owned", + }), + }), + }); + const outcomes = expected === null ? [() => expect(run).toThrow("refused")] : [run]; + outcomes.forEach((check) => check()); + expect(output).toBe(expected ?? ""); + expect(script).toContain("169.254.2.2"); + expect(script).not.toContain("169.254.*"); + }, + ); + + it.each([ + [1, "setpriv: setresuid failed: Operation not permitted", 0, 0], + [0, "", 0, 1], + [1, "missing entrypoint", 0, 1], + [1, "setpriv: setresuid failed: Operation not permitted", 1, 1], + ])( + "requires root denial and sandbox-user startup (%s/%s/%s)", + (rootStatus, rootOutput, userStatus, expected) => { + const script = fs.readFileSync(dcodeFreshReonboardCheck, "utf8"); + const block = script.slice( + script.indexOf(" # Native workloads"), + script.indexOf(" # shellcheck disable=SC2016 # Read"), + ); + const result = spawnSync( + "bash", + [ + "-c", + [ + "set -eu", + "resource_handle=owned", + "SANDBOX_NAME=fixture", + "HOSTILE_SHELL_ENV=/fixture-env", + "HOSTILE_PROFILE_MARKER=/fixture-marker", + "fail() { exit 1; }", + 'privileged_exec() { printf "%s\\n" "$ROOT_OUTPUT"; return "$ROOT_STATUS"; }', + 'openshell() { test "$5" = --no-login-shell || return 63; return "$USER_STATUS"; }', + block, + ].join("\n"), + ], + { + encoding: "utf8", + env: { + PATH: DEFAULT_TEST_PATH, + ROOT_OUTPUT: rootOutput, + ROOT_STATUS: String(rootStatus), + USER_STATUS: String(userStatus), + }, + }, + ); + expect(result.status, result.stderr).toBe(expected); + }, + ); - expect(script).toContain("169.254.2.2"); - expect(script).not.toContain("169.254.*"); + it("prepares a writable sentinel exclusively and refuses cleanup of another inode", () => { + const script = fs.readFileSync(dcodeLandlockCheck, "utf8"); + const source = script + .split("const source = String.raw`")[1] + .split("`;")[0] + .replaceAll("observed.st_uid == 0", `observed.st_uid == ${process.getuid?.()}`); + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "dcode-sentinel-")); + const probe = path.join(dir, "probe"); + const run = (operation: string, identity = "") => + spawnSync("python3", ["-I", "-c", source, operation, probe, identity], { encoding: "utf8" }); + try { + const created = run("prepare"); + expect(created.status, created.stderr).toBe(0); + expect(fs.statSync(probe).mode & 0o777).toBe(0o666); + expect(run("prepare").status).not.toBe(0); + expect(run("cleanup", "0:0").status).not.toBe(0); + expect(fs.existsSync(probe)).toBe(true); + expect(run("cleanup", created.stdout.trim()).status).toBe(0); + expect(fs.existsSync(probe)).toBe(false); + fs.symlinkSync(path.join(dir, "other"), probe); + expect(run("prepare").status).not.toBe(0); + expect(run("cleanup", created.stdout.trim()).status).not.toBe(0); + expect(fs.lstatSync(probe).isSymbolicLink()).toBe(true); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } }); it("preserves the repeated env-unset pairs from the failed observability invocation", async () => { @@ -221,7 +330,7 @@ describe("P0-E cloud-experimental parity guardrails", () => { expect(script).toMatch(/"\$CLI" "\$SANDBOX_NAME" exec -- \\\n\s+\/opt\/venv\/bin\/python3/); expect(script).not.toContain("env -u ALL_PROXY"); expect(script.match(/--noproxy '\*'/g)).toHaveLength(2); - expect(script).toContain("/usr/bin/curl --fail-with-body -sS"); + expect(script).toContain("/usr/bin/curl --verbose --fail-with-body -sS"); expect(script).toMatch( /run_deterministic_tool_trace\(\)[\s\S]*"\$CLI" "\$SANDBOX_NAME" exec --[\s\S]*\/opt\/venv\/bin\/python3/, ); @@ -361,46 +470,93 @@ describe("P0-E cloud-experimental parity guardrails", () => { it.each([ [ - "accepts an explicit non-empty success response", + "accepts a non-empty response", + { status_code: 200, content_length: 1234 }, "fetch-success-classification", - "FETCH_SUCCESS:200:1234", 0, "1 passed", ], [ - "accepts explicit denial evidence", + "accepts a policy denial", + { error: "network policy denied" }, "fetch-blocked-classification", - "FETCH_BLOCKED:network policy denied", 0, "1 passed", ], [ - "rejects an unclassified fetch error", + "accepts the managed URL rejection", + { error: "Fetch URL error: URL destination is blocked" }, + "fetch-blocked-classification", + 0, + "1 passed", + ], + [ + "rejects an opaque HTTP error", + { error: "opaque 403" }, "fetch-blocked-classification", - "FETCH_ERROR:opaque 403", 1, "lacked denial evidence", ], - ] as const)("%s from the fetch_url probe", (_label, selfTest, fixture, status, expected) => { - const result = spawnSync( - "bash", - [ - path.join( - process.cwd(), - "test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh", - ), - ], + [ + "rejects a DNS error", + { error: "Fetch URL error: URL hostname could not be resolved" }, + "fetch-blocked-classification", + 1, + "lacked denial evidence", + ], + [ + "rejects an unrelated blocked message", + { error: "remote URL destination is blocked temporarily" }, + "fetch-blocked-classification", + 1, + "lacked denial evidence", + ], + [ + "rejects success as denial", { - encoding: "utf8", - env: { - NEMOCLAW_E2E_PYTHON_EGRESS_SELF_TEST: selfTest, - NEMOCLAW_E2E_FETCH_URL_PROBE_FIXTURE: fixture, - PATH: process.env.PATH ?? "/usr/bin:/bin", - }, + status_code: 200, + content_length: 1234, + error: "Fetch URL error: URL destination is blocked", }, + "fetch-blocked-classification", + 1, + "reached fixture host unexpectedly", + ], + ] as const)("%s from the fetch_url probe", (_label, response, selfTest, status, expected) => { + const script = path.join(cloudChecksDir, "06-deepagents-code-python-egress.sh"); + const source = fs + .readFileSync(script, "utf8") + .split("fetch_url_probe_source() {\n cat <<'PY'\n")[1] + .split("\nPY\n}")[0]; + const probe = spawnSync( + "python3", + [ + "-c", + [ + "import json, sys, types", + "package = types.ModuleType('deepagents_code')", + "tools = types.ModuleType('deepagents_code.tools')", + "managed = types.ModuleType('deepagents_code._nemoclaw_managed')", + "tools.fetch_url = lambda *args, **kwargs: json.loads(sys.argv[2])", + "managed._managed_fetch_uses_openshell = lambda: True", + "sys.modules.update({'deepagents_code': package, 'deepagents_code.tools': tools, 'deepagents_code._nemoclaw_managed': managed})", + source, + ].join("\n"), + "https://fixture.example/", + JSON.stringify(response), + ], + { encoding: "utf8", env: { PATH: DEFAULT_TEST_PATH } }, ); - - expect(result.status).toBe(status); + expect(probe.status, probe.stderr).toBe(0); + const result = spawnSync("bash", [script], { + encoding: "utf8", + env: { + NEMOCLAW_E2E_PYTHON_EGRESS_SELF_TEST: selfTest, + NEMOCLAW_E2E_FETCH_URL_PROBE_FIXTURE: probe.stdout.trim(), + PATH: DEFAULT_TEST_PATH, + }, + }); + expect(result.status, result.stderr).toBe(status); expect(`${result.stdout}\n${result.stderr}`).toContain(expected); }); @@ -426,355 +582,6 @@ describe("P0-E cloud-experimental parity guardrails", () => { expect(result.stdout).toContain("ATOMIC_COMMAND"); }); - it.each([ - ["probe-command-shape", 0, "NATIVE_MULTILINE_ARGV\n"], - ["authenticated-probe-retry", 0, "REACHED:200\nCALLS:1\n"], - ["authenticated-probe-retry", 2, "REACHED:200\nCALLS:3\n"], - ["authenticated-probe-retry", 20, "REACHED:401\nCALLS:10\n"], - ])("passes the Deep Agents Tavily %s self-test (%i unresolved)", (mode, n, expected) => { - const result = spawnSync("bash", [dcodeTavilyCheck], { - encoding: "utf8", - env: { - NEMOCLAW_E2E_TAVILY_SELF_TEST: mode, - NEMOCLAW_E2E_TAVILY_UNRESOLVED_PROBES: String(n), - PATH: DEFAULT_TEST_PATH, - }, - }); - expect(result.status).toBe(0); - expect(result.stdout).toContain(expected); - }); - - it.each([ - [tavilyBlocked, "ok", observabilityEnabled, 0, denialRestored, observabilityPreserved], - ["REACHED:403", "ok", observabilityEnabled, 1, denialNotRestored, observabilityPreserved], - [tavilyBlocked, "fail", observabilityEnabled, 1, policyRemoveFailed, observabilityPreserved], - [ - tavilyBlocked, - "clear-marker", - observabilityEnabled, - 1, - denialRestored, - observabilityDriftedAfter, - ], - [tavilyBlocked, "ok", "disabled", 1, observabilityDriftedBefore, /registry=disabled, marker=1/], - ])( - "restores the default Tavily denial without observability drift (%s/%s/%s)", - (fixture, removeFixture, registryFixture, status, expected, observabilityExpected) => { - const result = spawnSync("bash", [dcodeTavilyCheck], { - encoding: "utf8", - env: { - NEMOCLAW_E2E_OBSERVABILITY_REGISTRY_FIXTURE: registryFixture, - NEMOCLAW_E2E_TAVILY_PROBE_FIXTURE: fixture, - NEMOCLAW_E2E_TAVILY_REMOVE_FIXTURE: removeFixture, - NEMOCLAW_E2E_TAVILY_SELF_TEST: "restore-denial", - PATH: DEFAULT_TEST_PATH, - SANDBOX_NAME: "deepagents-sandbox", - }, - }); - - expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(status); - expect(`${result.stdout}\n${result.stderr}`).toMatch(expected); - expect(`${result.stdout}\n${result.stderr}`).toMatch(observabilityExpected); - expect(fs.readFileSync(dcodeTavilyCheck, "utf8")).toContain( - "trap 'cleanup_tavily_check || exit 1' EXIT", - ); - }, - ); - - it.each([ - ["", "0", "ok", 0], - ["true", "0", "ok", 0], - ["false", "0", "ok", 0], - ["", "1", "ok", 1], - ["true", "1", "ok", 1], - ["false", "1", "ok", 1], - ["true", "0", "fail", 1], - ])( - "restores audit setting %s after removal status %s and restoration %s (#13082)", - (before, removeStatus, restoreStatus, expectedStatus) => { - const result = spawnSync("bash", [dcodeTavilyCheck], { - encoding: "utf8", - env: { - PATH: DEFAULT_TEST_PATH, - NEMOCLAW_E2E_TAVILY_SELF_TEST: "audit-cleanup", - NEMOCLAW_E2E_AUDIT_BEFORE: before as string, - NEMOCLAW_E2E_REMOVE_STATUS: removeStatus as string, - NEMOCLAW_E2E_AUDIT_RESTORE_FIXTURE: restoreStatus as string, - }, - }); - expect(result.status, result.stdout + result.stderr).toBe(expectedStatus); - expect(result.stdout).toContain(`RESTORED:${before}`); - }, - ); - - it.each([ - { stage: "register", status: 1, policy: "present", calls: ["register", "provider-reset"] }, - { - stage: "attach", - status: 1, - policy: "present", - calls: ["register", "attach", "provider-reset"], - }, - { - stage: "policy-add", - status: 1, - policy: "present", - calls: ["register", "attach", "policy-add", "provider-reset"], - }, - { - stage: "probe", - status: 1, - policy: "absent", - calls: ["register", "attach", "policy-add", "policy-remove", "provider-reset"], - }, - { - stage: "complete", - status: 0, - policy: "absent", - calls: ["register", "attach", "policy-add", "policy-remove", "provider-reset"], - }, - ])( - "limits Tavily cleanup to completed opt-in setup ($stage)", - ({ stage, status, policy, calls }) => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-tavily-setup-")); - const cli = path.join(root, "nemoclaw.cjs"); - const callLog = path.join(root, "calls"); - const audit = path.join(root, "audit"); - const policyState = path.join(root, "policy"); - try { - fs.mkdirSync(path.join(root, ".nemoclaw")); - fs.writeFileSync( - path.join(root, ".nemoclaw", "sandboxes.json"), - JSON.stringify({ - sandboxes: { - "deepagents-sandbox": { - agent: "langchain-deepagents-code", - observabilityEnabled: true, - }, - }, - }), - ); - fs.writeFileSync(callLog, ""); - fs.writeFileSync(audit, "false"); - fs.writeFileSync(policyState, "present"); - fs.writeFileSync( - cli, - String.raw`const fs = require("node:fs"); -const path = require("node:path"); -const root = process.env.FIXTURE_ROOT; -const args = process.argv.slice(2); -const record = (value) => fs.appendFileSync(path.join(root, "calls"), value + "\n"); -if (args.includes("--dry-run")) { console.log("api.tavily.com"); process.exit(0); } -switch (args.slice(0, 2).join(" ")) { - case "credentials add": - record("register"); - if (process.env.FAILURE_STAGE === "register") process.exit(1); - fs.writeFileSync(path.join(root, "provider"), args[2]); - break; - case "credentials reset": - record("provider-reset"); - fs.rmSync(path.join(root, "provider"), { force: true }); - break; - case "deepagents-sandbox policy-add": - record("policy-add"); - if (process.env.FAILURE_STAGE === "policy-add") process.exit(1); - fs.writeFileSync(path.join(root, "policy"), "present"); - break; - case "deepagents-sandbox policy-remove": - record("policy-remove"); - fs.writeFileSync(path.join(root, "policy"), "absent"); - break; - default: process.exit(2); -} -`, - ); - fs.writeFileSync( - path.join(root, "openshell"), - String.raw`#!/usr/bin/env node -const fs = require("node:fs"); -const path = require("node:path"); -const root = process.env.FIXTURE_ROOT; -const args = process.argv.slice(2); -const audit = path.join(root, "audit"); -const projectPython = "/sandbox/.nemoclaw-e2e-project-venv/bin/python3"; -switch (args.slice(0, 2).join(" ")) { - case "settings get": - console.log(JSON.stringify({ settings: { ocsf_json_enabled: { scope: "sandbox", value: fs.readFileSync(audit, "utf8") } } })); - break; - case "settings set": - fs.writeFileSync(audit, args[args.indexOf("--value") + 1]); - break; - case "sandbox provider": - fs.appendFileSync(path.join(root, "calls"), "attach\n"); - if (process.env.FAILURE_STAGE === "attach") process.exit(1); - break; - case "sandbox exec": { - const command = args.slice(args.indexOf("--") + 1); - if (command[0] === "sh") { console.log("1"); break; } - if (command[0] === "bash") { - console.log(command.join(" ").includes("/sandbox/.nemoclaw-e2e-project-venv") - ? projectPython : "/opt/venv/bin/python3"); - break; - } - const permitted = fs.readFileSync(path.join(root, "policy"), "utf8") === "present"; - if (process.env.FAILURE_STAGE === "probe" && permitted) process.exit(1); - console.log("PROBE_PID:4242"); - console.log(permitted && command[0] !== "/usr/bin/python3" && command[0] !== projectPython - ? "REACHED:200" : "BLOCKED:policy denied"); - break; - } - default: process.exit(2); -} -`, - { mode: 0o755 }, - ); - const result = spawnSync("bash", [dcodeTavilyCheck], { - encoding: "utf8", - env: { - PATH: `${root}:${DEFAULT_TEST_PATH}`, - HOME: root, - FIXTURE_ROOT: root, - FAILURE_STAGE: stage, - SANDBOX_NAME: "deepagents-sandbox", - NEMOCLAW_E2E_CLI: cli, - NEMOCLAW_E2E_POLICY_SETTLE_SECONDS: "0", - NEMOCLAW_E2E_TAVILY_AUTHENTICATED: "1", - TAVILY_API_KEY: "tavily-fixture-key", - NEMOCLAW_E2E_TAVILY_AUDIT_BEFORE_FIXTURE: "[100] old event", - NEMOCLAW_E2E_TAVILY_AUDIT_FIXTURE: - "[101] [sandbox] [OCSF] [ocsf] NET:OPEN [WARN] DENIED python3(4242) -> api.tavily.com:443", - }, - }); - expect(result.status, result.stdout + result.stderr).toBe(status); - expect(fs.readFileSync(policyState, "utf8")).toBe(policy); - expect(fs.readFileSync(callLog, "utf8").trim().split("\n")).toEqual(calls); - expect(fs.readFileSync(audit, "utf8")).toBe("false"); - expect(fs.existsSync(path.join(root, "provider"))).toBe(false); - } finally { - fs.rmSync(root, { force: true, recursive: true }); - } - }, - ); - - it("keeps Tavily credentials out of the cloud check environment (#13082)", () => { - const env = buildCloudExperimentalCommandEnv("sandbox", "inference-key", { - TAVILY_API_KEY: "ambient-key", - NEMOCLAW_E2E_TAVILY_AUTHENTICATED: "1", - }); - expect(env).not.toHaveProperty("TAVILY_API_KEY"); - expect(env).not.toHaveProperty("NEMOCLAW_E2E_TAVILY_AUTHENTICATED"); - expect( - canonicalTargets().find( - (target) => target.id === "ubuntu-repo-cloud-langchain-deepagents-code", - )?.requiredSecrets, - ).toEqual(["NVIDIA_INFERENCE_API_KEY"]); - }); - - it.each([undefined, "", "tavily-fixture-key"])( - "runs the credential-free Tavily check when its key is %s (#13082)", - async (tavilyKey) => { - const artifactRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-tavily-free-")); - const run = vi.fn(async (_command: TrustedShellCommand, _options?: ShellProbeRunOptions) => - shellResult(0, ""), - ); - const tavilyScript = - "test/e2e/e2e-cloud-experimental/checks/09-deepagents-code-tavily-opt-in.sh"; - const nextScript = - "test/e2e/e2e-cloud-experimental/checks/11-deepagents-code-observability.sh"; - try { - await runE2eCloudExperimentalChecks( - "ubuntu-repo-cloud-langchain-deepagents-code", - "sandbox", - [tavilyScript, nextScript], - { - artifacts: new ArtifactSink(artifactRoot), - cleanup: new CleanupRegistry(), - host: new HostCliClient({ run }), - secrets: new SecretStore({ TAVILY_API_KEY: tavilyKey }, (note) => { - throw new Error(note); - }), - }, - ); - expect(run.mock.calls.map(([command]) => command)).toEqual([ - expect.objectContaining({ command: "openshell" }), - expect.objectContaining({ - command: "bash", - args: [path.join(process.cwd(), tavilyScript)], - }), - expect.objectContaining({ - command: "bash", - args: [path.join(process.cwd(), nextScript)], - }), - ]); - const tavilyEnv = run.mock.calls[1]?.[1]?.env; - expect(tavilyEnv).not.toHaveProperty("TAVILY_API_KEY"); - expect(tavilyEnv).not.toHaveProperty("NEMOCLAW_E2E_TAVILY_AUTHENTICATED"); - } finally { - fs.rmSync(artifactRoot, { recursive: true, force: true }); - } - }, - ); - - it("fails the target when the credential-free Tavily check fails (#13082)", async () => { - const artifactRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-tavily-failure-")); - const run = vi - .fn() - .mockResolvedValueOnce(shellResult(0, "")) - .mockResolvedValueOnce(shellResult(1, "Tavily denial failed")); - try { - await expect( - runE2eCloudExperimentalChecks( - "ubuntu-repo-cloud-langchain-deepagents-code", - "sandbox", - ["test/e2e/e2e-cloud-experimental/checks/09-deepagents-code-tavily-opt-in.sh"], - { - artifacts: new ArtifactSink(artifactRoot), - cleanup: new CleanupRegistry(), - host: new HostCliClient({ run }), - secrets: new SecretStore({}, (note) => { - throw new Error(note); - }), - }, - ), - ).rejects.toThrow(); - } finally { - fs.rmSync(artifactRoot, { recursive: true, force: true }); - } - }); - - it.each([ - "denied", - "allowed", - "stale", - "wrong-host", - "wrong-pid", - "wrong-port", - "wrong-binary", - "missing-pid", - ])("requires post-removal Tavily audit evidence with %s events (#13082)", (event) => { - const cursor = Math.floor(Date.now() / 1000); - const epoch = event === "stale" ? cursor - 1 : cursor + 1; - const pid = event === "wrong-pid" ? 4243 : 4242; - const port = event === "wrong-port" ? 8443 : 443; - const binary = event === "wrong-binary" ? "node" : "python3.12"; - const host = event === "wrong-host" ? "other.example.com" : "api.tavily.com"; - const decision = event === "allowed" ? "ALLOWED" : "DENIED"; - const result = spawnSync("bash", [dcodeTavilyCheck], { - encoding: "utf8", - env: { - PATH: DEFAULT_TEST_PATH, - SANDBOX_NAME: "deepagents-sandbox", - NEMOCLAW_E2E_TAVILY_SELF_TEST: "restore-denial", - NEMOCLAW_E2E_TAVILY_PROBE_FIXTURE: - event === "missing-pid" ? tavilyBlocked : `PROBE_PID:4242\n${tavilyBlocked}`, - NEMOCLAW_E2E_TAVILY_AUTHENTICATED: "1", - NEMOCLAW_E2E_TAVILY_AUDIT_BEFORE_FIXTURE: `[${cursor}] old event`, - NEMOCLAW_E2E_TAVILY_AUDIT_FIXTURE: `[${epoch}] [sandbox] [OCSF] [ocsf] NET:OPEN [WARN] ${decision} ${binary}(${pid}) -> ${host}:${port}`, - }, - }); - expect(result.status, result.stdout + result.stderr).toBe(event === "denied" ? 0 : 1); - }); - it("skips the DCode auto-approval check before optional host prerequisites", () => { const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-dcode-approval-skip-")); try { @@ -942,7 +749,6 @@ exit 37 // The shell program is a fixed literal; repository and temporary-file paths are // positional arguments and are expanded only inside double quotes. - // lgtm[js/shell-command-injection-from-environment] const result = spawnSync( "/bin/bash", [ diff --git a/test/e2e/support/platform-parity-cloud-tavily.test.ts b/test/e2e/support/platform-parity-cloud-tavily.test.ts new file mode 100644 index 00000000000..bf7eaf64a07 --- /dev/null +++ b/test/e2e/support/platform-parity-cloud-tavily.test.ts @@ -0,0 +1,407 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import { ArtifactSink } from "../fixtures/artifacts.ts"; +import { HostCliClient } from "../fixtures/clients/host.ts"; +import { CleanupRegistry } from "../fixtures/cleanup.ts"; +import { SecretStore } from "../fixtures/secrets.ts"; +import { canonicalTargets } from "../registry/definitions/baseline.ts"; +import type { + ShellProbeResult, + ShellProbeRunOptions, + TrustedShellCommand, +} from "../fixtures/shell-probe.ts"; +import { + buildCloudExperimentalCommandEnv, + runE2eCloudExperimentalChecks, +} from "../live/cloud-experimental-checks.ts"; + +const dcodeTavilyCheck = path.join( + process.cwd(), + "test/e2e/e2e-cloud-experimental/checks/09-deepagents-code-tavily-opt-in.sh", +); +const DEFAULT_TEST_PATH = process.env.PATH ?? "/usr/bin:/bin"; +const tavilyBlocked = "BLOCKED:policy denied"; +const observabilityEnabled = "enabled"; +const denialRestored = /returns to the default Tavily denial/; +const denialNotRestored = /did not restore the default Tavily denial/; +const policyRemoveFailed = /policy-remove tavily failed/; +const observabilityPreserved = /preserve enabled observability after policy-remove/; +const observabilityDriftedAfter = /observability state drifted after policy-remove/; +const observabilityDriftedBefore = /observability state drifted before Tavily policy mutation/; + +function shellResult(exitCode: number, stdout: string, stderr = ""): ShellProbeResult { + return { + command: [], + exitCode, + signal: null, + timedOut: false, + stdout, + stderr, + artifacts: { + stdout: "stdout.txt", + stderr: "stderr.txt", + result: "result.json", + }, + }; +} + +describe("Deep Agents Tavily opt-in and revocation", () => { + it.each([ + ["probe-command-shape", 0, "NATIVE_MULTILINE_ARGV\n"], + ["authenticated-probe-retry", 0, "REACHED:200\nCALLS:1\n"], + ["authenticated-probe-retry", 2, "REACHED:200\nCALLS:3\n"], + ["authenticated-probe-retry", 20, "REACHED:401\nCALLS:10\n"], + ])("passes the Deep Agents Tavily %s self-test (%i unresolved)", (mode, n, expected) => { + const result = spawnSync("bash", [dcodeTavilyCheck], { + encoding: "utf8", + env: { + NEMOCLAW_E2E_TAVILY_SELF_TEST: mode, + NEMOCLAW_E2E_TAVILY_UNRESOLVED_PROBES: String(n), + PATH: DEFAULT_TEST_PATH, + }, + }); + expect(result.status).toBe(0); + expect(result.stdout).toContain(expected); + }); + + it.each([ + [tavilyBlocked, "ok", observabilityEnabled, 0, denialRestored, observabilityPreserved], + ["REACHED:403", "ok", observabilityEnabled, 1, denialNotRestored, observabilityPreserved], + [tavilyBlocked, "fail", observabilityEnabled, 1, policyRemoveFailed, observabilityPreserved], + [ + tavilyBlocked, + "clear-marker", + observabilityEnabled, + 1, + denialRestored, + observabilityDriftedAfter, + ], + [tavilyBlocked, "ok", "disabled", 1, observabilityDriftedBefore, /registry=disabled, marker=1/], + ])( + "restores the default Tavily denial without observability drift (%s/%s/%s)", + (fixture, removeFixture, registryFixture, status, expected, observabilityExpected) => { + const result = spawnSync("bash", [dcodeTavilyCheck], { + encoding: "utf8", + env: { + NEMOCLAW_E2E_OBSERVABILITY_REGISTRY_FIXTURE: registryFixture, + NEMOCLAW_E2E_TAVILY_PROBE_FIXTURE: fixture, + NEMOCLAW_E2E_TAVILY_REMOVE_FIXTURE: removeFixture, + NEMOCLAW_E2E_TAVILY_SELF_TEST: "restore-denial", + PATH: DEFAULT_TEST_PATH, + SANDBOX_NAME: "deepagents-sandbox", + }, + }); + + expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(status); + expect(`${result.stdout}\n${result.stderr}`).toMatch(expected); + expect(`${result.stdout}\n${result.stderr}`).toMatch(observabilityExpected); + expect(fs.readFileSync(dcodeTavilyCheck, "utf8")).toContain( + "trap 'cleanup_tavily_check || exit 1' EXIT", + ); + }, + ); + + it.each([ + ["", "0", "ok", 0], + ["true", "0", "ok", 0], + ["false", "0", "ok", 0], + ["", "1", "ok", 1], + ["true", "1", "ok", 1], + ["false", "1", "ok", 1], + ["true", "0", "fail", 1], + ])( + "restores audit setting %s after removal status %s and restoration %s (#13082)", + (before, removeStatus, restoreStatus, expectedStatus) => { + const result = spawnSync("bash", [dcodeTavilyCheck], { + encoding: "utf8", + env: { + PATH: DEFAULT_TEST_PATH, + NEMOCLAW_E2E_TAVILY_SELF_TEST: "audit-cleanup", + NEMOCLAW_E2E_AUDIT_BEFORE: before as string, + NEMOCLAW_E2E_REMOVE_STATUS: removeStatus as string, + NEMOCLAW_E2E_AUDIT_RESTORE_FIXTURE: restoreStatus as string, + }, + }); + expect(result.status, result.stdout + result.stderr).toBe(expectedStatus); + expect(result.stdout).toContain(`RESTORED:${before}`); + }, + ); + + it.each([ + { stage: "register", status: 1, policy: "present", calls: ["register", "provider-reset"] }, + { + stage: "attach", + status: 1, + policy: "present", + calls: ["register", "attach", "provider-reset"], + }, + { + stage: "policy-add", + status: 1, + policy: "present", + calls: ["register", "attach", "policy-add", "provider-reset"], + }, + { + stage: "probe", + status: 1, + policy: "absent", + calls: ["register", "attach", "policy-add", "policy-remove", "provider-reset"], + }, + { + stage: "complete", + status: 0, + policy: "absent", + calls: ["register", "attach", "policy-add", "policy-remove", "provider-reset"], + }, + ])( + "limits Tavily cleanup to completed opt-in setup ($stage)", + ({ stage, status, policy, calls }) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-tavily-setup-")); + const cli = path.join(root, "nemoclaw.cjs"); + const callLog = path.join(root, "calls"); + const audit = path.join(root, "audit"); + const policyState = path.join(root, "policy"); + try { + fs.mkdirSync(path.join(root, ".nemoclaw")); + fs.writeFileSync( + path.join(root, ".nemoclaw", "sandboxes.json"), + JSON.stringify({ + sandboxes: { + "deepagents-sandbox": { + agent: "langchain-deepagents-code", + observabilityEnabled: true, + }, + }, + }), + ); + fs.writeFileSync(callLog, ""); + fs.writeFileSync(audit, "false"); + fs.writeFileSync(policyState, "present"); + fs.writeFileSync( + cli, + String.raw`const fs = require("node:fs"); +const path = require("node:path"); +const root = process.env.FIXTURE_ROOT; +const args = process.argv.slice(2); +const record = (value) => fs.appendFileSync(path.join(root, "calls"), value + "\n"); +if (args.includes("--dry-run")) { console.log("api.tavily.com"); process.exit(0); } +switch (args.slice(0, 2).join(" ")) { + case "credentials add": + record("register"); + if (process.env.FAILURE_STAGE === "register") process.exit(1); + fs.writeFileSync(path.join(root, "provider"), args[2]); + break; + case "credentials reset": + record("provider-reset"); + fs.rmSync(path.join(root, "provider"), { force: true }); + break; + case "deepagents-sandbox policy-add": + record("policy-add"); + if (process.env.FAILURE_STAGE === "policy-add") process.exit(1); + fs.writeFileSync(path.join(root, "policy"), "present"); + break; + case "deepagents-sandbox policy-remove": + record("policy-remove"); + fs.writeFileSync(path.join(root, "policy"), "absent"); + break; + default: process.exit(2); +} +`, + ); + fs.writeFileSync( + path.join(root, "openshell"), + String.raw`#!/usr/bin/env node +const fs = require("node:fs"); +const path = require("node:path"); +const root = process.env.FIXTURE_ROOT; +const args = process.argv.slice(2); +const audit = path.join(root, "audit"); +const projectPython = "/sandbox/.nemoclaw-e2e-project-venv/bin/python3"; +switch (args.slice(0, 2).join(" ")) { + case "settings get": + console.log(JSON.stringify({ settings: { ocsf_json_enabled: { scope: "sandbox", value: fs.readFileSync(audit, "utf8") } } })); + break; + case "settings set": + fs.writeFileSync(audit, args[args.indexOf("--value") + 1]); + break; + case "sandbox provider": + fs.appendFileSync(path.join(root, "calls"), "attach\n"); + if (process.env.FAILURE_STAGE === "attach") process.exit(1); + break; + case "sandbox exec": { + const command = args.slice(args.indexOf("--") + 1); + if (command[0] === "sh") { console.log("1"); break; } + if (command[0] === "bash") { + console.log(command.join(" ").includes("/sandbox/.nemoclaw-e2e-project-venv") + ? projectPython : "/opt/venv/bin/python3"); + break; + } + const permitted = fs.readFileSync(path.join(root, "policy"), "utf8") === "present"; + if (process.env.FAILURE_STAGE === "probe" && permitted) process.exit(1); + console.log("PROBE_PID:4242"); + console.log(permitted && command[0] !== "/usr/bin/python3" && command[0] !== projectPython + ? "REACHED:200" : "BLOCKED:policy denied"); + break; + } + default: process.exit(2); +} +`, + { mode: 0o755 }, + ); + const result = spawnSync("bash", [dcodeTavilyCheck], { + encoding: "utf8", + env: { + PATH: `${root}:${DEFAULT_TEST_PATH}`, + HOME: root, + FIXTURE_ROOT: root, + FAILURE_STAGE: stage, + SANDBOX_NAME: "deepagents-sandbox", + NEMOCLAW_E2E_CLI: cli, + NEMOCLAW_E2E_POLICY_SETTLE_SECONDS: "0", + NEMOCLAW_E2E_TAVILY_AUTHENTICATED: "1", + TAVILY_API_KEY: "tavily-fixture-key", + NEMOCLAW_E2E_TAVILY_AUDIT_BEFORE_FIXTURE: "[100] old event", + NEMOCLAW_E2E_TAVILY_AUDIT_FIXTURE: + "[101] [sandbox] [OCSF] [ocsf] NET:OPEN [WARN] DENIED python3(4242) -> api.tavily.com:443", + }, + }); + expect(result.status, result.stdout + result.stderr).toBe(status); + expect(fs.readFileSync(policyState, "utf8")).toBe(policy); + expect(fs.readFileSync(callLog, "utf8").trim().split("\n")).toEqual(calls); + expect(fs.readFileSync(audit, "utf8")).toBe("false"); + expect(fs.existsSync(path.join(root, "provider"))).toBe(false); + } finally { + fs.rmSync(root, { force: true, recursive: true }); + } + }, + ); + + it("keeps Tavily credentials out of the cloud check environment (#13082)", () => { + const env = buildCloudExperimentalCommandEnv("sandbox", "inference-key", { + TAVILY_API_KEY: "ambient-key", + OTHER_SECRET: "unrelated-key", + NEMOCLAW_E2E_TAVILY_AUTHENTICATED: "1", + }); + expect(env).not.toHaveProperty("TAVILY_API_KEY"); + expect(env).not.toHaveProperty("OTHER_SECRET"); + expect(env).not.toHaveProperty("NEMOCLAW_E2E_TAVILY_AUTHENTICATED"); + expect( + canonicalTargets().find( + (target) => target.id === "ubuntu-repo-cloud-langchain-deepagents-code", + )?.requiredSecrets, + ).toEqual(["NVIDIA_INFERENCE_API_KEY"]); + }); + + it.each([undefined, "", "tavily-fixture-key"])( + "runs the credential-free Tavily check when its key is %s (#13082)", + async (tavilyKey) => { + const artifactRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-tavily-free-")); + const run = vi.fn(async (_command: TrustedShellCommand, _options?: ShellProbeRunOptions) => + shellResult(0, ""), + ); + const tavilyScript = + "test/e2e/e2e-cloud-experimental/checks/09-deepagents-code-tavily-opt-in.sh"; + const nextScript = + "test/e2e/e2e-cloud-experimental/checks/11-deepagents-code-observability.sh"; + try { + await runE2eCloudExperimentalChecks( + "ubuntu-repo-cloud-langchain-deepagents-code", + "sandbox", + [tavilyScript, nextScript], + { + artifacts: new ArtifactSink(artifactRoot), + cleanup: new CleanupRegistry(), + host: new HostCliClient({ run }), + secrets: new SecretStore({ TAVILY_API_KEY: tavilyKey }, (note) => { + throw new Error(note); + }), + }, + ); + expect(run.mock.calls.map(([command]) => command)).toEqual([ + expect.objectContaining({ command: "openshell" }), + expect.objectContaining({ + command: "bash", + args: [path.join(process.cwd(), tavilyScript)], + }), + expect.objectContaining({ + command: "bash", + args: [path.join(process.cwd(), nextScript)], + }), + ]); + expect(run.mock.calls[0]?.[1]?.env).not.toHaveProperty("TAVILY_API_KEY"); + expect(run.mock.calls[2]?.[1]?.env).not.toHaveProperty("TAVILY_API_KEY"); + const tavilyEnv = run.mock.calls[1]?.[1]?.env; + expect(tavilyEnv).not.toHaveProperty("TAVILY_API_KEY"); + expect(tavilyEnv).not.toHaveProperty("NEMOCLAW_E2E_TAVILY_AUTHENTICATED"); + } finally { + fs.rmSync(artifactRoot, { recursive: true, force: true }); + } + }, + ); + + it.each([shellResult(1, "Tavily denial failed"), shellResult(0, "SKIP: Tavily unavailable")])( + "rejects a credential-free Tavily check that fails or skips (%j)", + async (result) => { + const artifactRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-tavily-failure-")); + const run = vi.fn().mockResolvedValueOnce(shellResult(0, "")).mockResolvedValueOnce(result); + try { + await expect( + runE2eCloudExperimentalChecks( + "ubuntu-repo-cloud-langchain-deepagents-code", + "sandbox", + ["test/e2e/e2e-cloud-experimental/checks/09-deepagents-code-tavily-opt-in.sh"], + { + artifacts: new ArtifactSink(artifactRoot), + cleanup: new CleanupRegistry(), + host: new HostCliClient({ run }), + secrets: new SecretStore({}, (note) => { + throw new Error(note); + }), + }, + ), + ).rejects.toThrow(); + } finally { + fs.rmSync(artifactRoot, { recursive: true, force: true }); + } + }, + ); + + it.each([ + "denied", + "allowed", + "stale", + "wrong-host", + "wrong-pid", + "wrong-port", + "wrong-binary", + "missing-pid", + ])("requires post-removal Tavily audit evidence with %s events (#13082)", (event) => { + const cursor = Math.floor(Date.now() / 1000); + const epoch = event === "stale" ? cursor - 1 : cursor + 1; + const pid = event === "wrong-pid" ? 4243 : 4242; + const port = event === "wrong-port" ? 8443 : 443; + const binary = event === "wrong-binary" ? "node" : "python3.12"; + const host = event === "wrong-host" ? "other.example.com" : "api.tavily.com"; + const decision = event === "allowed" ? "ALLOWED" : "DENIED"; + const result = spawnSync("bash", [dcodeTavilyCheck], { + encoding: "utf8", + env: { + PATH: DEFAULT_TEST_PATH, + SANDBOX_NAME: "deepagents-sandbox", + NEMOCLAW_E2E_TAVILY_SELF_TEST: "restore-denial", + NEMOCLAW_E2E_TAVILY_PROBE_FIXTURE: + event === "missing-pid" ? tavilyBlocked : `PROBE_PID:4242\n${tavilyBlocked}`, + NEMOCLAW_E2E_TAVILY_AUTHENTICATED: "1", + NEMOCLAW_E2E_TAVILY_AUDIT_BEFORE_FIXTURE: `[${cursor}] old event`, + NEMOCLAW_E2E_TAVILY_AUDIT_FIXTURE: `[${epoch}] [sandbox] [OCSF] [ocsf] NET:OPEN [WARN] ${decision} ${binary}(${pid}) -> ${host}:${port}`, + }, + }); + expect(result.status, result.stdout + result.stderr).toBe(event === "denied" ? 0 : 1); + }); +}); diff --git a/test/e2e/support/podman-cpu-lifecycle-artifacts.test.ts b/test/e2e/support/podman-cpu-lifecycle-artifacts.test.ts index 5adc25a45e3..c4266790e80 100644 --- a/test/e2e/support/podman-cpu-lifecycle-artifacts.test.ts +++ b/test/e2e/support/podman-cpu-lifecycle-artifacts.test.ts @@ -10,6 +10,7 @@ import { describe, expect, it } from "vitest"; import type { ContainerEngine } from "../../../src/lib/adapters/container-engine.ts"; import { + PODMAN_ISOLATION_ROLE_LABEL, PODMAN_MANAGED_LABEL, PODMAN_SANDBOX_ID_LABEL, PODMAN_SANDBOX_NAME_LABEL, @@ -26,6 +27,7 @@ const SECRET = "nvapi-this-must-not-reach-artifacts"; function inspectOutput( labelOverrides: Readonly> = {}, status = "running", + stateOverrides: Readonly> = {}, ): string { return JSON.stringify([ { @@ -38,6 +40,7 @@ function inspectOutput( Env: [`NVIDIA_API_KEY=${SECRET}`], Labels: { [PODMAN_MANAGED_LABEL]: "true", + [PODMAN_ISOLATION_ROLE_LABEL]: "sandbox", [PODMAN_SANDBOX_ID_LABEL]: "sandbox-id", [PODMAN_SANDBOX_NAME_LABEL]: "podman-openclaw", [PODMAN_SANDBOX_NAMESPACE_LABEL]: PODMAN_SANDBOX_NAMESPACE, @@ -48,10 +51,13 @@ function inspectOutput( }, }, State: { + ExitCode: 0, + OOMKilled: false, Error: SECRET, Paused: false, Running: true, Status: status, + ...stateOverrides, }, }, ]); @@ -64,13 +70,14 @@ describe("Podman CPU proof artifact sanitization", () => { expect(summary).toEqual({ labels: { [PODMAN_MANAGED_LABEL]: "true", + [PODMAN_ISOLATION_ROLE_LABEL]: "sandbox", [PODMAN_SANDBOX_ID_LABEL]: "sandbox-id", [PODMAN_SANDBOX_NAME_LABEL]: "podman-openclaw", [PODMAN_SANDBOX_NAMESPACE_LABEL]: PODMAN_SANDBOX_NAMESPACE, [PODMAN_SANDBOX_WORKSPACE_LABEL]: PODMAN_SANDBOX_WORKSPACE, "nemoclaw.agent": "openclaw", }, - state: { paused: false, running: true, status: "running" }, + state: { exitCode: 0, oomKilled: false, paused: false, running: true, status: "running" }, }); expect(JSON.stringify(summary)).not.toContain(SECRET); expect(summary).not.toHaveProperty("Id"); @@ -79,6 +86,41 @@ describe("Podman CPU proof artifact sanitization", () => { expect(summary).not.toHaveProperty("LogPath"); }); + it("distinguishes a failed supervisor from its workload without exposing runtime errors", () => { + const summary = sanitizePodmanInspectArtifact( + inspectOutput({ [PODMAN_ISOLATION_ROLE_LABEL]: "supervisor" }, "exited", { + ExitCode: 137, + OOMKilled: true, + Running: false, + }), + ); + expect(summary.labels[PODMAN_ISOLATION_ROLE_LABEL]).toBe("supervisor"); + expect(summary.state).toEqual({ + exitCode: 137, + oomKilled: true, + paused: false, + running: false, + status: "exited", + }); + expect(JSON.stringify(summary)).not.toContain(SECRET); + }); + + it.each([undefined, null, SECRET, "137", -1, 256, 1.5])( + "omits invalid diagnostic state values: %s", + (value) => { + const summary = sanitizePodmanInspectArtifact( + inspectOutput({ [PODMAN_ISOLATION_ROLE_LABEL]: SECRET }, "exited", { + ExitCode: value, + OOMKilled: value, + }), + ); + expect(summary.labels[PODMAN_ISOLATION_ROLE_LABEL]).toBeNull(); + expect(summary.state.exitCode).toBeNull(); + expect(summary.state.oomKilled).toBeNull(); + expect(JSON.stringify(summary)).not.toContain(SECRET); + }, + ); + it("omits a secret-shaped value even when it occupies an allowlisted label", () => { const summary = sanitizePodmanInspectArtifact( inspectOutput({ [PODMAN_SANDBOX_NAME_LABEL]: SECRET }), diff --git a/test/e2e/support/podman-cpu-lifecycle-helpers.test.ts b/test/e2e/support/podman-cpu-lifecycle-helpers.test.ts index 7d947789dfa..f14840125ec 100644 --- a/test/e2e/support/podman-cpu-lifecycle-helpers.test.ts +++ b/test/e2e/support/podman-cpu-lifecycle-helpers.test.ts @@ -10,7 +10,11 @@ import { describe, expect, test, vi } from "vitest"; import { ArtifactSink } from "../fixtures/artifacts.ts"; import { startTestProgress, type TestProgress } from "../fixtures/progress.ts"; import { ShellProbe } from "../fixtures/shell-probe.ts"; -import { runCommand, startPinnedGateway } from "../live/podman-cpu-lifecycle-helpers.ts"; +import { + inspectContainer, + runCommand, + startPinnedGateway, +} from "../live/podman-cpu-lifecycle-helpers.ts"; const PHASES = ["exercise the Podman lifecycle helper", "verify helper cleanup"] as const; @@ -56,6 +60,89 @@ function killProcessIfAlive(pid: number | null): void { } describe("Podman CPU lifecycle helper", () => { + function inspectedContainer(identity = "a".repeat(64), namespace = "") { + return { + Id: identity, + Name: "openshell-default--candidate-sandbox-id", + Config: { + Labels: { + "openshell.ai/sandbox-id": "sandbox-id", + "openshell.ai/sandbox-namespace": namespace, + }, + User: "1000:1000", + Cmd: ["--bootstrap", "/.openshell/channel/sandbox/bootstrap.json"], + Entrypoint: ["/opt/openshell/bin/openshell-sandbox"], + }, + }; + } + + test("binds inspection to the scoped discovery result", () => { + const container = inspectedContainer(); + const capture = vi + .fn() + .mockReturnValueOnce({ status: 0, stderr: "", stdout: container.Id + "\n" }) + .mockReturnValueOnce({ status: 0, stderr: "", stdout: JSON.stringify([container]) }); + expect(inspectContainer({ capture } as never, "candidate", container.Id)).toEqual(container); + expect(capture.mock.calls).toEqual([ + [ + [ + "ps", + "--all", + "--quiet", + "--no-trunc", + "--filter", + "label=openshell.managed=true", + "--filter", + "label=openshell.ai/sandbox-name=candidate", + "--filter", + "label=openshell.ai/sandbox-workspace=default", + "--filter", + "label=openshell.ai/isolation-role=sandbox", + ], + ], + [["container", "inspect", container.Id]], + ]); + }); + + test.each([ + ["root identity", { User: "0:0" }], + ["another group", { User: "1000:0" }], + [ + "legacy launcher", + { + Cmd: [ + "launch-capability-free", + "1000", + "1000", + "/.openshell/channel/sandbox/bootstrap.json", + "/sandbox", + ], + }, + ], + ])("rejects a workload with %s", (_reason, config) => { + const container = inspectedContainer(); + Object.assign(container.Config, config); + const capture = vi + .fn() + .mockReturnValueOnce({ status: 0, stderr: "", stdout: container.Id + "\n" }) + .mockReturnValueOnce({ status: 0, stderr: "", stdout: JSON.stringify([container]) }); + expect(() => inspectContainer({ capture } as never, "candidate")).toThrow(); + }); + + test.each([ + ["another container", "b".repeat(64), ""], + ["another namespace", "a".repeat(64), "foreign"], + ])("rejects inspection of %s", (_reason, identity, namespace) => { + const capture = vi + .fn() + .mockReturnValueOnce({ status: 0, stderr: "", stdout: "a".repeat(64) + "\n" }) + .mockReturnValueOnce({ + status: 0, + stderr: "", + stdout: JSON.stringify([inspectedContainer(identity, namespace)]), + }); + expect(() => inspectContainer({ capture } as never, "candidate")).toThrow(); + }); test("reports CLI child lifecycle through the canonical ShellProbe boundary (#8497)", async () => { const root = await fs.mkdtemp(path.join(os.tmpdir(), "nemoclaw-podman-command-test-")); const logLines: string[] = []; @@ -126,6 +213,51 @@ describe("Podman CPU lifecycle helper", () => { } }); + test("retains redacted gateway errors after successful RPC logs replace the tail", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "nemoclaw-podman-gateway-log-test-")); + const bootstrapPath = path.join(root, "gateway-bootstrap.cjs"); + const artifactDir = path.join(root, "artifacts"); + const progress = testProgress([]); + await fs.writeFile( + bootstrapPath, + [ + 'process.stdout.write(("WARN earlier warning " + ".".repeat(1_024) + "\\n").repeat(40));', + 'process.stdout.write("2026-10-15T00:00:00Z ERROR restart failed OPENAI_API_KEY=nvapi-");', + 'setTimeout(() => process.stdout.write("gateway-secret\\n" + "INFO successful RPC\\n".repeat(8_000)), 20);', + ].join("\n"), + "utf8", + ); + try { + await expect( + startPinnedGateway( + process.execPath, + { NODE_OPTIONS: `--require=${bootstrapPath}` }, + progress, + artifactDir, + ), + ).rejects.toThrow("gateway exited"); + const tail = await fs.readFile( + path.join(artifactDir, "openshell-podman-gateway.log"), + "utf8", + ); + const errors = await fs.readFile( + path.join(artifactDir, "openshell-podman-gateway-errors.log"), + "utf8", + ); + expect(tail).not.toContain("restart failed"); + expect(errors).toContain("ERROR restart failed OPENAI_API_KEY="); + expect(errors).not.toContain("gateway-secret"); + expect(errors).not.toContain("INFO successful RPC"); + expect(errors.length).toBeLessThanOrEqual(32 * 1024); + expect( + (await fs.stat(path.join(artifactDir, "openshell-podman-gateway-errors.log"))).mode & 0o777, + ).toBe(0o600); + } finally { + progress.stop(); + await fs.rm(root, { force: true, recursive: true }); + } + }); + test("escalates a rejected gateway child to SIGKILL before returning (#8497)", async () => { const root = await fs.mkdtemp(path.join(os.tmpdir(), "nemoclaw-podman-gateway-test-")); const artifactDir = path.join(root, "artifacts"); diff --git a/test/e2e/support/portable-hermes-probe-diagnostics.test.ts b/test/e2e/support/portable-hermes-probe-diagnostics.test.ts index c206d7358de..350ddff108d 100644 --- a/test/e2e/support/portable-hermes-probe-diagnostics.test.ts +++ b/test/e2e/support/portable-hermes-probe-diagnostics.test.ts @@ -55,3 +55,25 @@ it("does not record a probe artifact when the route succeeds", async () => { fs.rmSync(root, { force: true, recursive: true }); } }); + +it("records an mTLS Health probe failure without a sandbox bearer token", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-mtls-probe-")); + try { + await recordPortableHermesGatewayProbeFailure(new ArtifactSink(root), "", { + status: 2, + stdout: '{"error":"connection refused"}', + stderr: "Health probe could not reach the gateway", + }); + const evidence = JSON.parse( + fs.readFileSync(path.join(root, "portable-hermes-gateway-probe-failure.json"), "utf-8"), + ); + expect(evidence).toEqual({ + schemaVersion: 1, + status: 2, + stdoutTail: '{"error":"connection refused"}', + stderrTail: "Health probe could not reach the gateway", + }); + } finally { + fs.rmSync(root, { force: true, recursive: true }); + } +}); diff --git a/test/e2e/support/portable-hermes-transaction-live.test.ts b/test/e2e/support/portable-hermes-transaction-live.test.ts new file mode 100644 index 00000000000..79bdc72d2cf --- /dev/null +++ b/test/e2e/support/portable-hermes-transaction-live.test.ts @@ -0,0 +1,48 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { expect, it } from "vitest"; +import { portableHermesTransactionSandboxIsPresent } from "../fixtures/portable-hermes-transaction-live.ts"; + +const sandbox = { + id: "sandbox-alpha", + name: "alpha", + labels: {}, + resource_version: 1, + created_at: "2026-10-29T00:00:00Z", + phase: "Ready", + current_policy_version: 1, +}; + +it.each([ + { inventory: [sandbox], present: true }, + { inventory: [], present: false }, + { inventory: { sandboxes: [sandbox], next_page_token: "" }, present: true }, + { inventory: { sandboxes: [], next_page_token: "" }, present: false }, + { + inventory: { sandboxes: [{ ...sandbox, name: "other" }], next_page_token: "" }, + present: false, + }, +])("reads complete transaction cleanup inventory $inventory (#12603)", ({ inventory, present }) => { + expect( + portableHermesTransactionSandboxIsPresent("alpha", { + status: 0, + stdout: JSON.stringify(inventory), + }), + ).toBe(present); +}); + +it.each([ + { status: 1, stdout: "[]" }, + { status: null, stdout: "[]" }, + { status: 0, stdout: "[]", error: new Error("transport lost") }, + { status: 0, stdout: "not-json" }, + { status: 0, stdout: JSON.stringify({ sandboxes: [], next_page_token: "more" }) }, + { status: 0, stdout: JSON.stringify({ sandboxes: [] }) }, + { status: 0, stdout: JSON.stringify([{ name: "alpha" }]) }, + { status: 0, stdout: JSON.stringify({ sandboxes: [sandbox, sandbox], next_page_token: "" }) }, +])("refuses ambiguous transaction cleanup evidence $stdout (#12603)", (result) => { + expect(() => portableHermesTransactionSandboxIsPresent("alpha", result)).toThrow( + /cleanup observation|duplicate sandbox names/, + ); +}); diff --git a/test/e2e/support/portable-profile-gateway-proof.test.ts b/test/e2e/support/portable-profile-gateway-proof.test.ts new file mode 100644 index 00000000000..823c6c6d056 --- /dev/null +++ b/test/e2e/support/portable-profile-gateway-proof.test.ts @@ -0,0 +1,126 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { ChildProcess } from "node:child_process"; +import * as childProcess from "node:child_process"; +import { PassThrough } from "node:stream"; +import { expect, test, vi } from "vitest"; + +import * as observedChild from "../fixtures/observed-child-process.ts"; +import { startTestProgress } from "../fixtures/progress.ts"; +import { + recordPortableSandboxDiagnostics, + verifyPinnedPodmanGatewayStarts, +} from "../live/portable-profile-gateway-proof.ts"; + +vi.mock("node:child_process", async () => ({ + ...(await vi.importActual("node:child_process")), + spawnSync: vi.fn(), +})); + +const id = "a".repeat(64); +const socketPath = "/run/user/1002/podman/podman.sock"; +const logsArgs = ["--url", `unix://${socketPath}`, "logs", id]; +const callbackFailure = new Error("sandbox entered Error"); + +test("does not inspect containers after sandbox creation succeeds (#12603)", () => { + recordPortableSandboxDiagnostics(socketPath, "hermes-portable-e2e", 0); + expect(childProcess.spawnSync).not.toHaveBeenCalled(); +}); + +test.each([ + { status: 0, remainingCalls: [logsArgs], output: /sandbox startup failure/u }, + { status: 1, remainingCalls: [], output: /Sandbox diagnostic inventory \(status 1\)/u }, +])( + "captures only selected fixture logs after inventory status $status (#12603)", + ({ status, remainingCalls, output }) => { + const secret = `nvapi-${"fixture".repeat(4)}`; + const capture = vi + .mocked(childProcess.spawnSync) + .mockReturnValueOnce({ + status, + stdout: `${id}\nnot-an-id\n`, + stderr: "", + pid: 1, + signal: null, + output: [], + }) + .mockReturnValue({ + status: 1, + stdout: "sandbox startup failure", + stderr: `Bearer ${secret}`, + pid: 1, + signal: null, + output: [], + }); + const diagnostic = vi.spyOn(console, "error").mockImplementation(() => undefined); + recordPortableSandboxDiagnostics(socketPath, "hermes-portable-e2e", 1); + expect(capture).toHaveBeenCalledTimes(remainingCalls.length + 1); + expect(capture.mock.calls[0]?.[1]).toEqual([ + "--url", + "unix:///run/user/1002/podman/podman.sock", + "ps", + "--all", + "--no-trunc", + "--filter", + "label=openshell.ai/sandbox-name=hermes-portable-e2e", + "--format", + "{{.ID}}", + ]); + expect(capture.mock.calls.slice(1).map((call) => call[1])).toEqual(remainingCalls); + expect(diagnostic.mock.calls.flat().join("\n")).toMatch(output); + expect(diagnostic.mock.calls.flat().join("\n")).not.toContain(secret); + }, +); + +test.each([ + { fails: false, callback: async () => undefined }, + { + fails: true, + callback: async () => { + throw callbackFailure; + }, + }, +])( + "retains callback failure diagnostics without changing gateway cleanup (failure=$fails) (#12603)", + async ({ fails, callback }) => { + vi.useFakeTimers(); + const progress = startTestProgress("gateway diagnostic", ["run gateway", "check result"], { + logLine: () => undefined, + }); + const child = new ChildProcess(); + child.stdout = new PassThrough(); + child.stderr = new PassThrough(); + vi.spyOn(observedChild, "spawnObservedChild").mockReturnValue(child); + const kill = vi.spyOn(child, "kill").mockReturnValue(true); + const diagnostic = vi.spyOn(console, "error").mockImplementation(() => undefined); + const secret = `nvapi-${"fixture".repeat(4)}`; + try { + const result = verifyPinnedPodmanGatewayStarts( + "openshell-gateway", + {}, + progress, + async () => { + child.stderr?.emit("data", `compute rejected; authorization: Bearer ${secret}\n`); + await callback(); + }, + ); + const checked = fails + ? expect(result).rejects.toBe(callbackFailure) + : expect(result).resolves.toBeUndefined(); + child.stdout.emit("data", "Using compute driver driver=podman\n"); + await vi.advanceTimersByTimeAsync(2200); + await checked; + expect(kill).toHaveBeenCalledExactlyOnceWith("SIGTERM"); + expect(diagnostic).toHaveBeenCalledExactlyOnceWith( + expect.stringContaining("compute rejected"), + ); + expect(diagnostic.mock.calls.flat().join("\n")).not.toContain(secret); + } finally { + progress.stop(); + child.stdout.destroy(); + child.stderr.destroy(); + vi.useRealTimers(); + } + }, +); diff --git a/test/e2e/support/portable-profile-rootless-runtime-workflow.test.ts b/test/e2e/support/portable-profile-rootless-runtime-workflow.test.ts index fa964e12de3..e13c12d29f5 100644 --- a/test/e2e/support/portable-profile-rootless-runtime-workflow.test.ts +++ b/test/e2e/support/portable-profile-rootless-runtime-workflow.test.ts @@ -185,10 +185,10 @@ describe("portable profile rootless runtime workflow", () => { ); expect(liveTest).toContain("preparePortableExperimentalHost(process.env, { home });"); expect(liveTest).toContain( - 'import { OPENSHELL_V0116_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts";', + 'import { OPENSHELL_V017_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts";', ); expect(liveTest).toContain( - "getDockerSupervisorImage: () => OPENSHELL_V0116_QUALIFICATION.supervisorImage", + "getDockerSupervisorImage: () => OPENSHELL_V017_QUALIFICATION.supervisorImage", ); expect(liveTest).not.toContain("OPENSHELL_V0106_QUALIFICATION"); expect(liveTest).toContain("createHermesPortableBuildContextPlan("); diff --git a/test/e2e/support/pr-self-hosted-llama-selector.test.ts b/test/e2e/support/pr-self-hosted-llama-selector.test.ts index b5330c467de..88d22c8881b 100644 --- a/test/e2e/support/pr-self-hosted-llama-selector.test.ts +++ b/test/e2e/support/pr-self-hosted-llama-selector.test.ts @@ -297,6 +297,14 @@ describe.concurrent("generic NVIDIA GPU PR selection", () => { await expect(selectHermesRootEntrypoint(["src/lib/hermes-managed-route.ts"])).resolves.toBe( "selected=true", ); + await expect( + selectHermesRootEntrypoint(["src/lib/inference/native-provider/catalog.ts"]), + ).resolves.toBe("selected=true"); + await expect( + selectHermesRootEntrypoint([ + "src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json", + ]), + ).resolves.toBe("selected=true"); }); // source-shape-contract: security -- Changed-file discovery failures must stop trusted copied-PR qualification instead of silently skipping it diff --git a/test/e2e/support/registered-inference.test.ts b/test/e2e/support/registered-inference.test.ts new file mode 100644 index 00000000000..0ba9db27a26 --- /dev/null +++ b/test/e2e/support/registered-inference.test.ts @@ -0,0 +1,598 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { resolveRegisteredInferenceProvider } from "../../../src/lib/inference/native-provider/registered.ts"; +import { inspectInferenceProviderProfile } from "../../../src/lib/adapters/openshell/provider-profile.ts"; +import type { OpenShellProviderAdapter } from "../../../src/lib/adapters/openshell/provider-adapter.ts"; +import type { HostCliClient } from "../fixtures/clients/host.ts"; +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; +import { expectOpenAiChatThroughSandbox } from "../live/inference-routing-helpers.ts"; +import { getSandbox } from "../../../src/lib/state/registry.ts"; +import { + nativeLocalIdentity, + type NativeLocalBinding, +} from "../../../src/lib/inference/native-local/contract.ts"; +import { customNativeProvider } from "../../../src/lib/inference/native-provider/custom.ts"; +import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; +import { + INFERENCE_CURL_DIAGNOSTIC_SCRIPT, + INFERENCE_CURL_HTTP_CODE_MARKER, +} from "../fixtures/curl-policy-denial.ts"; +import { + operatorInferenceSwitchFixture, + OPERATOR_SWITCH_PROVIDER, + OPERATOR_SWITCH_CREDENTIAL_ENV, + registeredCompatibleChatCommand, + registeredCompatibleResponseCommand, + readRegisteredLocalRoute, +} from "../fixtures/registered-inference.ts"; + +vi.mock("../fixtures/e2e-test.ts", async () => ({ expect: (await import("vitest")).expect })); + +vi.mock("../../../src/lib/state/registry.ts", () => ({ getSandbox: vi.fn() })); + +const localBinding: NativeLocalBinding = { + provider: "compatible-endpoint", + agent: "openclaw", + endpointUrl: "http://host.openshell.internal:8000/v1", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + authMode: "authenticated", + gatewayName: "test-gateway", + sandboxName: "test-sandbox", +}; +const local = { + ...localBinding, + ...nativeLocalIdentity(localBinding), + schemaVersion: 1 as const, + providerId: "provider-one", +}; +const selection = { + name: "test-sandbox", + agent: "openclaw" as const, + gatewayName: "test-gateway", + provider: "compatible-endpoint", + model: "test-model", + nativeLocalProviderAttachment: local, +}; +const payload = JSON.stringify({ + model: "test-model", + messages: [{ role: "user", content: "PONG $(not-a-command)" }], +}); +let directory: string; +beforeEach(() => { + vi.mocked(getSandbox).mockReturnValue(selection); + directory = fs.mkdtempSync(path.join(os.tmpdir(), "registered-inference-")); + fs.writeFileSync(path.join(directory, "curl"), '#!/bin/sh\nprintf "%s\\n" "$@"\n', { + mode: 0o755, + }); +}); +afterEach(() => fs.rmSync(directory, { recursive: true, force: true })); +function run(command: [string, ...string[]], env: Record) { + const options = { + encoding: "utf8", + env: { + ...process.env, + HOME: directory, + TMPDIR: directory, + PATH: `${directory}:${process.env.PATH}`, + ...env, + }, + } as const; + const executable = new Map([ + ["bash", "bash"], + ["/bin/bash", "/bin/bash"], + ["curl", "curl"], + ]).get(command[0]); + expect(executable).toBeDefined(); + return spawnSync(executable!, command.slice(1), options); +} + +it("executes the local probe with its exact endpoint, payload and scoped credential", () => { + const credential = "openshell:resolve:env:v17_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; + const result = run(registeredCompatibleChatCommand("test-sandbox", "test-model", payload), { + NEMOCLAW_LOCAL_INFERENCE_TOKEN: credential, + }); + expect(result.status).toBe(0); + expect(result.stdout.split("\n")).toEqual([ + "-fsS", + "--max-time", + "60", + `${local.endpointUrl}/chat/completions`, + "-H", + "Content-Type: application/json", + "-H", + `Authorization: Bearer ${credential}`, + "--data-raw", + payload, + "", + ]); +}); + +it.each([ + "raw-secret", + "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + "openshell:resolve:env:v17_NEMOCLAW_LOCAL_INFERENCE_TOKEN\nraw-secret", +])("rejects an unsafe local credential before invoking curl: %s", (credential) => { + const result = run(registeredCompatibleChatCommand("test-sandbox", "test-model", payload), { + NEMOCLAW_LOCAL_INFERENCE_TOKEN: credential, + }); + expect(result.status).toBe(2); + expect(result.stdout).toBe(""); +}); + +it.each([ + { model: "other" }, + { gatewayName: "other-gateway" }, + { agent: "hermes" }, + { nativeLocalProviderAttachment: { ...local, providerId: "" } }, + { nativeLocalProviderAttachment: undefined }, +])("rejects a stale or missing selection before constructing a probe: %j", (change) => { + vi.mocked(getSandbox).mockReturnValue({ ...selection, ...change } as typeof selection); + expect(() => registeredCompatibleChatCommand("test-sandbox", "test-model", payload)).toThrow(); +}); + +it("reads the replacement endpoint from the current registry after recreation", () => { + const binding = { ...localBinding, endpointUrl: "http://host.openshell.internal:8001/v1" }; + vi.mocked(getSandbox).mockReturnValue({ + ...selection, + nativeLocalProviderAttachment: { + ...binding, + ...nativeLocalIdentity(binding), + schemaVersion: 1, + providerId: "replacement", + }, + }); + const result = run(registeredCompatibleChatCommand("test-sandbox", "test-model", payload), { + NEMOCLAW_LOCAL_INFERENCE_TOKEN: "openshell:resolve:env:v18_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }); + expect(result.status).toBe(0); + expect(result.stdout).toContain(`${binding.endpointUrl}/chat/completions`); + expect(result.stdout).not.toContain(local.endpointUrl); +}); + +it.each([false, true])( + "preserves hosted error responses and the explicit redirect choice: %s", + (follow) => { + const definition = customNativeProvider({ + provider: "compatible-endpoint", + endpointUrl: "https://example.com/v1", + api: "openai-completions", + credentialEnv: "COMPATIBLE_API_KEY", + allowedIps: ["8.8.8.8"], + }); + vi.mocked(getSandbox).mockReturnValue({ + ...selection, + nativeLocalProviderAttachment: undefined, + nativeHostedProviderAttachment: { ...definition, schemaVersion: 1, providerId: "hosted-one" }, + }); + const credential = "openshell:resolve:env:v18_COMPATIBLE_API_KEY"; + const result = run( + registeredCompatibleResponseCommand("test-sandbox", "test-model", payload, follow), + { COMPATIBLE_API_KEY: credential }, + ); + expect(result.status).toBe(0); + expect(result.stdout.split("\n")).toEqual([ + "--verbose", + "--write-out", + "", + `${INFERENCE_CURL_HTTP_CODE_MARKER}%{http_code}`, + "", + "-sS", + "--include", + ...(follow ? ["--location", "--post302", "--max-redirs", "3"] : []), + "--max-time", + "60", + "https://example.com/v1/chat/completions", + "-H", + "Content-Type: application/json", + "-H", + `Authorization: Bearer ${credential}`, + "--data-raw", + payload, + "", + ]); + }, +); + +it.each([0, 7, 56, 127])("retains curl status %s and only connect diagnostics", (status) => { + fs.writeFileSync( + path.join(directory, "curl"), + `#!/bin/sh +printf '\nNEMOCLAW_INFERENCE_CURL_HTTP_CODE=000\n' +printf '%s\n' '> Authorization: Bearer scoped-handle' '> Cookie: private-cookie' '* Server auth using Basic with user secret-user' '* Immediate connect fail for 198.18.0.1: Permission denied' 'curl: (7) Failed to connect' >&2 +exit ${status} +`, + { mode: 0o755 }, + ); + const result = run( + [ + "/bin/bash", + "--noprofile", + "--norc", + "-c", + `${INFERENCE_CURL_DIAGNOSTIC_SCRIPT}; curl -sS https://example.com/v1/models`, + ], + {}, + ); + expect(result.status).toBe(status); + expect(result.stdout).toBe("\nNEMOCLAW_INFERENCE_CURL_HTTP_CODE=000\n"); + expect(result.stderr).toBe( + "* Immediate connect fail for 198.18.0.1: Permission denied\ncurl: (7) Failed to connect\n", + ); + expect(fs.readdirSync(directory)).toEqual(["curl"]); +}); + +function readback(providerId = local.providerId, attached = local.providerName) { + return vi + .fn() + .mockResolvedValueOnce({ + exitCode: 0, + stdout: `Name: ${local.providerName}\nId: ${providerId}\nResource version: 1\nType: ${local.profileId}\nCredential keys: ${local.credentialEnv}\nConfig keys: `, + stderr: "", + }) + .mockResolvedValueOnce({ exitCode: 0, stdout: attached, stderr: "" }); +} +it("reads the sibling provider identity and attachment on its recorded gateway", async () => { + const openshell = readback(); + const result = await readRegisteredLocalRoute( + { openshell } as unknown as SandboxClient, + "test-sandbox", + "before", + ); + expect(result).toEqual({ + exitCode: 0, + stdout: JSON.stringify({ + provider: selection.provider, + model: selection.model, + receipt: local, + }), + stderr: "", + }); + expect(openshell.mock.calls.map(([command]) => command)).toEqual([ + ["provider", "get", "-g", "test-gateway", local.providerName], + ["sandbox", "provider", "list", "-g", "test-gateway", "test-sandbox"], + ]); +}); +it.each([ + ["other-provider", local.providerName], + [local.providerId, "detached"], +])("rejects a replaced or detached sibling provider: %s %s", async (id, attached) => { + const openshell = readback(id, attached); + expect( + await readRegisteredLocalRoute( + { openshell } as unknown as SandboxClient, + "test-sandbox", + "after", + ), + ).toMatchObject({ + exitCode: 1, + stderr: "Sibling native provider identity or attachment mismatch", + }); +}); + +const gatewayResponse = { + ok: true, + capability: "model.run", + transport: "gateway", + provider: "inference", + model: "test-model", + outputs: [{ text: "PONG" }], +}; + +it("runs the real OpenClaw gateway consumer with no endpoint or credential override", async () => { + const exec = vi.fn(async () => ({ + exitCode: 0, + stdout: JSON.stringify(gatewayResponse), + stderr: "", + })); + await expectOpenAiChatThroughSandbox( + { exec } as unknown as SandboxClient, + "test-sandbox", + "test-model", + ["test-host-secret"], + "gateway-consumer", + "openclaw-gateway", + ); + expect(exec).toHaveBeenCalledWith( + "test-sandbox", + [ + "openclaw", + "infer", + "model", + "run", + "--gateway", + "--json", + "--prompt", + "Reply with exactly one word: PONG", + ], + expect.objectContaining({ + artifactName: "gateway-consumer", + redactionValues: ["test-host-secret"], + timeoutMs: 90000, + }), + ); + expect(vi.mocked(getSandbox)).not.toHaveBeenCalled(); +}); + +it.each([ + { transport: "embedded" }, + { provider: "other" }, + { model: "other" }, + { ok: false }, + { capability: "other" }, + { outputs: [] }, + { outputs: [{ text: "wrong response" }] }, +])("refuses a mismatched OpenClaw gateway response %j", async (override) => { + const exec = vi.fn(async () => ({ + exitCode: 0, + stdout: JSON.stringify({ ...gatewayResponse, ...override }), + stderr: "", + })); + await expect( + expectOpenAiChatThroughSandbox( + { exec } as unknown as SandboxClient, + "test-sandbox", + "test-model", + [], + "gateway-denial", + "openclaw-gateway", + ), + ).rejects.toThrow(); +}); + +it("propagates a failed OpenClaw gateway command even with success-shaped output", async () => { + const exec = vi.fn(async () => ({ + exitCode: 42, + stdout: JSON.stringify(gatewayResponse), + stderr: "command failed", + })); + await expect( + expectOpenAiChatThroughSandbox( + { exec } as unknown as SandboxClient, + "test-sandbox", + "test-model", + [], + "gateway-command-failure", + "openclaw-gateway", + ), + ).rejects.toThrow(); +}); + +it("preserves the registered native curl caller used by DNS and runtime identity checks", async () => { + const exec = vi.fn(async () => ({ + exitCode: 0, + stdout: JSON.stringify({ choices: [{ message: { content: "PONG" } }] }), + stderr: "", + })); + await expectOpenAiChatThroughSandbox( + { exec } as unknown as SandboxClient, + "test-sandbox", + "test-model", + [], + "native-curl", + ); + expect(exec).toHaveBeenCalledWith( + "test-sandbox", + registeredCompatibleChatCommand( + "test-sandbox", + "test-model", + JSON.stringify({ + model: "test-model", + messages: [{ role: "user", content: "Reply with exactly one word: PONG" }], + max_tokens: 1024, + }), + ), + expect.objectContaining({ artifactName: "native-curl" }), + ); +}); + +async function operatorFixture( + endpointUrl = "https://operator.example.com/v1", + addresses = ["8.8.8.8"], +) { + let profile: any; + let providerId: string | undefined; + let created = 0; + let attached = false; + const command = vi.fn(async (_executable: string, args: string[], options: any) => { + const result = (stdout = "", exitCode = 0) => ({ stdout, stderr: "", exitCode }); + const operation = args.find((arg) => + ["import", "export", "get", "create", "list", "detach", "delete"].includes(arg), + ); + switch (operation) { + case "import": + profile = JSON.parse(fs.readFileSync(args[args.indexOf("--file") + 1]!, "utf8")); + return result(); + case "export": + return result(JSON.stringify(profile)); + case "get": + return providerId + ? result( + [ + `Name: ${OPERATOR_SWITCH_PROVIDER}`, + `Type: ${profile.id}`, + `Id: ${providerId}`, + "Resource version: 1", + `Credential keys: ${OPERATOR_SWITCH_CREDENTIAL_ENV}`, + "Config keys: ", + ].join("\n"), + ) + : result("provider not found", 1); + case "create": + expect(options.env[OPERATOR_SWITCH_CREDENTIAL_ENV]).toBe("synthetic-key"); + providerId = `fixture-${++created}`; + return result(); + case "list": + return result(attached ? OPERATOR_SWITCH_PROVIDER : ""); + case "detach": + attached = false; + return result(); + case "delete": + providerId = undefined; + return result(); + default: + throw new Error("Unexpected fixture command"); + } + }); + const fixture = await operatorInferenceSwitchFixture( + { command, openshellCommandPath: "openshell" } as unknown as HostCliClient, + { + home: directory, + env: { HOME: directory, OPENSHELL_GATEWAY: "nemoclaw" }, + endpointUrl, + lookup: async () => addresses.map((address) => ({ address, family: 4 })), + credentialValue: "synthetic-key", + sandboxName: "test-sandbox", + model: "test-model", + }, + ); + return { + fixture, + command, + attach: () => { + attached = true; + }, + changeProfile: () => { + profile.endpoints[0].port = 18001; + }, + replaceOutsideFixture: () => { + providerId = "operator-owned-replacement"; + }, + }; +} + +it("creates an inference profile with exact endpoint and environment-only synthetic credentials", async () => { + const { fixture, command } = await operatorFixture(); + const id = await fixture.create(); + expect(id).toBe("fixture-1"); + const imported = command.mock.calls.find((call) => call[1].includes("import"))!; + const profile = JSON.parse( + fs.readFileSync(imported[1][imported[1].indexOf("--file") + 1]!, "utf8"), + ); + expect(profile).toMatchObject({ + inference_capable: true, + endpoints: [ + { + host: "operator.example.com", + port: 443, + allowed_ips: ["8.8.8.8"], + protocol: "rest", + enforcement: "enforce", + }, + ], + credentials: [{ env_vars: [OPERATOR_SWITCH_CREDENTIAL_ENV], auth_style: "bearer" }], + }); + await expect( + resolveRegisteredInferenceProvider({ + adapter: { + getProvider: async () => ({ + ok: true, + value: { + name: OPERATOR_SWITCH_PROVIDER, + type: profile.id, + credentialKeys: [OPERATOR_SWITCH_CREDENTIAL_ENV], + configKeys: [], + revision: { id, resourceVersion: 1 }, + }, + }), + inspectProviderProfile: async () => ({ + ok: true, + value: { + inference: inspectInferenceProviderProfile(JSON.stringify(profile)), + }, + }), + } as unknown as OpenShellProviderAdapter, + gatewayName: "nemoclaw", + sandboxName: "test-sandbox", + providerName: OPERATOR_SWITCH_PROVIDER, + endpointUrl: fixture.endpointUrl, + inferenceApi: "openai-completions", + credentialEnv: OPERATOR_SWITCH_CREDENTIAL_ENV, + }), + ).resolves.toMatchObject({ providerId: id, endpointUrl: "https://operator.example.com/v1" }); + expect(command.mock.calls.map((call) => call[1].join(" ")).join("\n")).not.toContain( + "synthetic-key", + ); + expect(imported[2].env[OPERATOR_SWITCH_CREDENTIAL_ENV]).toBeUndefined(); +}); + +it("replaces only its attached immutable provider and cleans the new unattached identity", async () => { + const { fixture, command, attach } = await operatorFixture(); + await fixture.create(); + attach(); + expect(await fixture.replace()).toBe("fixture-2"); + await fixture.cleanup(); + expect(command.mock.calls.filter((call) => call[1].includes("delete"))).toHaveLength(2); + expect(command.mock.calls.find((call) => call[1].includes("detach"))![1]).toEqual([ + "sandbox", + "provider", + "detach", + "test-sandbox", + OPERATOR_SWITCH_PROVIDER, + "-g", + "nemoclaw", + ]); +}); + +it("refuses replacement and cleanup when another operator owns the current provider identity", async () => { + const { fixture, command, replaceOutsideFixture } = await operatorFixture(); + await fixture.create(); + replaceOutsideFixture(); + await expect(fixture.replace()).rejects.toThrow("unowned"); + await expect(fixture.cleanup()).rejects.toThrow("unowned"); + expect( + command.mock.calls.filter((call) => call[1].includes("delete") || call[1].includes("detach")), + ).toEqual([]); +}); + +it("cleans its provider attachment before deletion after an interrupted selection", async () => { + const { fixture, command, attach } = await operatorFixture(); + await fixture.create(); + attach(); + await fixture.cleanup(); + expect( + command.mock.calls.map((call) => call[1][2]).filter((operation) => operation === "detach"), + ).toEqual(["detach"]); + expect(command.mock.calls.filter((call) => call[1].includes("delete"))).toHaveLength(1); +}); + +it("binds the recorded selection to the original provider, exported profile and live attachment", async () => { + const { fixture, attach } = await operatorFixture(); + const id = await fixture.create(); + attach(); + fs.mkdirSync(path.join(directory, ".nemoclaw")); + fs.writeFileSync( + path.join(directory, ".nemoclaw", "sandboxes.json"), + JSON.stringify({ sandboxes: { "test-sandbox": fixture.expected(id).selected } }), + ); + expect(await fixture.evidence()).toMatchObject(fixture.expected(id)); +}); + +it("does not recompute expected profile identity from a later changed export", async () => { + const { fixture, attach, changeProfile } = await operatorFixture(); + const id = await fixture.create(); + attach(); + fs.mkdirSync(path.join(directory, ".nemoclaw")); + fs.writeFileSync( + path.join(directory, ".nemoclaw", "sandboxes.json"), + JSON.stringify({ sandboxes: { "test-sandbox": fixture.expected(id).selected } }), + ); + changeProfile(); + expect(await fixture.evidence()).not.toMatchObject(fixture.expected(id)); +}); + +it("rejects an HTTP operator fixture before writing its profile", async () => { + const before = fs.readdirSync(directory); + await expect(operatorFixture("http://operator.example.com/v1")).rejects.toThrow("HTTPS"); + expect(fs.readdirSync(directory)).toEqual(before); +}); + +it("rejects private operator endpoint resolution before writing its profile", async () => { + const before = fs.readdirSync(directory); + await expect(operatorFixture(undefined, ["10.1.2.3"])).rejects.toThrow("public endpoint IP pins"); + expect(fs.readdirSync(directory)).toEqual(before); +}); diff --git a/test/e2e/support/routed-private-relay.test.ts b/test/e2e/support/routed-private-relay.test.ts index 94aa198e0d5..5d34f6fe4ee 100644 --- a/test/e2e/support/routed-private-relay.test.ts +++ b/test/e2e/support/routed-private-relay.test.ts @@ -35,6 +35,7 @@ function snapshot(file: string) { } afterEach(() => { vi.restoreAllMocks(); + vi.unstubAllEnvs(); dirs.splice(0).forEach((dir) => fs.rmSync(dir, { recursive: true, force: true })); }); @@ -204,6 +205,8 @@ function runtimeFixture( snapshotNonzero?: boolean; inspectThrows?: boolean; removalNonzero?: boolean; + network?: string; + hostProbeOutput?: string; } = {}, ) { vi.spyOn(RuntimeProviderPrerequisite.prototype, "resolveSandboxResourceHandle").mockResolvedValue( @@ -228,7 +231,10 @@ function runtimeFixture( ); const effects: Record void> = { "container cp": copySummary }; effects[key]?.(); - const stdout = args.at(-1) === "sandbox-id" ? '{"owned-network":{}}' : "172.18.0.3"; + const stdout = + args.at(-1) === "sandbox-id" + ? JSON.stringify({ [options.network ?? "owned-network"]: {} }) + : "172.18.0.3"; return { command: [], exitCode: @@ -245,16 +251,19 @@ function runtimeFixture( }; }); const hostCommand = vi.fn( - async (command: string, args: string[], options?: { timeoutMs?: number }) => ({ + async (command: string, args: string[], runOptions?: { timeoutMs?: number }) => ({ command: [], exitCode: 0, signal: null, timedOut: false, - stdout: execFileSync(command, args, { - encoding: "utf8", - timeout: options?.timeoutMs ?? 10_000, - killSignal: "SIGKILL", - }), + stdout: + command === "bash" && options.hostProbeOutput !== undefined + ? options.hostProbeOutput + : execFileSync(command, args, { + encoding: "utf8", + timeout: runOptions?.timeoutMs ?? 10_000, + killSignal: "SIGKILL", + }), stderr: "", artifacts: { stdout: "", stderr: "", result: "" }, }), @@ -271,6 +280,36 @@ function runtimeFixture( }), }; } +describe("host-network supervisor private endpoint", () => { + it.each(["docker", "podman"])( + "uses the existing host listener for %s without starting a network-none relay", + async (provider) => { + vi.stubEnv("NEMOCLAW_GATEWAY_RUNTIME", provider); + vi.stubEnv("OPENSHELL_PODMAN_SOCKET", "/tmp/podman.sock"); + const f = runtimeFixture({ network: "none", hostProbeOutput: "route 10.20.30.40\n" }); + const endpoint = await f.start(); + expect(endpoint).toMatchObject({ address: "10.20.30.40", port: 1234 }); + await endpoint.close(); + expect(f.command.mock.calls.map(([args]) => args.slice(0, 2))).toEqual([ + ["container", "inspect"], + ]); + expect(f.hostCommand).toHaveBeenCalledExactlyOnceWith("bash", expect.any(Array), { + artifactName: "routed-private-host-address", + env: expect.any(Object), + timeoutMs: 30_000, + }); + }, + ); + + it.each(["1.1.1.1", "127.0.0.1", "169.254.2.2"])( + "rejects host address %s without creating or removing containers", + async (address) => { + const f = runtimeFixture({ network: "none", hostProbeOutput: `route ${address}\n` }); + await expect(f.start()).rejects.toThrow("operator-trustable private"); + expect(f.command).toHaveBeenCalledTimes(1); + }, + ); +}); describe("relay diagnostic cleanup ownership", () => { it.each([{ snapshotThrows: true }, { snapshotNonzero: true }, {}])( "always removes the exact relay after snapshot outcome %j", diff --git a/test/e2e/support/runtime-provider-fixture.test.ts b/test/e2e/support/runtime-provider-fixture.test.ts index 8d6fed169d3..ee3c28c8597 100644 --- a/test/e2e/support/runtime-provider-fixture.test.ts +++ b/test/e2e/support/runtime-provider-fixture.test.ts @@ -169,7 +169,7 @@ describe("configured E2E runtime provider fixture", () => { "--filter", "label=openshell.ai/sandbox-name=alpha", "--format", - "{{.ID}}", + '{{.ID}}\t{{.Label "openshell.ai/isolation-role"}}', ], expect.objectContaining({ artifactName: "privileged-id-resource" }), ); @@ -180,6 +180,7 @@ describe("configured E2E runtime provider fixture", () => { "--url", "unix:///run/user/1001/podman/podman.sock", "container", + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "exec", "--user", "root", @@ -190,4 +191,44 @@ describe("configured E2E runtime provider fixture", () => { expect.objectContaining({ artifactName: "privileged-id" }), ); }); + + it.each(["docker", "podman"])( + "selects the workload without its %s supervisor", + async (provider) => { + const workload = "a".repeat(64); + const supervisor = "b".repeat(64); + const command = vi + .fn() + .mockResolvedValue( + successfulProbeWithStdout(`${workload}\tsandbox\n${supervisor}\tsupervisor\n`), + ); + const runtime = new RuntimeProviderPrerequisite({ command }, undefined, { + NEMOCLAW_GATEWAY_RUNTIME: provider, + OPENSHELL_PODMAN_SOCKET: "/run/user/1001/podman/podman.sock", + }); + await expect(runtime.resolveSandboxResourceHandle("alpha")).resolves.toBe(workload); + expect(command).toHaveBeenCalledOnce(); + }, + ); + + it.each([ + "", + `${"a".repeat(64)}\n${"c".repeat(64)}`, + "--privileged", + `${"a".repeat(64)}\tsupervisor`, + `${"a".repeat(64)}\tunknown`, + `${"a".repeat(64)}\tsandbox\textra`, + ])( + "refuses root execution for missing, ambiguous, or malformed workloads: %j", + async (stdout) => { + const command = vi + .fn() + .mockResolvedValue(successfulProbeWithStdout(stdout)); + const runtime = new RuntimeProviderPrerequisite({ command }, undefined, { + NEMOCLAW_GATEWAY_RUNTIME: "docker", + }); + await expect(runtime.execSandboxAsRoot("alpha", ["id", "-u"])).rejects.toThrow(/resource/); + expect(command).toHaveBeenCalledOnce(); + }, + ); }); diff --git a/test/e2e/support/security-container-observation.test.ts b/test/e2e/support/security-container-observation.test.ts new file mode 100644 index 00000000000..e01f18daf2b --- /dev/null +++ b/test/e2e/support/security-container-observation.test.ts @@ -0,0 +1,190 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; +import { observeSecurityContainerPair } from "../fixtures/security-posture-containers.ts"; +import { OPENSHELL_V017_QUALIFICATION } from "../fixtures/openshell-v0116-qualification.ts"; + +const WORKLOAD = "a".repeat(64); +const SUPERVISOR = "b".repeat(64); +const IMAGE = "c".repeat(64); + +function harness(provider = "docker") { + const common = { + running: true, + paused: false, + pid: 100, + startedAt: "2026-10-15T00:00:00Z", + privileged: false, + capAdd: null, + capDrop: ["ALL"], + securityOpt: ["no-new-privileges:true"], + name: "sandbox", + sandboxId: "sandbox-id", + namespace: "", + managed: provider === "docker" ? "openshell" : "true", + effectiveCaps: null, + boundingCaps: null, + }; + const workload: Record = { + ...common, + id: WORKLOAD, + image: `sha256:${"d".repeat(64)}`, + user: "998:999", + role: "sandbox", + entrypoint: [ + provider === "docker" + ? "/.openshell/runtime/openshell-sandbox" + : "/opt/openshell/bin/openshell-sandbox", + ], + command: ["--bootstrap", "/.openshell/channel/sandbox/bootstrap.json"], + network: "none", + readonlyRootfs: false, + }; + const supervisor: Record = { + ...common, + id: SUPERVISOR, + image: `sha256:${IMAGE}`, + user: provider === "docker" ? "65534:65534" : "998:999", + role: "supervisor", + entrypoint: ["/openshell-supervisor"], + network: "host", + readonlyRootfs: provider === "docker", + }; + const state = { discovery: `${SUPERVISOR}\n`, health: 0, reads: 0, restart: false }; + const capture = vi.fn((args: readonly string[]) => { + const operation = args[0]; + expect(["container", "ps", "image", "exec"]).toContain(operation); + state.reads += Number(operation === "container"); + const observation = args.at(-1) === WORKLOAD ? workload : supervisor; + const stdoutByOperation: Record = { + container: JSON.stringify( + state.restart && state.reads > 2 ? { ...observation, pid: 200 } : observation, + ), + ps: state.discovery, + image: JSON.stringify({ id: `sha256:${IMAGE}` }), + exec: "", + }; + const stdout = stdoutByOperation[operation ?? ""] ?? ""; + return { status: args[0] === "exec" ? state.health : 0, stdout, stderr: "not-for-diagnostics" }; + }); + const run = () => + observeSecurityContainerPair( + capture, + { providerId: provider, resourceHandle: WORKLOAD }, + "sandbox", + { uid: 998, gid: 999 }, + ); + return { capture, workload, supervisor, state, run }; +} + +describe("OpenShell security container observations", () => { + it.each(["docker", "podman"])( + "binds %s launch metadata and native health to the pinned pair", + (provider) => { + const { capture, run } = harness(provider); + const result = run(); + expect(result).toMatchObject({ + evidence: "container-launch-configuration-and-supervisor-health", + workload: { id: WORKLOAD, user: "998:999" }, + supervisor: { + id: SUPERVISOR, + image: IMAGE, + user: provider === "docker" ? "65534:65534" : "998:999", + }, + }); + expect(capture.mock.calls.map(([args]) => args[0])).toEqual([ + "container", + "ps", + "container", + "image", + "exec", + "container", + "container", + ]); + expect(capture.mock.calls[3]?.[0].at(-1)).toBe(OPENSHELL_V017_QUALIFICATION.supervisorImage); + expect(capture.mock.calls[4]?.[0]).toEqual([ + "exec", + SUPERVISOR, + "/openshell-supervisor", + "health", + "--socket", + provider === "docker" + ? "/run/openshell/health.sock" + : "/run/openshell/supervisor-health.sock", + ]); + const formats = capture.mock.calls + .filter(([args]) => args[0] === "container") + .map(([args]) => args[3]); + expect(formats.join(" ")).not.toMatch(/\.Env|\.Args|\.Mounts|\.Health|\.Config\}\}/u); + expect(formats[1]).not.toContain(".Config.Cmd"); + }, + ); + + it.each([ + ["root workload", "workload", "user", "0:0"], + ["root supervisor", "supervisor", "user", "0:0"], + ["stopped supervisor", "supervisor", "running", false], + ["paused workload", "workload", "paused", true], + ["wrong namespace", "supervisor", "namespace", "other"], + ["wrong sandbox ID", "supervisor", "sandboxId", "other"], + ["wrong image", "supervisor", "image", `sha256:${"e".repeat(64)}`], + ["wrong entrypoint", "workload", "entrypoint", ["/bin/sh"]], + ["wrong bootstrap", "workload", "command", ["--bootstrap", "/tmp/untrusted.json"]], + ["privileged supervisor", "supervisor", "privileged", true], + ["added capability", "supervisor", "capAdd", ["SYS_ADMIN"]], + ["missing capability drop", "workload", "capDrop", []], + ["missing no-new-privileges", "supervisor", "securityOpt", []], + ["host-network workload", "workload", "network", "host"], + ["writable Docker supervisor root", "supervisor", "readonlyRootfs", false], + ] as const)("rejects %s", (_name, target, field, value) => { + const fixture = harness(); + fixture[target][field] = value; + expect(fixture.run).toThrow(/container security observation/u); + }); + + it("rejects nonempty effective Podman capabilities", () => { + const fixture = harness("podman"); + fixture.supervisor.effectiveCaps = ["CAP_NET_ADMIN"]; + expect(fixture.run).toThrow(/container security observation/u); + }); + + it.each(["", `${SUPERVISOR}\n${"e".repeat(64)}\n`, WORKLOAD, "--host=attacker"])( + "rejects missing or ambiguous companion identity: %s", + (discovery) => { + const fixture = harness(); + fixture.state.discovery = discovery; + expect(fixture.run).toThrow(/container security observation/u); + expect(fixture.capture.mock.calls).toHaveLength(2); + }, + ); + + it.each(["health", "restart"] as const)( + "rejects %s failure without printing engine diagnostics", + (failure) => { + const fixture = harness(); + Object.assign(fixture.state, failure === "health" ? { health: 1 } : { restart: true }); + expect(fixture.run).toThrow("OpenShell container security observation failed"); + }, + ); + + it.each(["malformed", "error", "oversized"])( + "rejects %s metadata without exposing output", + (failure) => { + const capture = vi.fn(() => ({ + status: failure === "error" ? 1 : 0, + stdout: failure === "oversized" ? "x".repeat(16_384) : "sensitive-invalid-json", + stderr: "secret", + })); + expect(() => + observeSecurityContainerPair( + capture, + { providerId: "docker", resourceHandle: WORKLOAD }, + "sandbox", + { uid: 998, gid: 999 }, + ), + ).toThrow(/OpenShell container security observation/u); + expect(capture).toHaveBeenCalledOnce(); + }, + ); +}); diff --git a/test/e2e/support/security-posture.test.ts b/test/e2e/support/security-posture.test.ts index a1e11a7b8e8..bedc8d35b2e 100644 --- a/test/e2e/support/security-posture.test.ts +++ b/test/e2e/support/security-posture.test.ts @@ -17,10 +17,8 @@ import type { HostCliClient } from "../fixtures/clients/host.ts"; import type { SandboxClient } from "../fixtures/clients/sandbox.ts"; import { assertSecurityPosture, - OPENSHELL_SUPERVISOR_CAPABILITY_MASK, parseCapabilitySurfaceReport, parseExpectedOpenShellVersion, - PODMAN_OPENSHELL_SUPERVISOR_CAPABILITY_MASK, type ProcessSecurityIdentity, parseSplitProcessSecurityReport, SPLIT_PROCESS_SECURITY_PROBE, @@ -31,9 +29,34 @@ import { validateSplitProcessSecurityReport, } from "../fixtures/security-posture.ts"; import type { ShellProbeResult } from "../fixtures/shell-probe.ts"; +import type { SecurityContainerPair } from "../fixtures/security-posture-containers.ts"; + +const CONTAINER_PAIR: SecurityContainerPair = { + evidence: "container-launch-configuration-and-supervisor-health", + workload: { + id: "a".repeat(64), + image: "b".repeat(64), + startedAt: "2026-10-15T00:00:00Z", + pid: 100, + user: "998:998", + entrypoint: ["/.openshell/runtime/openshell-sandbox"], + sandboxId: "sandbox-id", + namespace: "default", + }, + supervisor: { + id: "c".repeat(64), + image: "d".repeat(64), + startedAt: "2026-10-15T00:00:00Z", + pid: 101, + user: "65534:65534", + entrypoint: ["/openshell-supervisor"], + sandboxId: "sandbox-id", + namespace: "default", + }, +}; const ZERO_CAPABILITIES = "0000000000000000"; -const SUPERVISOR_EXECUTABLE = "/opt/openshell/bin/openshell-sandbox"; +const BOUNDARY_EXECUTABLE = "/.openshell/runtime/openshell-sandbox"; const SANDBOX_NAME = "secure-sandbox"; const RESOURCE_HANDLE = "opaque-runtime-resource"; const MANAGED_IMAGE_REVISION = "a".repeat(40); @@ -46,10 +69,12 @@ import pwd import sys import types -probe, proc_root_arg, censuses_json = sys.argv[1:4] +probe, proc_root_arg, censuses_json, options_json = sys.argv[1:5] +options = json.loads(options_json) real_path = type(pathlib.Path()) real_scandir = os.scandir real_stat = os.stat +real_readlink = os.readlink proc_root = real_path(proc_root_arg) censuses = json.loads(censuses_json) scan_index = 0 @@ -91,6 +116,20 @@ def controlled_stat(path, *args, **kwargs): os.symlink(target, link, target_is_directory=True) return real_stat(path, *args, **kwargs) +def controlled_readlink(path, *args, **kwargs): + if str(path) == str(proc_root / "1" / "exe"): + if options.get("boundaryRead") == "missing": + raise FileNotFoundError(2, "fixture missing process") + if options.get("boundaryRead") == "readable": + return "/.openshell/runtime/openshell-sandbox" + raise PermissionError(13, "fixture permission denied") + if options.get("denyChildRead"): + raise PermissionError(13, "fixture child permission denied") + return real_readlink(path, *args, **kwargs) + +os.readlink = controlled_readlink +os.getresuid = lambda: tuple(options.get("observerUids", [1000, 1000, 1000])) +os.getresgid = lambda: tuple(options.get("observerGids", [1000, 1000, 1000])) pathlib.Path = controlled_path os.scandir = controlled_scandir os.stat = controlled_stat @@ -147,6 +186,12 @@ function validNemoclawStartProcess({ }; } +function setProcessIdentity(process: ProcessSecurityIdentity, id: number): void { + process.status.uid = repeatedId(id); + process.status.gid = repeatedId(id); + process.status.groups = [String(id)]; +} + function validReport(): SplitProcessSecurityReport { const childSupervisor = validNemoclawStartProcess(); return { @@ -155,26 +200,26 @@ function validReport(): SplitProcessSecurityReport { observedProcEntries: 12, sandboxGid: 1000, sandboxUid: 1000, - supervisor: { - argv: [SUPERVISOR_EXECUTABLE, "--workdir", "/sandbox"], - executable: SUPERVISOR_EXECUTABLE, + workloadBoundary: { + argv: [BOUNDARY_EXECUTABLE, "--bootstrap", "/.openshell/channel/sandbox/bootstrap.json"], + executable: null, pid: 1, ppid: 0, state: "S", startTime: "101", status: { capAmb: ZERO_CAPABILITIES, - capBnd: OPENSHELL_SUPERVISOR_CAPABILITY_MASK, - capEff: OPENSHELL_SUPERVISOR_CAPABILITY_MASK, + capBnd: ZERO_CAPABILITIES, + capEff: ZERO_CAPABILITIES, capInh: ZERO_CAPABILITIES, - capPrm: OPENSHELL_SUPERVISOR_CAPABILITY_MASK, - gid: repeatedId(0), - groups: ["0", "1000"], + capPrm: ZERO_CAPABILITIES, + gid: repeatedId(1000), + groups: ["1000"], noNewPrivs: "1", - uid: repeatedId(0), + uid: repeatedId(1000), }, }, - version: 2, + version: 3, }; } @@ -220,14 +265,25 @@ function writeProcProcess(root: string, process: ProcessSecurityIdentity): void path.join(processDirectory, "cmdline"), Buffer.from(`${process.argv.join("\0")}\0`, "utf8"), ); - symlinkSync(process.executable, path.join(processDirectory, "exe")); + process.executable !== null && + symlinkSync(process.executable, path.join(processDirectory, "exe")); } -function runSplitProcessProbeWithCensuses(censuses: ControlledCensus[]) { +function runSplitProcessProbeWithCensuses( + censuses: ControlledCensus[], + options: { + boundaryRead?: "missing" | "readable"; + boundaryArgv?: string[]; + denyChildRead?: boolean; + observerUids?: number[]; + observerGids?: number[]; + } = {}, +) { const procRoot = mkdtempSync(path.join(os.tmpdir(), "nemoclaw-security-posture-proc-")); const report = validReport(); + report.workloadBoundary.argv = options.boundaryArgv ?? report.workloadBoundary.argv; try { - writeProcProcess(procRoot, report.supervisor); + writeProcProcess(procRoot, report.workloadBoundary); const controlledCensuses = censuses.map((census, index) => { const snapshotRoot = path.join(procRoot, "snapshots", String(index)); const stableRoot = path.join(snapshotRoot, "stable"); @@ -255,6 +311,7 @@ function runSplitProcessProbeWithCensuses(censuses: ControlledCensus[]) { SPLIT_PROCESS_SECURITY_PROBE, procRoot, JSON.stringify(controlledCensuses), + JSON.stringify(options), ], { encoding: "utf8", killSignal: "SIGKILL", timeout: 10_000 }, ); @@ -342,12 +399,12 @@ afterEach(() => vi.unstubAllEnvs()); describe("security posture fixture", () => { it("accepts only the exact stable OpenShell version token", () => { - expect(parseExpectedOpenShellVersion(successfulProbe("openshell 0.0.116\n"))).toBe("0.0.116"); - expect(() => - parseExpectedOpenShellVersion(successfulProbe("openshell 0.0.116-rc.1\n")), - ).toThrow(/expected OpenShell 0\.0\.116/u); - expect(() => parseExpectedOpenShellVersion(successfulProbe("openshell 0.0.116+dev\n"))).toThrow( - /expected OpenShell 0\.0\.116/u, + expect(parseExpectedOpenShellVersion(successfulProbe("openshell 0.1.7\n"))).toBe("0.1.7"); + expect(() => parseExpectedOpenShellVersion(successfulProbe("openshell 0.1.7-rc.1\n"))).toThrow( + /expected OpenShell 0\.1\.7/u, + ); + expect(() => parseExpectedOpenShellVersion(successfulProbe("openshell 0.1.7+dev\n"))).toThrow( + /expected OpenShell 0\.1\.7/u, ); }); @@ -377,10 +434,74 @@ describe("security posture fixture", () => { expect(result.error, "python3 is required to run the embedded probe").toBeUndefined(); expect(result.status, result.stderr).toBe(0); const report = parseSplitProcessSecurityReport(result.stdout); + expect(report.workloadBoundary.executable).toBeNull(); expect(report.childSupervisors.map(({ pid }) => pid)).toEqual([42, 43]); expect(report.observedChildSupervisors.map(({ pid }) => pid)).toEqual([42, 43]); }); + it.each([ + { + options: { observerUids: [0, 0, 0] }, + error: /observer must run as the sandbox owner/u, + name: "root observer", + }, + { + options: { observerUids: [1000, 1000, 0] }, + error: /observer must run as the sandbox owner/u, + name: "saved root identity", + }, + { + options: { observerGids: [1001, 1001, 1001] }, + error: /observer must run as the sandbox owner/u, + name: "wrong observer group", + }, + { + options: { boundaryRead: "missing" as const }, + error: /FileNotFoundError/u, + name: "missing PID 1 executable", + }, + { + options: { denyChildRead: true }, + error: /child permission denied/u, + name: "denied child executable", + }, + ])("rejects $name without fabricating a process report", ({ options, error }) => { + const result = runSplitProcessProbeWithCensuses( + [{ children: [validNemoclawStartProcess()] }], + options, + ); + expect(result.status).toBe(1); + expect(result.stderr).toMatch(error); + expect(result.stdout).toBe(""); + }); + + it("rejects a readable PID 1 executable instead of claiming nondumpable protection", () => { + const result = runSplitProcessProbeWithCensuses([{ children: [validNemoclawStartProcess()] }], { + boundaryRead: "readable", + }); + expect(result.status, result.stderr).toBe(0); + expect(() => parseSplitProcessSecurityReport(result.stdout)).toThrow( + /must deny executable-link access/u, + ); + }); + + it("rejects an unobserved child executable", () => { + const report = validReport(); + report.childSupervisors[0]!.executable = null; + expect(() => validateSplitProcessSecurityReport(report)).toThrow( + /expected the system Bash executable/u, + ); + }); + + it("rejects unexpected boundary arguments without printing them", () => { + const result = runSplitProcessProbeWithCensuses([{ children: [validNemoclawStartProcess()] }], { + boundaryArgv: [BOUNDARY_EXECUTABLE, "credential=DO_NOT_LOG"], + }); + expect(result.status).toBe(1); + expect(result.stderr).toMatch(/unexpected OpenShell workload boundary command/u); + expect(result.stdout + result.stderr).not.toContain("DO_NOT_LOG"); + }); + it("separates the final stable census from every process observed during acquisition", () => { const direct = validNemoclawStartProcess(); const firstDescendant = validNemoclawStartProcess({ @@ -597,10 +718,9 @@ describe("security posture fixture", () => { }); }); - it("accepts the expected OpenShell supervisor and non-root nemoclaw-start child supervisor", () => { + it("accepts the non-root workload boundary and nemoclaw-start child supervisor", () => { const report = validReport(); - expect(OPENSHELL_SUPERVISOR_CAPABILITY_MASK).toBe("00000004a82c35fb"); expect(validateSplitProcessSecurityReport(report)).toEqual(report); expect(parseSplitProcessSecurityReport(JSON.stringify(report))).toEqual(report); }); @@ -620,18 +740,18 @@ describe("security posture fixture", () => { }, ); - it("accepts the exact OpenShell supervisor groups in either report order", () => { - const report = validReport(); - report.supervisor.status.groups.reverse(); - - expect(validateSplitProcessSecurityReport(report)).toEqual(report); + it.each(["unknown", "toString"])("rejects unreviewed runtime provider %s", (providerId) => { + expect(() => validateSplitProcessSecurityReport(validReport(), providerId)).toThrow( + /no reviewed workload boundary/u, + ); }); - it("accepts a root-only OpenShell supervisor without an engine-added sandbox group", () => { + it("rejects a workload boundary with a root supplementary group", () => { const report = validReport(); - report.supervisor.status.groups = ["0"]; - - expect(validateSplitProcessSecurityReport(report)).toEqual(report); + report.workloadBoundary.status.groups = ["0"]; + expect(() => validateSplitProcessSecurityReport(report)).toThrow( + /Groups expected exactly 1000/u, + ); }); it.each([ @@ -662,7 +782,9 @@ describe("security posture fixture", () => { report.observedProcEntries = observedProcEntries; report.sandboxUid = 998; report.sandboxGid = sandboxGid; - report.supervisor.status.groups = ["0", String(sandboxGid)]; + report.workloadBoundary.status.uid = repeatedId(998); + report.workloadBoundary.status.gid = repeatedId(sandboxGid); + report.workloadBoundary.status.groups = [String(sandboxGid)]; setCurrentChildSupervisors( report, processes.map(({ pid, ppid, startTime }) => @@ -690,109 +812,109 @@ describe("security posture fixture", () => { { error: /expected pid=1 ppid=0/u, mutate: (report) => { - report.supervisor.ppid = 2; + report.workloadBoundary.ppid = 2; }, name: "parent process", }, { - error: /does not have the expected OpenShell supervisor command/u, + error: /must deny executable-link access/u, mutate: (report) => { - report.supervisor.executable = "/usr/bin/bash"; + report.workloadBoundary.executable = "/usr/bin/bash"; }, name: "executable", }, { - error: /does not have the expected OpenShell supervisor command/u, + error: /does not have the expected OpenShell workload boundary command/u, mutate: (report) => { - report.supervisor.argv.push("--unexpected"); + report.workloadBoundary.argv.push("--unexpected"); }, name: "command arguments", }, { - error: /supervisor Uid expected 0/u, + error: /workload boundary Uid expected 1000/u, mutate: (report) => { - report.supervisor.status.uid = repeatedId(1000); + report.workloadBoundary.status.uid = repeatedId(0); }, name: "user identity", }, { - error: /supervisor Gid expected 0/u, + error: /workload boundary Gid expected 1000/u, mutate: (report) => { - report.supervisor.status.gid = repeatedId(1000); + report.workloadBoundary.status.gid = repeatedId(0); }, name: "group identity", }, { - error: /supervisor Groups expected exactly 0 or 0 1000/u, + error: /workload boundary Groups expected exactly 1000/u, mutate: (report) => { - report.supervisor.status.groups = ["0", "44"]; + report.workloadBoundary.status.groups = ["0", "44"]; }, name: "wrong sandbox supplementary group", }, { - error: /supervisor Groups expected exactly 0 or 0 1000/u, + error: /workload boundary Groups expected exactly 1000/u, mutate: (report) => { - report.supervisor.status.groups = ["0", "1000", "44"]; + report.workloadBoundary.status.groups = ["0", "1000", "44"]; }, name: "extra supplementary group", }, { - error: /supervisor Groups expected exactly 0 or 0 1000/u, + error: /workload boundary Groups expected exactly 1000/u, mutate: (report) => { - report.supervisor.status.groups = ["0", "0"]; + report.workloadBoundary.status.groups = ["0", "0"]; }, name: "duplicate supplementary group", }, { - error: /supervisor\.state must be one of D, R, S/u, + error: /workloadBoundary\.state must be one of D, R, S/u, mutate: (report) => { - report.supervisor.state = "T"; + report.workloadBoundary.state = "T"; }, name: "process state", }, { - error: /supervisor CapInh drifted/u, + error: /workload boundary\.capInh expected 0/u, mutate: (report) => { - report.supervisor.status.capInh = "0000000000000001"; + report.workloadBoundary.status.capInh = "0000000000000001"; }, name: "inheritable capability", }, { - error: /supervisor capPrm expected/u, + error: /workload boundary\.capPrm expected/u, mutate: (report) => { - report.supervisor.status.capPrm = ZERO_CAPABILITIES; + report.workloadBoundary.status.capPrm = "0000000000000001"; }, name: "permitted capability", }, { - error: /supervisor capEff expected/u, + error: /workload boundary\.capEff expected/u, mutate: (report) => { - report.supervisor.status.capEff = ZERO_CAPABILITIES; + report.workloadBoundary.status.capEff = "0000000000000001"; }, name: "effective capability", }, { - error: /supervisor capBnd expected/u, + error: /workload boundary\.capBnd expected/u, mutate: (report) => { - report.supervisor.status.capBnd = ZERO_CAPABILITIES; + report.workloadBoundary.status.capBnd = "0000000000000001"; }, name: "bounding capability", }, { - error: /supervisor CapAmb drifted/u, + error: /workload boundary\.capAmb expected 0/u, mutate: (report) => { - report.supervisor.status.capAmb = "0000000000000001"; + report.workloadBoundary.status.capAmb = "0000000000000001"; }, name: "ambient capability", }, { - error: /supervisor expected NoNewPrivs=1/u, + error: /workload boundary expected NoNewPrivs=1/u, mutate: (report) => { - report.supervisor.status.noNewPrivs = "0"; + report.workloadBoundary.status.noNewPrivs = "0"; }, name: "NoNewPrivs", }, - ])("rejects OpenShell supervisor $name drift", ({ error, mutate }) => { + ])("rejects OpenShell workload boundary $name drift", ({ error, mutate }) => { const report = validReport(); mutate(report); @@ -863,7 +985,7 @@ describe("security posture fixture", () => { it.each([ { - error: /must not replace the OpenShell supervisor at PID 1/u, + error: /must not replace the OpenShell workload boundary at PID 1/u, mutate: (report) => { report.childSupervisors[0]!.pid = 1; }, @@ -960,10 +1082,10 @@ describe("security posture fixture", () => { it("rejects malformed and overflowing split-process reports", () => { expect(() => parseSplitProcessSecurityReport("not-json")).toThrow(/emitted invalid JSON/u); expect(() => validateSplitProcessSecurityReport({ childSupervisors: [] })).toThrow( - /version must be 2/u, + /version must be 3/u, ); expect(() => validateSplitProcessSecurityReport({ ...validReport(), version: 1 })).toThrow( - /version must be 2/u, + /version must be 3/u, ); const malformed = { ...validReport(), childSupervisors: "one" }; @@ -1024,34 +1146,51 @@ describe("security posture fixture", () => { }); it.each([ - { capabilityMask: OPENSHELL_SUPERVISOR_CAPABILITY_MASK, providerId: "docker" }, - { capabilityMask: PODMAN_OPENSHELL_SUPERVISOR_CAPABILITY_MASK, providerId: "podman" }, - ])( - "checks the split-process report through the selected $providerId provider", - async ({ capabilityMask, providerId }) => { + ["docker", 0, 0, "local", 0], + ["docker", 7, 0, "local", 7], + ["docker", 0, 9, "local", 9], + ["docker", 0, 0, "remote", 1], + ["podman", 0, 0, "local", 0], + ["podman", 7, 0, "local", 7], + ["podman", 0, 9, "local", 9], + ["podman", 0, 0, "remote", 1], + ] as const)( + "checks %s security with validation status %s, read status %s and mode %s", + async (providerId, validateStatus, readStatus, mode, expectedStatus) => { + const executable = + providerId === "docker" ? BOUNDARY_EXECUTABLE : "/opt/openshell/bin/openshell-sandbox"; vi.stubEnv("NEMOCLAW_E2E_SECURITY_POSTURE", "1"); vi.stubEnv("NEMOCLAW_E2E_EXPECT_OPENSHELL_SPLIT_PROCESS", "1"); const report = validReport(); - report.supervisor.status.capBnd = capabilityMask; - report.supervisor.status.capEff = capabilityMask; - report.supervisor.status.capPrm = capabilityMask; + report.workloadBoundary.argv[0] = executable; + report.sandboxUid = report.sandboxGid = 998; + setProcessIdentity(report.workloadBoundary, 998); + setProcessIdentity(report.childSupervisors[0]!, 998); const directChildSupervisor = report.childSupervisors[0]!; setCurrentChildSupervisors(report, [ validNemoclawStartProcess({ pid: 43, ppid: directChildSupervisor.pid, startTime: "203", + sandboxUid: 998, + sandboxGid: 998, }), directChildSupervisor, ]); const command = vi .fn() .mockResolvedValueOnce(successfulProbe("uid=1000 gid=1000\n")) - .mockResolvedValueOnce(successfulProbe("openshell 0.0.116\n")) - .mockResolvedValueOnce(successfulProbe(capabilitySurfaceProof("connect"))); + .mockResolvedValueOnce(successfulProbe("openshell 0.1.7\n")) + .mockResolvedValueOnce( + successfulProbe( + capabilitySurfaceProof("connect").replace("uid=1000 gid=1000", "uid=998 gid=998"), + ), + ); const execShell = vi.fn(async (_name, script) => String(script).includes("surface=exec") - ? successfulProbe(capabilitySurfaceProof("exec")) + ? successfulProbe( + capabilitySurfaceProof("exec").replace("uid=1000 gid=1000", "uid=998 gid=998"), + ) : successfulProbe(), ); const host = { @@ -1070,19 +1209,23 @@ describe("security posture fixture", () => { stdout: Buffer.from(JSON.stringify(report), "utf8"), stderr: Buffer.alloc(0), })); + const inspectContainerPair = vi.fn(() => CONTAINER_PAIR); const summary = await assertSecurityPosture(host, sandbox, SANDBOX_NAME, "openclaw", { environment: inlineManagedImageCatalogEnvironment(), + inspectContainerPair, executePrivilegedCommand, resolvePrivilegedTarget, }); expect(summary).toEqual({ + processScope: "workload-container", + containerPair: CONTAINER_PAIR, capabilitySurfaces: { connect: { surface: "connect", - uid: 1000, - gid: 1000, + uid: 998, + gid: 998, capInh: ZERO_CAPABILITIES, capPrm: ZERO_CAPABILITIES, capEff: ZERO_CAPABILITIES, @@ -1091,8 +1234,8 @@ describe("security posture fixture", () => { }, entrypoint: { surface: "entrypoint", - uid: 1000, - gid: 1000, + uid: 998, + gid: 998, capInh: ZERO_CAPABILITIES, capPrm: ZERO_CAPABILITIES, capEff: ZERO_CAPABILITIES, @@ -1101,8 +1244,8 @@ describe("security posture fixture", () => { }, exec: { surface: "exec", - uid: 1000, - gid: 1000, + uid: 998, + gid: 998, capInh: ZERO_CAPABILITIES, capPrm: ZERO_CAPABILITIES, capEff: ZERO_CAPABILITIES, @@ -1116,22 +1259,35 @@ describe("security posture fixture", () => { runtimeProxyEnvLocked: true, runtimeVersions: { managedImageRevision: MANAGED_IMAGE_REVISION, - openshell: "0.0.116", + openshell: "0.1.7", }, splitProcess: { childSupervisor: directChildSupervisor, - supervisor: report.supervisor, + workloadBoundary: report.workloadBoundary, }, startupLogClean: true, }); expect(command).toHaveBeenCalledTimes(3); expect(command.mock.calls[2]?.[1]?.[3]).toMatch(/\nexit 0$/u); expect(resolvePrivilegedTarget).toHaveBeenCalledTimes(2); + expect(inspectContainerPair.mock.calls).toEqual([ + [ + SANDBOX_NAME, + { providerId, resourceHandle: RESOURCE_HANDLE }, + { uid: 998, gid: 998, workdir: "/sandbox" }, + ], + [ + SANDBOX_NAME, + { providerId, resourceHandle: RESOURCE_HANDLE }, + { uid: 998, gid: 998, workdir: "/sandbox" }, + ], + ]); expect(executePrivilegedCommand).toHaveBeenCalledWith( SANDBOX_NAME, ["/usr/bin/python3", "-I", "-c", SPLIT_PROCESS_SECURITY_PROBE], { expectedResourceHandle: RESOURCE_HANDLE, + sandboxUser: "998:998", sanitizeEnvironment: true, timeout: 30_000, }, @@ -1144,16 +1300,105 @@ describe("security posture fixture", () => { expect( sandboxScripts.some((script) => script.includes("cannot modify config inside the sandbox")), ).toBe(false); + + const configurationScript = sandboxScripts.find((script) => + script.includes("openclaw config validate"), + )!; + const fixtureDirectory = mkdtempSync(path.join(os.tmpdir(), "nemoclaw-config-proof-")); + try { + const proxyFile = path.join(fixtureDirectory, "proxy-env.sh"); + writeFileSync( + proxyFile, + `openclaw() {\nif [ "$2" = validate ]; then return ${validateStatus}; fi\nprintf '%s\\n' '${mode}'\nreturn ${readStatus}\n}\n`, + ); + const result = spawnSync( + "sh", + [ + "-c", + configurationScript.replace("/tmp/nemoclaw-proxy-env.sh", '"$1"'), + "security-posture-config-proof", + proxyFile, + ], + { encoding: "utf8", timeout: 5_000 }, + ); + expect(result.error).toBeUndefined(); + expect(result.status, `validate=${validateStatus} read=${readStatus} mode=${mode}`).toBe( + expectedStatus, + ); + } finally { + rmSync(fixtureDirectory, { recursive: true, force: true }); + } + + command + .mockReset() + .mockResolvedValueOnce(successfulProbe("uid=1000 gid=1000\n")) + .mockResolvedValueOnce(successfulProbe("openshell 0.1.7\n")); + execShell.mockClear(); + inspectContainerPair + .mockReset() + .mockReturnValueOnce(CONTAINER_PAIR) + .mockReturnValueOnce({ + ...CONTAINER_PAIR, + supervisor: { ...CONTAINER_PAIR.supervisor, pid: 999 }, + }); + await expect( + assertSecurityPosture(host, sandbox, SANDBOX_NAME, "openclaw", { + environment: inlineManagedImageCatalogEnvironment(), + inspectContainerPair, + executePrivilegedCommand, + resolvePrivilegedTarget, + }), + ).rejects.toThrow(/container pair changed/u); + expect(execShell).not.toHaveBeenCalled(); }, ); + it("rejects a report for a different managed-image owner before child commands", async () => { + vi.stubEnv("NEMOCLAW_E2E_SECURITY_POSTURE", "1"); + vi.stubEnv("NEMOCLAW_E2E_EXPECT_OPENSHELL_SPLIT_PROCESS", "1"); + const command = vi + .fn() + .mockResolvedValueOnce(successfulProbe("uid=1000 gid=1000\n")) + .mockResolvedValueOnce(successfulProbe("openshell 0.1.7\n")); + const execShell = vi.fn(); + const executePrivilegedCommand = vi.fn(() => ({ + status: 0, + signal: null, + stdout: Buffer.from(JSON.stringify(validReport())), + stderr: Buffer.alloc(0), + })); + await expect( + assertSecurityPosture( + { command, openshellCommandPath: "/tmp/openshell" } as unknown as HostCliClient, + { execShell } as unknown as SandboxClient, + SANDBOX_NAME, + "hermes", + { + environment: inlineManagedImageCatalogEnvironment(), + inspectContainerPair: () => CONTAINER_PAIR, + executePrivilegedCommand, + resolvePrivilegedTarget: () => ({ + providerId: "docker", + resourceHandle: RESOURCE_HANDLE, + }), + }, + ), + ).rejects.toThrow(/different managed-image sandbox identity/u); + expect(executePrivilegedCommand).toHaveBeenCalledWith( + SANDBOX_NAME, + expect.any(Array), + expect.objectContaining({ sandboxUser: "998:999", expectedResourceHandle: RESOURCE_HANDLE }), + ); + expect(execShell).not.toHaveBeenCalled(); + }); + it("rejects runtime provider resource identity drift during privileged inspection", async () => { vi.stubEnv("NEMOCLAW_E2E_SECURITY_POSTURE", "1"); vi.stubEnv("NEMOCLAW_E2E_EXPECT_OPENSHELL_SPLIT_PROCESS", "1"); const command = vi .fn() .mockResolvedValueOnce(successfulProbe("uid=1000 gid=1000\n")) - .mockResolvedValueOnce(successfulProbe("openshell 0.0.116\n")); + .mockResolvedValueOnce(successfulProbe("openshell 0.1.7\n")); const execShell = vi.fn(); const resolvePrivilegedTarget = vi .fn() @@ -1174,6 +1419,7 @@ describe("security posture fixture", () => { "openclaw", { environment: inlineManagedImageCatalogEnvironment(), + inspectContainerPair: () => CONTAINER_PAIR, executePrivilegedCommand, resolvePrivilegedTarget, }, @@ -1191,7 +1437,7 @@ describe("security posture fixture", () => { const command = vi .fn() .mockResolvedValueOnce(successfulProbe("uid=1000 gid=1000\n")) - .mockResolvedValueOnce(successfulProbe("openshell 0.0.116\n")); + .mockResolvedValueOnce(successfulProbe("openshell 0.1.7\n")); const execShell = vi.fn(); const resolvePrivilegedTarget = vi.fn(() => ({ providerId: "podman", @@ -1212,6 +1458,7 @@ describe("security posture fixture", () => { "openclaw", { environment: inlineManagedImageCatalogEnvironment(), + inspectContainerPair: () => CONTAINER_PAIR, executePrivilegedCommand, resolvePrivilegedTarget, }, diff --git a/test/e2e/support/spark-express-vllm-safety.test.ts b/test/e2e/support/spark-express-vllm-safety.test.ts index 52b798a473d..8894354b6dd 100644 --- a/test/e2e/support/spark-express-vllm-safety.test.ts +++ b/test/e2e/support/spark-express-vllm-safety.test.ts @@ -20,7 +20,7 @@ const result = (exitCode: number, stdout = "", stderr = "") => ({ }); describe("DGX Spark Express vLLM qualification safety", () => { - it("probes both allowed addresses and every unintended IPv4 and IPv6 address (#14083)", () => { + it("probes loopback and every unintended IPv4 and IPv6 address (#14083)", () => { const ipv4 = (address: string, internal = false) => ({ address, internal, @@ -39,15 +39,16 @@ describe("DGX Spark Express vLLM qualification safety", () => { cidr: null, }); expect( - vllmPublicationProbeUrls("172.18.0.1", { + vllmPublicationProbeUrls({ lo: [ipv4("127.0.0.1", true), ipv6("::1")], openshell: [ipv4("172.18.0.1")], eth0: [ipv4("192.0.2.10"), ipv6("2001:db8::10"), ipv6("fe80::10", 2)], }), ).toEqual({ - allowed: ["http://127.0.0.1:8000/health", "http://172.18.0.1:8000/health"], + allowed: ["http://127.0.0.1:8000/health"], denied: [ "http://[::1]:8000/health", + "http://172.18.0.1:8000/health", "http://192.0.2.10:8000/health", "http://[2001:db8::10]:8000/health", "http://[fe80::10%25eth0]:8000/health", @@ -65,8 +66,8 @@ describe("DGX Spark Express vLLM qualification safety", () => { mac: "", cidr: null, }; - expect(vllmPublicationProbeUrls("172.18.0.1", { eth0: [entry] }).ipv6).toBe("unconfigured"); - expect(() => vllmPublicationProbeUrls("172.18.0.1", {})).toThrow("unintended non-loopback"); + expect(vllmPublicationProbeUrls({ eth0: [entry] }).ipv6).toBe("unconfigured"); + expect(() => vllmPublicationProbeUrls({})).toThrow("unintended non-loopback"); }); it("accepts only local Docker selectors (#8379)", () => { diff --git a/test/e2e/support/spark-express-vllm-safety.ts b/test/e2e/support/spark-express-vllm-safety.ts index f2da1b2ef7d..c5d1c947df6 100644 --- a/test/e2e/support/spark-express-vllm-safety.ts +++ b/test/e2e/support/spark-express-vllm-safety.ts @@ -30,11 +30,12 @@ export function assertLocalDockerEnvironment(env: NodeJS.ProcessEnv): void { } } -export function vllmPublicationProbeUrls( - bridgeHost: string, - interfaces = networkInterfaces(), -): { allowed: string[]; denied: string[]; ipv6: "configured" | "unconfigured" } { - const allowedHosts = new Set(["127.0.0.1", bridgeHost]); +export function vllmPublicationProbeUrls(interfaces = networkInterfaces()): { + allowed: string[]; + denied: string[]; + ipv6: "configured" | "unconfigured"; +} { + const allowedHosts = new Set(["127.0.0.1"]); const addresses = Object.entries(interfaces).flatMap(([name, entries]) => (entries ?? []).map((entry) => ({ ...entry, name })), ); diff --git a/test/e2e/support/weather-response.test.ts b/test/e2e/support/weather-response.test.ts new file mode 100644 index 00000000000..bfeb5c7ec7c --- /dev/null +++ b/test/e2e/support/weather-response.test.ts @@ -0,0 +1,57 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { WEATHER_RESPONSE_VALIDATION_PYTHON } from "../live/common-egress-agent-helpers.ts"; + +describe("weather response proof", () => { + let directory: string; + beforeEach(() => { + directory = mkdtempSync(join(tmpdir(), "nemoclaw-weather-proof-")); + }); + afterEach(() => rmSync(directory, { recursive: true, force: true })); + + function validate(body: unknown): void { + const path = join(directory, "response.json"); + writeFileSync(path, JSON.stringify(body)); + execFileSync("python3", ["-c", WEATHER_RESPONSE_VALIDATION_PYTHON, path], { stdio: "pipe" }); + } + + function response() { + return { + latitude: 37.7749, + longitude: -122.4194, + current: { temperature_2m: 12.3, time: "2026-11-02T12:00" }, + current_units: { temperature_2m: "°C" }, + }; + } + + it("accepts a finite Celsius temperature for the requested location", () => { + expect(() => validate(response())).not.toThrow(); + }); + + it.each([ + ["a reply token", "WEATHER_AGENT_OK"], + ["an empty object", {}], + ["another location", { ...response(), latitude: 40 }], + ["another longitude", { ...response(), longitude: 122.4194 }], + [ + "a string temperature", + { ...response(), current: { ...response().current, temperature_2m: "12" } }, + ], + [ + "a Boolean temperature", + { ...response(), current: { ...response().current, temperature_2m: true } }, + ], + ["a missing time", { ...response(), current: { temperature_2m: 12 } }], + ["an empty time", { ...response(), current: { ...response().current, time: "" } }], + ["a different unit", { ...response(), current_units: { temperature_2m: "°F" } }], + ["a missing temperature", { ...response(), current: { time: response().current.time } }], + ])("rejects %s", (_label, body) => { + expect(() => validate(body)).toThrow(); + }); +}); diff --git a/test/fixtures/openshell-provider-profile.ts b/test/fixtures/openshell-provider-profile.ts index bdb4b7ccb0e..7c22902792a 100644 --- a/test/fixtures/openshell-provider-profile.ts +++ b/test/fixtures/openshell-provider-profile.ts @@ -26,9 +26,9 @@ export function managedBraveProfile() { port: 443, ports: [], protocol: "rest", - tls: "", - enforcement: "enforce", - access: "read-write", + tls: 0, + enforcement: 1, + access: 2, rules: [], allowedIps: [], denyRules: [], @@ -69,7 +69,7 @@ export function managedTavilyProfile(agent: "openclaw" | "hermes" = "openclaw") { ...base.endpoints[0], host: "api.tavily.com", - access: "", + access: 0, requestBodyCredentialRewrite: true, rules: ["/search", "/extract"].map((path) => ({ allow: { diff --git a/test/generation/agent-variant-docs.test.ts b/test/generation/agent-variant-docs.test.ts index e08ba0f283d..42afee5755b 100644 --- a/test/generation/agent-variant-docs.test.ts +++ b/test/generation/agent-variant-docs.test.ts @@ -169,7 +169,7 @@ describe("agent variant docs", () => { }); it.each(["reference/commands.mdx", "manage-sandboxes/gateway-lifecycle-control.mdx"])( - "limits the supervised recovery exception to Hermes in %s", + "documents stopped native gateways without promising supervised recovery in %s", (sourcePath) => { const pageSource = readFileSync(path.join(repoRoot, "docs", sourcePath), "utf8"); const render = (variant: (typeof agentVariants)[number]) => @@ -177,14 +177,9 @@ describe("agent variant docs", () => { const supervisedRecovery = "`recover` requests a supervised gateway relaunch before verification."; - expect(render("hermes")).toContain(supervisedRecovery); - expect(render("hermes")).toContain("after a clean exit or exit status `79`"); - expect(render("hermes")).toContain("authenticated gated recovery"); - expect(render("hermes")).not.toContain( - "If a built-in gateway is stopped, `recover` reports that state instead of relaunching the process.", - ); + expect(render("hermes")).not.toContain(supervisedRecovery); expect(render("hermes")).toContain( - "Otherwise, `recover` reports a stopped gateway without relaunching it.", + "If the Hermes gateway is stopped, `recover` reports it without relaunching it.", ); expect(render("openclaw")).toContain( "If the OpenClaw gateway is stopped, `recover` reports it without relaunching it.", diff --git a/test/generation/generate-hermes-config.test.ts b/test/generation/generate-hermes-config.test.ts index 2d1807e5ef4..8ce5c607a63 100644 --- a/test/generation/generate-hermes-config.test.ts +++ b/test/generation/generate-hermes-config.test.ts @@ -43,13 +43,7 @@ import { withLegacyMessagingPlanEnvDirect, } from "../messaging-plan-test-helper"; -const SCRIPT_PATH = path.join( - import.meta.dirname, - "../..", - "agents", - "hermes", - "generate-config.ts", -); +const SCRIPT_PATH = path.resolve(import.meta.dirname, "../../agents/hermes/generate-config.ts"); const SCRIPT_DIR = path.dirname(SCRIPT_PATH); const CONFIG_MODULE_DIR = path.join(import.meta.dirname, "../..", "agents", "hermes", "config"); @@ -289,12 +283,8 @@ function copyConfigGeneratorFixture(fixtureRoot: string): string { path.join(fixtureRoot, nativeConfigDir, "agent-config.ts"), ); fs.copyFileSync( - path.join(import.meta.dirname, "../..", nativeConfigDir, "hosted.ts"), - path.join(fixtureRoot, nativeConfigDir, "hosted.ts"), - ); - fs.copyFileSync( - path.join(import.meta.dirname, "../..", nativeConfigDir, "contract.ts"), - path.join(fixtureRoot, nativeConfigDir, "contract.ts"), + path.join(import.meta.dirname, "../..", nativeConfigDir, "catalog.ts"), + path.join(fixtureRoot, nativeConfigDir, "catalog.ts"), ); const nativeNvidiaConfigDir = path.join("src", "lib", "inference", "native-nvidia"); fs.mkdirSync(path.join(fixtureRoot, nativeNvidiaConfigDir), { recursive: true }); @@ -1491,3 +1481,15 @@ describe("agents/hermes/generate-config.ts", () => { ); }); }); + +it("projects the attached Router credential into Hermes without a host blueprint", () => { + const { config } = runConfigScript({ + NEMOCLAW_UPSTREAM_PROVIDER: "team-router", + NEMOCLAW_INFERENCE_BASE_URL: "http://host.openshell.internal:43123/team/v1", + NEMOCLAW_LOCAL_INFERENCE_TOKEN: "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }); + expect(config.model.base_url).toBe("http://host.openshell.internal:43123/team/v1"); + expect(config.model.api_key).toBe("${NEMOCLAW_LOCAL_INFERENCE_TOKEN}"); + expect(config.providers["team-router"].api_key).toBe("${NEMOCLAW_LOCAL_INFERENCE_TOKEN}"); + expect(JSON.stringify(config)).not.toContain("v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"); +}); diff --git a/test/generation/generate-native-hosted-config.test.ts b/test/generation/generate-native-hosted-config.test.ts index e3e424317d7..0c254ecda4b 100644 --- a/test/generation/generate-native-hosted-config.test.ts +++ b/test/generation/generate-native-hosted-config.test.ts @@ -115,3 +115,66 @@ describe("native provider protocol details", () => { expect(config.model).toEqual(expect.objectContaining({ api_mode: "anthropic_messages" })); }); }); + +it("projects the attached Router credential into OpenClaw without loading a host blueprint", () => { + const config = buildConfig({ + ...baseOpenClawGenerationEnv(), + NEMOCLAW_UPSTREAM_PROVIDER: "team-router", + NEMOCLAW_INFERENCE_BASE_URL: "http://host.openshell.internal:43123/team/v1", + NEMOCLAW_INFERENCE_API: "openai-completions", + NEMOCLAW_LOCAL_INFERENCE_TOKEN: "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }); + expect(config.models.providers["test-provider"]).toEqual( + expect.objectContaining({ + baseUrl: "http://host.openshell.internal:43123/team/v1", + apiKey: "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }), + ); +}); + +it.each([ + ["openai-completions", undefined], + ["openai-responses", undefined], + ["openai-completions", "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"], + ["openai-responses", "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"], +])( + "projects the local compatible credential for %s with generation handle %s", + (api, generationHandle) => { + const handle = "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"; + const config = buildConfig({ + ...baseOpenClawGenerationEnv(), + NEMOCLAW_UPSTREAM_PROVIDER: "compatible-endpoint", + NEMOCLAW_INFERENCE_PROVIDER_ID: "inference", + NEMOCLAW_INFERENCE_BASE_URL: "http://host.openshell.internal:45173/v1", + NEMOCLAW_INFERENCE_API: api, + ...(generationHandle ? { NEMOCLAW_LOCAL_INFERENCE_TOKEN: generationHandle } : {}), + COMPATIBLE_API_KEY: "host-secret-must-not-appear", + }); + expect(config.models.providers.inference).toEqual( + expect.objectContaining({ + baseUrl: "http://host.openshell.internal:45173/v1", + api, + apiKey: "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }), + ); + expect(JSON.stringify(config)).not.toContain("COMPATIBLE_API_KEY"); + expect(JSON.stringify(config)).not.toContain("host-secret-must-not-appear"); + expect(JSON.stringify(config)).not.toContain(handle); + }, +); + +it.each([ + ["http://10.1.2.3:45173/v1", undefined], + ["http://10.1.2.3:45173/v1", "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"], + ["https://custom.example/team/v1", "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN"], +])("keeps the hosted credential when no local projection applies at %s", (endpoint, handle) => { + const config = buildConfig({ + ...baseOpenClawGenerationEnv(), + NEMOCLAW_UPSTREAM_PROVIDER: "compatible-endpoint", + NEMOCLAW_INFERENCE_PROVIDER_ID: "inference", + NEMOCLAW_INFERENCE_BASE_URL: endpoint, + NEMOCLAW_INFERENCE_API: "openai-completions", + ...(handle ? { NEMOCLAW_LOCAL_INFERENCE_TOKEN: handle } : {}), + }); + expect(config.models.providers.inference.apiKey).toBe("openshell:resolve:env:COMPATIBLE_API_KEY"); +}); diff --git a/test/generation/generate-native-local-image-config.test.ts b/test/generation/generate-native-local-image-config.test.ts new file mode 100644 index 00000000000..7599ab4c6f5 --- /dev/null +++ b/test/generation/generate-native-local-image-config.test.ts @@ -0,0 +1,116 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; + +const temporary: string[] = []; +afterEach(() => { + for (const root of temporary.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +// Execute only the TypeScript files declared by the image, without the checkout, +// compiled CLI, package dependencies, or a TypeScript module-resolution loader. +function imagePayload(agent: "pi" | "langchain-deepagents-code") { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-config-payload-")); + temporary.push(root); + const dockerfile = fs.readFileSync(`agents/${agent}/Dockerfile`, "utf8"); + copyImageSource(root, dockerfile, `agents/${agent}/generate-config.ts`); + copyImageSource(root, dockerfile, "src/lib/core/endpoint-url-safety.ts"); + copyImageSource(root, dockerfile, "src/lib/inference/native-local/agent-config.ts"); + copyImageSource(root, dockerfile, "src/lib/inference/native-provider/agent-config.ts"); + copyImageSource(root, dockerfile, "src/lib/inference/native-provider/catalog.ts"); + copyImageSource(root, dockerfile, "src/lib/inference/native-nvidia/contract.ts"); + copyImageSource( + root, + dockerfile, + agent === "pi" + ? "agents/pi/check-native-credentials.ts" + : "src/lib/inference/managed-dcode/identity.ts", + ); + return root; +} + +function copyImageSource(root: string, dockerfile: string, source: string) { + const fields = dockerfile + .split("\n") + .map((line) => line.trim().split(/\s+/u)) + .find((parts) => parts[0] === "COPY" && parts.includes(source)); + assert(fields, `Image must copy ${source}`); + const target = fields.at(-1)!; + const destination = path.join( + root, + target, + ...(target.endsWith("/") ? [path.basename(source)] : []), + ); + fs.mkdirSync(path.dirname(destination), { recursive: true }); + fs.copyFileSync(source, destination); +} + +describe.each(["pi", "langchain-deepagents-code"] as const)( + "%s image configuration payload", + (agent) => { + it.each([ + ["nvidia-prod", "https://inference.local/v1", undefined, false], + ["ollama-local", "http://host.openshell.internal:11434/v1", undefined, true], + [ + "team-router", + "http://host.openshell.internal:43123/team/v1", + "openshell:resolve:env:v7_NEMOCLAW_LOCAL_INFERENCE_TOKEN", + true, + ], + ["team-router", "http://host.openshell.internal:43123/team/v1", "raw-key", false], + ] as const)( + "generates %s configuration with only image sources", + (provider, endpoint, handle, native) => { + const root = imagePayload(agent); + const script = + agent === "pi" + ? "opt/nemoclaw-pi/generate-config.ts" + : "opt/nemoclaw-deepagents-code/agents/langchain-deepagents-code/generate-config.ts"; + const home = path.join(root, "sandbox"); + const result = spawnSync(process.execPath, [path.join(root, script)], { + cwd: root, + encoding: "utf8", + env: { + PATH: process.env.PATH, + HOME: home, + NEMOCLAW_MODEL: "selected-model", + NEMOCLAW_INFERENCE_PROVIDER_ID: "inference", + NEMOCLAW_UPSTREAM_PROVIDER: provider, + NEMOCLAW_INFERENCE_BASE_URL: endpoint, + NEMOCLAW_INFERENCE_API: "openai-completions", + ...(handle ? { NEMOCLAW_LOCAL_INFERENCE_TOKEN: handle } : {}), + }, + }); + expect(result.status, result.stderr).toBe(0); + const config = fs.readFileSync( + path.join(home, agent === "pi" ? ".pi/agent/models.json" : ".deepagents/config.toml"), + "utf8", + ); + expect(config).toContain(endpoint); + expect(config.includes("NEMOCLAW_LOCAL_INFERENCE_TOKEN")).toBe(native); + expect(config).not.toContain("raw-key"); + expect(config).not.toContain(handle ?? "raw-key"); + }, + ); + }, +); + +// Reaching the explicit runtime lookup proves copied imports load; credential +// resolution itself remains the pinned Pi image build's acceptance check. +it("loads the Pi credential checker from image sources before resolving its pinned runtime", () => { + const root = imagePayload("pi"); + const absentRuntime = path.join(root, "uninstalled-pinned-pi-runtime"); + const result = spawnSync( + process.execPath, + [path.join(root, "opt/nemoclaw-pi/check-native-credentials.ts"), absentRuntime], + { cwd: root, encoding: "utf8", env: { PATH: process.env.PATH } }, + ); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain(path.join(absentRuntime, "dist/core/resolve-config-value.js")); +}); diff --git a/test/generation/generate-openclaw-config-web-search.test.ts b/test/generation/generate-openclaw-config-web-search.test.ts index aeba3701b79..24030e8e30e 100644 --- a/test/generation/generate-openclaw-config-web-search.test.ts +++ b/test/generation/generate-openclaw-config-web-search.test.ts @@ -66,7 +66,7 @@ describe("generate-openclaw-config.mts: web search", () => { expect(config.plugins?.entries?.brave).toBeUndefined(); expect(config.plugins?.allow).toBeUndefined(); expect(config.tools?.web?.search?.apiKey).toBeUndefined(); - expect(config.tools?.web?.fetch).toEqual({ enabled: true, useTrustedEnvProxy: true }); + expect(config.tools?.web?.fetch).toEqual({ enabled: true, useTrustedEnvProxy: false }); }); it("allows the enabled Brave plugin (#8975)", () => { diff --git a/test/generation/generate-openclaw-config.test.ts b/test/generation/generate-openclaw-config.test.ts index acca748ab16..636fbdbf124 100644 --- a/test/generation/generate-openclaw-config.test.ts +++ b/test/generation/generate-openclaw-config.test.ts @@ -515,78 +515,32 @@ describe("generate-openclaw-config.mts: config generation", () => { it("omits the Discord token from generated config", async () => { const channels = Buffer.from(JSON.stringify(["telegram", "discord"])).toString("base64"); const config = await runMessagingConfig(channels); - expect(config.proxy).toMatchObject({ - enabled: true, - proxyUrl: "http://10.200.0.1:3128", - loopbackMode: "gateway-only", - }); + expect(config.proxy).toBeUndefined(); expect(config.channels.telegram.accounts.default.botToken).toBeUndefined(); expect(config.channels.discord.accounts.default.token).toBeUndefined(); - expect(config.channels.telegram.accounts.default.proxy).toBe("http://10.200.0.1:3128"); - expect(config.channels.discord.accounts.default.proxy).toBeUndefined(); - }); - - it("routes Discord gateway traffic through OpenClaw's managed proxy (#3894)", async () => { - const channels = Buffer.from(JSON.stringify(["discord"])).toString("base64"); - const config = await runMessagingConfig(channels, { - NEMOCLAW_PROXY_HOST: "10.201.0.9", - NEMOCLAW_PROXY_PORT: "43128", - }); - - expect(config.proxy).toEqual({ - enabled: true, - proxyUrl: "http://10.201.0.9:43128", - loopbackMode: "gateway-only", - }); - expect(config.channels.discord.accounts.default).toMatchObject({ - enabled: true, - }); - expect(config.channels.discord.accounts.default.token).toBeUndefined(); - expect(config.channels.discord.accounts.default.proxy).toBeUndefined(); - }); - - it("does not write a Discord account proxy when the managed proxy is configured", async () => { - const channels = Buffer.from(JSON.stringify(["discord"])).toString("base64"); - const config = await runMessagingConfig(channels, { - NEMOCLAW_PROXY_PORT: "43128", - }); - - expect(config.proxy.proxyUrl).toBe("http://10.200.0.1:43128"); - expect(config.channels.discord.accounts.default.proxy).toBeUndefined(); - }); - - it("can defer OpenClaw managed proxy config for build-time doctor", async () => { - const channels = Buffer.from(JSON.stringify(["discord"])).toString("base64"); - const config = await runMessagingConfig(channels, { - NEMOCLAW_OPENCLAW_MANAGED_PROXY: "0", - }); - - expect(config.proxy).toBeUndefined(); + expect(config.channels.telegram.accounts.default.proxy).toBeUndefined(); expect(config.channels.discord.accounts.default.proxy).toBeUndefined(); }); - it("ignores the OpenShell loopback proxy env var when using OpenClaw managed proxy", async () => { - const channels = Buffer.from(JSON.stringify(["discord"])).toString("base64"); - const config = await runMessagingConfig(channels, { - OPENSHELL_LOOPBACK_PROXY_URL: "http://127.0.0.1:45211", - NEMOCLAW_DISCORD_PROXY_PORT: "43129", - }); - - expect(config.proxy.proxyUrl).toBe("http://10.200.0.1:3128"); - expect(config.channels.discord.accounts.default.proxy).toBeUndefined(); - }); - - it("keeps Telegram on the OpenShell proxy while Discord relies on the managed proxy", async () => { - const channels = Buffer.from(JSON.stringify(["telegram", "discord"])).toString("base64"); - const config = await runMessagingConfig(channels, { - NEMOCLAW_PROXY_HOST: "10.201.0.9", - NEMOCLAW_PROXY_PORT: "43128", - }); - - expect(config.proxy.proxyUrl).toBe("http://10.201.0.9:43128"); - expect(config.channels.telegram.accounts.default.proxy).toBe("http://10.201.0.9:43128"); - expect(config.channels.discord.accounts.default.proxy).toBeUndefined(); - }); + it.each([ + {}, + { NEMOCLAW_PROXY_HOST: "10.201.0.9", NEMOCLAW_PROXY_PORT: "43128" }, + { NEMOCLAW_PROXY_HOST: "", NEMOCLAW_PROXY_PORT: "" }, + { NEMOCLAW_OPENCLAW_MANAGED_PROXY: "1" }, + { OPENSHELL_LOOPBACK_PROXY_URL: "http://127.0.0.1:45211" }, + ] as Record[])( + "uses native messaging egress despite obsolete proxy inputs %#", + async (env) => { + const channels = Buffer.from(JSON.stringify(["telegram", "discord"])).toString("base64"); + const config = await runMessagingConfig(channels, env); + expect(config.proxy).toBeUndefined(); + expect(config.tools.web.fetch.useTrustedEnvProxy).toBe(false); + expect(config.channels.telegram.accounts.default.proxy).toBeUndefined(); + expect(config.channels.discord.accounts.default).toMatchObject({ enabled: true }); + expect(config.channels.discord.accounts.default.token).toBeUndefined(); + expect(config.channels.discord.accounts.default.proxy).toBeUndefined(); + }, + ); it("emits Bolt-shape placeholders for Slack so the SDK's prefix regex passes", async () => { const channels = Buffer.from(JSON.stringify(["slack"])).toString("base64"); @@ -689,7 +643,7 @@ describe("generate-openclaw-config.mts: config generation", () => { const config = runConfigScript(); expect(config.tools?.web?.fetch).toEqual({ enabled: true, - useTrustedEnvProxy: true, + useTrustedEnvProxy: false, }); expect(config.tools?.web?.search).toEqual({ enabled: false }); }); @@ -704,7 +658,7 @@ describe("generate-openclaw-config.mts: config generation", () => { enabled: true, config: { webSearch: { apiKey: "openshell:resolve:env:BRAVE_API_KEY" } }, }); - expect(config.tools?.web?.fetch).toEqual({ enabled: true, useTrustedEnvProxy: true }); + expect(config.tools?.web?.fetch).toEqual({ enabled: true, useTrustedEnvProxy: false }); }); it("propagates the agent timeout to the run and provider request (#8468)", () => { @@ -1749,22 +1703,6 @@ describe("generate-openclaw-config.mts: empty-string env vars fall back to defau expect(config.gateway.controlUi.allowedOrigins).toEqual(["http://127.0.0.1:18789"]); }); - it("treats empty NEMOCLAW_PROXY_HOST as unset and uses the documented default", async () => { - const channelB64 = Buffer.from(JSON.stringify(["telegram"])).toString("base64"); - const cfg = await runMessagingConfig(channelB64, { - NEMOCLAW_PROXY_HOST: "", - }); - expect(cfg.channels.telegram.accounts.default.proxy).toBe("http://10.200.0.1:3128"); - }); - - it("treats empty NEMOCLAW_PROXY_PORT as unset and uses the documented default", async () => { - const channelB64 = Buffer.from(JSON.stringify(["telegram"])).toString("base64"); - const cfg = await runMessagingConfig(channelB64, { - NEMOCLAW_PROXY_PORT: "", - }); - expect(cfg.channels.telegram.accounts.default.proxy).toBe("http://10.200.0.1:3128"); - }); - it("treats empty NEMOCLAW_CONTEXT_WINDOW as unset and uses the documented default", () => { const cfg = runConfigScript({ NEMOCLAW_CONTEXT_WINDOW: "" }); expect(cfg.models.providers["test-provider"].models[0].contextWindow).toBe(131072); diff --git a/test/generation/generate-openclaw-native-nvidia-config.test.ts b/test/generation/generate-openclaw-native-nvidia-config.test.ts index 3ccee50bfc7..c9a79dd1636 100644 --- a/test/generation/generate-openclaw-native-nvidia-config.test.ts +++ b/test/generation/generate-openclaw-native-nvidia-config.test.ts @@ -18,7 +18,7 @@ it("uses an opaque native Model Router credential without serializing the key (# NEMOCLAW_UPSTREAM_PROVIDER: "nvidia-router", NEMOCLAW_INFERENCE_BASE_URL: endpoint, NVIDIA_INFERENCE_API_KEY: "test-only-secret", - NEMOCLAW_LOCAL_INFERENCE_TOKEN: "test-only-secret", + NEMOCLAW_LOCAL_INFERENCE_TOKEN: "openshell:resolve:env:v1_NEMOCLAW_LOCAL_INFERENCE_TOKEN", }); expect(config.models.providers.inference.baseUrl).toBe(endpoint); expect(config.models.providers.inference.apiKey).toBe( diff --git a/test/helpers/candidate-openshell-trust.ts b/test/helpers/candidate-openshell-trust.ts index f407b432701..d09a65b2242 100644 --- a/test/helpers/candidate-openshell-trust.ts +++ b/test/helpers/candidate-openshell-trust.ts @@ -31,6 +31,8 @@ export function fixture() { ) as Record; const current = validateInstallerSources(sources); const release = structuredClone(current.installerReleases[0]!); + // The registry stores revocations globally; resolved releases carry a derived copy. + delete release.revokedTemplateSha256; const prior = release.version; release.version = version; release.formula.url = `https://github.com/NVIDIA/OpenShell/releases/download/v${version}/openshell.rb`; @@ -154,7 +156,11 @@ export function runCoveredRegistry( JSON.stringify({ ...oldRegistry, releases: scenario === "absent-untrusted" ? [] : [...oldRegistry.releases, nextRelease], - revokedTemplateSha256: scenario.endsWith("revoked") ? f.release.installerTemplateSha256 : [], + revokedTemplateSha256: scenario.endsWith("revoked") + ? [...new Set([...oldRegistry.revokedTemplateSha256, ...f.release.installerTemplateSha256])] + : scenario === "absent-untrusted" + ? [] + : oldRegistry.revokedTemplateSha256, }), ); const candidateRegistry = structuredClone(oldRegistry); diff --git a/test/helpers/credential-window-http-fixture.ts b/test/helpers/credential-window-http-fixture.ts index b62ae41c020..140d88eec08 100644 --- a/test/helpers/credential-window-http-fixture.ts +++ b/test/helpers/credential-window-http-fixture.ts @@ -2,23 +2,46 @@ // SPDX-License-Identifier: Apache-2.0 import { EventEmitter } from "node:events"; +import https from "node:https"; +import { runInNewContext } from "node:vm"; +import { createMcpFixtureTls } from "../e2e/fixtures/mcp-fixture-tls.ts"; +import { startFakeMcpHttpsServer } from "../e2e/live/mcp-bridge-servers.ts"; +import { + buildCredentialWindowChildScript, + buildCredentialWindowOneShotScript, + CREDENTIAL_WINDOW_PATHS, + CREDENTIAL_WINDOW_STEPS, +} from "../e2e/live/openshell-credential-generation-window.ts"; export function credentialWindowHttpFixture(result: number | string) { + const requests: unknown[] = []; return { - request: (_options: unknown, callback: (response: unknown) => void) => { + requests, + request: (options: unknown, callback: (response: unknown) => void) => { + requests.push(options); const outbound = new EventEmitter(); return Object.assign(outbound, { setTimeout: () => {}, - end: () => + end: (body: string) => queueMicrotask(() => { - if (typeof result === "string") { + const method = JSON.parse(body).method; + const negotiating = method === "initialize" || method === "notifications/initialized"; + if (!negotiating && typeof result === "string") { outbound.emit("error", { code: result, message: "private-error-secret" }); } else { const response = Object.assign(new EventEmitter(), { - statusCode: result, + statusCode: method === "initialize" ? 200 : negotiating ? 202 : result, + headers: { "mcp-session-id": "fixture-session" }, + setEncoding: () => {}, resume: () => {}, }); callback(response); + if (method === "initialize") { + response.emit( + "data", + JSON.stringify({ result: { protocolVersion: "2025-11-25" } }), + ); + } response.emit("end"); } }), @@ -26,3 +49,83 @@ export function credentialWindowHttpFixture(result: number | string) { }, }; } + +export async function runCredentialWindowSessionFixture( + scenario: "retained" | "fresh", + authorization = "fixture-secret", +) { + const certificate = createMcpFixtureTls(); + const server = await startFakeMcpHttpsServer({ + secret: "fixture-secret", + tls: certificate.tls, + }); + try { + const writes = new Map(); + let summary = ""; + let diagnostic = ""; + const url = `https://127.0.0.1:${server.port}/mcp`; + const script = + scenario === "fresh" + ? buildCredentialWindowOneShotScript() + : buildCredentialWindowChildScript({ mcpUrl: url, maxRuntimeMs: 3000 }); + const childProcess = { + argv: ["node", url, "fresh-request"], + env: { FAKE_MCP_SECRET: "openshell:resolve:env:v42_FAKE_MCP_SECRET" }, + stdout: { + write: (value: string) => { + summary += value; + }, + }, + stderr: { + write: (value: string) => { + diagnostic += value; + }, + }, + exitCode: 0, + }; + await runInNewContext(script, { + Buffer, + URL, + setTimeout, + process: childProcess, + require: (name: string) => + name === "node:fs" + ? { + existsSync: () => true, + readFileSync: () => + writes.has(CREDENTIAL_WINDOW_PATHS.acknowledgement) + ? CREDENTIAL_WINDOW_STEPS.stop + : CREDENTIAL_WINDOW_STEPS.allowedBeforeExpiry, + writeFileSync: (file: string, value: string) => writes.set(file, value), + } + : { + request: ( + options: https.RequestOptions, + callback: (response: import("node:http").IncomingMessage) => void, + ) => + https.request( + { + ...options, + ca: certificate.tls.cert, + // Model the proxy rewrite while exercising the real fixture's session validation. + headers: { + ...options.headers, + authorization: `Bearer ${authorization}`, + }, + }, + callback, + ), + }, + }); + return { + exitCode: childProcess.exitCode, + writes, + summary, + diagnostic, + requests: server.requests, + }; + } finally { + await server.close(); + certificate.close(); + } +} diff --git a/test/helpers/destroy-flow-test-harness.ts b/test/helpers/destroy-flow-test-harness.ts index ae5cc892757..a07a835c022 100644 --- a/test/helpers/destroy-flow-test-harness.ts +++ b/test/helpers/destroy-flow-test-harness.ts @@ -141,6 +141,7 @@ type DestroyHarnessOptions = { promptResponses?: string[]; provider?: string; registryEntryPresent?: boolean; + registryGatewayPort?: number; registryEntryOverrides?: Partial; registeredSandboxCount?: number; recoveredGatewayRuntimeProviderId?: string | null; @@ -307,6 +308,10 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr vi.spyOn(forwardRecovery, "teardownSandboxDashboardForward").mockResolvedValue(true); const destroyCommand = requireSource("../../../commands/sandbox/destroy.js").default; const destroyPreflight = requireSource("./destroy-preflight.js"); + const googlechatLifecycle = requireSource( + "../../messaging/channels/googlechat/tunnel/lifecycle.js", + ); + vi.spyOn(googlechatLifecycle, "stopGooglechatWebhookTunnel").mockReturnValue(""); const crossPortRegistry = requireSource("../../state/registry/cross-port.js"); let listCrossRootGatewayCleanupOwnersSpy: MockInstance | null = null; const sandboxSession = requireSource("../../state/sandbox-session.js"); @@ -435,6 +440,20 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr } return true; }); + const registryGatewayPort = options.registryGatewayPort; + if (registryGatewayPort !== undefined) { + const { resolveNemoclawStateDir } = requireSource("../../state/paths.js"); + vi.spyOn(destroyPreflight, "resolveSandboxDestroyRegistryAuthority").mockImplementation( + (name: unknown) => ({ + entry: registry.getSandbox(name), + gatewayPort: registryGatewayPort, + stateDir: resolveNemoclawStateDir(undefined, registryGatewayPort), + getSandbox: registry.getSandbox, + listSandboxes: registry.listSandboxes, + removeSandbox: registry.removeSandbox, + }), + ); + } const stopModelRouterForDestroyedSandboxSpy = vi.spyOn( destroyPreflight, "stopModelRouterForDestroyedSandbox", @@ -645,12 +664,14 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr options.onDockerRun?.(identityProbeCall); const defaultIdentityResult = { status: 0, - stdout: sandboxPresent ? "aaaaaaaaaaaa\topenshell\tdefault\tsb-alpha" : "", + stdout: sandboxPresent ? "aaaaaaaaaaaa\topenshell\tdefault\tsb-alpha\t\t" : "", }; const sequencedResult = options.dockerRunResultSequence?.[identityProbeCall - 1]; const exactCleanupResult = { status: options.dockerOrphanQueryStatus ?? 0, - stdout: dockerNameLabeledIds.map((id) => `${id}\topenshell\tdefault\tsb-alpha`).join("\n"), + stdout: dockerNameLabeledIds + .map((id) => `${id}\topenshell\tdefault\tsb-alpha\t\t`) + .join("\n"), stderr: "", }; const result = diff --git a/test/helpers/fetch-guard-patch-harness.ts b/test/helpers/fetch-guard-patch-harness.ts index 77c6fc6648d..b6f6d1e3321 100644 --- a/test/helpers/fetch-guard-patch-harness.ts +++ b/test/helpers/fetch-guard-patch-harness.ts @@ -15,6 +15,7 @@ const OPENCLAW_VERSION_EXTRACTOR = path.join( ); export const CURRENT_REVIEWED_OPENCLAW_PATCH_CLASSIFIER_VERSION = "2026.9.2"; +export const LEGACY_OPENCLAW_PATCH_FIXTURE_VERSION = "2026.7.1"; export function dockerRunCommandBetween(startMarker: string, endMarker: string): string { const dockerfile = fs.readFileSync(DOCKERFILE, "utf-8"); @@ -84,12 +85,22 @@ export function runDockerfilePatchBlock( dist: string, tmp: string, endMarker: string, - version = CURRENT_REVIEWED_OPENCLAW_PATCH_CLASSIFIER_VERSION, + version = LEGACY_OPENCLAW_PATCH_FIXTURE_VERSION, + legacyFixture = true, ) { const command = dockerRunCommandBetween( - "# Patch OpenClaw media fetch for proxy-only sandbox", + "# Patch OpenClaw network boundaries for OpenShell 0.1.3.", endMarker, - ).replaceAll("/usr/local/lib/node_modules/openclaw/dist", dist); + ) + .replaceAll("/usr/local/lib/node_modules/openclaw/dist", dist) + .replaceAll( + "/usr/local/lib/nemoclaw/patch-openclaw-native-network.mts", + "'" + + path + .join(import.meta.dirname, "../../scripts/lib/patch-openclaw-native-network.mts") + .replaceAll("'", "'\\''") + + "'", + ); const scriptPath = path.join(tmp, "patch.sh"); fs.writeFileSync( scriptPath, @@ -104,7 +115,11 @@ export function runDockerfilePatchBlock( const fakeBin = createSedWrapper(tmp); return spawnSync("bash", [scriptPath], { encoding: "utf-8", - env: { ...process.env, PATH: `${fakeBin}:${process.env.PATH || ""}` }, + env: { + ...process.env, + NEMOCLAW_E2E_FIXTURE_LEGACY_OPENCLAW: legacyFixture ? "1" : "0", + PATH: `${fakeBin}:${process.env.PATH || ""}`, + }, timeout: 10000, }); } @@ -112,7 +127,7 @@ export function runDockerfilePatchBlock( export function runFetchGuardPatchBlock( dist: string, tmp: string, - version = CURRENT_REVIEWED_OPENCLAW_PATCH_CLASSIFIER_VERSION, + version = LEGACY_OPENCLAW_PATCH_FIXTURE_VERSION, ) { return runDockerfilePatchBlock( dist, diff --git a/test/helpers/hermes-portable-ollama-test-harness.ts b/test/helpers/hermes-portable-ollama-test-harness.ts index 15efe3142ce..ce9a57ce97e 100644 --- a/test/helpers/hermes-portable-ollama-test-harness.ts +++ b/test/helpers/hermes-portable-ollama-test-harness.ts @@ -1,6 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +import fs from "node:fs"; + import type { ContainerEngineCommandCapture, ContainerEngineCommandResult, @@ -170,6 +172,7 @@ export interface PortableGatewayProviderHarness { readonly setCredentialEnv: (value: string) => void; readonly setLookupFailure: (value: boolean) => void; readonly setMalformed: (value: boolean) => void; + readonly setIdentity: (name: string, type: string, id?: string) => void; readonly setPresent: (value: boolean) => void; readonly setProfileState: (value: "exact" | "missing" | "incompatible" | "import-failed") => void; } @@ -178,6 +181,10 @@ export function createPortableGatewayProviderHarness( events: string[], ): PortableGatewayProviderHarness { let present = false; + let providerName = "ollama-local"; + let providerId = "portable-ollama-provider"; + let providerType = "openai"; + let profileDocument: Record | null = null; let malformed = false; let deleteFailure = false; let createTransportAmbiguity = false; @@ -212,6 +219,11 @@ export function createPortableGatewayProviderHarness( setMalformed: (value: boolean) => { malformed = value; }, + setIdentity: (name: string, type: string, id = providerId) => { + providerName = name; + providerType = type; + providerId = id; + }, setPresent: (value: boolean) => { present = value; }, @@ -221,19 +233,36 @@ export function createPortableGatewayProviderHarness( run(args: string[], options: Parameters[1]) { calls.push(Object.freeze({ args: Object.freeze([...args]), timeout: options.timeout })); events.push(`openshell:${args.join(" ")}`); + if (args[0] === "policy" && args[1] === "list") + return { + status: 0, + stdout: JSON.stringify({ revisions: [], next_page_token: "" }), + stderr: "", + }; if (args[0] === "provider" && args[1] === "profile" && args[2] === "export") { - if (profileState === "missing" || profileState === "import-failed") { + if ( + profileState === "missing" || + profileState === "import-failed" || + (args[3] !== "openai" && !profileDocument) + ) { return { status: 1, stdout: "", stderr: "provider profile not found" }; } return { status: 0, - stdout: JSON.stringify({ - id: "openai", - credentials: [], - endpoints: profileState === "exact" ? [] : ["https://example.invalid"], - binaries: [], - inference_capable: true, - }), + stdout: JSON.stringify( + profileDocument + ? { + ...profileDocument, + ...(profileState === "incompatible" ? { binaries: ["/bin/sh"] } : {}), + } + : { + id: "openai", + credentials: [], + endpoints: profileState === "exact" ? [] : ["https://example.invalid"], + binaries: [], + inference_capable: true, + }, + ), stderr: "", }; } @@ -241,6 +270,9 @@ export function createPortableGatewayProviderHarness( if (profileState === "import-failed") { return { status: 1, stdout: "", stderr: "profile import failed" }; } + const file = args.find((arg) => arg.endsWith("profile.yaml")); + if (!file) throw new Error("Missing profile import file"); + profileDocument = JSON.parse(fs.readFileSync(file, "utf8")); profileState = "exact"; return { status: 0, stdout: "", stderr: "" }; } @@ -255,12 +287,12 @@ export function createPortableGatewayProviderHarness( ? { status: 0, stdout: [ - "\u001b[2mId:\u001b[0m portable-ollama-provider", + `\u001b[2mId:\u001b[0m ${providerId}`, `\u001b[2mResource version:\u001b[0m ${String(resourceVersion)}`, - "Name: ollama-local", - "Type: openai", + `Name: ${providerName}`, + `Type: ${providerType}`, `Credential keys: ${credentialEnv}`, - "Config keys: OPENAI_BASE_URL", + `Config keys: ${providerType === "openai" ? "OPENAI_BASE_URL" : ""}`, ].join("\n"), stderr: "", } @@ -273,6 +305,8 @@ export function createPortableGatewayProviderHarness( } if (args[0] === "provider" && args[1] === "create") { if (present) return { status: 1, stdout: "", stderr: "provider already exists" }; + providerName = args[args.indexOf("--name") + 1] ?? "ollama-local"; + providerType = args[args.indexOf("--type") + 1] ?? "openai"; if (foreignCreateCredentialEnv !== null) { credentialEnv = foreignCreateCredentialEnv; present = true; diff --git a/test/helpers/hermes-portable-onboarding-fixture.ts b/test/helpers/hermes-portable-onboarding-fixture.ts index 6bce8b0739c..fe63bb1c591 100644 --- a/test/helpers/hermes-portable-onboarding-fixture.ts +++ b/test/helpers/hermes-portable-onboarding-fixture.ts @@ -134,7 +134,7 @@ export function unexpectedHermesPortablePodmanArgs(args: readonly string[]): nev export function hermesPortableTestOpenShellAuthority(): HermesPortableOpenShellExecutableAuthority { return { - version: "0.0.116", + version: "0.1.7", executable: { executablePath: "/usr/bin/openshell", device: "1", diff --git a/test/helpers/hermes-portable-uninstall-fixture.ts b/test/helpers/hermes-portable-uninstall-fixture.ts index 9c4c62bc69f..af5d6d84762 100644 --- a/test/helpers/hermes-portable-uninstall-fixture.ts +++ b/test/helpers/hermes-portable-uninstall-fixture.ts @@ -22,7 +22,6 @@ import type { SandboxEntry } from "../../src/lib/state/registry/types"; import { createPortableOnboardEnvironmentScope } from "../../src/lib/onboard/session-bootstrap"; import { prepareHostLocalInferenceStartup, - type HostLocalInferenceGatewayMutation, type HostLocalInferenceStartupRequest, } from "../../src/lib/onboard/runtime-provider/host-local-inference-routing"; import { HOST_LOCAL_INFERENCE_APPLICATION_BASE_URL } from "../../src/lib/onboard/runtime-provider/host-local-inference-routing"; @@ -254,16 +253,6 @@ function sandboxInspect( ]); } -function createGatewayMutation(mutation: HostLocalInferenceGatewayMutation): void { - mutation.upsertProvider!( - "ollama-local", - "openai", - "NEMOCLAW_OLLAMA_PROXY_TOKEN", - "http://host.openshell.internal:11434/v1", - { NEMOCLAW_OLLAMA_PROXY_TOKEN: "ollama" }, - ); -} - export interface HermesPortableUninstallFixture { readonly cleanupInput: PortableRuntimeCleanupInput; readonly deps: HermesPortableUninstallDeps; @@ -397,7 +386,7 @@ export async function createHermesPortableUninstallFixture( model: "qwen3-vl:4b", providerBaseUrl: "http://host.openshell.internal:11434/v1", }); - createGatewayMutation(mutation); + const nativeLocalProviderAttachment = await mutation.registerNativeProvider!(); await mutation.commit(); route.prepared.commit(); const inferenceStateRoot = path.join(stateDir, "portable-inference"); @@ -434,6 +423,7 @@ export async function createHermesPortableUninstallFixture( endpointSource: null, preferredInferenceApi: null, hostLocalInferenceReceipt: serializedInferenceReceipt, + nativeLocalProviderAttachment, }; const siblingRow: SandboxEntry = { ...targetRow, @@ -497,7 +487,9 @@ export async function createHermesPortableUninstallFixture( timeout, ) => { if (args[0] === "provider") { - return gatewayProvider.run([...args], { + if (args[2] !== "-g" || args[3] !== GATEWAY_NAME) + throw new Error("Unexpected provider gateway scope"); + return gatewayProvider.run([...args.slice(0, 2), ...args.slice(4)], { ignoreError: true, suppressOutput: true, stdio: ["ignore", "pipe", "pipe"], diff --git a/test/helpers/hermes-restart-config-seal-fixture.ts b/test/helpers/hermes-restart-config-seal-fixture.ts index 3c10eb3466c..0d49a985cd3 100644 --- a/test/helpers/hermes-restart-config-seal-fixture.ts +++ b/test/helpers/hermes-restart-config-seal-fixture.ts @@ -99,6 +99,8 @@ export function hashInputs(configPath: string, envPath: string): string { export function createRestartFixture(): RestartFixture { const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-hermes-restart-seal-")); + // Match the nonroot sandbox identity before descendants inherit this group. + fs.chownSync(root, process.geteuid!(), process.getegid!()); const sandboxDir = path.join(root, "sandbox"); const hermesDir = path.join(sandboxDir, ".hermes"); const configPath = path.join(hermesDir, "config.yaml"); diff --git a/test/helpers/inference-set-openclaw-native-local.ts b/test/helpers/inference-set-openclaw-native-local.ts index 648e4162407..5c36f4f5cc9 100644 --- a/test/helpers/inference-set-openclaw-native-local.ts +++ b/test/helpers/inference-set-openclaw-native-local.ts @@ -4,6 +4,7 @@ import { nativeLocalIdentity, type NativeLocalProviderAttachment, + type NativeLocalBinding, } from "../../src/lib/inference/native-local/contract"; import type { ConfigObject } from "../../src/lib/security/credential-filter"; import { createDeps } from "./inference-set"; @@ -24,15 +25,27 @@ export const attachment = { providerId: "owned-local-provider", }; -export function fixture(selectedAttachment: NativeLocalProviderAttachment = attachment) { +export function fixture( + override: Partial = {}, + credentialEnv?: string, + endpointUrl?: string, + recordedAttachment?: NativeLocalProviderAttachment, +) { + const selected = { ...binding, ...override }; + const recorded = { + ...selected, + ...nativeLocalIdentity(selected), + schemaVersion: 1 as const, + providerId: attachment.providerId, + }; const config: ConfigObject = { agents: { defaults: { model: { primary: "inference/old-model" } } }, models: { providers: { inference: { - baseUrl: selectedAttachment.endpointUrl, + baseUrl: selected.endpointUrl, apiKey: "openshell:resolve:env:NEMOCLAW_LOCAL_INFERENCE_TOKEN", - api: "openai-completions", + api: selected.api ?? "openai-completions", models: [{ id: "old-model" }], }, }, @@ -44,25 +57,26 @@ export function fixture(selectedAttachment: NativeLocalProviderAttachment = atta name: "alpha", agent: "openclaw", gatewayName: "nemoclaw", - provider: selectedAttachment.provider, + provider: selected.provider, model: "old-model", - endpointUrl: selectedAttachment.endpointUrl, - preferredInferenceApi: "openai-completions", - nativeLocalProviderAttachment: selectedAttachment, + endpointUrl: endpointUrl ?? selected.endpointUrl, + ...(credentialEnv ? { credentialEnv } : {}), + preferredInferenceApi: selected.api ?? "openai-completions", + nativeLocalProviderAttachment: recordedAttachment ?? recorded, }, }); } export function switchingFixture( - previous = attachment, - provider: "ollama-local" | "vllm-local" = "vllm-local", + override: Partial = {}, + prior: Parameters = [], ) { - const deps = fixture(previous); + const deps = fixture(...prior); const nextBinding = { ...binding, - provider, - endpointUrl: - provider === "ollama-local" ? binding.endpointUrl : "http://host.openshell.internal:8000/v1", + provider: "vllm-local" as const, + endpointUrl: "http://host.openshell.internal:8000/v1", + ...override, }; const next = { ...nextBinding, @@ -79,9 +93,9 @@ export function switchingFixture( }; deps.resolveLocalSwitchRuntime = () => ({ endpointUrl: next.endpointUrl, - credentialValue: provider === "ollama-local" ? "ollama" : "dummy", + credentialValue: nextBinding.provider === "ollama-local" ? "ollama" : "dummy", authMode: "sentinel", - ownedProxy: provider === "ollama-local", + ownedProxy: nextBinding.provider === "ollama-local", }); const adapter = deps.providerAdapter; const original = { ...adapter }; diff --git a/test/helpers/inference-set.ts b/test/helpers/inference-set.ts index 4bf3c885c9a..f04868342bf 100644 --- a/test/helpers/inference-set.ts +++ b/test/helpers/inference-set.ts @@ -346,7 +346,6 @@ export function createDeps(options: { captureOpenshell: ReturnType; setOpenClawConfigValues: ReturnType; writeSandboxConfig: ReturnType; - recomputeSandboxConfigHash: ReturnType; updateSandbox: ReturnType; readSandboxConfig: ReturnType; updateSession: ReturnType; @@ -386,7 +385,6 @@ export function createDeps(options: { ), setOpenClawConfigValues: vi.fn(), writeSandboxConfig: vi.fn(), - recomputeSandboxConfigHash: vi.fn(), updateSandbox: vi.fn(options.updateSandbox ?? (() => true)), readSandboxConfig: vi.fn(() => options.config), updateSession: vi.fn((mutator: (value: Session) => Session | void) => { @@ -548,7 +546,6 @@ export function createDeps(options: { readSandboxConfig: calls.readSandboxConfig, setOpenClawConfigValues: calls.setOpenClawConfigValues, writeSandboxConfig: calls.writeSandboxConfig, - recomputeSandboxConfigHash: calls.recomputeSandboxConfigHash, prepareRunOpenshell: calls.prepareRunOpenshell, inferenceRouteMutator, inferenceRouteObserver: diff --git a/test/helpers/installer-run-fixture.ts b/test/helpers/installer-run-fixture.ts index e5738d56b35..5fbf80d7f8e 100644 --- a/test/helpers/installer-run-fixture.ts +++ b/test/helpers/installer-run-fixture.ts @@ -231,7 +231,7 @@ fi`, }); } -/** Writes the package files that make a temporary root a source checkout. */ +/** Writes package identities for a source checkout. */ export function writeSourceCheckoutPackages(root: string): void { fs.writeFileSync( path.join(root, "package.json"), diff --git a/test/helpers/langchain-deepagents-code-patch-fixture.ts b/test/helpers/langchain-deepagents-code-patch-fixture.ts index d101decfd38..244cce2d1da 100644 --- a/test/helpers/langchain-deepagents-code-patch-fixture.ts +++ b/test/helpers/langchain-deepagents-code-patch-fixture.ts @@ -720,8 +720,87 @@ def _get_provider_kwargs(provider, *, model_name=None): ` from __future__ import annotations +import contextlib +import ipaddress +import threading from urllib.parse import urljoin, urlparse +_dns_pin_lock = threading.Lock() + +def _is_blocked_ip(ip: ipaddress.IPv4Address | ipaddress.IPv6Address) -> bool: + """Return True if \`ip\` belongs to a non-publicly-routable range. + + Rejects: private (RFC1918/ULA), loopback, link-local (including cloud + IMDS at \`169.254.169.254\`), reserved, multicast, unspecified + (\`0.0.0.0\`/\`::\`), and anything \`ipaddress\` does not consider globally + routable (catches benchmarking, documentation, and similar ranges the + explicit predicates miss). + + IPv4-mapped IPv6 (\`::ffff:a.b.c.d\`) and 6to4 (\`2002::/16\`) are unwrapped + to their underlying IPv4 address before the checks so that private + space tunneled inside an IPv6 wrapper is still caught — e.g., + \`::ffff:127.0.0.1\` and \`2002:a9fe:a9fe::1\` (6to4 over IMDS) both + evaluate as blocked. + """ + if isinstance(ip, ipaddress.IPv6Address): + if ip.ipv4_mapped is not None: + ip = ip.ipv4_mapped + elif ip.sixtofour is not None: + ip = ip.sixtofour + return ( + not ip.is_global + or ip.is_private + or ip.is_loopback + or ip.is_link_local + or ip.is_reserved + or ip.is_multicast + or ip.is_unspecified + ) + + +@contextlib.contextmanager +def _pinned_dns(hostname: str, allowed_ips: list[str]) -> Iterator[None]: + """Force outgoing urllib3 connections for \`hostname\` to use \`allowed_ips\`. + + Patches \`urllib3.util.connection.create_connection\` for the duration of + the context so that \`requests\` cannot re-resolve \`hostname\` to a + different IP than the one \`_validate_url\` vetted (defends against DNS + rebinding TOCTOU). The patch is process-global, so the module lock + serializes concurrent fetches. + + Args: + hostname: The exact hostname (already IDNA-encoded by the caller) + whose resolution must be pinned. + allowed_ips: The IPs \`_validate_url\` confirmed are safe to connect + to. Tried in order; the first that accepts the connection wins. + """ + from urllib3.util import connection as urllib3_connection + + with _dns_pin_lock: + original = urllib3_connection.create_connection + + def patched( + address: tuple[str, int], *args: Any, **kwargs: Any + ) -> socket.socket: + host, port = address[0], address[1] + if host != hostname: + return original(address, *args, **kwargs) + last_exc: OSError | None = None + for ip in allowed_ips: + try: + return original((ip, port), *args, **kwargs) + except OSError as exc: + last_exc = exc + assert last_exc is not None # noqa: S101 # loop body guarantees this + raise last_exc + + urllib3_connection.create_connection = patched # ty: ignore[invalid-assignment] # signature matches at runtime + try: + yield + finally: + urllib3_connection.create_connection = original + + _ALLOWED_URL_SCHEMES = frozenset({"http", "https"}) _MAX_FETCH_REDIRECTS = 5 diff --git a/test/helpers/managed-startup-root-replay-filesystem.ts b/test/helpers/managed-startup-root-replay-filesystem.ts index a20e215b6fa..414c6065532 100644 --- a/test/helpers/managed-startup-root-replay-filesystem.ts +++ b/test/helpers/managed-startup-root-replay-filesystem.ts @@ -55,7 +55,12 @@ export function mockRootReplayFilesystem( runtimeWrites: string[], seededFiles: ReadonlyMap< string, - { readonly contents: string | Buffer; readonly mode: number } + { + readonly contents: string | Buffer; + readonly mode: number; + readonly uid?: number; + readonly gid?: number; + } > = new Map(), ): { readonly beforeRename: (callback: ((source: string, target: string) => void) | null) => void; @@ -87,7 +92,10 @@ export function mockRootReplayFilesystem( "/var/lib", "/var/lib/nemoclaw", ]); - const fixtureFiles = new Map([ + const fixtureFiles = new Map< + string, + { contents: string | Buffer; mode: number; uid?: number; gid?: number } + >([ [ "/usr/sbin/update-ca-certificates", { contents: "managed startup test executable", mode: 0o555 }, @@ -112,9 +120,12 @@ export function mockRootReplayFilesystem( const fileModes = new Map([...fixtureFiles].map(([target, file]) => [target, file.mode])); let nextFileInode = 2n; const fileInodes = new Map(); + const fileOwners = new Map(); const fileCtimes = new Map(); for (const target of files.keys()) { fileInodes.set(target, nextFileInode); + const seed = fixtureFiles.get(target); + fileOwners.set(nextFileInode, { uid: seed?.uid ?? 0, gid: seed?.gid ?? 0 }); fileCtimes.set(target, 1n); nextFileInode += 1n; } @@ -185,7 +196,7 @@ export function mockRootReplayFilesystem( ({ ctimeNs, dev: 1n, - gid: 0n, + gid: BigInt(fileOwners.get(ino)?.gid ?? 0), ino, isDirectory: () => false, isFile: () => true, @@ -194,7 +205,7 @@ export function mockRootReplayFilesystem( mtimeNs: 1n, nlink, size: BigInt(bytes.length), - uid: 0n, + uid: BigInt(fileOwners.get(ino)?.uid ?? 0), }) as fs.BigIntStats; const missing = (): never => { throw Object.assign(new Error("missing"), { code: "ENOENT" }); @@ -243,7 +254,12 @@ export function mockRootReplayFilesystem( fileCtimes.get(resolved) ?? missing(), fileLinkCount(fileInodes.get(resolved) ?? missing()), ) - : stat("file", mode); + : stat( + "file", + mode, + fileOwners.get(fileInodes.get(resolved)!)?.uid ?? 0, + fileOwners.get(fileInodes.get(resolved)!)?.gid ?? 0, + ); }) as typeof fs.lstatSync); vi.spyOn(fs, "mkdirSync").mockImplementation((( target: fs.PathLike, diff --git a/test/helpers/onboard-created-sandbox-fixture.test.ts b/test/helpers/onboard-created-sandbox-fixture.test.ts index 481b04bdbfb..74e3ad98543 100644 --- a/test/helpers/onboard-created-sandbox-fixture.test.ts +++ b/test/helpers/onboard-created-sandbox-fixture.test.ts @@ -8,6 +8,7 @@ import { NEMOCLAW_CREATE_ATTEMPT_LABEL, NEMOCLAW_CREATE_ATTEMPT_NONCE_HEX_LENGTH, isOpenShellSandboxId, + observeCreatedOpenShellSandboxId, parseOpenShellSandboxId, parseStrictOpenShellSandboxListJson, } from "../../src/lib/adapters/openshell/sandbox-identity"; @@ -49,7 +50,7 @@ function selectorListCommand(gatewayName: string | null, nonce = CREATE_ATTEMPT_ `${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}`, "--output", "json", - "--limit", + "--page-size", "2", ]; } @@ -77,6 +78,18 @@ describe("created sandbox fixture", () => { fixture.create(createCommand()); const createdSandboxId = fixture.state.sandboxId; + expect( + observeCreatedOpenShellSandboxId( + { + sandboxName: "alpha", + gatewayName: "gateway-alpha", + createAttemptNonce: CREATE_ATTEMPT_NONCE, + runCaptureOpenshell: (args) => fixture.capture(["openshell", ...args]) ?? "", + }, + 1000, + ), + ).toEqual({ state: "matched", sandboxId: createdSandboxId }); + const selectorOutput = fixture.capture(selectorListCommand("gateway-alpha")); const rows = parseStrictOpenShellSandboxListJson(selectorOutput ?? ""); expect(rows).toHaveLength(1); diff --git a/test/helpers/onboard-final-flow-phases.ts b/test/helpers/onboard-final-flow-phases.ts index 11e4d023c35..2c10e3393b5 100644 --- a/test/helpers/onboard-final-flow-phases.ts +++ b/test/helpers/onboard-final-flow-phases.ts @@ -217,7 +217,7 @@ export function createPhases( handleAgentSetup: vi.fn(async () => { order.push("agent-setup"); }), - agentSetupContext: () => ({ gatewayName: "nemoclaw-19090" }), + agentSetupContext: () => ({ gatewayName: "nemoclaw-19090", getSandbox: () => null }), ensureAgentDashboardForward: vi.fn(() => { order.push("agent-forward"); return 45123; diff --git a/test/helpers/onboard-script-mocks-openshell-capture.test.ts b/test/helpers/onboard-script-mocks-openshell-capture.test.ts index 1c43a99ea81..68fcf2476c6 100644 --- a/test/helpers/onboard-script-mocks-openshell-capture.test.ts +++ b/test/helpers/onboard-script-mocks-openshell-capture.test.ts @@ -23,7 +23,7 @@ const exactCreateQuery = [ selector, "--output", "json", - "--limit", + "--page-size", "2", ] as const; const exactCreateCommand = ["openshell", "sandbox", "create", "--label", selector] as const; @@ -51,7 +51,17 @@ describe("created sandbox fixture selector observations", () => { it.each([ [ "an unscoped query", - ["openshell", "sandbox", "list", "--selector", selector, "--output", "json", "--limit", "2"], + [ + "openshell", + "sandbox", + "list", + "--selector", + selector, + "--output", + "json", + "--page-size", + "2", + ], ], [ "a different gateway", @@ -67,7 +77,11 @@ describe("created sandbox fixture selector observations", () => { ], [ "missing the two-row bound", - exactCreateQuery.filter((part) => part !== "--limit" && part !== "2"), + exactCreateQuery.filter((part) => part !== "--page-size" && part !== "2"), + ], + [ + "the unsupported legacy row-bound flag", + exactCreateQuery.map((part) => (part === "--page-size" ? "--limit" : part)), ], ["a different row bound", exactCreateQuery.map((part) => (part === "2" ? "3" : part))], [ diff --git a/test/helpers/onboard-script-mocks.cjs b/test/helpers/onboard-script-mocks.cjs index e895e716955..ff5ddf42ae1 100644 --- a/test/helpers/onboard-script-mocks.cjs +++ b/test/helpers/onboard-script-mocks.cjs @@ -7,6 +7,20 @@ const Module = require("node:module"); const path = require("node:path"); +function readMessagingProfile() { + const fs = require("node:fs"); + const YAML = require("yaml"); + return YAML.parse( + fs.readFileSync( + path.resolve( + __dirname, + "../../nemoclaw-blueprint/provider-profiles/nemoclaw-messaging-v1.yaml", + ), + "utf8", + ), + ); +} + // Onboarding fixtures use synthetic Discord tokens and must not contact Discord. const fixtureFetch = global.fetch; global.fetch = (url, options) => @@ -269,7 +283,8 @@ function mockEndpointlessProviderProfileRun(command, profileId, inferenceCapable status: 0, stdout: JSON.stringify({ id: profileId, - credentials: [], + credentials: + profileId === "nemoclaw-messaging-v1" ? readMessagingProfile().credentials : [], endpoints: [], binaries: [], inference_capable: inferenceCapable, @@ -290,7 +305,7 @@ function mockEndpointlessProviderProfileRun(command, profileId, inferenceCapable } function mockManagedEndpointlessProviderProfileRun(command) { - return mockEndpointlessProviderProfileRun(command, "nemoclaw-mcp-v1", false); + return mockEndpointlessProviderProfileRun(command, "nemoclaw-messaging-v1", false); } function mockProviderPreparationRun(command, gatewayName, profileId, inferenceCapable) { @@ -315,13 +330,7 @@ function createStatefulMessagingProviderRunner({ const providers = new Map( initialProviders.map(([name, type, credential]) => [name, { type, credential }]), ); - const messagingProfile = JSON.stringify({ - id: "nemoclaw-mcp-v1", - credentials: [], - endpoints: [], - binaries: [], - inference_capable: false, - }); + const messagingProfileOutput = JSON.stringify(readMessagingProfile()); let messagingProfileImported = false; let lifecycleReleased = false; return (command, options = {}) => { @@ -339,7 +348,7 @@ function createStatefulMessagingProviderRunner({ args[profileActionIndex] === "-g" ? args[profileActionIndex + 2] : args[profileActionIndex]; if (profileAction === "export") { return messagingProfileImported - ? { status: 0, stdout: messagingProfile, stderr: "" } + ? { status: 0, stdout: messagingProfileOutput, stderr: "" } : { status: 1, stdout: "", stderr: "provider profile not found" }; } const fileIndex = args.indexOf("--file"); @@ -685,7 +694,7 @@ function createCreatedSandboxFixture(options = {}) { !exactArgs[5].startsWith(prefix) || exactArgs[6] !== "--output" || exactArgs[7] !== "json" || - exactArgs[8] !== "--limit" || + exactArgs[8] !== "--page-size" || exactArgs[9] !== "2" ) { return null; @@ -1312,7 +1321,7 @@ function mockDockerSandboxLifecycleReleaseFromRunner() { state.finalCommitReleased || state.legacyRecoverySandboxId ? ONBOARD_SANDBOX_NEW_CONTAINER_ID : ONBOARD_SANDBOX_OLD_CONTAINER_ID; - return `${containerId}\topenshell\tdefault\t${state.sandboxId || ONBOARD_READY_SANDBOX_ID}\n`; + return `${containerId}\topenshell\tdefault\t${state.sandboxId || ONBOARD_READY_SANDBOX_ID}\t\t\n`; } if ( (state.finalCommitReleased || state.legacyRecoverySandboxId) && diff --git a/test/helpers/openclaw-nested-fetch-evidence.ts b/test/helpers/openclaw-nested-fetch-evidence.ts new file mode 100644 index 00000000000..1e8b8d7822c --- /dev/null +++ b/test/helpers/openclaw-nested-fetch-evidence.ts @@ -0,0 +1,123 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// OpenClaw 2026.9.2 records nested tool results as openclaw.nested-tool.v1. +// Its model.completed snapshot contains the wrapper, without a projected target. +export function nestedFetchEvidence(variant: string) { + const url = "https://www.wikidata.org/w/api.php?entity=Q30&secret=must-not-remain"; + const input = { url, maxChars: 8000 }; + const payload = { url, finalUrl: url, status: 200, text: "United States" }; + const wrapper = { + role: "assistant", + content: [ + { + type: "toolCall", + id: "wrapper-1", + name: "tool_call", + arguments: { id: "openclaw:core:web_fetch", args: input }, + }, + ], + }; + const result = { + role: "toolResult", + toolCallId: "wrapper-1", + toolName: "tool_call", + content: [{ type: "text", text: "{}" }], + isError: false, + }; + const details: Record = { + runId: "run-1", + scopeId: "scope-1", + afterEntryId: "entry-1", + startOrder: 0, + parentToolCallId: "wrapper-1", + toolCallId: "tool_search_code:wrapper-1:web_fetch:1", + toolName: "web_fetch", + input, + result: { content: [{ type: "text", text: JSON.stringify(payload) }], details: payload }, + isError: false, + startedAt: 1, + timestamp: 2, + }; + const nested = { + role: "custom", + customType: "openclaw.nested-tool.v1", + display: true, + excludeFromContext: true, + content: "", + details, + timestamp: 1, + }; + switch (variant) { + case "wrong run": + details.runId = "another-run"; + break; + case "wrong parent": + details.parentToolCallId = "another-wrapper"; + break; + case "wrong generated ID": + details.toolCallId = "tool_search_code:another-wrapper:web_fetch:1"; + break; + case "wrong wrapper target": + wrapper.content[0]!.arguments.id = "openclaw:core:exec"; + break; + case "tool failure": + details.isError = true; + break; + case "missing error flag": + delete details.isError; + break; + case "missing result": + delete details.result; + break; + case "omitted result": + details.result = { + content: [{ type: "text", text: "[Nested tool output omitted: exceeds display limit]" }], + }; + break; + case "oversized request": + input.maxChars = 8001; + break; + case "wrong URL": + input.url = "https://example.com/"; + break; + case "wrong role": + nested.role = "assistant"; + break; + case "context-visible record": + nested.excludeFromContext = false; + break; + } + const messages: unknown[] = [wrapper, result, nested]; + if (variant === "duplicate record") messages.push(nested); + if (variant === "hidden forbidden tool") + messages.push({ + ...nested, + details: { ...details, toolCallId: "tool_search_code:wrapper-1:exec:2", toolName: "exec" }, + }); + const trajectory = [ + { type: "model.completed", runId: "run-1", data: { messagesSnapshot: [wrapper, result] } }, + { + type: "trace.artifacts", + runId: "run-1", + data: { + finalStatus: "success", + toolMetas: [ + { toolName: "web_fetch", meta: url }, + { toolName: "tool_call", meta: {} }, + ], + }, + }, + ]; + if (variant === "parent outside snapshot") { + const snapshot = structuredClone([wrapper, result]); + (snapshot[0] as typeof wrapper).content[0]!.id = "current-wrapper"; + (snapshot[1] as typeof result).toolCallId = "current-wrapper"; + trajectory[0]!.data.messagesSnapshot = snapshot; + } + return { + session: messages.map((message) => JSON.stringify({ type: "message", message })).join("\n"), + trajectory: trajectory.map((event) => JSON.stringify(event)).join("\n"), + expectation: { url, content: "United States" }, + }; +} diff --git a/test/helpers/openclaw-pairing-fixture.ts b/test/helpers/openclaw-pairing-fixture.ts index 879dedf4b8e..4d93e9d254f 100644 --- a/test/helpers/openclaw-pairing-fixture.ts +++ b/test/helpers/openclaw-pairing-fixture.ts @@ -81,3 +81,83 @@ export function createOpenClawPairingFixture() { writeJson(path.join(stateDirectory, "devices", "pending.json"), {}); return { root, stateDirectory, deviceId, publicKey, privateKeyPem }; } + +export const SQLITE_PAIRING_FIXTURE_SCRIPT = String.raw` +import json +import os +import sqlite3 +import sys + +state_dir = sys.argv[1] +value = json.load(sys.stdin) +os.umask(0o007) +sqlite_state_dir = os.path.join(state_dir, 'state') +os.makedirs(sqlite_state_dir, mode=0o770, exist_ok=True) +database = os.path.join(sqlite_state_dir, 'openclaw.sqlite') +connection = sqlite3.connect(database) +connection.execute('PRAGMA journal_mode = WAL') +connection.execute('PRAGMA wal_autocheckpoint = 0') +connection.executescript(''' +CREATE TABLE device_identities ( + identity_key TEXT PRIMARY KEY, device_id TEXT NOT NULL, public_key_pem TEXT NOT NULL, + private_key_pem TEXT NOT NULL, created_at_ms INTEGER NOT NULL, updated_at_ms INTEGER NOT NULL +); +CREATE TABLE device_pairing_paired ( + device_id TEXT PRIMARY KEY, public_key TEXT NOT NULL, display_name TEXT, + operator_label TEXT, platform TEXT, device_family TEXT, client_id TEXT, client_mode TEXT, + browser_origin TEXT, role TEXT, roles_json TEXT, scopes_json TEXT, + approved_scopes_json TEXT, remote_ip TEXT, tokens_json TEXT, approved_via TEXT, + node_surface_json TEXT, pending_node_surface_json TEXT, created_at_ms INTEGER NOT NULL, + approved_at_ms INTEGER NOT NULL, last_seen_at_ms INTEGER, last_seen_reason TEXT +); +CREATE TABLE device_pairing_pending ( + request_id TEXT PRIMARY KEY, device_id TEXT NOT NULL, public_key TEXT NOT NULL, + display_name TEXT, platform TEXT, device_family TEXT, client_id TEXT, client_mode TEXT, + browser_origin TEXT, role TEXT, roles_json TEXT, scopes_json TEXT, remote_ip TEXT, + silent INTEGER, is_repair INTEGER, ts INTEGER NOT NULL, refreshed_at_ms INTEGER +); +CREATE TABLE device_auth_tokens ( + device_id TEXT NOT NULL, role TEXT NOT NULL, token TEXT NOT NULL, scopes_json TEXT NOT NULL, + updated_at_ms INTEGER NOT NULL, + PRIMARY KEY (device_id, role) +); +''') +connection.execute(f"PRAGMA user_version = {value.get('userVersion', 15)}") +connection.execute( + 'INSERT INTO device_identities VALUES (?, ?, ?, ?, ?, ?)', + ('primary', value['deviceId'], value['publicKeyPem'], value['privateKeyPem'], 1, 1), +) +operator = { + 'operator': { + 'token': value['token'], + 'role': 'operator', + 'scopes': value['pairedTokenScopes'], + }, +} +connection.execute( + 'INSERT INTO device_pairing_paired VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', + ( + value['deviceId'], value['publicKey'], None, None, None, None, 'cli', 'cli', None, + 'operator', json.dumps(['operator']), json.dumps(value['pairedScopes']), + json.dumps(value['pairedScopes']), None, json.dumps(operator), None, None, None, + 1, 1, None, None, + ), +) +connection.execute( + 'INSERT INTO device_auth_tokens VALUES (?, ?, ?, ?, ?)', + (value['deviceId'], 'operator', value['token'], json.dumps(value['authScopes']), 1), +) +for request in value.get('pending') or []: + connection.execute( + 'INSERT INTO device_pairing_pending VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', + ( + request['requestId'], request['deviceId'], request['publicKey'], None, None, None, + request.get('clientId'), request.get('clientMode'), None, request.get('role'), + json.dumps(request['roles']) if 'roles' in request else None, + json.dumps(request['scopes']) if 'scopes' in request else None, + None, None, int(request['isRepair']) if 'isRepair' in request else None, 1, None, + ), + ) +connection.commit() +os._exit(0) +`; diff --git a/test/helpers/openshell-installer-template.ts b/test/helpers/openshell-installer-template.ts index db6a4911e4c..76298a025b9 100644 --- a/test/helpers/openshell-installer-template.ts +++ b/test/helpers/openshell-installer-template.ts @@ -4,9 +4,23 @@ import assert from "node:assert/strict"; import fs from "node:fs"; import path from "node:path"; -import { v00116Pins, v012Pins } from "./openshell-release-fixtures"; +import { extractInstallerPins } from "../../scripts/checks/extract-installer-pins.mts"; +import { v00116Pins } from "./openshell-release-fixtures"; import { selectHistoricalGatewayProcessReader } from "./prepared-gateway-runtime"; +export function npmReplacementFixtureVersion(source: string): string { + const versions = new Set( + extractInstallerPins(source, { + functionName: "openshell_cli_pinned_sha256", + sourceLabel: "Brev launchable", + }).map((pin) => pin.releaseVersion), + ); + assert.equal(versions.size, 1, "npm replacement fixture must select one OpenShell release"); + const [version] = versions; + assert.ok(version); + return version; +} + /** Stage the verifier and its inert registry for both candidate and separate trusted checkouts. */ export function copyTrustedInstallerVerifier(repoRoot: string, targetRoot: string): void { [ @@ -126,6 +140,79 @@ const MACOS_METHOD_START = `MACOS_INSTALL_METHOD="\${_NEMOCLAW_OPENSHELL_INSTALL const MACOS_METHOD_END = "esac\n"; export function installerReleaseTemplate(source: string, version: string): string { + if (version !== "0.1.7") { + const v017SandboxBuild = String.raw` # OpenShell v0.1.7 standalone sandbox binaries. + b4ae1d60b238c84e290bcd08a962997e5c36ed04237fa59b09b5799b11363feb | \ + d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21) + printf '%s\n' "0.1.7" + ;; +`; + assert.ok(source.includes(v017SandboxBuild), "OpenShell 0.1.7 sandbox fixture boundary"); + source = source + .replace(v017SandboxBuild, "") + .replace( + "OpenShell 0.1.2, 0.1.3, 0.1.5, and 0.1.7 enforce", + "OpenShell 0.1.2, 0.1.3, and 0.1.5 enforce", + ) + .replace("0.1.2 | 0.1.3 | 0.1.5 | 0.1.7)", "0.1.2 | 0.1.3 | 0.1.5)"); + } + if (version !== "0.1.5" && version !== "0.1.7") { + const v015SandboxBuild = String.raw` # OpenShell v0.1.5 standalone sandbox binaries. + a048b1fd2acd78f44f50a4cf0e02cdefec691656f3941e92c5ec0e09a69990e2 | \ + 65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc) + printf '%s\n' "0.1.5" + ;; +`; + assert.ok(source.includes(v015SandboxBuild), "OpenShell 0.1.5 sandbox fixture boundary"); + source = source + .replace(v015SandboxBuild, "") + .replace("OpenShell 0.1.2, 0.1.3, and 0.1.5 enforce", "OpenShell 0.1.2 and 0.1.3 enforce") + .replace("0.1.2 | 0.1.3 | 0.1.5)", "0.1.2 | 0.1.3)"); + } + // Historical fixtures must retain their reviewed pre-0.1.3 feature gate. + if (version !== "0.1.3" && version !== "0.1.5" && version !== "0.1.7") { + const v013SandboxBuild = String.raw` # OpenShell v0.1.3 standalone sandbox binaries. + 38fe063f19c72680a1bc4b0c343a3ec5b7a5cb087a627faab5b2aefd9d50d094 | \ + 23147b42d31bd0b13e1a9ed7b3834b8e6075836cba0872fffa9e7d481f6dd281) + printf '%s\n' "0.1.3" + ;; +`; + assert.ok(source.includes(v013SandboxBuild), "OpenShell 0.1.3 sandbox fixture boundary"); + source = source.replace(v013SandboxBuild, ""); + source = source.replace( + " # OpenShell 0.1.2 and 0.1.3 enforce MCP policy in their supervisor images.\n" + + " # Require a pinned sandbox digest matching the CLI, not version text alone.\n", + " # OpenShell 0.1.2 enforces MCP policy in its supervisor image. Recognize\n" + + " # only its exact pinned sandbox artifacts here, not version text alone.\n", + ); + const fallbackStart = source.indexOf( + " local sandbox_digest sandbox_version openshell_version\n", + ); + if (fallbackStart !== -1) { + const fallbackEnd = source.indexOf(" esac\n", fallbackStart); + assert.ok(fallbackEnd > fallbackStart, "OpenShell 0.1.3 sandbox fixture boundary"); + source = + source.slice(0, fallbackStart) + + " local sandbox_digest\n" + + ' sandbox_digest="$(file_sha256 "$sandbox_bin")" || return 1\n' + + ' if [ "$(pinned_sandbox_build_version "$sandbox_digest")" = "0.1.2" ]; then\n' + + " return 0\n" + + " fi\n" + + source.slice(fallbackEnd + " esac\n".length); + } + } + // Historical trust fixtures must use their own reviewed guard. Keep the + // current guard only in the current-release fixture. + const guardStart = source.indexOf("# Prove absence through searchable parents;"); + if (version !== "0.1.3" && version !== "0.1.5" && version !== "0.1.7" && guardStart !== -1) { + const guardEnd = source.indexOf( + "# invalidState: a consumed OpenShell release asset", + guardStart, + ); + assert.ok(guardEnd > guardStart, "OpenShell 0.1.2 retirement guard fixture boundary"); + source = source.slice(0, guardStart) + source.slice(guardEnd); + source = source.replace("verify_legacy_openshell_state\n\n", ""); + } // Historical trust fixtures retain the installer bytes authorized before the // direct-install permission fix. The current candidate is checked separately. const safetyHelper = "user_local_executable_directory_safe() {\n"; @@ -192,7 +279,7 @@ fi ) .replace(fallback, ""); } - if (version === "0.0.106" || version === "0.0.116" || version === "0.1.2") return source; + if (["0.0.106", "0.0.116", "0.1.2", "0.1.3", "0.1.5", "0.1.7"].includes(version)) return source; const start = source.indexOf(MACOS_METHOD_START); const end = source.indexOf(MACOS_METHOD_END, start); if (start === -1 || end === -1 || source.indexOf(MACOS_METHOD_START, start + 1) !== -1) { @@ -273,10 +360,13 @@ export function prepareReleaseFixtureRuntime( // currentProcessReader selects the checkout's thread-group reader for 0.1.2 instead. const source = options.currentProcessReader ? checkoutSource.replace( - 'const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.0.116";', + /const QUALIFIED_STABLE_OPENSHELL_VERSION = "[0-9.]+";/, 'const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.1.2";', ) - : selectHistoricalGatewayProcessReader(checkoutSource); + : selectHistoricalGatewayProcessReader(checkoutSource).replace( + /const QUALIFIED_STABLE_OPENSHELL_VERSION = "[0-9.]+";/, + 'const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.1.2";', + ); const prepared = source.replace( /const OPENSHELL_SUPERVISOR_MANIFEST_DIGESTS: Readonly> = \{[\s\S]*?\n\};/, candidatePins.trim(), @@ -388,7 +478,7 @@ export type FixtureMode = | "trusted-formula-mismatch"; // Historical prospective templates are authorized for 0.0.116 only. -// Keep their fixtures on that release even when the checkout selects 0.1.2. +// Keep their fixtures on that release when the checkout selects a newer version. export function supervisorV00116Fixtures(selected: { blueprint: string; brevInstaller: string; @@ -396,23 +486,28 @@ export function supervisorV00116Fixtures(selected: { supervisorRuntime: string; }) { const pins = v00116Pins("installer"); + const selectedPins = extractInstallerPins(selected.installer, { + functionName: "openshell_pinned_sha256", + sourceLabel: "installer", + }); + const selectedVersion = npmReplacementFixtureVersion(selected.brevInstaller); const restorePins = (source: string) => - v012Pins("installer").reduce((result, pin) => { + selectedPins.reduce((result, pin) => { const original = pins.find((entry) => entry.asset === pin.asset); assert.ok(original, "0.0.116 fixture pin must exist"); return result.replaceAll(pin.sha256, original.sha256); }, source); return { blueprint: selected.blueprint.replaceAll( - 'openshell_version: "0.1.2"', + `openshell_version: "${selectedVersion}"`, 'openshell_version: "0.0.116"', ), - brevInstaller: restorePins(selected.brevInstaller).replaceAll("0.1.2", "0.0.116"), + brevInstaller: restorePins(selected.brevInstaller).replaceAll(selectedVersion, "0.0.116"), installer: restorePins(installerReleaseTemplate(selected.installer, "0.0.116")) - .replaceAll('VERSION="0.1.2"', 'VERSION="0.0.116"') - .replaceAll("v0.1.2:", "v0.0.116:"), + .replaceAll(`VERSION="${selectedVersion}"`, 'VERSION="0.0.116"') + .replaceAll(`v${selectedVersion}:`, "v0.0.116:"), supervisorRuntime: selected.supervisorRuntime.replace( - 'const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.1.2";', + `const QUALIFIED_STABLE_OPENSHELL_VERSION = "${selectedVersion}";`, 'const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.0.116";', ), }; diff --git a/test/helpers/openshell-release-fixtures.ts b/test/helpers/openshell-release-fixtures.ts index 89852f4f4ee..db34fccf9ed 100644 --- a/test/helpers/openshell-release-fixtures.ts +++ b/test/helpers/openshell-release-fixtures.ts @@ -13,16 +13,16 @@ export const CANDIDATE_RUNTIME = { export const CANDIDATE_RUNTIME_ENABLED = Object.values(CANDIDATE_RUNTIME).every(Boolean); export const PINNED_OPEN_SHELL_SHA256 = { - cliDarwinArm64: "e582f2374053bebac8e6aaeb4a369931b7d4bb97bd55055e2c02e85502627e22", - cliLinuxArm64: "7a949c48d1e000cd280869eea1e203e24816b9cfefc575b68a8b72b939cb3f43", - cliLinuxX64: "4fb4476d80a1875a0b83547ec3aba999cf0a2e2d75f95f2f709b622e2103520e", - formula: "cf00a9441589702ffe006720fd6a9dffc0f0745b337036aad26dc53eb94c1558", - gatewayDarwinArm64: "f192d3d737c125264e13ef73458541df2ca6a9eb2fa599736a7f2587d5d2ce8d", - gatewayLinuxArm64: "292c379193a339220234ffea585350901468bb8f4076e2076bc074e8ed18974b", - gatewayLinuxX64: "59c6da724eae7a00c28826f9191efbdf4fbaa5c768afdc8dea6a80a949ebcc89", - sandboxLinuxArm64: "959d9a88270e0336f04342560df750591da603424d0a9bfb481ee29670342557", - sandboxLinuxX64: "0bb160f73e5007338b94e3c868f66f50c71cd65c27c932ed9a4fa67c49e6d423", - sandboxBinaryLinuxX64: "326ee26df8f8575ba761470757a12fe5c1cdc904ba064b81946692dd0328dd40", + cliDarwinArm64: "c5899a2fe6b40c371e7d660bd8983cefb766d04658bd9193b12a80525b12c742", + cliLinuxArm64: "696b523c263895b69eaa7c7e6afbb19b8af6c65cd79c7ffac7b685b6e3aa23da", + cliLinuxX64: "5c1adfcae797eac9b2e8b9280c66c1f75e4e84d9e31ce6938c5b994314ef6098", + formula: "daa268c73d627cc72fb5c36b041e0e8625b1125cc83706f76772d52b5da081d7", + gatewayDarwinArm64: "fe47a385c16c10cde8905bf1bf627e4a9fbf4a81bcf44b388dec4d22618fbb48", + gatewayLinuxArm64: "eb0ba91efebbcc61e4cfae5fb18934c5fe308ece1a09ba16c999d00c452109dc", + gatewayLinuxX64: "e1714dcfd85c949e2f53260bcfbf04b203f6f5cbd3a61e61c11c38cbc91cb102", + sandboxLinuxArm64: "ed60eab7612f372a0f669186ac2637fe3223fe90a1c20ad51d747e1a87ca5c6d", + sandboxLinuxX64: "c1b3166d1c232948293ff84b0fac2d0419a24e4b694d354ba6aeb100efa2f039", + sandboxBinaryLinuxX64: "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21", } as const; export const ZERO_SHA256 = "0000000000000000000000000000000000000000000000000000000000000000"; @@ -88,6 +88,9 @@ export const INSTALLER_HASH_SUPERVISOR_MANIFEST_DIGESTS = new Map([ ["0.0.106", V00106_SUPERVISOR_MANIFEST_DIGEST], ["0.0.116", V00116_SUPERVISOR_MANIFEST_DIGEST], ["0.1.2", V012_SUPERVISOR_MANIFEST_DIGEST], + ["0.1.3", "sha256:4ab2b970cdb02c7518ba4ccbbadb4861ae9a8d0b8770e5b160697f0453b0a8e4"], + ["0.1.5", "sha256:fabcd01f656e9d23df4386f82e80b4f48b90f3452d7d5c58889007643a069161"], + ["0.1.7", "sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06"], ["9.9.9", `sha256:${"c".repeat(64)}`], ]); @@ -517,6 +520,7 @@ export const V012_TEMPLATE_DIGESTS = new Map([ [ "6808b7c667aef5c9ebdfe269ae1f9b4c181b5de6a6e62bdb526fac4338f5ee4f", "610fa58bc4242f23e6ce5e22444dd595995c5a9466bf406a34e5a548869377bc", + "4d5970db104adeddcb90e38ff13f482c5e41e47897007c14b181c5ab2c3f9ff2", ], ], [ diff --git a/test/helpers/rebuild-dcode-flow-helpers.ts b/test/helpers/rebuild-dcode-flow-helpers.ts index fc46c172dd5..0c978691e46 100644 --- a/test/helpers/rebuild-dcode-flow-helpers.ts +++ b/test/helpers/rebuild-dcode-flow-helpers.ts @@ -4,9 +4,10 @@ import { expect } from "vitest"; import { expectNoSandboxDelete } from "./rebuild-delete-assertions"; +import { customProviderRebuildEntry } from "../../src/lib/actions/sandbox/rebuild-flow-test-fixtures"; import type { RebuildFlowHarness } from "./rebuild-flow-generic-harness"; -export function makeDcodeSandboxEntry(): Record { +export function makeDcodeSandboxEntry(native = true): Record { return { name: "alpha", agent: "langchain-deepagents-code", @@ -25,6 +26,7 @@ export function makeDcodeSandboxEntry(): Record { gpuEnabled: false, sandboxGpuEnabled: false, sandboxGpuMode: "0", + ...(native ? customProviderRebuildEntry() : {}), }; } @@ -59,9 +61,10 @@ export function setGatewayProviderMetadata(harness: RebuildFlowHarness, stdout: }; } if (argv[0] === "provider" && argv[1] === "list") { - const output = JSON.stringify([ - { name: "compatible-endpoint", credential_keys: ["COMPATIBLE_API_KEY"] }, - ]); + const output = JSON.stringify({ + providers: [{ name: "compatible-endpoint", credential_keys: ["COMPATIBLE_API_KEY"] }], + next_page_token: "", + }); return { status: 0, output, stdout: output, stderr: "" }; } return argv[0] === "provider" && argv[1] === "get" diff --git a/test/helpers/rebuild-flow-generic-harness.ts b/test/helpers/rebuild-flow-generic-harness.ts index 61a3faa4c31..f0a37bec703 100644 --- a/test/helpers/rebuild-flow-generic-harness.ts +++ b/test/helpers/rebuild-flow-generic-harness.ts @@ -7,6 +7,8 @@ import { execFileSync } from "node:child_process"; import path from "node:path"; import { vi } from "vitest"; import { makePreparedRecoveryManifest } from "../../src/lib/actions/sandbox/rebuild-flow-test-fixtures"; +import { recordedNativeProviderAttachment } from "../../src/lib/inference/native-provider/recorded-selection"; +import { fixedNativeProviderForAttachment } from "../../src/lib/inference/native-provider/fixed"; import type { OpenShellRuntimeSelection } from "../../src/lib/adapters/openshell/runtime-selection"; import type { RebuildRecreateOnboardOpts } from "../../src/lib/actions/sandbox/rebuild-gpu-opt-out"; import { @@ -56,7 +58,7 @@ import { rebuildCustomImagePreflight, rebuildFlowHelpers, rebuildInference, - rebuildProviderPreflight, + rebuildHostCredentialPreflight, rebuildManagedImage, rebuildMessagingConflict, rebuildOnboardDependencies, @@ -118,7 +120,10 @@ export function createRebuildFlowHarness(overrides: RebuildFlowOverrides = {}): const nativeSource = createHarnessTempDir("nemoclaw-rebuild-native-"); fs.mkdirSync(path.join(nativeSource, ".hermes")); const nativeArchive = path.join(backupPath, "native-home.tar"); - execFileSync("tar", ["-C", nativeSource, "-cf", nativeArchive, "--", "."]); + execFileSync("tar", ["-C", nativeSource, "-cf", nativeArchive, "--", "."], { + // Linux sandbox archives have no macOS AppleDouble metadata entries. + env: { ...process.env, COPYFILE_DISABLE: "1" }, + }); const nativeArchiveSha256 = createHash("sha256") .update(fs.readFileSync(nativeArchive)) .digest("hex"); @@ -399,9 +404,10 @@ export function createRebuildFlowHarness(overrides: RebuildFlowOverrides = {}): ); const defaultHydrateCredentialEnv = onboardCredentialEnv.hydrateCredentialEnv.bind(onboardCredentialEnv); - vi.spyOn(rebuildProviderPreflight, "validateRebuildHostInferenceCredential").mockResolvedValue( - true, - ); + vi.spyOn( + rebuildHostCredentialPreflight, + "validateRebuildHostInferenceCredential", + ).mockResolvedValue(true); const hydrateCredentialEnvSpy = vi .spyOn(rebuildOnboardDependencies, "hydrateCredentialEnv") .mockImplementation((...args: unknown[]) => { @@ -788,6 +794,31 @@ export function createRebuildFlowHarness(overrides: RebuildFlowOverrides = {}): stderr: "sandbox alpha not found", }; } + const attachment = recordedNativeProviderAttachment(currentSandboxEntry); + if (argv[0] === "provider" && attachment) { + const definition = fixedNativeProviderForAttachment(attachment); + const stdout = + argv[1] === "list" + ? JSON.stringify({ + providers: [ + { + name: attachment.providerName, + credential_keys: [definition.credentialEnv], + credential_expires_at_ms: {}, + }, + ], + next_page_token: "", + }) + : [ + `Name: ${attachment.providerName}`, + `Type: ${attachment.profileId}`, + `Credential keys: ${definition.credentialEnv}`, + "Config keys: ", + `Id: ${attachment.providerId}`, + "Resource version: 1", + ].join("\n"); + return { status: 0, stdout, output: stdout, stderr: "" }; + } if (argv[0] === "provider" && argv[1] === "list") { const provider = String(currentSandboxEntry.provider ?? "compatible-endpoint"); const credentialEnv = @@ -795,12 +826,15 @@ export function createRebuildFlowHarness(overrides: RebuildFlowOverrides = {}): typeof currentSandboxEntry.credentialEnv === "string" ? currentSandboxEntry.credentialEnv : "COMPATIBLE_API_KEY"; - const output = JSON.stringify([ - { - name: provider, - credential_keys: credentialEnv ? [credentialEnv] : [], - }, - ]); + const output = JSON.stringify({ + providers: [ + { + name: provider, + credential_keys: credentialEnv ? [credentialEnv] : [], + }, + ], + next_page_token: "", + }); return { status: 0, output, stdout: output, stderr: "" }; } return argv[0] === "provider" && argv[1] === "get" diff --git a/test/helpers/rebuild-flow-harness.ts b/test/helpers/rebuild-flow-harness.ts index 60eab2d60db..1c07e73f4c7 100644 --- a/test/helpers/rebuild-flow-harness.ts +++ b/test/helpers/rebuild-flow-harness.ts @@ -74,6 +74,9 @@ export const onboardSession = requireDist("../../state/onboard-session.js"); export const removedImmutabilityMigration = requireDist( "../../state/migrations/removed-immutability.js", ); +export const { cliOpenShellInstalledVersionObserver } = requireDist( + "../../adapters/openshell/installed-version-cli.js", +) as typeof import("../../src/lib/adapters/openshell/installed-version-cli"); export const openshellRuntime = requireDist("../../adapters/openshell/runtime.js"); export const providerCommand = requireDist("../../adapters/openshell/provider-command.js"); export const policies = requireDist("../../policy/index.js"); @@ -105,6 +108,7 @@ export const rebuildFlowHelpers = requireDist("./rebuild-flow-helpers.js"); export const snapshotBackup = requireDist("./snapshot/backup-authority.js"); export const rebuildInference = requireDist("./inference-invocation-probe.js"); export const rebuildProviderPreflight = requireDist("./rebuild-provider-preflight.js"); +export const rebuildHostCredentialPreflight = requireDist("./rebuild/host-credential-preflight.js"); export const rebuildManagedImage = requireDist("./rebuild-managed-image-preflight.js"); export const rebuildMessagingConflict = requireDist("./rebuild-messaging-conflict-preflight.js"); export const rebuildPreparedImageContext = requireDist("./rebuild-prepared-image-context.js"); diff --git a/test/helpers/registered-inference.ts b/test/helpers/registered-inference.ts new file mode 100644 index 00000000000..61441180848 --- /dev/null +++ b/test/helpers/registered-inference.ts @@ -0,0 +1,92 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { vi } from "vitest"; +import type { OpenShellProviderAdapter } from "../../src/lib/adapters/openshell/provider-adapter"; +import { inspectInferenceProviderProfile } from "../../src/lib/adapters/openshell/provider-profile"; +import type { RegisteredInferenceProviderAttachment } from "../../src/lib/inference/native-provider/registered"; + +export function registeredInferenceFixture(attached = false) { + const state = { attached, id: "operator-provider-id", revision: 4 }; + const profile = { + id: "operator-inference", + inference_capable: true, + credentials: [ + { + name: "api_key", + env_vars: ["VENDOR_API_KEY"], + auth_style: "bearer", + header_name: "authorization", + required: true, + }, + ], + endpoints: [ + { + host: "native.example", + port: 443, + protocol: "rest", + access: "read-write", + enforcement: "enforce", + }, + ], + binaries: ["/usr/bin/curl", "/usr/local/bin/node"], + }; + const projection = () => inspectInferenceProviderProfile(JSON.stringify(profile))!; + const adapter = { + ensureProviderPolicyComposition: vi.fn(async () => ({ ok: true as const, value: undefined })), + listProviders: vi.fn(async () => ({ ok: true as const, value: { names: ["native-extra"] } })), + getProvider: vi.fn(async () => ({ + ok: true as const, + value: { + name: "native-extra", + type: profile.id, + credentialKeys: ["VENDOR_API_KEY"], + configKeys: [], + revision: { id: state.id, resourceVersion: state.revision }, + }, + })), + inspectProviderProfile: vi.fn(async () => ({ + ok: true as const, + value: { credentialKeys: ["VENDOR_API_KEY"], inference: projection() }, + })), + listProviderAttachments: vi.fn(async () => ({ + ok: true as const, + value: { names: state.attached ? ["native-extra"] : [] }, + })), + attachProvider: vi.fn(async () => { + state.attached = true; + return { ok: true as const }; + }), + detachProvider: vi.fn(async () => { + state.attached = false; + return { ok: true as const, value: { changed: true } }; + }), + createProvider: vi.fn(), + updateProvider: vi.fn(), + deleteProvider: vi.fn(), + importProviderProfile: vi.fn(), + configureProviderRefresh: vi.fn(), + getProviderRefreshStatus: vi.fn(), + } satisfies OpenShellProviderAdapter; + const binding: RegisteredInferenceProviderAttachment = { + schemaVersion: 1, + gatewayName: "nemoclaw-18080", + sandboxName: "alpha", + providerName: "native-extra", + providerId: state.id, + profileId: profile.id, + profileDigest: projection().digest, + endpointUrl: "https://native.example/v1", + inferenceApi: "openai-completions", + credentialEnv: "VENDOR_API_KEY", + attachedBySelection: false, + }; + const handle = "openshell:resolve:env:s" + "a".repeat(64) + "_VENDOR_API_KEY"; + return { state, profile, adapter, binding, handle }; +} + +export function registeredIdentityFailureAt(operation: string, failure: Error) { + return (observedOperation: string): void => { + if (observedOperation === operation) throw failure; + }; +} diff --git a/test/helpers/vitest-watch-triggers.ts b/test/helpers/vitest-watch-triggers.ts index 03e44ab5f6b..4496f39b557 100644 --- a/test/helpers/vitest-watch-triggers.ts +++ b/test/helpers/vitest-watch-triggers.ts @@ -45,6 +45,20 @@ function runTests(...tests: string[]): () => string[] { } export const vitestWatchTriggerPatterns: VitestWatchTriggerPattern[] = [ + { + pattern: /(?:^|\/)scripts\/install-openshell\.sh$/, + testsToRun: runTests( + "test/installer-integration/install-openshell-version-check.test.ts", + "test/installer-integration/install-openshell-retained-state-size.test.ts", + "test/installer-integration/install-openshell-version-pin.test.ts", + "test/installer-integration/install-openshell-directory-safety.test.ts", + "test/install/installer-directory-bsd-stat.test.ts", + ), + }, + { + pattern: /(?:^|\/)scripts\/install\.sh$/, + testsToRun: runTests("test/install/installer-directory-bsd-stat.test.ts"), + }, { pattern: /(?:^|\/)Dockerfile(?:\.base)?$/, testsToRun: (file) => { @@ -254,6 +268,8 @@ export const vitestWatchTriggerPatterns: VitestWatchTriggerPattern[] = [ pattern: /(?:^|\/)agents\/hermes\/runtime-config-guard\.py$/, testsToRun: runTests( "src/lib/actions/sandbox/gateway-restart-hermes-drift.test.ts", + "test/agents/hermes/hermes-runtime-config-guard.test.ts", + "test/agents/hermes/hermes-capability-free-supervisor.test.ts", "test/agents/hermes/hermes-restart-config-seal-write-lock.test.ts", "test/agents/hermes/hermes-nvidia-restore-validation.test.ts", ), diff --git a/test/inference/managed/managed-image-direct-e2e-inputs.test.ts b/test/inference/managed/managed-image-direct-e2e-inputs.test.ts index 82a0c53e0de..bec0ffddc69 100644 --- a/test/inference/managed/managed-image-direct-e2e-inputs.test.ts +++ b/test/inference/managed/managed-image-direct-e2e-inputs.test.ts @@ -6,24 +6,140 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, describe, expect, it } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +vi.mock("node:child_process", async (importOriginal) => { + const original = await importOriginal(); + return { ...original, spawnSync: vi.fn(original.spawnSync) }; +}); import { managedImageDirectNativeStartupCommand, + managedImageStartupFailureSummary, parseManagedImageDirectE2eInputs, + runManagedImageDirectE2e, } from "../../../scripts/checks/run-managed-image-direct-e2e"; import { MANAGED_STARTUP_EXECUTABLE } from "../../../src/lib/onboard/managed-startup/hold"; const IMMUTABLE_IMAGE_ID = `sha256:${"a".repeat(64)}`; const temporaryDirectories: string[] = []; +beforeEach(async () => { + const original = await vi.importActual("node:child_process"); + vi.mocked(spawnSync).mockReset().mockImplementation(original.spawnSync); +}); + afterEach(() => { + vi.restoreAllMocks(); for (const directory of temporaryDirectories.splice(0)) { fs.rmSync(directory, { recursive: true, force: true }); } }); describe("managed-image direct E2E inputs", () => { + it("retains classified startup failure and attempts container removal", () => { + const containerId = "b".repeat(64); + const commands: string[][] = []; + vi.mocked(spawnSync).mockImplementation(((command: string, args: string[]) => { + commands.push([command, ...args]); + const responses: Record = { + image: { status: 0, stdout: IMMUTABLE_IMAGE_ID, stderr: "" }, + run: { status: 0, stdout: containerId, stderr: "" }, + exec: { status: args.includes("test") ? 1 : 0, stdout: "", stderr: "" }, + inspect: { + status: 0, + stdout: JSON.stringify({ + Status: "exited", + Running: false, + ExitCode: 1, + OOMKilled: false, + }), + stderr: "", + }, + logs: { + status: 0, + stdout: "fixedHostedNativeProvider is not a function\nsynthetic-private-value", + stderr: "", + }, + }; + return ( + responses[command === "docker" ? args[0]! : ""] ?? { status: 0, stdout: "", stderr: "" } + ); + }) as typeof spawnSync); + let failure: unknown; + try { + runManagedImageDirectE2e({ + agent: "openclaw", + image: IMMUTABLE_IMAGE_ID, + platform: "linux/amd64", + }); + } catch (error) { + failure = error; + } + expect(failure).toBeInstanceOf(Error); + expect(String(failure)).toContain('"missingProviderFunction":true'); + expect(String(failure)).toContain('"exitCode":1'); + expect(String(failure)).not.toContain("synthetic-private-value"); + expect(commands.filter((args) => args.includes("test"))).toHaveLength(60); + expect(commands.at(-1)).toEqual(["docker", "rm", "-f", containerId]); + }); + + it("classifies startup failure without exposing container output or runtime handles", () => { + const secret = "synthetic-private-value"; + const summary = managedImageStartupFailureSummary( + { + status: 0, + stdout: JSON.stringify({ + Status: "exited", + Running: false, + ExitCode: 1, + OOMKilled: false, + Error: secret, + Env: [`PRIVATE=${secret}`], + }), + stderr: secret, + }, + { + status: 0, + stdout: `TypeError: fixedHostedNativeProvider is not a function\n${secret}`, + stderr: `fixedHostedNativeProvider returned non-zero exit status 1: ${secret}`, + }, + ); + expect(summary).toEqual({ + stateReadSucceeded: true, + containerStatus: "exited", + running: false, + exitCode: 1, + oomKilled: false, + logsReadSucceeded: true, + missingProviderFunction: true, + providerRefreshFailed: true, + managedStartupFailed: false, + permissionDenied: false, + }); + expect(JSON.stringify(summary)).not.toContain(secret); + }); + + it.each([ + "not JSON", + "null", + "[]", + '{"Status":"private-value","Running":"false","ExitCode":"secret"}', + ])("keeps invalid container state unknown: %s", (stdout) => { + const summary = managedImageStartupFailureSummary( + { status: 0, stdout, stderr: "" }, + { status: 1, stdout: "", stderr: "" }, + ); + expect(summary).toMatchObject({ + containerStatus: null, + running: null, + exitCode: null, + oomKilled: null, + logsReadSucceeded: false, + }); + expect(JSON.stringify(summary)).not.toContain("private-value"); + }); + it("runs the native marker through the image-declared entrypoint", () => { expect(managedImageDirectNativeStartupCommand()).toEqual([ MANAGED_STARTUP_EXECUTABLE, diff --git a/test/inference/managed/managed-image-interrupted-recovery.test.ts b/test/inference/managed/managed-image-interrupted-recovery.test.ts index 1e86b43f007..46dd0023259 100644 --- a/test/inference/managed/managed-image-interrupted-recovery.test.ts +++ b/test/inference/managed/managed-image-interrupted-recovery.test.ts @@ -13,6 +13,7 @@ import { prepareManagedImageOpenShellState, } from "../../../scripts/checks/run-managed-image-openshell-e2e"; import { NEMOCLAW_CREATE_ATTEMPT_LABEL } from "../../../src/lib/adapters/openshell/sandbox-identity"; +import { gatewayIdForStateDir } from "../../../src/lib/onboard/gateway/process-environment"; const mocks = vi.hoisted(() => ({ stopGateway: vi.fn(), @@ -75,7 +76,7 @@ let scope: { owner: string; stateDir: string; networkName: string }; let commands: ReturnType; beforeEach(async () => { - root = fs.mkdtempSync(path.join(os.tmpdir(), "managed-recovery-")); + root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "managed-recovery-"))); const owner = "llama-cpp-dgx-spark-42-2"; scope = { owner, @@ -130,7 +131,7 @@ beforeEach(async () => { stdout: `sha256:${"b".repeat(64)}`, stderr: "", } as any) - .mockReturnValueOnce({ status: 0, stdout: "openshell 0.0.116", stderr: "" } as any) + .mockReturnValueOnce({ status: 0, stdout: "openshell 0.1.7", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ @@ -140,12 +141,12 @@ beforeEach(async () => { } as any); mocks.capture.mockImplementation((args: readonly string[]) => args[0] === "--version" - ? "openshell 0.0.116" + ? "openshell 0.1.7" : commands.mock.calls.some( ([command, args]: [string, readonly string[]]) => command === "openshell" && args[0] === "sandbox" && args[1] === "delete", ) - ? "[]" + ? JSON.stringify({ sandboxes: [], next_page_token: "" }) : sandboxObservation(), ); }); @@ -179,6 +180,8 @@ it("recovers only the marked isolated sandbox, gateway, volumes and network", as "label=openshell.ai/managed-by=openshell", "--filter", `label=openshell.ai/sandbox-name=${input.sandbox}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", ], ["docker", "network", "rm", scope.networkName], ["docker", "network", "inspect", scope.networkName], @@ -216,6 +219,7 @@ it.each([ ["0.0.116", "--limit"], ["0.1.3", "--page-size"], ["0.1.5", "--page-size"], + ["0.1.7", "--page-size"], ])("confirms strict absence with OpenShell %s before continuing cleanup", async (version, flag) => { mocks.capture .mockReset() @@ -368,7 +372,7 @@ it("rejects ambiguous network inspection instead of deleting recovery state", as stdout: `sha256:${"b".repeat(64)}`, stderr: "", } as any) - .mockReturnValueOnce({ status: 0, stdout: "openshell 0.0.116", stderr: "" } as any) + .mockReturnValueOnce({ status: 0, stdout: "openshell 0.1.7", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any); await expect(cleanupInterruptedManagedImageOpenShellE2e(input, scope)).rejects.toThrow( @@ -449,7 +453,7 @@ it("continues cleanup after a post-deletion failure without deleting the sandbox stdout: `sha256:${"b".repeat(64)}`, stderr: "", } as any) - .mockReturnValueOnce({ status: 0, stdout: "openshell 0.0.116", stderr: "" } as any) + .mockReturnValueOnce({ status: 0, stdout: "openshell 0.1.7", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ @@ -475,6 +479,8 @@ it("continues cleanup after a post-deletion failure without deleting the sandbox "label=openshell.ai/managed-by=openshell", "--filter", `label=openshell.ai/sandbox-name=${input.sandbox}`, + "--filter", + "label=openshell.ai/isolation-role=sandbox", ], ["docker", "network", "rm", scope.networkName], ["docker", "network", "inspect", scope.networkName], @@ -502,8 +508,17 @@ it("rejects a forged deletion marker before any external cleanup", async () => { expect(mocks.stopGateway).not.toHaveBeenCalled(); }); -it("retains recovery state if absence cannot be confirmed after deletion", async () => { - mocks.capture.mockReturnValue(sandboxObservation()); +it.each([ + ["a remaining sandbox", sandboxObservation()], + ["an incomplete page", JSON.stringify({ sandboxes: [], next_page_token: "more" })], + ["missing pagination metadata", JSON.stringify({ sandboxes: [] })], + ["invalid JSON", "not-json"], +])("retains recovery state after deletion with %s", async (_reason, observation) => { + const capture = mocks.capture.getMockImplementation()!; + mocks.capture.mockImplementation((...args) => { + const result = capture(...args); + return result === JSON.stringify({ sandboxes: [], next_page_token: "" }) ? observation : result; + }); await expect(cleanupInterruptedManagedImageOpenShellE2e(input, scope)).rejects.toThrow( "sandbox identity or deletion is unproven", ); @@ -534,7 +549,7 @@ it("records the nonce-bound immutable identity from successful creation", () => `${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${createAttemptNonce}`, "--output", "json", - "--limit", + "--page-size", "2", ], expect.objectContaining({ ignoreError: false, timeout: 30000 }), @@ -646,7 +661,7 @@ it("cleans setup before any create submission without issuing sandbox deletion", stdout: `sha256:${"b".repeat(64)}`, stderr: "", } as any) - .mockReturnValueOnce({ status: 0, stdout: "openshell 0.0.116", stderr: "" } as any) + .mockReturnValueOnce({ status: 0, stdout: "openshell 0.1.7", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ @@ -722,7 +737,7 @@ it("cleans partial setup when no exact sandbox container was created", async () stdout: `sha256:${"b".repeat(64)}`, stderr: "", } as any) - .mockReturnValueOnce({ status: 0, stdout: "openshell 0.0.116", stderr: "" } as any) + .mockReturnValueOnce({ status: 0, stdout: "openshell 0.1.7", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ @@ -744,9 +759,13 @@ it("removes the exact leftover container without removing a foreign container", Labels: { "openshell.ai/managed-by": "openshell", "openshell.ai/sandbox-name": input.sandbox, + "openshell.ai/isolation-role": "sandbox", + "openshell.ai/sandbox-namespace": gatewayIdForStateDir( + path.join(scope.stateDir, "gateway-state"), + ), }, }, - NetworkSettings: { Networks: { [scope.networkName]: {} } }, + HostConfig: { NetworkMode: "none" }, }, ]); const foreign = JSON.stringify([ @@ -756,9 +775,11 @@ it("removes the exact leftover container without removing a foreign container", Labels: { "openshell.ai/managed-by": "openshell", "openshell.ai/sandbox-name": input.sandbox, + "openshell.ai/isolation-role": "sandbox", + "openshell.ai/sandbox-namespace": gatewayIdForStateDir(scope.stateDir), }, }, - NetworkSettings: { Networks: { "foreign-network": {} } }, + HostConfig: { NetworkMode: "none" }, }, ]); commands @@ -766,7 +787,7 @@ it("removes the exact leftover container without removing a foreign container", .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: content, stderr: "" } as any) - .mockReturnValueOnce({ status: 0, stdout: "openshell 0.0.116", stderr: "" } as any) + .mockReturnValueOnce({ status: 0, stdout: "openshell 0.1.7", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "owned-container foreign-container", @@ -776,7 +797,7 @@ it("removes the exact leftover container without removing a foreign container", .mockReturnValueOnce({ status: 0, stdout: foreign, stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: content, stderr: "" } as any) - .mockReturnValueOnce({ status: 0, stdout: "openshell 0.0.116", stderr: "" } as any) + .mockReturnValueOnce({ status: 0, stdout: "openshell 0.1.7", stderr: "" } as any) .mockReturnValueOnce({ status: 0, stdout: "foreign-container", @@ -790,6 +811,21 @@ it("removes the exact leftover container without removing a foreign container", stderr: `Error: No such network: ${scope.networkName}`, } as any); await cleanupInterruptedManagedImageOpenShellE2e(input, scope); + expect(mocks.capture).toHaveBeenLastCalledWith( + [ + "sandbox", + "list", + "-g", + "nemoclaw", + "--selector", + `${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${createAttemptNonce}`, + "--output", + "json", + "--page-size", + "2", + ], + expect.objectContaining({ ignoreError: false, timeout: 30000 }), + ); expect(commands).toHaveBeenCalledWith( "docker", ["rm", "--force", "owned-container"], diff --git a/test/inference/managed/managed-image-protected-runtime-contract.test.ts b/test/inference/managed/managed-image-protected-runtime-contract.test.ts index d1244c589fd..53d49d5694e 100644 --- a/test/inference/managed/managed-image-protected-runtime-contract.test.ts +++ b/test/inference/managed/managed-image-protected-runtime-contract.test.ts @@ -127,7 +127,7 @@ function nativeQualificationFixture( beforeEach(() => { runtime.version = "0.1.3"; }); -describe.each(["0.1.3", "0.1.5"])("OpenShell %s native inference", (version) => { +describe.each(["0.1.3", "0.1.5", "0.1.7"])("OpenShell %s native inference", (version) => { beforeEach(() => { runtime.version = version; }); diff --git a/test/inference/managed/managed-image-publication-workflow.test.ts b/test/inference/managed/managed-image-publication-workflow.test.ts index 79a988dff32..96c8a1f853d 100644 --- a/test/inference/managed/managed-image-publication-workflow.test.ts +++ b/test/inference/managed/managed-image-publication-workflow.test.ts @@ -470,6 +470,7 @@ describe("complete managed-image publication workflow", () => { expect.arrayContaining([ ".github/actions/ci-reviewed-npm-audit/**", "ci/reviewed-npm-audit.json", + "src/lib/inference/native-provider/catalog.ts", "test/helpers/openclaw-real-agent-reload-proof.ts", ]), ); diff --git a/test/install/install-onboard-yes.test.ts b/test/install/install-onboard-yes.test.ts index f9804f02b2a..9e08ba90352 100644 --- a/test/install/install-onboard-yes.test.ts +++ b/test/install/install-onboard-yes.test.ts @@ -577,6 +577,7 @@ describe("install.sh run_onboard — _CLI_PATH precedence (#3276)", () => { type DeferredOnboardingMainOptions = { agent?: string; + cliPath?: string; deferEnv?: boolean; deferFlag?: boolean; inferenceKey?: string; @@ -591,6 +592,7 @@ type DeferredOnboardingMainOptions = { function runDeferredOnboardingMain(options: DeferredOnboardingMainOptions = {}) { const registeredSandboxCount = options.registeredSandboxCount ?? 0; const agent = options.agent ?? "hermes"; + const cliPath = options.cliPath ?? path.resolve(import.meta.dirname, "../../bin/nemoclaw.js"); const result = runInstallerSourcedBody( ` set -e @@ -611,7 +613,7 @@ fix_npm_permissions() { :; } preinstall_backup_and_retire_legacy_gateway() { :; } install_nemoclaw() { record install-nemoclaw; } verify_nemoclaw() { - _CLI_PATH=${JSON.stringify(path.resolve(import.meta.dirname, "../../bin/nemoclaw.js"))} + _CLI_PATH="$TEST_CLI_PATH" NEMOCLAW_READY_NOW=true } require_reportable_openshell_version() { :; } @@ -646,6 +648,7 @@ main --non-interactive --yes-i-accept-third-party-software ${options.deferFlag ? NEMOCLAW_PROVIDER_KEY: options.providerKey ?? "", ONBOARD_STATUS: String(options.onboardStatus ?? 0), REGISTERED_SANDBOX_COUNT: String(registeredSandboxCount), + TEST_CLI_PATH: cliPath, }, includeNodeOnPath: true, timeoutMs: 15_000, @@ -664,6 +667,20 @@ main --non-interactive --yes-i-accept-third-party-software ${options.deferFlag ? } describe("deferred onboarding", () => { + it("treats the resolved CLI path as shell data", () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-install-cli-path-")); + const marker = path.join(tempRoot, "executed"); + onTestFinished(() => fs.rmSync(tempRoot, { recursive: true, force: true })); + + const result = runDeferredOnboardingMain({ + cliPath: `$(touch ${marker})`, + deferFlag: true, + }); + + expect(result.result.status, result.output).toBe(0); + expect(fs.existsSync(marker)).toBe(false); + }); + it("lists the installer option and environment setting (#10288)", () => { const result = runInstallerSourcedBody("usage", { extraEnv: { NEMOCLAW_AGENT: "hermes" }, diff --git a/test/install/install-openshell-upgrade-help.test.ts b/test/install/install-openshell-upgrade-help.test.ts index ac7a4c1c8a4..b52c89ed042 100644 --- a/test/install/install-openshell-upgrade-help.test.ts +++ b/test/install/install-openshell-upgrade-help.test.ts @@ -9,20 +9,20 @@ import { expect, it } from "vitest"; const installer = path.join(import.meta.dirname, "../..", "scripts", "install.sh"); -it("documents gateway process retirement before accepting prepared upgrade state", async () => { +it("limits prepared upgrade guidance to eligible gateways and preserves legacy state", async () => { const { stderr, stdout } = await promisify(execFile)("bash", [installer, "--help"], { encoding: "utf-8", env: process.env, }); - const backup = - "NEMOCLAW_REQUIRE_ALL_SANDBOX_BACKUPS=1 nemoclaw backup-all --retire-legacy-forwards"; - const destroy = "openshell gateway destroy -g nemoclaw || openshell gateway destroy"; - expect(stderr).toBe(""); - expect(stdout).toContain(backup); - expect(stdout).toContain(destroy); - expect(stdout.indexOf(backup)).toBeLessThan(stdout.indexOf(destroy)); expect(stdout).toContain( - "For NEMOCLAW_GATEWAY_PORT=, destroy nemoclaw- with -g and omit the unnamed fallback", + "the installed OpenShell version must be verifiable and within the supported range", + ); + expect(stdout).toContain( + "This flag does not migrate registered sandboxes from an out-of-range OpenShell version", ); + expect(stdout).toContain("Retained NemoClaw sandbox records still block component replacement"); + expect(stdout).toContain("Preserve the legacy gateway, registrations, sandbox records, and data"); + expect(stdout).not.toContain("openshell gateway destroy"); + expect(stdout).not.toContain("nemoclaw backup-all --retire-legacy-forwards"); }); diff --git a/test/install/install-openshell-upgrade-prompt.test.ts b/test/install/install-openshell-upgrade-prompt.test.ts index b2f22f18d79..ca898808ec0 100644 --- a/test/install/install-openshell-upgrade-prompt.test.ts +++ b/test/install/install-openshell-upgrade-prompt.test.ts @@ -36,91 +36,6 @@ function writeExecutable(target: string, contents: string): void { fs.writeFileSync(target, contents, { mode: 0o755 }); } -async function runInstallerOpenshellVersionFlow( - setupOpenshell: (bin: string) => void, - installedOpenshellBody = '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell 0.0.85"\nexit 0\n', -) { - const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-version-flow-")); - const home = path.join(tmp, "home"); - const bin = path.join(tmp, "bin"); - const setupDir = path.join(tmp, "setup"); - const registry = path.join(home, ".nemoclaw", "sandboxes.json"); - const gatewayState = path.join(tmp, "gateway.state"); - const backupLog = path.join(tmp, "backup.log"); - const installLog = path.join(tmp, "install.log"); - const healthyOpenshell = path.join(tmp, "healthy-openshell"); - - fs.mkdirSync(path.dirname(registry), { recursive: true }); - fs.mkdirSync(bin, { recursive: true }); - fs.mkdirSync(path.join(setupDir, "lib"), { recursive: true }); - fs.writeFileSync(registry, '{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); - fs.writeFileSync(gatewayState, "gateway-original\n"); - writeExecutable(path.join(setupDir, "setup-jetson.sh"), "#!/usr/bin/env bash\nexit 0\n"); - writeExecutable( - path.join(setupDir, "lib", "station-vllm-conflict.sh"), - `#!/usr/bin/env bash -handle_station_vllm_conflict() { :; } -consume_station_local_vllm_resume() { return 1; } -`, - ); - writeExecutable(healthyOpenshell, installedOpenshellBody); - setupOpenshell(bin); - - const result = await runCommand( - "bash", - [ - "-c", - `source "${INSTALLER_PAYLOAD}" >/dev/null 2>&1 -SCRIPT_DIR="${setupDir}" -_CLI_PATH="" -_CLI_BIN="nemoclaw-test-missing" -resolve_nemoclaw_gateway_port() { printf '8080\\n'; } -preflight_explicit_express_flags() { :; } -print_banner() { :; } -preflight_usage_notice_prompt() { :; } -prepare_installer_host() { :; } -step() { :; } -install_nodejs() { :; } -ensure_supported_runtime() { :; } -fix_npm_permissions() { :; } -preinstall_backup_and_retire_legacy_gateway() { - command_exists openshell || return 0 - printf 'backup\\n' >>"${backupLog}" - [ -n "$(installed_openshell_version 2>/dev/null || true)" ] || - error "Could not determine the installed OpenShell version. The installer stopped after backup without retiring the gateway." - printf 'gateway-retired\\n' >"${gatewayState}" - printf '{"sandboxes":{}}\\n' >"${registry}" -} -install_nemoclaw() { - printf 'install\\n' >>"${installLog}" - if ! command_exists openshell; then - cp "${healthyOpenshell}" "${bin}/openshell" - fi -} -verify_nemoclaw() { :; } -print_done() { :; } -main --non-interactive --yes-i-accept-third-party-software`, - ], - { - encoding: "utf-8", - env: { - ...process.env, - HOME: home, - PATH: `${bin}:${path.dirname(process.execPath)}:/usr/bin:/bin`, - }, - }, - ); - - return { - result, - backupLog: fs.existsSync(backupLog) ? fs.readFileSync(backupLog, "utf-8") : "", - gatewayState: fs.readFileSync(gatewayState, "utf-8"), - registry: fs.readFileSync(registry, "utf-8"), - installLog: fs.existsSync(installLog) ? fs.readFileSync(installLog, "utf-8") : "", - openshellBody: fs.readFileSync(path.join(bin, "openshell"), "utf-8"), - }; -} - async function runPreinstallUpgradeGuard( env: Record = {}, options: { @@ -142,6 +57,7 @@ async function runPreinstallUpgradeGuard( hasOldCli?: boolean; openshellOnPath?: boolean; openshellVersion?: string; + ambiguousOpenshellVersionOutput?: boolean; openshellVersionCommandFails?: boolean; registryJson?: string; userLocalOpenshell?: boolean; @@ -184,7 +100,8 @@ async function runPreinstallUpgradeGuard( const finishPreparedInstallSucceeds = options.finishPreparedInstallSucceeds === false ? "0" : "1"; const openshellVersionCommandFails = options.openshellVersionCommandFails === true ? "1" : "0"; const installedOpenshellVersionOverride = - options.openshellVersionCommandFails === true + options.openshellVersionCommandFails === true || + options.ambiguousOpenshellVersionOutput === true ? "" : `installed_openshell_version() { printf '${openshellVersion}'; }`; @@ -220,6 +137,10 @@ printf '%s\\n' "$*" >> "${openshellLog}" if [ "\${1:-}" = "--version" ] && [ "${openshellVersionCommandFails}" = "1" ]; then exit 7 fi +if [ "\${1:-}" = "--version" ] && [ "${options.ambiguousOpenshellVersionOutput === true ? "1" : "0"}" = "1" ]; then + printf 'openshell 0.1.5\nopenshell 0.0.116\n' + exit 0 +fi if [ "\${1:-} \${2:-}" = "gateway remove" ] && [ "${gatewayRemoveSucceeds}" != "1" ]; then exit 4 fi @@ -411,81 +332,6 @@ require_reportable_openshell_version`, const brokenOpenshell = "#!/usr/bin/env bash\nexit 1\n"; const healthyOpenshell = '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell 0.0.85"\nexit 0\n'; - const invalidVersionOpenshell = - '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell unknown"\nexit 0\n'; - const versionPrintingBrokenOpenshell = - '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell 0.0.85"\nexit 1\n'; - - it("backs up before rejecting a broken OpenShell without gateway or sandbox mutation (#7300)", async () => { - const { result, backupLog, gatewayState, registry, installLog } = - await runInstallerOpenshellVersionFlow((bin) => { - writeExecutable(path.join(bin, "openshell"), brokenOpenshell); - }); - - expect(result.status, result.stdout + result.stderr).not.toBe(0); - expect(result.stderr + result.stdout).toContain("stopped after backup"); - expect(backupLog).toBe("backup\n"); - expect(gatewayState).toBe("gateway-original\n"); - expect(registry).toBe('{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); - expect(installLog).toBe(""); - }); - - it("backs up before rejecting a failed OpenShell version command without mutation (#7300)", async () => { - const { result, backupLog, gatewayState, registry, installLog } = - await runInstallerOpenshellVersionFlow((bin) => { - writeExecutable(path.join(bin, "openshell"), versionPrintingBrokenOpenshell); - }); - - expect(result.status, result.stdout + result.stderr).not.toBe(0); - expect(result.stderr + result.stdout).toContain("stopped after backup"); - expect(backupLog).toBe("backup\n"); - expect(gatewayState).toBe("gateway-original\n"); - expect(registry).toBe('{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); - expect(installLog).toBe(""); - }); - - it("backs up before rejecting invalid OpenShell version output without mutation (#7300)", async () => { - const { result, backupLog, gatewayState, registry, installLog } = - await runInstallerOpenshellVersionFlow((bin) => { - writeExecutable(path.join(bin, "openshell"), invalidVersionOpenshell); - }); - - expect(result.status, result.stdout + result.stderr).not.toBe(0); - expect(result.stderr + result.stdout).toContain("stopped after backup"); - expect(backupLog).toBe("backup\n"); - expect(gatewayState).toBe("gateway-original\n"); - expect(registry).toBe('{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); - expect(installLog).toBe(""); - }); - - it("preserves a reportable OpenShell through the installer flow (#7300)", async () => { - const { result, openshellBody } = await runInstallerOpenshellVersionFlow((bin) => { - writeExecutable(path.join(bin, "openshell"), healthyOpenshell); - }); - - expect(result.status, result.stdout + result.stderr).toBe(0); - expect(openshellBody).toBe(healthyOpenshell); - }); - - it("installs OpenShell when no binary is present (#7300)", async () => { - const { result, installLog } = await runInstallerOpenshellVersionFlow(() => undefined); - - expect(result.status, result.stdout + result.stderr).toBe(0); - expect(installLog).toBe("install\n"); - }); - - it("rejects an installed OpenShell whose version command fails before onboarding (#7300)", async () => { - const { result, installLog, openshellBody } = await runInstallerOpenshellVersionFlow( - () => undefined, - versionPrintingBrokenOpenshell, - ); - - expect(result.status, result.stdout + result.stderr).not.toBe(0); - expect(result.stderr + result.stdout).toContain("could not report its version"); - expect(installLog).toBe("install\n"); - expect(openshellBody).toBe(versionPrintingBrokenOpenshell); - }); - it.skipIf(process.platform !== "linux")( "fails closed before onboarding when a present openshell cannot report its version (#7300)", async () => { @@ -660,6 +506,85 @@ esac`, expect(openshellLog).toBe(""); }); + it.each( + ["0.1.5", "0.1.7"].flatMap((targetVersion) => + [ + "0.0.116", + "0.1.2", + "0.1.4", + ...(targetVersion === "0.1.7" ? ["0.1.5", "0.1.6"] : []), + ].flatMap((version) => + ["automatic", "prepared"].map((mode) => ({ version, mode, targetVersion })), + ), + ), + )( + "preserves a registered $version gateway before the $targetVersion $mode cutover (#12558)", + async ({ version, mode, targetVersion }) => { + const { result, cliLog, openshellLog, registry } = await runPreinstallUpgradeGuard( + { + NON_INTERACTIVE: "1", + NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE: "1", + NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE: '["alpha"]', + ...(mode === "prepared" ? { NEMOCLAW_OPENSHELL_UPGRADE_PREPARED: "1" } : {}), + }, + { + currentMinOpenshellVersion: targetVersion, + currentMaxOpenshellVersion: targetVersion, + finishInstallMode: "source", + openshellVersion: version, + }, + ); + + expect(result.status).not.toBe(0); + expect(result.stdout + result.stderr).toContain( + 'Refusing to retire gateway nemoclaw with registered sandboxes ["alpha"]', + ); + expect(cliLog).toBe(""); + expect(openshellLog).toBe(""); + expect(registry).toBe('{"sandboxes":{"alpha":{"name":"alpha"}}}'); + }, + ); + + it("preserves a registered gateway when its installed version is unreadable before 0.1.5 (#12558)", async () => { + const { result, cliLog, openshellLog } = await runPreinstallUpgradeGuard( + { + NON_INTERACTIVE: "1", + NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE: "1", + NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE: '["alpha"]', + }, + { + currentMinOpenshellVersion: "0.1.5", + currentMaxOpenshellVersion: "0.1.5", + openshellVersionCommandFails: true, + }, + ); + + expect(result.status).not.toBe(0); + expect(result.stdout + result.stderr).toContain( + "Could not verify the installed OpenShell version before the 0.1.x cutover", + ); + expect(cliLog).toBe(""); + expect(openshellLog).not.toContain("gateway "); + }); + + it("preserves a registered gateway when version output contains two releases before 0.1.5 (#12558)", async () => { + const { result, cliLog, openshellLog } = await runPreinstallUpgradeGuard( + { NON_INTERACTIVE: "1", NEMOCLAW_ACCEPT_EXPERIMENTAL_OPENSHELL_UPGRADE: "1" }, + { + ambiguousOpenshellVersionOutput: true, + currentMinOpenshellVersion: "0.1.5", + currentMaxOpenshellVersion: "0.1.5", + }, + ); + + expect(result.status).not.toBe(0); + expect(result.stdout + result.stderr).toContain( + "Could not verify the installed OpenShell version before the 0.1.x cutover", + ); + expect(cliLog).toBe(""); + expect(openshellLog).not.toContain("gateway "); + }); + it("uses only the current CLI for strict backup before legacy gateway retirement (#6114)", async () => { const { result, cliLog, openshellLog } = await runPreinstallUpgradeGuard({ NON_INTERACTIVE: "1", @@ -828,22 +753,27 @@ esac`, expect(openshellLog).toContain("gateway remove nemoclaw"); }); - it("keeps a backed-up gateway whose OpenShell version is already supported", async () => { - const { result, cliLog, openshellLog } = await runPreinstallUpgradeGuard( - { NON_INTERACTIVE: "1" }, - { - finishInstallMode: "source", - hasOldCli: false, - openshellVersion: "0.0.85", - registryJson: - '{"sandboxes":{"alpha":{"name":"alpha","nemoclawVersion":"0.0.85","fromDockerfile":false}}}', - }, - ); + it.each(["0.0.85", "0.1.5", "0.1.7"])( + "keeps a backed-up gateway on supported OpenShell %s", + async (version) => { + const { result, cliLog, openshellLog } = await runPreinstallUpgradeGuard( + { NON_INTERACTIVE: "1" }, + { + finishInstallMode: "source", + hasOldCli: false, + openshellVersion: version, + currentMinOpenshellVersion: version, + currentMaxOpenshellVersion: version, + registryJson: + '{"sandboxes":{"alpha":{"name":"alpha","nemoclawVersion":"0.0.85","fromDockerfile":false}}}', + }, + ); - expect(result.status).toBe(0); - expect(cliLog).toContain("current:backup-all"); - expect(openshellLog).toBe("openshell install-mode if-missing defer=\n"); - }); + expect(result.status).toBe(0); + expect(cliLog).toContain("current:backup-all"); + expect(openshellLog).toBe("openshell install-mode if-missing defer=\n"); + }, + ); it("forces OpenShell installation for a managed CLI install mode (#8800)", async () => { const { result, openshellLog } = await runPreinstallUpgradeGuard( @@ -1013,7 +943,7 @@ esac`, }, ); - it("fails closed before gateway retirement when the supported range is invalid", async () => { + it("fails closed before backup when the supported range is invalid", async () => { const { result, cliLog, openshellLog } = await runPreinstallUpgradeGuard( { NON_INTERACTIVE: "1", @@ -1030,7 +960,7 @@ esac`, expect(result.stdout + result.stderr).toContain( "Could not resolve the current OpenShell version range", ); - expect(cliLog).toContain("current:backup-all"); + expect(cliLog).toBe(""); expect(openshellLog).toBe(""); }); diff --git a/test/install/install-openshell-version-flow.test.ts b/test/install/install-openshell-version-flow.test.ts new file mode 100644 index 00000000000..7b8397b4f06 --- /dev/null +++ b/test/install/install-openshell-version-flow.test.ts @@ -0,0 +1,200 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { execFile, type ExecFileOptionsWithStringEncoding } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; + +import { describe, expect, it } from "vitest"; + +const INSTALLER_PAYLOAD = path.join(import.meta.dirname, "../..", "scripts", "install.sh"); +const execFileAsync = promisify(execFile); + +async function runCommand( + command: string, + args: string[], + options: ExecFileOptionsWithStringEncoding, +) { + try { + const { stderr, stdout } = await execFileAsync(command, args, options); + return { status: 0, stderr, stdout }; + } catch (error) { + const result = error as { code?: unknown; stderr?: string; stdout?: string }; + return { + status: typeof result.code === "number" ? result.code : null, + stderr: result.stderr ?? "", + stdout: result.stdout ?? "", + }; + } +} + +function writeExecutable(target: string, contents: string): void { + fs.writeFileSync(target, contents, { mode: 0o755 }); +} + +async function runInstallerOpenshellVersionFlow( + setupOpenshell: (bin: string) => void, + installedOpenshellBody = '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell 0.0.85"\nexit 0\n', +) { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-version-flow-")); + const home = path.join(tmp, "home"); + const bin = path.join(tmp, "bin"); + const setupDir = path.join(tmp, "setup"); + const registry = path.join(home, ".nemoclaw", "sandboxes.json"); + const gatewayState = path.join(tmp, "gateway.state"); + const backupLog = path.join(tmp, "backup.log"); + const installLog = path.join(tmp, "install.log"); + const healthyOpenshell = path.join(tmp, "healthy-openshell"); + + fs.mkdirSync(path.dirname(registry), { recursive: true }); + fs.mkdirSync(bin, { recursive: true }); + fs.mkdirSync(path.join(setupDir, "lib"), { recursive: true }); + fs.writeFileSync(registry, '{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); + fs.writeFileSync(gatewayState, "gateway-original\n"); + writeExecutable(path.join(setupDir, "setup-jetson.sh"), "#!/usr/bin/env bash\nexit 0\n"); + writeExecutable( + path.join(setupDir, "lib", "station-vllm-conflict.sh"), + `#!/usr/bin/env bash +handle_station_vllm_conflict() { :; } +consume_station_local_vllm_resume() { return 1; } +`, + ); + writeExecutable(healthyOpenshell, installedOpenshellBody); + setupOpenshell(bin); + + const result = await runCommand( + "bash", + [ + "-c", + `source "${INSTALLER_PAYLOAD}" >/dev/null 2>&1 +SCRIPT_DIR="${setupDir}" +_CLI_PATH="" +_CLI_BIN="nemoclaw-test-missing" +resolve_nemoclaw_gateway_port() { printf '8080\\n'; } +preflight_explicit_express_flags() { :; } +print_banner() { :; } +preflight_usage_notice_prompt() { :; } +prepare_installer_host() { :; } +step() { :; } +install_nodejs() { :; } +ensure_supported_runtime() { :; } +fix_npm_permissions() { :; } +preinstall_backup_and_retire_legacy_gateway() { + command_exists openshell || return 0 + printf 'backup\\n' >>"${backupLog}" + [ -n "$(installed_openshell_version 2>/dev/null || true)" ] || + error "Could not determine the installed OpenShell version. The installer stopped after backup without retiring the gateway." + printf 'gateway-retired\\n' >"${gatewayState}" + printf '{"sandboxes":{}}\\n' >"${registry}" +} +install_nemoclaw() { + printf 'install\\n' >>"${installLog}" + if ! command_exists openshell; then + cp "${healthyOpenshell}" "${bin}/openshell" + fi +} +verify_nemoclaw() { :; } +print_done() { :; } +main --non-interactive --yes-i-accept-third-party-software`, + ], + { + encoding: "utf-8", + env: { + ...process.env, + HOME: home, + PATH: `${bin}:${path.dirname(process.execPath)}:/usr/bin:/bin`, + }, + }, + ); + + return { + result, + backupLog: fs.existsSync(backupLog) ? fs.readFileSync(backupLog, "utf-8") : "", + gatewayState: fs.readFileSync(gatewayState, "utf-8"), + registry: fs.readFileSync(registry, "utf-8"), + installLog: fs.existsSync(installLog) ? fs.readFileSync(installLog, "utf-8") : "", + openshellBody: fs.readFileSync(path.join(bin, "openshell"), "utf-8"), + }; +} + +describe.concurrent("install.sh OpenShell version flow", () => { + const brokenOpenshell = "#!/usr/bin/env bash\nexit 1\n"; + const healthyOpenshell = + '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell 0.0.85"\nexit 0\n'; + const invalidVersionOpenshell = + '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell unknown"\nexit 0\n'; + const versionPrintingBrokenOpenshell = + '#!/usr/bin/env bash\n[ "$1" = "--version" ] && echo "openshell 0.0.85"\nexit 1\n'; + + it("backs up before rejecting a broken OpenShell without gateway or sandbox mutation (#7300)", async () => { + const { result, backupLog, gatewayState, registry, installLog } = + await runInstallerOpenshellVersionFlow((bin) => { + writeExecutable(path.join(bin, "openshell"), brokenOpenshell); + }); + + expect(result.status, result.stdout + result.stderr).not.toBe(0); + expect(result.stderr + result.stdout).toContain("stopped after backup"); + expect(backupLog).toBe("backup\n"); + expect(gatewayState).toBe("gateway-original\n"); + expect(registry).toBe('{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); + expect(installLog).toBe(""); + }); + + it("backs up before rejecting a failed OpenShell version command without mutation (#7300)", async () => { + const { result, backupLog, gatewayState, registry, installLog } = + await runInstallerOpenshellVersionFlow((bin) => { + writeExecutable(path.join(bin, "openshell"), versionPrintingBrokenOpenshell); + }); + + expect(result.status, result.stdout + result.stderr).not.toBe(0); + expect(result.stderr + result.stdout).toContain("stopped after backup"); + expect(backupLog).toBe("backup\n"); + expect(gatewayState).toBe("gateway-original\n"); + expect(registry).toBe('{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); + expect(installLog).toBe(""); + }); + + it("backs up before rejecting invalid OpenShell version output without mutation (#7300)", async () => { + const { result, backupLog, gatewayState, registry, installLog } = + await runInstallerOpenshellVersionFlow((bin) => { + writeExecutable(path.join(bin, "openshell"), invalidVersionOpenshell); + }); + + expect(result.status, result.stdout + result.stderr).not.toBe(0); + expect(result.stderr + result.stdout).toContain("stopped after backup"); + expect(backupLog).toBe("backup\n"); + expect(gatewayState).toBe("gateway-original\n"); + expect(registry).toBe('{"sandboxes":{"alpha":{"name":"alpha"}}}\n'); + expect(installLog).toBe(""); + }); + + it("preserves a reportable OpenShell through the installer flow (#7300)", async () => { + const { result, openshellBody } = await runInstallerOpenshellVersionFlow((bin) => { + writeExecutable(path.join(bin, "openshell"), healthyOpenshell); + }); + + expect(result.status, result.stdout + result.stderr).toBe(0); + expect(openshellBody).toBe(healthyOpenshell); + }); + + it("installs OpenShell when no binary is present (#7300)", async () => { + const { result, installLog } = await runInstallerOpenshellVersionFlow(() => undefined); + + expect(result.status, result.stdout + result.stderr).toBe(0); + expect(installLog).toBe("install\n"); + }); + + it("rejects an installed OpenShell whose version command fails before onboarding (#7300)", async () => { + const { result, installLog, openshellBody } = await runInstallerOpenshellVersionFlow( + () => undefined, + versionPrintingBrokenOpenshell, + ); + + expect(result.status, result.stdout + result.stderr).not.toBe(0); + expect(result.stderr + result.stdout).toContain("could not report its version"); + expect(installLog).toBe("install\n"); + expect(openshellBody).toBe(versionPrintingBrokenOpenshell); + }); +}); diff --git a/test/install/installer-directory-bsd-stat.test.ts b/test/install/installer-directory-bsd-stat.test.ts new file mode 100644 index 00000000000..a7996bb510b --- /dev/null +++ b/test/install/installer-directory-bsd-stat.test.ts @@ -0,0 +1,87 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; + +describe.each(["install.sh", "install-openshell.sh"])("%s BSD directory metadata", (installer) => { + const source = fs.readFileSync( + path.join(import.meta.dirname, "../../scripts", installer), + "utf8", + ); + const guard = source.match(/^user_local_executable_directory_safe\(\) \{[\s\S]*?^\}/m)![0]; + + it.skipIf(process.platform === "win32").each([ + { + name: "accepts a root-owned sticky ancestor", + owner: 0, + mode: "1777", + target: false, + status: 0, + }, + { + name: "rejects a root-owned writable ancestor without sticky permission", + owner: 0, + mode: "0777", + target: false, + status: 1, + }, + { + name: "rejects a user-owned writable sticky ancestor", + owner: 501, + mode: "1777", + target: false, + status: 1, + }, + { + name: "rejects a root-owned writable sticky target", + owner: 0, + mode: "1777", + target: true, + status: 1, + }, + ])("$name", ({ owner, mode, target, status }) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-bsd-directory-")); + const child = path.join(root, "bin"); + fs.mkdirSync(child); + try { + const result = spawnSync( + "bash", + [ + "-c", + `${guard} +id() { printf '501\n'; } +stat() { + [ "$1" = -f ] || return 1 + local owner=501 mode=0755 + if [ "$3" = "$TEST_ANCESTOR" ]; then owner="$TEST_OWNER"; mode="$TEST_MODE"; fi + # BSD's low permission field excludes the sticky/set-ID bits. + case "$2" in + '%u %Lp') printf '%s %s\n' "$owner" "\${mode#?}" ;; + '%u %Mp%03Lp') printf '%s %s\n' "$owner" "$mode" ;; + *) return 1 ;; + esac +} +user_local_executable_directory_safe "$TEST_TARGET" +`, + ], + { + encoding: "utf8", + env: { + ...process.env, + TEST_ANCESTOR: root, + TEST_TARGET: target ? root : child, + TEST_OWNER: String(owner), + TEST_MODE: mode, + }, + }, + ); + expect(result.status, result.stderr).toBe(status); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/test/install/installer-hash-check.test.ts b/test/install/installer-hash-check.test.ts index fc638d7831e..aa2bd9dcc69 100644 --- a/test/install/installer-hash-check.test.ts +++ b/test/install/installer-hash-check.test.ts @@ -39,6 +39,7 @@ import { addV00106OperationalTrust, alterRequiredReleaseValue, brevMutationFixtures, + npmReplacementFixtureVersion, copyTrustedInstallerVerifier, prepareTrustedFormulaMismatchFixture, preparedReleaseArgs, @@ -47,8 +48,8 @@ import { retainBaseInstallerTrust, removeV00106OperationalTrust, } from "../helpers/openshell-installer-template"; - import { selectPreparedGatewayRuntime } from "../helpers/prepared-gateway-runtime"; +import { prepareInstallerBin } from "../support/openshell-installed-version"; import { registerV012TemplateCases } from "./installer-hash-v012-suite"; const REPO_ROOT = path.join(import.meta.dirname, "../.."); @@ -398,7 +399,6 @@ const PARSER_MUTATIONS: Partial string>> "trusted-formula-mismatch": (source) => source.replace(FORMULA_DIGEST, "0".repeat(64)), }; const tempDirs: string[] = []; - afterEach(() => { for (const tempDir of tempDirs.splice(0)) { fs.rmSync(tempDir, { recursive: true, force: true }); @@ -535,8 +535,8 @@ function renderInstallerTemplate(openshellVersion: string, pinFunction: string): function renderBrevTemplate(openshellVersion: string, pinFunction: string): string { const template = NPM_CLEANUP_TEMPLATE.replaceAll( - /0\.0\.116|0\.1\.2/g, - openshellVersion === "0.1.2" ? "0.1.2" : "0.0.116", + /0\.0\.116|0\.1\.2|0\.1\.3|0\.1\.5|0\.1\.7/g, + ["0.1.2", "0.1.3", "0.1.5", "0.1.7"].includes(openshellVersion) ? openshellVersion : "0.0.116", ); const selected = template.replace( /^(\s*stable\s*\|\s*auto\)\s*OPENSHELL_VERSION=")v[0-9]+\.[0-9]+\.[0-9]+("\s*;;\s*)$/m, @@ -554,6 +554,7 @@ function renderSupervisorRuntime(openshellVersion: string): string { assert(INSTALLER_HASH_SUPERVISOR_MANIFEST_DIGESTS.has(openshellVersion)); const pins = [...INSTALLER_HASH_SUPERVISOR_MANIFEST_DIGESTS] .filter(([version]) => version !== "9.9.9" || openshellVersion === version) + .filter(([version]) => !["0.1.3", "0.1.5"].includes(version) || openshellVersion === version) .map(([version, digest]) => ` "${version}": "${digest}",`) .join("\n"); return `const OPENSHELL_SUPERVISOR_MANIFEST_DIGESTS: Readonly> = { @@ -600,7 +601,7 @@ function createFixture( tempDirs.push(fixtureRoot); copyTrustedInstallerVerifier(REPO_ROOT, fixtureRoot); retainBaseInstallerTrust(fixtureRoot); - fs.mkdirSync(binDir, { recursive: true }); + prepareInstallerBin(binDir); fs.mkdirSync(path.join(fixtureRoot, "nemoclaw-blueprint"), { recursive: true, }); @@ -751,7 +752,9 @@ function runFixture( ) { const fixtureRoot = createFixture(openshellVersion, formatting); const targetChecker = path.join(fixtureRoot, "scripts", "check-installer-hash.sh"); - const trustedRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-trusted-hash-check-")); + const trustedRoot = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-trusted-hash-check-")), + ); const trustedCheckerPath = path.join(trustedRoot, "scripts", "check-installer-hash.sh"); const trustedParserPath = path.join( trustedRoot, @@ -818,7 +821,7 @@ function expectTrustedRelease( manifests: ReadonlyMap, assets: ReadonlyMap, ): void { - expect(result.status, result.stdout).toBe(0); + expect(result.status, result.stdout + result.stderr).toBe(0); expect(result.stdout).toContain(`Checking OpenShell v${version} release assets`); manifests.forEach((contents, manifest) => { const digest = createHash("sha256").update(contents).digest("hex"); @@ -829,7 +832,7 @@ function expectTrustedRelease( } function parseNpmReplacement(source: string, digest: string, trustedDigest = digest) { - const root = createFixture(); + const root = createFixture(npmReplacementFixtureVersion(source)); const parser = path.join(root, "scripts/checks/extract-installer-pins.mts"); const parserSource = fs.readFileSync(parser, "utf8"); fs.writeFileSync(parser, parserSource.replace(digest, trustedDigest)); @@ -1058,7 +1061,6 @@ describe("installer hash verification", () => { ), ]); }); - registerV012TemplateCases(REPO_ROOT, createFixture); it.each(V012_TRUST_MUTATIONS)( "rejects an altered OpenShell 0.1.2 %s", @@ -1090,7 +1092,7 @@ describe("installer hash verification", () => { const source = fs .readFileSync(path.join(REPO_ROOT, runtimePath), "utf8") .replace( - 'const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.1.2";', + /const QUALIFIED_STABLE_OPENSHELL_VERSION = "[0-9.]+";/, 'const QUALIFIED_STABLE_OPENSHELL_VERSION = "0.0.116";', ); const prepared = selectPreparedGatewayRuntime(source).replace( @@ -1160,10 +1162,8 @@ describe("installer hash verification", () => { }); it("requires the trusted release-record prerequisite before a newer pin PR", () => { - // The first invocation keeps the base trust records unchanged while the - // separate target tree selects 9.9.9. The target cannot authorize itself. - // The second models the complete trust prerequisite already present in - // base code; only then may the otherwise identical pin tree pass. + // The candidate cannot authorize itself. Accept the same pins only after + // their trust records exist in the separate base checkout. const beforePrerequisite = runFixture("complete", "9.9.9", true); expect(beforePrerequisite.status).toBe(1); expect(beforePrerequisite.stdout).toContain( diff --git a/test/install/installer-hash-v012-suite.ts b/test/install/installer-hash-v012-suite.ts index 70c03d1eeda..62ae2880819 100644 --- a/test/install/installer-hash-v012-suite.ts +++ b/test/install/installer-hash-v012-suite.ts @@ -42,6 +42,10 @@ function guardedInstaller(repoRoot: string, source: string): string { const featureCheck = ' if [[ "$sandbox_strings" != *"$OPENSHELL_SANDBOX_MCP_FEATURE"* ]]; then\n'; const installStart = "# invalidState: a consumed"; + if (source.includes(after)) { + for (const required of [featureCheck, installStart]) assert.ok(source.includes(required)); + return source.replace(installStart, guard + installStart); + } for (const required of [before, featureCheck, installStart]) assert.ok(source.includes(required)); return source .replace(before, after) diff --git a/test/install/installer-homebrew-formula-reuse-trust.test.ts b/test/install/installer-homebrew-formula-reuse-trust.test.ts index 45407f872a8..fea53ad5755 100644 --- a/test/install/installer-homebrew-formula-reuse-trust.test.ts +++ b/test/install/installer-homebrew-formula-reuse-trust.test.ts @@ -13,7 +13,17 @@ const INSTALLER_SOURCE = fs.readFileSync( path.join(REPO_ROOT, "scripts/install-openshell.sh"), "utf8", ); -const CURRENT_TEMPLATE_DIGEST = "229888660c962fb85db9e4604485b1b9bd2db23b30a5a31ed7348c9c5a04b179"; +const TRUSTED_V017_TEMPLATE_DIGEST = + "6f21070fe62f7556fcdb440c70fa0b792d4c1ab71441d2b9dcc74d21dadda252"; +const originalCurl = + 'curl -fL "${curl_progress[@]}" "https://github.com/NVIDIA/OpenShell/releases/download/${RELEASE_TAG}/$name" \\'; +const boundedCurl = [ + 'curl -fL "${curl_progress[@]}" --connect-timeout 10 --retry 3 --retry-delay 2 \\', + " --speed-limit 1024 --speed-time 60 \\", + ' "https://github.com/NVIDIA/OpenShell/releases/download/${RELEASE_TAG}/$name" \\', +].join("\n"); +const originalInstaller = INSTALLER_SOURCE.replace(boundedCurl, originalCurl); +const boundedInstaller = originalInstaller.replace(originalCurl, boundedCurl); const tempDirs: string[] = []; afterEach(() => { @@ -64,30 +74,35 @@ function expectTrustedTemplate(source: string, digest: string): void { describe("installer Homebrew formula reuse trust", () => { const supersededV00106Template = INSTALLER_SOURCE.replace( - 'MIN_VERSION="0.0.116"', + 'MIN_VERSION="0.1.7"', 'MIN_VERSION="0.0.106"', ) - .replace('MAX_VERSION="0.0.116"', 'MAX_VERSION="0.0.106"') - .replace('DEV_MIN_VERSION="0.0.116"', 'DEV_MIN_VERSION="0.0.106"'); + .replace('MAX_VERSION="0.1.7"', 'MAX_VERSION="0.0.106"') + .replace('DEV_MIN_VERSION="0.1.7"', 'DEV_MIN_VERSION="0.0.106"'); const untrustedTemplate = INSTALLER_SOURCE.replace( 'info "Detected $OS_LABEL ($ARCH_LABEL)"', 'info "Detected $OS_LABEL ($ARCH_LABEL)"\n# unlisted installer template', ); - // source-shape-contract: security -- The current installer template must match its reviewed trust record - it("accepts only the reviewed OpenShell 0.0.116 installer template", () => { - expect(INSTALLER_SOURCE).toContain('MIN_VERSION="0.0.116"'); - expect(INSTALLER_SOURCE).toContain('MAX_VERSION="0.0.116"'); - expectTrustedTemplate(INSTALLER_SOURCE, CURRENT_TEMPLATE_DIGEST); + // source-shape-contract: security -- Exact current installer bytes must be base-authorized before trusted CI can admit the dependent runtime change + it("accepts only the reviewed OpenShell 0.1.7 installer template", () => { + expect(INSTALLER_SOURCE).toContain('MIN_VERSION="0.1.7"'); + expect(INSTALLER_SOURCE).toContain('MAX_VERSION="0.1.7"'); + expectTrustedTemplate(boundedInstaller, TRUSTED_V017_TEMPLATE_DIGEST); }); it.each([ ["existing template", untrustedTemplate], + ["unbounded download", originalInstaller], + [ + "retained sandbox registry guard", + boundedInstaller.replace(" verifyRetainedNemoClawRegistry();\n", ""), + ], [ "download retry policy", INSTALLER_SOURCE.replace("--retry 3 --retry-delay 2", "--retry 4 --retry-delay 2"), ], - ])("rejects an unreviewed mutation of the OpenShell 0.0.116 %s", (_name, source) => { + ])("rejects an unreviewed mutation of the OpenShell 0.1.7 %s", (_name, source) => { const result = runTrustCheck(source); expect(result.status).toBe(1); @@ -99,7 +114,7 @@ describe("installer Homebrew formula reuse trust", () => { expect(result.status).toBe(1); expect(result.stderr).toContain( - "installer pin-table release 0.0.116 must match installer MIN_VERSION 0.0.106", + "installer pin-table release 0.1.7 must match installer MIN_VERSION 0.0.106", ); }); }); diff --git a/test/install/installer-supervisor-manifest-trust.test.ts b/test/install/installer-supervisor-manifest-trust.test.ts index fcc7cfcae3e..db1de94e4aa 100644 --- a/test/install/installer-supervisor-manifest-trust.test.ts +++ b/test/install/installer-supervisor-manifest-trust.test.ts @@ -12,6 +12,7 @@ import { afterEach, describe, expect, it } from "vitest"; import { V00103_SUPERVISOR_MANIFEST_DIGEST, V00116_SUPERVISOR_MANIFEST_DIGEST, + V012_SUPERVISOR_MANIFEST_DIGEST, } from "../helpers/openshell-release-fixtures"; import { supervisorV00116Fixtures } from "../helpers/openshell-installer-template"; @@ -224,43 +225,57 @@ describe("OpenShell supervisor manifest trust", () => { expect(result.status, result.stderr).toBe(0); }); - it("accepts the same-thread-group executable runtime (#12614)", () => { - const result = runParser(); - expect(result.status, result.stderr).toBe(0); + it.each([false, true])( + "accepts the thread-group runtime with selectedRelease=%s (#12928)", + (selectedRelease) => { + const result = runParser({ selectedRelease }); + expect(result.status, result.stderr).toBe(0); - const missingReader = runParser({ - transformSupervisor: (source) => - source.replace('return readGatewayProcEntry(pid, "exe");', "return null;"), - }); - expect(missingReader.status).toBe(1); - expect(missingReader.stderr).toContain( - "supervisor runtime operational template is not base-trusted", - ); - }); + const missingReader = runParser({ + selectedRelease, + transformSupervisor: (source) => + source.replace('return readGatewayProcEntry(pid, "exe");', "return null;"), + }); + expect(missingReader.status).toBe(1); + expect(missingReader.stderr).toContain( + "supervisor runtime operational template is not base-trusted", + ); + }, + ); - it("rejects a repository mutation of the thread-group executable runtime", () => { - const result = runParser({ - transformSupervisor: (source) => - selectThreadGroupExecutableRuntime(source).replace( - "ghcr.io/nvidia/openshell/supervisor@${manifestDigest}", - "registry.invalid/openshell/supervisor@${manifestDigest}", - ), - }); - expect(result.status).toBe(1); - expect(result.stderr).toContain("supervisor runtime operational template is not base-trusted"); - }); + it.each([false, true])( + "rejects a thread-group runtime mutation with selectedRelease=%s", + (selectedRelease) => { + const result = runParser({ + selectedRelease, + transformSupervisor: (source) => + selectThreadGroupExecutableRuntime(source).replace( + "ghcr.io/nvidia/openshell/supervisor@${manifestDigest}", + "registry.invalid/openshell/supervisor@${manifestDigest}", + ), + }); + expect(result.status).toBe(1); + expect(result.stderr).toContain( + "supervisor runtime operational template is not base-trusted", + ); + }, + ); - it("rejects a replacement image digest in the thread-group executable runtime", () => { - const result = runParser({ - transformSupervisor: (source) => - selectThreadGroupExecutableRuntime(source).replace( - V00116_SUPERVISOR_MANIFEST_DIGEST, - REPLACEMENT_SUPERVISOR_MANIFEST_DIGEST, - ), - }); - expect(result.status).toBe(1); - expect(result.stderr).toContain("must use only base-trusted identities"); - }); + it.each([false, true])( + "rejects a thread-group image substitution with selectedRelease=%s", + (selectedRelease) => { + const result = runParser({ + selectedRelease, + transformSupervisor: (source) => + selectThreadGroupExecutableRuntime(source).replace( + selectedRelease ? V012_SUPERVISOR_MANIFEST_DIGEST : V00116_SUPERVISOR_MANIFEST_DIGEST, + REPLACEMENT_SUPERVISOR_MANIFEST_DIGEST, + ), + }); + expect(result.status).toBe(1); + expect(result.stderr).toContain("must use only base-trusted identities"); + }, + ); it("rejects an operational mutation of the base-trusted prospective supervisor fixture", () => { const result = runParser({ diff --git a/test/install/native-security-packages.test.ts b/test/install/native-security-packages.test.ts index 756571325c2..60e23800087 100644 --- a/test/install/native-security-packages.test.ts +++ b/test/install/native-security-packages.test.ts @@ -67,37 +67,31 @@ function runLibssh2Harness(nestedFailure = false) { { mode: 0o700 }, ); - const result = spawnSync( - "bash", - [ - "-c", - [ - "set -euo pipefail", - 'source "$1"', - 'calls="$HARNESS_LOG/calls"', - 'mapfile() { local target="$2" line; eval "$target=()"; while IFS= read -r line; do eval "$target+=(\\"\\$line\\")"; done; }', - 'chmod() { printf "chmod %s\\n" "$*" >>"$calls"; }', - "id() { return 0; }", - 'useradd() { printf "useradd %s\\n" "$*" >>"$calls"; }', - 'chpasswd() { cat >/dev/null; printf "chpasswd\\n" >>"$calls"; }', - 'install() { printf "install %s\\n" "$*" >>"$calls"; }', - 'sed() { printf "sed %s\\n" "$*" >>"$calls"; }', - 'make() { printf "make %s\\n" "$*" >>"$calls"; }', - 'run_libssh2_tests "$2"', - ].join("\n"), - "libssh2-harness", - BUILD_SCRIPT, - fixture, - ], - { - encoding: "utf-8", - env: { - ...process.env, - HARNESS_LOG: harnessLog, - NESTED_FAILURE: nestedFailure ? "1" : "0", - }, + const result = spawnSync("bash", ["-s", "--", BUILD_SCRIPT, fixture], { + input: [ + "set -euo pipefail", + // Keep the sourced builder's temporary files inside this owned fixture. + 'mktemp() { command mktemp -d "$HARNESS_LOG/build.XXXXXX"; }', + 'source "$1"', + "unset -f mktemp", + 'calls="$HARNESS_LOG/calls"', + 'mapfile() { local target="$2" line; eval "$target=()"; while IFS= read -r line; do eval "$target+=(\\"\\$line\\")"; done; }', + 'chmod() { printf "chmod %s\\n" "$*" >>"$calls"; }', + "id() { return 0; }", + 'useradd() { printf "useradd %s\\n" "$*" >>"$calls"; }', + 'chpasswd() { cat >/dev/null; printf "chpasswd\\n" >>"$calls"; }', + 'install() { printf "install %s\\n" "$*" >>"$calls"; }', + 'sed() { printf "sed %s\\n" "$*" >>"$calls"; }', + 'make() { printf "make %s\\n" "$*" >>"$calls"; }', + 'run_libssh2_tests "$2"', + ].join("\n"), + encoding: "utf-8", + env: { + ...process.env, + HARNESS_LOG: harnessLog, + NESTED_FAILURE: nestedFailure ? "1" : "0", }, - ); + }); return { fixture, harnessLog, result }; } @@ -117,7 +111,10 @@ function runPythonFixPackageHarness(architecture: "amd64" | "arm64", sourceHashF 'harness_output="$1"', 'architecture="$2"', 'set -- "$harness_output"', + // Keep the sourced builder's temporary files inside this owned fixture. + 'mktemp() { command mktemp -d "$HARNESS_LOG/build.XXXXXX"; }', "source scripts/security/build-native-security-packages.sh", + "unset -f mktemp", 'download() { printf "download %s\\n" "$1" >>"$HARNESS_LOG/calls"; : >"$2"; }', 'verify_sha256() { printf "verify %s %s\\n" "$1" "${2##*/}" >>"$HARNESS_LOG/calls"; if [[ "${FAIL_SOURCE_HASH:-0}" == "1" && "$2" == *python-stdlib-original.deb ]]; then return 1; fi; }', "dpkg-deb() {", @@ -194,11 +191,19 @@ describe("native security package remediation", () => { }); it.each([ - ["amd64", "db161322a3481d2c0c3b9a3b9a03c3ab0e2b1718f54b88755fb4a3f939165b84"], - ["arm64", "d1178d24e4d143cc6c577d9dc0f26dd982efccc2e600d25ce86361f168a22be0"], + [ + "amd64", + "db161322a3481d2c0c3b9a3b9a03c3ab0e2b1718f54b88755fb4a3f939165b84", + "c8d9cc73aab111fac7dc1b26c6e2ffda7af08f04", + ], + [ + "arm64", + "d1178d24e4d143cc6c577d9dc0f26dd982efccc2e600d25ce86361f168a22be0", + "35cc8669f33410e90648c1dee9ed1395920b9b9e", + ], ] as const)( "builds the Python fix from the reviewed %s snapshot artifact", - (architecture, hash) => { + (architecture, hash, snapshotFile) => { const { fixture, harnessLog, outputDir, result } = runPythonFixPackageHarness(architecture); try { expect({ status: result.status, stderr: result.stderr }).toEqual({ @@ -206,9 +211,7 @@ describe("native security package remediation", () => { stderr: "", }); const calls = fs.readFileSync(path.join(harnessLog, "calls"), "utf-8"); - expect(calls).toContain( - `download https://snapshot.debian.org/archive/debian/20260906T023042Z/pool/main/p/python3.13/libpython3.13-stdlib_3.13.5-2+deb13u5_${architecture}.deb`, - ); + expect(calls).toContain(`download https://snapshot.debian.org/file/${snapshotFile}`); expect(calls).toContain(`verify ${hash} python-stdlib-original.deb`); expect(calls).toContain( "verify f91ec3de6331206bbe2ec3e54a05f646bd23d3c61a18d4a01b25164e070bacc9 parser.py", diff --git a/test/install/update-hermes-agent-script.test.ts b/test/install/update-hermes-agent-script.test.ts index 77b37a26a0b..c3e46bf4021 100644 --- a/test/install/update-hermes-agent-script.test.ts +++ b/test/install/update-hermes-agent-script.test.ts @@ -40,7 +40,7 @@ const CURRENT_INSTALLED_DOCKERFILE = [ "COPY agents/hermes/validate-hermes-env-secret-boundary.py /usr/local/lib/nemoclaw/validate-hermes-env-secret-boundary.py", "RUN sha256sum /sandbox/.hermes/config.yaml /sandbox/.hermes/.env > /etc/nemoclaw/hermes.config-hash", "COPY agents/hermes/mcp-config-transaction.py /usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py", - "COPY src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.116.json /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.116.json", + "COPY src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json /usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.1.7.json", "RUN HERMES_HOME=/sandbox/.hermes /usr/local/bin/hermes doctor --fix \\", " && node /opt/nemoclaw-hermes-config/generate-config.ts", "", @@ -73,6 +73,10 @@ describe("scripts/update-hermes-agent.sh", () => { fs.mkdirSync(path.dirname(script), { recursive: true }); fs.mkdirSync(path.join(repo, "agents", "hermes"), { recursive: true }); fs.mkdirSync(fakeBin, { recursive: true }); + writeExecutable( + path.join(fakeBin, "mktemp"), + '#!/bin/sh\nexec /usr/bin/mktemp "$@" "$TMPDIR/hermes.XXXXXXXX"\n', + ); fs.copyFileSync(SCRIPT, script); fs.chmodSync(script, 0o755); fs.copyFileSync(HERMES_BASE_DOCKERFILE, path.join(repo, "agents", "hermes", "Dockerfile.base")); @@ -134,6 +138,7 @@ fi ...process.env, PATH: `${fakeBin}:${process.env.PATH}`, HOME: path.join(tmp, "home"), + TMPDIR: tmp, HERMES_BASE_REF: baseRef, FAKE_DOCKER_LOG: dockerLog, FAKE_NEMOHERMES_LOG: nemohermesLog, @@ -169,6 +174,10 @@ fi fs.mkdirSync(path.dirname(script), { recursive: true }); fs.mkdirSync(path.dirname(dockerfile), { recursive: true }); fs.mkdirSync(fakeBin, { recursive: true }); + writeExecutable( + path.join(fakeBin, "mktemp"), + '#!/bin/sh\nexec /usr/bin/mktemp "$@" "$TMPDIR/hermes.XXXXXXXX"\n', + ); fs.copyFileSync(SCRIPT, script); fs.chmodSync(script, 0o755); fs.copyFileSync(HERMES_BASE_DOCKERFILE, dockerfile); @@ -210,6 +219,7 @@ printf 'fake archive' > "$output" ...process.env, PATH: `${fakeBin}:${process.env.PATH}`, HOME: path.join(tmp, "home"), + TMPDIR: tmp, NEMOCLAW_SOURCE_ROOT: undefined, }, timeout: 10_000, @@ -441,7 +451,7 @@ fi ); const installedAgentDockerfile = path.join(path.dirname(installedDockerfile), "Dockerfile"); const preMcpDockerfile = CURRENT_INSTALLED_DOCKERFILE.replace( - /^(?:COPY (?:agents\/hermes\/mcp-config-transaction\.py|src\/lib\/actions\/sandbox\/openshell-child-visible-credentials\.v0\.0\.116\.json) .*)\n/gm, + /^(?:COPY (?:agents\/hermes\/mcp-config-transaction\.py|src\/lib\/actions\/sandbox\/openshell-child-visible-credentials\.v0\.1\.7\.json) .*)\n/gm, "", ); fs.mkdirSync(path.dirname(installedDockerfile), { recursive: true }); @@ -466,7 +476,7 @@ fi expect(run.status).toBe(1); expect(run.stdout).toContain("INVALID: installed copy"); expect(run.stdout).toContain("marker hermes-mcp-config-transaction.py"); - expect(run.stdout).toContain("marker openshell-child-visible-credentials.v0.0.116.json"); + expect(run.stdout).toContain("marker openshell-child-visible-credentials.v0.1.7.json"); expect(fs.readFileSync(installedDockerfile, "utf-8")).toBe(CURRENT_INSTALLED_BASE); expect(fs.readFileSync(installedAgentDockerfile, "utf-8")).toBe(preMcpDockerfile); } finally { diff --git a/test/installer-integration/install-openshell-directory-safety.test.ts b/test/installer-integration/install-openshell-directory-safety.test.ts new file mode 100644 index 00000000000..17ece5bf6ab --- /dev/null +++ b/test/installer-integration/install-openshell-directory-safety.test.ts @@ -0,0 +1,200 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { createLinuxArm64DownloadFixture } from "../support/openshell-arm64-download"; +import { writeExecutable } from "../support/openshell-installed-version"; + +const SCRIPT = path.join(import.meta.dirname, "../..", "scripts", "install-openshell.sh"); + +describe("OpenShell installer directory safety", { timeout: 15_000 }, () => { + it("creates owner-safe user-local directories in a direct install under umask 0002 (#12496)", () => { + const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); + try { + fs.rmSync(path.join(fakeBin, "openshell")); + const bashEnv = path.join(tmp, "force-user-local-install.sh"); + fs.writeFileSync( + bashEnv, + `function [ { + if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then + return 1 + fi + builtin [ "$@" +} +`, + ); + const previousUmask = process.umask(0o002); + let directInstall; + try { + directInstall = spawnSync("bash", [SCRIPT], { + env: { + ...process.env, + BASH_ENV: bashEnv, + HOME: tmp, + XDG_BIN_HOME: "", + NEMOCLAW_NON_INTERACTIVE: "1", + NEMOCLAW_OPENSHELL_CHANNEL: "stable", + NEMOCLAW_TEST_USER_LOCAL_ONLY: "1", + PATH: `${fakeBin}:/usr/bin:/bin`, + }, + encoding: "utf8", + }); + } finally { + process.umask(previousUmask); + } + expect(directInstall.status, `${directInstall.stdout}\n${directInstall.stderr}`).toBe(0); + expect({ + local: fs.statSync(path.join(tmp, ".local")).mode & 0o777, + bin: fs.statSync(path.join(tmp, ".local", "bin")).mode & 0o777, + }).toEqual({ local: 0o755, bin: 0o755 }); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }); + it("falls back from a writable unsafe system default in interactive mode (#12496)", () => { + const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); + try { + fs.rmSync(path.join(fakeBin, "openshell")); + const localBin = path.join(tmp, ".local", "bin"); + const bashEnv = path.join(tmp, "unsafe-system-target.sh"); + fs.writeFileSync( + bashEnv, + `function [ { + if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then return 0; fi + if [[ "$#" -eq 4 && "$1" = "!" && "$2" = "-t" && "$3" = "0" && "$4" = "]" ]]; then return 1; fi + builtin [ "$@" +} +function stat { + if [[ " $* " == *" /usr/local/bin "* ]]; then printf "0 775\\n"; return 0; fi + command stat "$@" +} +`, + ); + const result = spawnSync("bash", [SCRIPT], { + env: { + ...process.env, + BASH_ENV: bashEnv, + HOME: tmp, + XDG_BIN_HOME: "", + NEMOCLAW_NON_INTERACTIVE: "", + NEMOCLAW_OPENSHELL_CHANNEL: "stable", + PATH: `${fakeBin}:/usr/bin:/bin`, + }, + encoding: "utf8", + }); + expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0); + expect(fs.existsSync(path.join(localBin, "openshell"))).toBe(true); + expect(fs.existsSync(path.join(localBin, "openshell-gateway"))).toBe(true); + expect(fs.statSync(localBin).mode & 0o777).toBe(0o755); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }); + it("rejects an unsafe system directory before invoking sudo (#12496)", () => { + const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); + try { + fs.rmSync(path.join(fakeBin, "openshell")); + const sudoLog = path.join(tmp, "sudo.log"); + writeExecutable( + path.join(fakeBin, "sudo"), + '#!/usr/bin/env bash\nprintf "%s\\n" "$*" >> "$NEMOCLAW_TEST_SUDO_LOG"\nexit 99\n', + ); + const bashEnv = path.join(tmp, "unsafe-privileged-system-target.sh"); + fs.writeFileSync( + bashEnv, + `function [ { + if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then return 1; fi + if [[ "$#" -eq 4 && "$1" = "!" && "$2" = "-t" && "$3" = "0" && "$4" = "]" ]]; then return 1; fi + builtin [ "$@" +} +function stat { + if [[ " $* " == *" /usr/local/bin "* ]]; then printf "0 775\\n"; return 0; fi + command stat "$@" +} +`, + ); + const result = spawnSync("bash", [SCRIPT], { + env: { + ...process.env, + BASH_ENV: bashEnv, + HOME: tmp, + NEMOCLAW_NON_INTERACTIVE: "", + NEMOCLAW_OPENSHELL_CHANNEL: "stable", + NEMOCLAW_TEST_SUDO_LOG: sudoLog, + PATH: `${fakeBin}:/usr/bin:/bin`, + }, + encoding: "utf8", + }); + expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(1); + expect(result.stderr).toContain("Unsafe executable directory /usr/local/bin"); + expect(fs.existsSync(sudoLog)).toBe(false); + expect(fs.existsSync(path.join(tmp, ".local", "bin", "openshell"))).toBe(false); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }); + it.each(["writable ancestor", "symlinked bin"])( + "rejects a %s before publishing a direct user-local binary (#12496)", + (unsafePath) => { + const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); + try { + fs.rmSync(path.join(fakeBin, "openshell")); + const local = path.join(tmp, ".local"); + const bin = path.join(local, "bin"); + const outside = path.join(tmp, "outside"); + fs.mkdirSync(local); + const makeUnsafe: Record void> = { + "writable ancestor": () => { + fs.mkdirSync(bin); + fs.chmodSync(local, 0o775); + }, + "symlinked bin": () => { + fs.mkdirSync(outside); + fs.symlinkSync(outside, bin); + }, + }; + makeUnsafe[unsafePath](); + const bashEnv = path.join(tmp, "force-user-local-install.sh"); + fs.writeFileSync( + bashEnv, + `function [ { + if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then + return 1 + fi + builtin [ "$@" +} +`, + ); + const result = spawnSync("bash", [SCRIPT], { + env: { + ...process.env, + BASH_ENV: bashEnv, + HOME: tmp, + XDG_BIN_HOME: "", + NEMOCLAW_NON_INTERACTIVE: "1", + NEMOCLAW_OPENSHELL_CHANNEL: "stable", + NEMOCLAW_TEST_USER_LOCAL_ONLY: "1", + PATH: `${fakeBin}:/usr/bin:/bin`, + }, + encoding: "utf8", + }); + + expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(1); + expect(result.stderr).toContain(unsafePath === "writable ancestor" ? local : bin); + expect( + fs.existsSync(path.join(unsafePath === "writable ancestor" ? bin : outside, "openshell")), + ).toBe(false); + const assertUnchanged: Record void> = { + "writable ancestor": () => expect(fs.statSync(local).mode & 0o777).toBe(0o775), + "symlinked bin": () => expect(fs.lstatSync(bin).isSymbolicLink()).toBe(true), + }; + assertUnchanged[unsafePath](); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }, + ); +}); diff --git a/test/installer-integration/install-openshell-registration.test.ts b/test/installer-integration/install-openshell-registration.test.ts new file mode 100644 index 00000000000..9a9583a787c --- /dev/null +++ b/test/installer-integration/install-openshell-registration.test.ts @@ -0,0 +1,18 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import { runCommandWithInstalledVersion } from "../support/openshell-installed-version"; + +describe("OpenShell installer gateway registrations", { timeout: 15_000 }, () => { + it.each(["user", "system", "fixed-system"] as const)( + "preserves components when the %s gateway registration is a regular file", + (legacyRegistrationFile) => { + const result = runCommandWithInstalledVersion("0.0.116", {}, { legacyRegistrationFile }); + expect(result.status).toBe(1); + expect(result.stderr).toContain("a legacy gateway registration is not a directory"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); +}); diff --git a/test/installer-integration/install-openshell-retained-state-size.test.ts b/test/installer-integration/install-openshell-retained-state-size.test.ts new file mode 100644 index 00000000000..dbb898b3cfe --- /dev/null +++ b/test/installer-integration/install-openshell-retained-state-size.test.ts @@ -0,0 +1,55 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { runCommandWithInstalledVersion } from "../support/openshell-installed-version"; + +describe("OpenShell installer retained state size", { timeout: 15_000 }, () => { + it.each([ + { record: "registry", staleSize: false }, + { record: "metadata", staleSize: false }, + { record: "registry", staleSize: true }, + { record: "metadata", staleSize: true }, + ])( + "preserves oversized retained $record when the size snapshot is stale=$staleSize", + ({ record, staleSize }) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-state-size-")); + try { + const preload = path.join(root, "stale-size.cjs"); + fs.writeFileSync( + preload, + ` +const fs = require("node:fs"); +const original = fs.fstatSync; +fs.fstatSync = (...args) => { + const stat = original(...args); + if (stat.size > 16 * 1024 * 1024) stat.size = 0; + return stat; +}; +require("node:module").syncBuiltinESMExports(); +`, + ); + const oversized = " ".repeat(16 * 1024 * 1024 + 1); + const result = runCommandWithInstalledVersion( + "0.0.116", + staleSize ? { NODE_OPTIONS: `--require=${preload}` } : {}, + { + driverBins: false, + ...(record === "registry" + ? { retainedRegistry: oversized } + : { legacyGateway: "empty", legacyMetadata: oversized }), + }, + ); + expect(result.status).toBe(1); + expect(result.stderr).toContain("exceeds the 16777216 byte limit"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, + ); +}); diff --git a/test/installer-integration/install-openshell-sandbox-digest.test.ts b/test/installer-integration/install-openshell-sandbox-digest.test.ts new file mode 100644 index 00000000000..5262f3c5f79 --- /dev/null +++ b/test/installer-integration/install-openshell-sandbox-digest.test.ts @@ -0,0 +1,42 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import credentialBoundaryManifest from "../../src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json"; +import { runCommandWithInstalledVersion } from "../support/openshell-installed-version"; + +const REQUIRED_OPENSHELL_VERSION = credentialBoundaryManifest.openshellVersion; + +describe("OpenShell split sandbox installation", () => { + it.each([ + ["0.1.2", "5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1", true], + ["0.1.2", "9b527c257e7917d11cee34075369cdfb69a57764198da6e72cc0847cb9b427aa", true], + ["0.1.2", "0".repeat(64), false], + ["0.1.2", "326ee26df8f8575ba761470757a12fe5c1cdc904ba064b81946692dd0328dd40", false], + ["0.1.3", "38fe063f19c72680a1bc4b0c343a3ec5b7a5cb087a627faab5b2aefd9d50d094", true], + ["0.1.3", "23147b42d31bd0b13e1a9ed7b3834b8e6075836cba0872fffa9e7d481f6dd281", true], + ["0.1.3", "5b2178f3b64a6c96eff9ed61bd7feeada4b4a4b3c68f3664e3b8f4f2b264a9b1", false], + ["0.1.2", "23147b42d31bd0b13e1a9ed7b3834b8e6075836cba0872fffa9e7d481f6dd281", false], + ["0.1.3", "0".repeat(64), false], + ["0.1.5", "a048b1fd2acd78f44f50a4cf0e02cdefec691656f3941e92c5ec0e09a69990e2", true], + ["0.1.5", "65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc", true], + ["0.1.5", "23147b42d31bd0b13e1a9ed7b3834b8e6075836cba0872fffa9e7d481f6dd281", false], + ["0.1.7", "b4ae1d60b238c84e290bcd08a962997e5c36ed04237fa59b09b5799b11363feb", true], + ["0.1.7", "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21", true], + ["0.1.7", "65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc", false], + ] as const)( + "checks the sandbox digest and claimed version before deferring supervisor policy verification [%s, %s]", + (version, digest, accepted) => { + const result = runCommandWithInstalledVersion( + version, + {}, + { + entrypoint: version === REQUIRED_OPENSHELL_VERSION ? "direct" : "feature-check", + sandboxMarkers: "", + sandboxBinaryDigest: digest, + }, + ); + expect(result.status === 0, result.stdout + result.stderr).toBe(accepted); + }, + ); +}); diff --git a/test/installer-integration/install-openshell-version-check.test.ts b/test/installer-integration/install-openshell-version-check.test.ts index a1543f7a86c..4a4b0594483 100644 --- a/test/installer-integration/install-openshell-version-check.test.ts +++ b/test/installer-integration/install-openshell-version-check.test.ts @@ -7,16 +7,19 @@ import os from "node:os"; import path from "node:path"; import { describe, expect, it } from "vitest"; +import credentialBoundaryManifest from "../../src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json"; +import { + prepareInstallerBin, + runCommandWithInstalledVersion, + writeExecutable, +} from "../support/openshell-installed-version"; import { createLinuxArm64DownloadFixture } from "../support/openshell-arm64-download"; - -import credentialBoundaryManifest from "../../src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.116.json"; import { BREW_OUTCOMES, CANDIDATE_RUNTIME, CANDIDATE_RUNTIME_ENABLED, OPENSHELL_FEATURE_MARKERS, OPENSHELL_MCP_FEATURE_MARKER, - OPENSHELL_REWRITE_FEATURE_MARKERS, PINNED_OPEN_SHELL_SHA256, trustedFormulaBoundaryEvents, ZERO_SHA256, @@ -25,241 +28,300 @@ import { const SCRIPT = path.join(import.meta.dirname, "../..", "scripts", "install-openshell.sh"); const REQUIRED_OPENSHELL_VERSION = credentialBoundaryManifest.openshellVersion; const LEGACY_OPENSHELL_VERSION = "0.0.44"; -type OpenShellFeaturePlacement = "openshell" | "gateway" | "split-mcp-gateway" | "none"; - -function writeExecutable(target: string, contents: string) { - fs.writeFileSync(target, contents, { mode: 0o755 }); -} -function runWithInstalledVersion( - version: string, - extraEnv: NodeJS.ProcessEnv = {}, - options: { - capability?: boolean; - curlExitCode?: number; - featurePlacement?: OpenShellFeaturePlacement; - driverBins?: boolean | "gateway" | "gateway-vm"; - driverLocation?: "path" | "explicit" | "symlink"; - driverVersion?: string; - sandboxVersion?: string; - sandboxVersionExit?: number; - sandboxBinaryDigest?: string; - driverVersionExit?: number; - driverReadable?: boolean; - homebrewAvailable?: boolean; - homebrewFormulaDownload?: boolean; - homebrewFormulaDigest?: string; - homebrewOperationLog?: string; - os?: string; - arch?: string; - } = {}, -) { - const capability = options.capability ?? true; - const featurePlacement: OpenShellFeaturePlacement = capability - ? (options.featurePlacement ?? "openshell") - : "none"; - const openshellMarkers = - featurePlacement === "openshell" - ? OPENSHELL_FEATURE_MARKERS - : featurePlacement === "split-mcp-gateway" - ? OPENSHELL_REWRITE_FEATURE_MARKERS - : ""; - const gatewayMarkers = - featurePlacement === "gateway" - ? OPENSHELL_FEATURE_MARKERS - : featurePlacement === "split-mcp-gateway" - ? OPENSHELL_MCP_FEATURE_MARKER - : ""; - const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-ver-")); - try { - const fakeBin = path.join(tmp, "bin"); - const driverBin = options.driverLocation ? path.join(tmp, "driver-bin") : fakeBin; - fs.mkdirSync(fakeBin); - fs.mkdirSync(driverBin, { recursive: true }); - - writeExecutable( - path.join(fakeBin, "uname"), - `#!/usr/bin/env bash -if [ "\${1:-}" = "-m" ]; then echo "${options.arch ?? "x86_64"}"; else echo "${options.os ?? "Linux"}"; fi`, - ); - // Fake openshell that reports the given version - writeExecutable( - path.join(fakeBin, "openshell"), - `#!/usr/bin/env bash -if [ "\${1:-}" = "--version" ]; then echo "openshell ${version}"; exit 0; fi -${openshellMarkers ? `# ${openshellMarkers}` : ""} -exit 99`, +describe("install-openshell.sh version check", { timeout: 15_000 }, () => { + it("starts a fresh OpenShell install without Node.js when no gateway state exists", () => { + const result = runCommandWithInstalledVersion( + "", + {}, + { + missingCli: true, + missingNode: true, + driverBins: false, + }, ); + expect(result.downloads, `${result.stdout}\n${result.stderr}`).not.toBe(""); + expect(result.stderr).not.toContain("Node.js is required"); + }); - const driverFixtures: Array<{ name: string; markers: string }> = - options.driverBins === false - ? [] - : [ - { name: "openshell-gateway", markers: gatewayMarkers }, - ...(options.driverBins === "gateway" - ? [] - : [ - { - name: "openshell-sandbox", - markers: OPENSHELL_MCP_FEATURE_MARKER, - }, - ]), - ...(options.driverBins === "gateway-vm" - ? [ - { - name: "openshell-driver-vm", - markers: OPENSHELL_MCP_FEATURE_MARKER, - }, - ] - : []), - ]; - for (const fixture of driverFixtures) { - writeExecutable( - path.join(driverBin, fixture.name), - `#!/usr/bin/env bash -if [ "\${1:-}" = "--version" ]; then echo "${fixture.name} ${fixture.name === "openshell-sandbox" ? (options.sandboxVersion ?? options.driverVersion ?? version) : (options.driverVersion ?? version)}"; exit ${fixture.name === "openshell-sandbox" ? (options.sandboxVersionExit ?? options.driverVersionExit ?? 0) : (options.driverVersionExit ?? 0)}; fi -# ${fixture.markers} -exit 0`, + it.each([ + { missingCli: true }, + { missingCli: false, legacyGateway: "empty" as const }, + { missingCli: true, registryGatewayPort: "8990" }, + { missingCli: false, version: REQUIRED_OPENSHELL_VERSION, legacyGateway: "empty" as const }, + ])( + "preserves retained sandbox records before replacement %#", + ({ version = "0.0.116", ...options }) => { + const result = runCommandWithInstalledVersion( + version, + { NEMOCLAW_OPENSHELL_UPGRADE_PREPARED: "1" }, + { + ...options, + driverBins: false, + retainedRegistry: + '{"sandboxes":{"retained-box":{"name":"retained-box","apiKey":"fixture-secret-do-not-print"}}}', + }, ); - if (options.driverReadable === false) fs.chmodSync(path.join(driverBin, fixture.name), 0o111); - if (options.driverLocation === "symlink") { - fs.symlinkSync(path.join(driverBin, fixture.name), path.join(fakeBin, fixture.name)); - } - } + expect(result.status).toBe(1); + expect(result.stderr).toContain("retained NemoClaw sandbox records"); + const scope = + "registryGatewayPort" in options ? `/gateways/${options.registryGatewayPort}` : ""; + expect(result.stderr).toContain(`/.nemoclaw${scope}/sandboxes.json": "retained-box"`); + expect(result.stderr).not.toContain("fixture-secret-do-not-print"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); - switch (options.sandboxBinaryDigest) { - case undefined: - break; - default: - writeExecutable( - path.join(fakeBin, "sha256sum"), - `#!/usr/bin/env bash -case "\${1:-}" in - */openshell-sandbox) - printf '%s %s\\n' '${options.sandboxBinaryDigest}' "$1" - exit 0 - ;; -esac -exit 1`, - ); - } + it("requires Node.js when sandbox records remain without the old CLI", () => { + const result = runCommandWithInstalledVersion( + "", + {}, + { + missingCli: true, + missingNode: true, + driverBins: false, + retainedRegistry: '{"sandboxes":{"retained-box":{"name":"retained-box"}}}', + }, + ); + expect(result.stderr).toContain("Node.js is required"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }); - writeExecutable( - path.join(fakeBin, "curl"), - options.homebrewFormulaDownload - ? `#!/usr/bin/env bash -printf '%s\n' "$*" >&2; out="" -while [ "$#" -gt 0 ]; do - if [ "$1" = "-o" ]; then shift; out="$1"; fi - shift || true -done -[ -n "$out" ] || exit 1 -printf '%s\n' '# downloaded OpenShell formula' 'class Openshell < Formula' > "$out" -exit 0` - : `#!/usr/bin/env bash -echo "curl stub: $*" >&2 -exit ${options.curlExitCode ?? 1}`, + it.each([ + "invalid JSON", + '{"sandboxes":[]}', + '{"sandboxes":{"retained-box":null}}', + '{"sandboxes":{"retained-box":{"name":"different-box"}}}', + '{"sandboxes":{" ":{"name":" "}}}', + ])("preserves unreadable or malformed retained registry %#", (retainedRegistry) => { + const result = runCommandWithInstalledVersion( + "", + {}, + { + missingCli: true, + driverBins: false, + retainedRegistry, + }, ); + expect(result.status).toBe(1); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }); - writeExecutable( - path.join(fakeBin, "gh"), - `#!/usr/bin/env bash -exit 1`, + it("allows a fresh install with a verified empty NemoClaw registry", () => { + const result = runCommandWithInstalledVersion( + "", + {}, + { + missingCli: true, + driverBins: false, + retainedRegistry: '{"sandboxes":{}}', + }, ); + expect(result.downloads, result.stderr).not.toBe(""); + expect(result.stderr).not.toContain("Cannot replace OpenShell"); + expect(result.retainedFilesUnchanged).toBe(true); + }); - if ((options.os ?? "Linux") === "Darwin") { - const tap = path.join(tmp, "tap"); - const formula = path.join(tap, "Formula", "openshell.rb"); - fs.mkdirSync(path.dirname(formula), { recursive: true }); - fs.writeFileSync(formula, "class Openshell < Formula\nend\n"); - writeExecutable( - path.join(fakeBin, "sha256sum"), - `#!/usr/bin/env bash -case "\${1:-}" in - ${JSON.stringify(formula)}) - if grep -q '^# downloaded OpenShell formula$' "$1"; then - printf '%s %s\\n' '${PINNED_OPEN_SHELL_SHA256.formula}' "$1" - else - printf '%s %s\\n' '${options.homebrewFormulaDigest ?? PINNED_OPEN_SHELL_SHA256.formula}' "$1" - fi - ;; - */openshell.rb) printf '%s %s\\n' '${PINNED_OPEN_SHELL_SHA256.formula}' "$1" ;; - *) /usr/bin/sha256sum "$@" ;; -esac`, + it.each([undefined, true, "system"] as const)( + "requires Node.js to inspect existing CLI or gateway state %s", + (legacyGateway) => { + const result = runCommandWithInstalledVersion( + "0.0.116", + {}, + { + missingCli: legacyGateway !== undefined, + missingNode: true, + legacyGateway, + }, ); - writeExecutable( - path.join(fakeBin, "codesign"), - `#!/usr/bin/env bash -state="\${NEMOCLAW_FAKE_CODESIGN_STATE:-}" -if [ "\${1:-}" = "-d" ]; then - if [ "\${NEMOCLAW_FAKE_CODESIGN_HAS_ENTITLEMENT:-1}" = "1" ] || { [ -n "$state" ] && [ -f "$state" ]; }; then - printf '%s\\n' 'com.apple.security.hypervisor' - fi - exit 0 -fi -if [ -n "\${NEMOCLAW_FAKE_CODESIGN_LOG:-}" ]; then - printf '%s\\n' "$*" >> "$NEMOCLAW_FAKE_CODESIGN_LOG" -fi -if [ -n "$state" ]; then - : > "$state" -fi -exit 0`, + expect(result.status).toBe(1); + expect(result.stderr).toContain("Node.js is required to verify legacy OpenShell state"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); + + it("requires legacy inspection when the gateway configuration path is relative", () => { + const result = runCommandWithInstalledVersion( + "", + {}, + { + missingCli: true, + missingNode: true, + driverBins: false, + systemGatewayDirectory: "relative", + }, + ); + expect(result.status).toBe(1); + expect(result.stderr).toContain("Node.js is required"); + expect(result.downloads).toBe(""); + }); + + it("requires legacy inspection when a gateway parent is a dangling symbolic link", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-gateway-parent-")); + try { + const parent = path.join(root, "linked-config"); + fs.symlinkSync(path.join(root, "missing"), parent); + const result = runCommandWithInstalledVersion( + "", + {}, + { + missingCli: true, + missingNode: true, + driverBins: false, + systemGatewayDirectory: parent, + }, ); - switch (options.homebrewAvailable ?? true) { - case true: - writeExecutable( - path.join(fakeBin, "brew"), - `#!/usr/bin/env bash -${options.homebrewOperationLog ? `printf '%s\\n' "$*" >> ${JSON.stringify(options.homebrewOperationLog)}` : ""} -case "$*" in - "--repository nvidia/openshell") printf '%s\\n' ${JSON.stringify(tap)}; exit 0 ;; - "list --formula openshell") - exit 0 - ;; - "info --json=v2 openshell") - printf '%s\n' '{"formulae":[{"name":"openshell","tap":"nvidia/openshell"}]}' - exit 0 - ;; - "services restart openshell") - exit 0 - ;; - "install --formula nvidia/openshell/openshell"|"reinstall --formula nvidia/openshell/openshell") - exit "\${NEMOCLAW_TEST_BREW_INSTALL_STATUS:-0}" - ;; - "--prefix") - printf '%s\\n' ${JSON.stringify(tmp)} - exit 0 - ;; -esac -exit 0`, - ); - break; - } + expect(result.status).toBe(1); + expect(result.stderr).toContain("Node.js is required"); + expect(result.downloads).toBe(""); + expect(fs.lstatSync(parent).isSymbolicLink()).toBe(true); + } finally { + fs.rmSync(root, { recursive: true, force: true }); } + }); - const explicitDriverEnv = - options.driverLocation === "explicit" - ? { - NEMOCLAW_OPENSHELL_GATEWAY_BIN: path.join(driverBin, "openshell-gateway"), - NEMOCLAW_OPENSHELL_SANDBOX_BIN: path.join(driverBin, "openshell-sandbox"), - } - : {}; - return spawnSync("bash", [SCRIPT], { - env: { - ...process.env, - NEMOCLAW_OPENSHELL_CHANNEL: "stable", - ...explicitDriverEnv, - ...extraEnv, - PATH: `${fakeBin}:${driverBin}:/usr/bin:/bin`, + it.each(["Linux", "Darwin"] as const)( + "preserves the legacy gateway before component replacement on %s", + (os) => { + const result = runCommandWithInstalledVersion( + "0.0.116", + {}, + { + legacyGateway: true, + os, + arch: "arm64", + }, + ); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("legacy gateway"); + expect(result.stderr).toContain('gateway "nemoclaw" has sandboxes: "retained-box"'); + expect(result.downloads, `${result.stdout}\n${result.stderr}`).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); + + it.each([ + "partial", + "invalid", + "unreachable", + "stderr", + "corrupt", + "system", + "retired", + "metadata-link", + "metadata-directory", + ] as const)("does not replace components with %s legacy state", (legacyGateway) => { + const result = runCommandWithInstalledVersion( + "0.0.116", + { + NEMOCLAW_OPENSHELL_UPGRADE_PREPARED: "1", + NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE: "1", }, - encoding: "utf8", - }); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } -} + { legacyGateway }, + ); + expect(result.status).toBe(1); + expect(result.stderr).toContain("legacy gateway"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }); + + it("allows replacement with a verified empty legacy inventory and preserves user mTLS", () => { + const result = runCommandWithInstalledVersion("0.0.116", {}, { legacyGateway: "empty" }); + expect(result.downloads, result.stderr).not.toBe(""); + expect(result.stderr).not.toContain("Cannot replace OpenShell"); + expect(result.retainedFilesUnchanged).toBe(true); + }); + + it.each([false, true])( + "preserves default system gateway state with an empty override and missing CLI=%s", + (missingCli) => { + const result = runCommandWithInstalledVersion( + "0.0.116", + {}, + { + legacyGateway: "fixed-system", + missingCli, + missingNode: missingCli, + driverBins: false, + }, + ); + expect(result.status).toBe(1); + expect(result.stderr).toContain(missingCli ? "Node.js is required" : 'gateway "nemoclaw"'); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); + + it.each([ + { legacyInventory: [{ name: "alpha" }, { name: "beta" }] }, + { legacyInventory: { sandboxes: [{ name: "alpha" }, { name: "beta" }], next_page_token: "" } }, + ])("names all sandboxes in a complete legacy inventory %#", ({ legacyInventory }) => { + const result = runCommandWithInstalledVersion( + "0.0.116", + {}, + { legacyGateway: true, legacyInventory }, + ); + expect(result.status).toBe(1); + expect(result.stderr).toContain('gateway "nemoclaw" has sandboxes: "alpha", "beta"'); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }); + + it.each([[null], [{}], [{ name: "" }], [{ name: "a".repeat(513) }]])( + "refuses malformed sandbox names without replacing legacy state %#", + (...rows) => { + const result = runCommandWithInstalledVersion( + "0.0.116", + {}, + { legacyGateway: true, legacyInventory: rows }, + ); + expect(result.status).toBe(1); + expect(result.stderr).toContain("invalid or incomplete sandbox inventory"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); + + it.each([ + ["missing", { missingCli: true }], + ["failing", { versionExit: 1 }], + ] as const)("preserves registered legacy state with a %s CLI", (_kind, options) => { + const result = runCommandWithInstalledVersion( + "0.0.116", + {}, + { ...options, legacyGateway: true }, + ); + expect(result.status).toBe(1); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }); + + it("rejects an endpoint override instead of inventorying a different gateway", () => { + const result = runCommandWithInstalledVersion( + "0.0.116", + { OPENSHELL_GATEWAY_ENDPOINT: "https://127.0.0.1:9999" }, + { legacyGateway: "empty" }, + ); + expect(result.status).toBe(1); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }); + + it.each(["unknown", "0.0.116 could not load installation"])( + "does not replace an installed CLI that reports %s", + (version) => { + const result = runCommandWithInstalledVersion(version); + expect(result.status).toBe(1); + expect(result.stderr).toContain("installed CLI version is unknown"); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); -describe("install-openshell.sh version check", { timeout: 15_000 }, () => { it.runIf(CANDIDATE_RUNTIME_ENABLED)( "validates the receipt-bound candidate through the installer path (#6691)", () => { @@ -283,13 +345,51 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { ); it("exits cleanly when the required OpenShell and driver binaries are already installed", () => { - const result = runWithInstalledVersion(REQUIRED_OPENSHELL_VERSION); + const result = runCommandWithInstalledVersion(REQUIRED_OPENSHELL_VERSION); expect(result.status).toBe(0); expect(result.stdout).toContain(`already installed: ${REQUIRED_OPENSHELL_VERSION}`); }); + it("preserves registered sandboxes when the selected OpenShell release is already installed", () => { + const result = runCommandWithInstalledVersion( + REQUIRED_OPENSHELL_VERSION, + {}, + { legacyGateway: true }, + ); + expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0); + expect(result.stdout).toContain(`already installed: ${REQUIRED_OPENSHELL_VERSION}`); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }); + + it.each(["Linux", "Darwin"] as const)( + "refuses component replacement with a current CLI and registered sandboxes on %s", + (os) => { + const result = runCommandWithInstalledVersion( + REQUIRED_OPENSHELL_VERSION, + {}, + { legacyGateway: true, driverBins: false, os, arch: "arm64" }, + ); + expect(result.status).toBe(1); + expect(result.stderr).toContain('legacy gateway "nemoclaw" has sandboxes'); + expect(result.downloads).toBe(""); + expect(result.retainedFilesUnchanged).toBe(true); + }, + ); + + it("allows current-release component repair after verifying an empty gateway inventory", () => { + const result = runCommandWithInstalledVersion( + REQUIRED_OPENSHELL_VERSION, + {}, + { legacyGateway: "empty", driverBins: false }, + ); + expect(result.downloads, result.stderr).not.toBe(""); + expect(result.stderr).not.toContain("Cannot replace OpenShell"); + expect(result.retainedFilesUnchanged).toBe(true); + }); + it("accepts MCP L7 support from the installed gateway sidecar", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { featurePlacement: "split-mcp-gateway" }, @@ -299,7 +399,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("does not combine the OpenShell CLI with driver binaries from another PATH root", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { driverLocation: "path" }, @@ -312,7 +412,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("accepts cross-prefix driver binaries only through explicit overrides", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { driverLocation: "explicit" }, @@ -322,7 +422,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("rejects mixed release components hidden behind one symlink directory", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { driverLocation: "symlink" }, @@ -332,7 +432,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("rejects stale components copied into the active install root", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { driverVersion: "0.0.71" }, @@ -342,7 +442,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("rejects a component whose version probe fails after printing a version", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { driverVersionExit: 42 }, @@ -352,7 +452,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("accepts the exact pinned sandbox when its host-side version probe cannot load", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { @@ -365,7 +465,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("rejects a non-runnable sandbox whose digest is not a pinned release artifact", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { sandboxVersionExit: 127, sandboxBinaryDigest: ZERO_SHA256 }, @@ -375,7 +475,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("rejects a selected component that cannot be scanned", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { driverReadable: false }, @@ -389,7 +489,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { path.join(os.tmpdir(), "nemoclaw-openshell-component-dir-"), ); try { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, { NEMOCLAW_OPENSHELL_GATEWAY_BIN: explicitDirectory, @@ -405,7 +505,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("requests the Darwin Arm64 gateway archive instead of the VM-driver archive when driver binaries are missing", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { @@ -421,14 +521,18 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("fails closed when the required OpenShell lacks required messaging rewrite support", () => { - const result = runWithInstalledVersion(REQUIRED_OPENSHELL_VERSION, {}, { capability: false }); + const result = runCommandWithInstalledVersion( + REQUIRED_OPENSHELL_VERSION, + {}, + { capability: false }, + ); expect(result.status).toBe(1); // `fail()` writes to stderr as of #3446; previously stdout. expect(result.stderr).toMatch(/missing request-body-credential-rewrite support/); }); it("accepts macOS OpenShell when the gateway binary is installed", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { @@ -442,7 +546,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("ignores a stale sibling sandbox binary for a macOS VM-driver install", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { os: "Darwin", arch: "arm64", sandboxVersion: LEGACY_OPENSHELL_VERSION }, @@ -456,7 +560,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { try { const state = path.join(tmp, "codesign-state"); const log = path.join(tmp, "codesign.log"); - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, { NEMOCLAW_FAKE_CODESIGN_HAS_ENTITLEMENT: "0", @@ -482,7 +586,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { }); it("reuses the standalone macOS gateway when Homebrew is unavailable", () => { - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, {}, { @@ -505,7 +609,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-stale-formula-")); try { const brewLog = path.join(tmp, "brew.log"); - const result = runWithInstalledVersion( + const result = runCommandWithInstalledVersion( REQUIRED_OPENSHELL_VERSION, { NEMOCLAW_TEST_BREW_INSTALL_STATUS: "1" }, { @@ -536,7 +640,7 @@ describe("install-openshell.sh version check", { timeout: 15_000 }, () => { const fakeBin = path.join(tmp, "bin"); const downloadLog = path.join(tmp, "downloads.log"); const brewLog = path.join(tmp, "brew.log"); - fs.mkdirSync(fakeBin); + prepareInstallerBin(fakeBin); writeExecutable( path.join(fakeBin, "uname"), @@ -579,6 +683,8 @@ exit 0`, env: { ...process.env, HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", PATH: `${fakeBin}:/usr/bin:/bin`, }, @@ -606,7 +712,7 @@ exit 0`, const brewLog = path.join(tmp, "brew.log"); const formulaTmpDir = path.join(tmp, "formula-tmp"); const untrustCount = path.join(tmp, "untrust-count"); - fs.mkdirSync(fakeBin); + prepareInstallerBin(fakeBin); fs.mkdirSync(path.join(homebrewPrefix, "bin"), { recursive: true }); writeExecutable( @@ -726,6 +832,8 @@ exit 1`, env: { ...process.env, HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), XDG_BIN_HOME: path.join(tmp, "local-bin"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", PATH: `${fakeBin}:/usr/bin:/bin`, @@ -779,6 +887,8 @@ exit 1`, env: { ...process.env, HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), XDG_BIN_HOME: path.join(tmp, "local-bin"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", NEMOCLAW_TEST_BREW_TRUST_STATUS: "1", @@ -804,6 +914,8 @@ exit 1`, env: { ...process.env, HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), XDG_BIN_HOME: path.join(tmp, "local-bin"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", NEMOCLAW_TEST_BREW_TRUST_HELP_STATUS: "1", @@ -835,6 +947,8 @@ exit 1`, env: { ...process.env, HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), XDG_BIN_HOME: path.join(tmp, "local-bin"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", PATH: `${fakeBin}:/usr/bin:/bin`, @@ -867,193 +981,6 @@ exit 1`, } }); - it("creates owner-safe user-local directories in a direct install under umask 0002 (#12496)", () => { - const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); - try { - fs.rmSync(path.join(fakeBin, "openshell")); - const bashEnv = path.join(tmp, "force-user-local-install.sh"); - fs.writeFileSync( - bashEnv, - `function [ { - if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then - return 1 - fi - builtin [ "$@" -} -`, - ); - const previousUmask = process.umask(0o002); - let directInstall; - try { - directInstall = spawnSync("bash", [SCRIPT], { - env: { - ...process.env, - BASH_ENV: bashEnv, - HOME: tmp, - XDG_BIN_HOME: "", - NEMOCLAW_NON_INTERACTIVE: "1", - NEMOCLAW_OPENSHELL_CHANNEL: "stable", - NEMOCLAW_TEST_USER_LOCAL_ONLY: "1", - PATH: `${fakeBin}:/usr/bin:/bin`, - }, - encoding: "utf8", - }); - } finally { - process.umask(previousUmask); - } - expect(directInstall.status, `${directInstall.stdout}\n${directInstall.stderr}`).toBe(0); - expect({ - local: fs.statSync(path.join(tmp, ".local")).mode & 0o777, - bin: fs.statSync(path.join(tmp, ".local", "bin")).mode & 0o777, - }).toEqual({ local: 0o755, bin: 0o755 }); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } - }); - it("falls back from a writable unsafe system default in interactive mode (#12496)", () => { - const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); - try { - fs.rmSync(path.join(fakeBin, "openshell")); - const localBin = path.join(tmp, ".local", "bin"); - const bashEnv = path.join(tmp, "unsafe-system-target.sh"); - fs.writeFileSync( - bashEnv, - `function [ { - if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then return 0; fi - if [[ "$#" -eq 4 && "$1" = "!" && "$2" = "-t" && "$3" = "0" && "$4" = "]" ]]; then return 1; fi - builtin [ "$@" -} -function stat { - if [[ " $* " == *" /usr/local/bin "* ]]; then printf "0 775\\n"; return 0; fi - command stat "$@" -} -`, - ); - const result = spawnSync("bash", [SCRIPT], { - env: { - ...process.env, - BASH_ENV: bashEnv, - HOME: tmp, - XDG_BIN_HOME: "", - NEMOCLAW_NON_INTERACTIVE: "", - NEMOCLAW_OPENSHELL_CHANNEL: "stable", - PATH: `${fakeBin}:/usr/bin:/bin`, - }, - encoding: "utf8", - }); - expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0); - expect(fs.existsSync(path.join(localBin, "openshell"))).toBe(true); - expect(fs.existsSync(path.join(localBin, "openshell-gateway"))).toBe(true); - expect(fs.statSync(localBin).mode & 0o777).toBe(0o755); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } - }); - it("rejects an unsafe system directory before invoking sudo (#12496)", () => { - const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); - try { - fs.rmSync(path.join(fakeBin, "openshell")); - const sudoLog = path.join(tmp, "sudo.log"); - writeExecutable( - path.join(fakeBin, "sudo"), - '#!/usr/bin/env bash\nprintf "%s\\n" "$*" >> "$NEMOCLAW_TEST_SUDO_LOG"\nexit 99\n', - ); - const bashEnv = path.join(tmp, "unsafe-privileged-system-target.sh"); - fs.writeFileSync( - bashEnv, - `function [ { - if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then return 1; fi - if [[ "$#" -eq 4 && "$1" = "!" && "$2" = "-t" && "$3" = "0" && "$4" = "]" ]]; then return 1; fi - builtin [ "$@" -} -function stat { - if [[ " $* " == *" /usr/local/bin "* ]]; then printf "0 775\\n"; return 0; fi - command stat "$@" -} -`, - ); - const result = spawnSync("bash", [SCRIPT], { - env: { - ...process.env, - BASH_ENV: bashEnv, - HOME: tmp, - NEMOCLAW_NON_INTERACTIVE: "", - NEMOCLAW_OPENSHELL_CHANNEL: "stable", - NEMOCLAW_TEST_SUDO_LOG: sudoLog, - PATH: `${fakeBin}:/usr/bin:/bin`, - }, - encoding: "utf8", - }); - expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(1); - expect(result.stderr).toContain("Unsafe executable directory /usr/local/bin"); - expect(fs.existsSync(sudoLog)).toBe(false); - expect(fs.existsSync(path.join(tmp, ".local", "bin", "openshell"))).toBe(false); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } - }); - it.each(["writable ancestor", "symlinked bin"])( - "rejects a %s before publishing a direct user-local binary (#12496)", - (unsafePath) => { - const { tmp, fakeBin } = createLinuxArm64DownloadFixture(); - try { - fs.rmSync(path.join(fakeBin, "openshell")); - const local = path.join(tmp, ".local"); - const bin = path.join(local, "bin"); - const outside = path.join(tmp, "outside"); - fs.mkdirSync(local); - const makeUnsafe: Record void> = { - "writable ancestor": () => { - fs.mkdirSync(bin); - fs.chmodSync(local, 0o775); - }, - "symlinked bin": () => { - fs.mkdirSync(outside); - fs.symlinkSync(outside, bin); - }, - }; - makeUnsafe[unsafePath](); - const bashEnv = path.join(tmp, "force-user-local-install.sh"); - fs.writeFileSync( - bashEnv, - `function [ { - if [[ "$#" -eq 3 && "$1" = "-w" && "$2" = "/usr/local/bin" && "$3" = "]" ]]; then - return 1 - fi - builtin [ "$@" -} -`, - ); - const result = spawnSync("bash", [SCRIPT], { - env: { - ...process.env, - BASH_ENV: bashEnv, - HOME: tmp, - XDG_BIN_HOME: "", - NEMOCLAW_NON_INTERACTIVE: "1", - NEMOCLAW_OPENSHELL_CHANNEL: "stable", - NEMOCLAW_TEST_USER_LOCAL_ONLY: "1", - PATH: `${fakeBin}:/usr/bin:/bin`, - }, - encoding: "utf8", - }); - - expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(1); - expect(result.stderr).toContain(unsafePath === "writable ancestor" ? local : bin); - expect( - fs.existsSync(path.join(unsafePath === "writable ancestor" ? bin : outside, "openshell")), - ).toBe(false); - const assertUnchanged: Record void> = { - "writable ancestor": () => expect(fs.statSync(local).mode & 0o777).toBe(0o775), - "symlinked bin": () => expect(fs.lstatSync(bin).isSymbolicLink()).toBe(true), - }; - assertUnchanged[unsafePath](); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } - }, - ); - it("upgrades into the active writable openshell directory to avoid PATH shadowing", () => { const tmp = fs.mkdtempSync( path.join(fs.realpathSync(os.tmpdir()), "nemoclaw-openshell-active-dir-"), @@ -1063,7 +990,7 @@ function stat { const fakeBin = path.join(tmp, "fake-bin"); const installLog = path.join(tmp, "install.log"); fs.mkdirSync(activeBin); - fs.mkdirSync(fakeBin); + prepareInstallerBin(fakeBin); writeExecutable( path.join(activeBin, "openshell"), @@ -1175,6 +1102,8 @@ exit 0`, env: { ...process.env, HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", PATH: `${fakeBin}:${activeBin}:/usr/bin:/bin`, }, @@ -1198,7 +1127,7 @@ exit 0`, const fakeBin = path.join(tmp, "bin"); const tarLog = path.join(tmp, "tar.log"); const installLog = path.join(tmp, "install.log"); - fs.mkdirSync(fakeBin); + prepareInstallerBin(fakeBin); writeExecutable( path.join(fakeBin, "uname"), @@ -1270,6 +1199,8 @@ exit 0`, env: { ...process.env, HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), NEMOCLAW_OPENSHELL_CHANNEL: "stable", PATH: `${fakeBin}:/usr/bin:/bin`, }, @@ -1290,7 +1221,7 @@ exit 0`, it.each([22, 28])( "stops before verification when curl exits with status %i (#11281)", (status) => { - const result = runWithInstalledVersion("0.0.38", {}, { curlExitCode: status }); + const result = runCommandWithInstalledVersion("0.0.38", {}, { curlExitCode: status }); expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(status); expect(result.stderr).toContain("curl stub:"); @@ -1300,32 +1231,32 @@ exit 0`, ); it("triggers upgrade when openshell 0.0.38 is installed (below current floor)", () => { - const result = runWithInstalledVersion("0.0.38"); + const result = runCommandWithInstalledVersion("0.0.38"); expect(result.status).not.toBe(0); expect(result.stdout).toMatch(/below minimum.*upgrading/); }); it("triggers upgrade when openshell 0.0.28 is installed (below MIN_VERSION)", () => { - const result = runWithInstalledVersion("0.0.28"); + const result = runCommandWithInstalledVersion("0.0.28"); // Script should warn about upgrade then fail at the download step (curl stub fails) expect(result.status).not.toBe(0); expect(result.stdout).toMatch(/below minimum.*upgrading/); }); it("triggers upgrade when openshell 0.0.26 is installed (Landlock-vulnerable version)", () => { - const result = runWithInstalledVersion("0.0.26"); + const result = runCommandWithInstalledVersion("0.0.26"); expect(result.status).not.toBe(0); expect(result.stdout).toMatch(/below minimum.*upgrading/); }); it("triggers upgrade when openshell 0.0.24 is installed (old minimum)", () => { - const result = runWithInstalledVersion("0.0.24"); + const result = runCommandWithInstalledVersion("0.0.24"); expect(result.status).not.toBe(0); expect(result.stdout).toMatch(/below minimum.*upgrading/); }); - it("reinstalls the pinned release when openshell 0.0.117 is above MAX_VERSION", () => { - const result = runWithInstalledVersion("0.0.117"); + it("reinstalls the pinned release when openshell 0.1.8 is above MAX_VERSION", () => { + const result = runCommandWithInstalledVersion("0.1.8"); expect(result.status).not.toBe(0); expect(result.stdout).toContain( `above the maximum (${REQUIRED_OPENSHELL_VERSION}) supported by this NemoClaw release`, @@ -1338,7 +1269,7 @@ exit 0`, }); it("reinstalls the pinned release when openshell is at a much newer version", () => { - const result = runWithInstalledVersion("0.1.0"); + const result = runCommandWithInstalledVersion("1.0.0"); expect(result.status).not.toBe(0); expect(result.stdout).toContain( `above the maximum (${REQUIRED_OPENSHELL_VERSION}) supported by this NemoClaw release`, @@ -1351,13 +1282,13 @@ exit 0`, }); it("rejects the OpenShell dev channel even with the former risk-acceptance flag", () => { - const result = runWithInstalledVersion("0.0.116.dev84+g6b2180425", { + const result = runCommandWithInstalledVersion("0.0.116.dev84+g6b2180425", { NEMOCLAW_OPENSHELL_CHANNEL: "dev", NEMOCLAW_ACCEPT_DEV_UNVERIFIED_INSTALL: "1", }); expect(result.status).toBe(1); expect(result.stderr).toContain( - "NemoClaw requires exact stable OpenShell 0.0.116; the dev channel is not supported.", + "NemoClaw requires exact stable OpenShell 0.1.7; the dev channel is not supported.", ); expect(result.stdout).not.toContain("Installing OpenShell from release 'dev'"); }); @@ -1366,8 +1297,8 @@ exit 0`, "NEMOCLAW_OPENSHELL_MIN_VERSION", "NEMOCLAW_OPENSHELL_MAX_VERSION", "NEMOCLAW_OPENSHELL_PIN_VERSION", - ] as const)("rejects a non-0.0.116 %s override before installation", (variable) => { - const result = runWithInstalledVersion(REQUIRED_OPENSHELL_VERSION, { + ] as const)("rejects a non-0.1.2 %s override before installation", (variable) => { + const result = runCommandWithInstalledVersion(REQUIRED_OPENSHELL_VERSION, { [variable]: "0.0.115", }); expect(result.status).toBe(1); @@ -1378,7 +1309,7 @@ exit 0`, }); it("reconciles an installed development build to the exact stable release", () => { - const result = runWithInstalledVersion("0.0.116-dev.8+g7bce1223d"); + const result = runCommandWithInstalledVersion("0.0.116-dev.8+g7bce1223d"); expect(result.status).not.toBe(0); expect(result.stdout).toContain("OpenShell development builds are unsupported"); expect(result.stdout).toContain( @@ -1387,7 +1318,7 @@ exit 0`, }); it("keeps auto on the stable release-selection contract", () => { - const result = runWithInstalledVersion("0.0.36", { + const result = runCommandWithInstalledVersion("0.0.36", { NEMOCLAW_OPENSHELL_CHANNEL: "auto", }); @@ -1399,7 +1330,7 @@ exit 0`, }); it("rejects the removed artifact channel", () => { - const result = runWithInstalledVersion("0.0.72", { + const result = runCommandWithInstalledVersion("0.0.72", { NEMOCLAW_OPENSHELL_CHANNEL: "artifact", }); @@ -1411,7 +1342,7 @@ exit 0`, const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-noop-")); try { const fakeBin = path.join(tmp, "bin"); - fs.mkdirSync(fakeBin); + prepareInstallerBin(fakeBin); // No openshell binary — just stub curl/gh to fail fast writeExecutable( @@ -1431,11 +1362,13 @@ exit 1`, ...process.env, NEMOCLAW_OPENSHELL_CHANNEL: "stable", PATH: `${fakeBin}:/usr/bin:/bin`, + HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), }, encoding: "utf8", }); - // Should attempt install (not exit 0 early) and fail at the download step expect(result.stdout).toMatch(/Installing OpenShell from release/); expect(result.status).not.toBe(0); } finally { diff --git a/test/installer-integration/install-openshell-version-pin.test.ts b/test/installer-integration/install-openshell-version-pin.test.ts index d9463f8ae50..e11f7496a23 100644 --- a/test/installer-integration/install-openshell-version-pin.test.ts +++ b/test/installer-integration/install-openshell-version-pin.test.ts @@ -8,6 +8,8 @@ import path from "node:path"; import { expect, test } from "vitest"; +import { PINNED_OPEN_SHELL_SHA256 } from "../helpers/openshell-release-fixtures"; + const REPO_ROOT = path.join(import.meta.dirname, "../.."); const INSTALL_SCRIPT = path.join(REPO_ROOT, "scripts", "install-openshell.sh"); const TEST_TIMEOUT_MS = 2 * 60_000; @@ -26,8 +28,8 @@ test( `#!/bin/sh printf '%s\\n' '${JSON.stringify([ { tagName: "v0.0.99" }, - { tagName: "v0.0.117" }, - { tagName: "v0.0.116" }, + { tagName: "v0.1.8" }, + { tagName: "v0.1.7" }, ])}'`, ); @@ -42,8 +44,8 @@ printf '%s\\n' '${JSON.stringify([ const pin = require(${JSON.stringify(path.join(REPO_ROOT, "src/lib/onboard/openshell-pin.ts"))}); const version = require(${JSON.stringify(path.join(REPO_ROOT, "src/lib/onboard/openshell-version.ts"))}); const deps = { - getBlueprintMinOpenshellVersion: () => "0.0.116", - getBlueprintMaxOpenshellVersion: () => "0.0.116", + getBlueprintMinOpenshellVersion: () => "0.1.7", + getBlueprintMaxOpenshellVersion: () => "0.1.7", versionGte: version.versionGte, }; const resolution = pin.resolveOpenshellInstallPin(deps); @@ -68,12 +70,12 @@ process.stdout.write(JSON.stringify({ expect(result.status, result.stderr).toBe(0); expect(JSON.parse(result.stdout)).toEqual({ installed: "0.0.99", - resolution: { kind: "pin", version: "0.0.116", latest: "0.0.117", reason: "max-cap" }, + resolution: { kind: "pin", version: "0.1.7", latest: "0.1.8", reason: "max-cap" }, replacement: { INSTALLED_OPENSHELL_VERSION: "0.0.99", - NEMOCLAW_OPENSHELL_MIN_VERSION: "0.0.116", - NEMOCLAW_OPENSHELL_MAX_VERSION: "0.0.116", - NEMOCLAW_OPENSHELL_PIN_VERSION: "0.0.116", + NEMOCLAW_OPENSHELL_MIN_VERSION: "0.1.7", + NEMOCLAW_OPENSHELL_MAX_VERSION: "0.1.7", + NEMOCLAW_OPENSHELL_PIN_VERSION: "0.1.7", }, }); } finally { @@ -82,12 +84,6 @@ process.stdout.write(JSON.stringify({ }, ); -const PINNED_OPEN_SHELL_SHA256 = { - cliLinuxX64: "4fb4476d80a1875a0b83547ec3aba999cf0a2e2d75f95f2f709b622e2103520e", - gatewayLinuxX64: "59c6da724eae7a00c28826f9191efbdf4fbaa5c768afdc8dea6a80a949ebcc89", - sandboxLinuxX64: "0bb160f73e5007338b94e3c868f66f50c71cd65c27c932ed9a4fa67c49e6d423", -}; - type GhDownloadMode = "success" | "fail"; function writeExecutable(target: string, contents: string): void { @@ -95,7 +91,7 @@ function writeExecutable(target: string, contents: string): void { } // Bash helpers shared by the gh and curl stubs: write a fake archive and emit -// the same pinned digest lines the real OpenShell v0.0.116 release uses. A fake +// the same pinned digest lines the real OpenShell v0.1.7 release uses. A fake // sha256sum below keeps this test self-contained even though the tarball bytes are // synthetic. const SHARED_DOWNLOAD_BASH_HELPERS = `\ @@ -336,6 +332,13 @@ function runVersionPinTarget(options: { const fakeBin = path.join(tmp, "bin"); const downloadLog = path.join(tmp, "downloads.log"); fs.mkdirSync(fakeBin); + fs.symlinkSync(process.execPath, path.join(fakeBin, "node")); + writeExecutable( + path.join(fakeBin, "mktemp"), + `#!/usr/bin/env bash +if [ "$*" = "-d" ]; then exec /usr/bin/mktemp -d ${JSON.stringify(path.join(tmp, "install.XXXXXX"))}; fi +exec /usr/bin/mktemp "$@"`, + ); fs.writeFileSync(downloadLog, ""); createFakeUname(fakeBin); @@ -343,7 +346,7 @@ function runVersionPinTarget(options: { createFakeHelperBinaries(fakeBin); createFakeGh(fakeBin, downloadLog, options.ghDownloadMode); createFakeCurl(fakeBin, downloadLog); - createFakeTar(fakeBin, "0.0.116"); + createFakeTar(fakeBin, "0.1.7"); createFakeStrings(fakeBin); createFakeSha256sum(fakeBin); const unsafeDirectory = options.unsafeActiveDirectory?.location === "parent" ? tmp : fakeBin; @@ -367,6 +370,9 @@ function runVersionPinTarget(options: { ...process.env, NEMOCLAW_OPENSHELL_CHANNEL: "stable", PATH: `${fakeBin}:/usr/bin:/bin`, + HOME: tmp, + XDG_CONFIG_HOME: path.join(tmp, "config"), + OPENSHELL_SYSTEM_GATEWAY_DIR: path.join(tmp, "system-openshell"), ...(unsafeActiveDirectory?.pretendRootOwned ? { BASH_ENV: rootStickyEnv, NEMOCLAW_TEST_ROOT_STICKY_DIR: unsafeDirectory } : {}), @@ -392,10 +398,10 @@ function runVersionPinTarget(options: { expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0); expect(result.stdout).toMatch(options.expectedDecision); - // Assertion 2: download-log-contains-v0.0.116 — pinned release tag was + // Assertion 2: download-log-contains-v0.1.7 — pinned release tag was // requested from the release host. const downloads = fs.readFileSync(downloadLog, "utf-8"); - expect(downloads).toContain("v0.0.116"); + expect(downloads).toContain("v0.1.7"); // Assertion 3: download-log-excludes-installed-version — the existing // release is never re-fetched in place of the pinned replacement. @@ -404,23 +410,23 @@ function runVersionPinTarget(options: { if (options.ghDownloadMode === "fail") { // Assertion 3b: curl-fallback-observed — the installer must recover from // gh download failure by re-requesting the pinned assets via curl. - expect(downloads).toContain("gh download-fail v0.0.116"); + expect(downloads).toContain("gh download-fail v0.1.7"); expect(downloads).toContain("curl "); } else { - expect(downloads).toContain("gh download v0.0.116"); + expect(downloads).toContain("gh download v0.1.7"); expect(downloads).not.toContain("curl "); } - // Assertion 4: replaced-openshell-reports-0.0.116 — the binary on disk in + // Assertion 4: replaced-openshell-reports-0.1.7 — the binary on disk in // the active install dir (== fakeBin, since ACTIVE_OPENSHELL_BIN resolved - // there and it is writable) was overwritten with the pinned 0.0.116 build. + // there and it is writable) was overwritten with the pinned 0.1.7 build. const replacedVersion = spawnSync(path.join(fakeBin, "openshell"), ["--version"], { encoding: "utf8", killSignal: "SIGKILL", timeout: 30_000, }); expect(replacedVersion.status).toBe(0); - expect(replacedVersion.stdout).toContain("0.0.116"); + expect(replacedVersion.stdout).toContain("0.1.7"); expect(replacedVersion.stdout).not.toContain(options.installedVersion); }; (unsafeActiveDirectory ? assertUnsafe : assertSuccessful)(); @@ -436,9 +442,9 @@ test( }, () => { runVersionPinTarget({ - expectedDecision: /above the maximum.*reinstalling pinned OpenShell 0\.0\.116/u, + expectedDecision: /above the maximum.*reinstalling pinned OpenShell 0\.1\.7/u, ghDownloadMode: "success", - installedVersion: "0.0.117", + installedVersion: "0.1.8", }); }, ); @@ -450,21 +456,21 @@ test( }, () => { runVersionPinTarget({ - expectedDecision: /above the maximum.*reinstalling pinned OpenShell 0\.0\.116/u, + expectedDecision: /above the maximum.*reinstalling pinned OpenShell 0\.1\.7/u, ghDownloadMode: "fail", - installedVersion: "0.0.117", + installedVersion: "0.1.8", }); }, ); test( - "replaces an installed OpenShell 0.0.106 with the pinned 0.0.116 release (#11229)", + "replaces an installed OpenShell 0.0.106 with the pinned 0.1.7 release (#11229)", { timeout: TEST_TIMEOUT_MS, }, () => { runVersionPinTarget({ - expectedDecision: /below minimum 0\.0\.116.*upgrading/u, + expectedDecision: /below minimum 0\.1\.7.*upgrading/u, ghDownloadMode: "success", installedVersion: "0.0.106", }); diff --git a/test/installer-integration/install-preflight.test.ts b/test/installer-integration/install-preflight.test.ts index 260f800e2c0..d8ec6180eb5 100644 --- a/test/installer-integration/install-preflight.test.ts +++ b/test/installer-integration/install-preflight.test.ts @@ -567,6 +567,7 @@ exit 0 const openshellLog = path.join(tmp, "install-openshell.log"); fs.mkdirSync(path.join(tmp, ".git")); + fs.mkdirSync(path.join(tmp, "scripts")); writeNodeStub(fakeBin); writeExecutable( path.join(fakeBin, "openshell"), @@ -579,7 +580,6 @@ exit 0 writeSourceCheckoutPackages(tmp); - fs.mkdirSync(path.join(tmp, "scripts"), { recursive: true }); writeExecutable( path.join(tmp, "scripts", "install-openshell.sh"), `#!/usr/bin/env bash diff --git a/test/installer-integration/openshell-sdk-install-handoff.test.ts b/test/installer-integration/openshell-sdk-install-handoff.test.ts index 08527278fd6..33797f920f4 100644 --- a/test/installer-integration/openshell-sdk-install-handoff.test.ts +++ b/test/installer-integration/openshell-sdk-install-handoff.test.ts @@ -192,7 +192,7 @@ exit 0 path.join(binDir, "openshell"), `#!/usr/bin/env bash if [ "\${1:-}" = "--version" ] || [ "\${1:-}" = "version" ]; then - echo "openshell 0.0.116" + echo "openshell 0.1.7" fi exit 0 `, @@ -222,8 +222,8 @@ exit 0 const output = `${result.stdout}${result.stderr}`; expect(result.status, output).toBe(0); - expect(output).toContain("OpenShell SDK 0.0.116: verified archive prepared"); - expect(output).toContain("OpenShell SDK 0.0.116: import OK"); + expect(output).toContain("OpenShell SDK 0.1.7: verified archive prepared"); + expect(output).toContain("OpenShell SDK 0.1.7: import OK"); expect(fs.readFileSync(path.join(root, "build-saw-sdk"), "utf8")).toBe("yes"); }, ); diff --git a/test/mcp/mcp-bridge-servers.test.ts b/test/mcp/mcp-bridge-servers.test.ts index 0cee082590f..9fd74185213 100644 --- a/test/mcp/mcp-bridge-servers.test.ts +++ b/test/mcp/mcp-bridge-servers.test.ts @@ -91,20 +91,20 @@ afterEach(async () => { }); describe("authenticated MCP live fixtures", () => { - it("builds a bounded public HTTPS quick-tunnel origin without embedding credentials", () => { - expect(buildCloudflaredQuickTunnelArgs(43123)).toEqual([ + it("builds a loopback tunnel origin from the listening HTTPS fixture", async () => { + const server = await startFakeMcpHttpsServer({ secret: "fixture-only", tls: fixtureTls }); + servers.push(server); + expect(buildCloudflaredQuickTunnelArgs(server.port)).toEqual([ "tunnel", "--no-autoupdate", "--protocol", "http2", "--url", - "https://127.0.0.1:43123", + `https://127.0.0.1:${server.port}`, "--no-tls-verify", "--loglevel", "info", ]); - expect(() => buildCloudflaredQuickTunnelArgs(0)).toThrow(/invalid local MCP HTTPS port/); - expect(() => buildCloudflaredQuickTunnelArgs(65_536)).toThrow(/invalid local MCP HTTPS port/); }); it("accepts only an exact public trycloudflare origin from tunnel output", () => { diff --git a/test/mcp/mcp-provider-detach-retry.test.ts b/test/mcp/mcp-provider-detach-retry.test.ts index 980b97db6ce..3c7c825966e 100644 --- a/test/mcp/mcp-provider-detach-retry.test.ts +++ b/test/mcp/mcp-provider-detach-retry.test.ts @@ -26,13 +26,13 @@ providerCommands.runOpenshellProviderCommand = (args, options) => { } if (args[0] === "sandbox" && args[1] === "provider" && args[2] === "list") { return attached - ? { status: 0, stdout: "NAME TYPE CREDENTIAL_KEYS CONFIG_KEYS\nalpha-mcp-fake nemoclaw-mcp-v1 1 0\n", stderr: "" } + ? { status: 0, stdout: "NAME TYPE CREDENTIAL_KEYS CONFIG_KEYS\nalpha-mcp-fake " + providerType + " 1 0\n", stderr: "" } : { status: 0, stdout: "No providers attached to sandbox alpha.\n", stderr: "" }; } if (args[0] === "provider" && args[1] === "get") { return { status: 0, - stdout: "Name: " + args[2] + "\nId: " + liveId + "\nType: nemoclaw-mcp-v1\nResource version: 4\nCredential keys: EXPECTED_TOKEN\nConfig keys: \n", + stdout: "Name: " + args[2] + "\nId: " + liveId + "\nType: " + providerType + "\nResource version: 4\nCredential keys: EXPECTED_TOKEN\nConfig keys: \n", stderr: "", }; } @@ -55,6 +55,7 @@ providerCommands.runOpenshellProviderCommand = (args, options) => { throw new Error("unexpected call: " + args.join(" ")); }; const providerActions = require("./src/lib/actions/sandbox/mcp-bridge-provider.js"); +const providerType = providerActions.mcpBridgeProviderType("EXPECTED_TOKEN"); const entry = { server: "fake", agent: "openclaw", diff --git a/test/mcp/mcp-tool-discovery-image-contract.test.ts b/test/mcp/mcp-tool-discovery-image-contract.test.ts index ba4626e89a9..a0399cbcd83 100644 --- a/test/mcp/mcp-tool-discovery-image-contract.test.ts +++ b/test/mcp/mcp-tool-discovery-image-contract.test.ts @@ -214,7 +214,7 @@ describe("MCP tool discovery image contract", () => { // source-shape-contract: security -- Exact reviewed runtime digests reject substituted executable and license artifacts before managed image construction. it.each([ { - expectedHash: "06731230bced718dd7b3fb15b2801f0b679dfbeca69b0e1b41cd72f38b7fae2d", + expectedHash: "c87f1bbc69fdede89e24fe48279bfa73bacb1dce88b6b2866330bfd33c2b483a", relativePath: "managed-startup-direct-image-runtime.bundle", }, { diff --git a/test/networking/http-proxy-fix-e2e.test.ts b/test/networking/http-proxy-fix-e2e.test.ts deleted file mode 100644 index 9bc26cb5151..00000000000 --- a/test/networking/http-proxy-fix-e2e.test.ts +++ /dev/null @@ -1,271 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 -// -// End-to-end test for the FORWARD-mode rewrite in -// nemoclaw-blueprint/scripts/http-proxy-fix.js. -// -// Spins up a local HTTPS mock server on 127.0.0.1 that pretends to be a -// custom OpenAI-compatible upstream (deepinfra / OpenRouter / vLLM behind -// an OpenShell-routed inference.local). Constructs a FORWARD-mode -// http.request the way axios + HTTPS_PROXY would produce, including the -// forward-proxy http.Agent, proxy basic-auth, and proxy-pointing Host -// header — i.e. exactly the request shape that hit the deepinfra users -// on NemoClaw 0.0.24. Asserts the wrapper's rewrite produces a request -// that actually completes against the upstream. -// -// Self-signed cert is generated in-memory via openssl at test setup. -// Skipped if openssl is not on PATH. - -import { execSync } from "node:child_process"; -import fs from "node:fs"; -import http from "node:http"; -import https from "node:https"; -import os from "node:os"; -import path from "node:path"; -import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -const FIX_PATH = path.resolve( - import.meta.dirname, - "../..", - "nemoclaw-blueprint", - "scripts", - "http-proxy-fix.js", -); -const PROXY_HOST = "10.200.0.1"; -const TLS_VALIDATION_OPTION = ["reject", "Unauthorized"].join("") as "rejectUnauthorized"; - -// Cert setup at module load (not inside beforeAll) so the result is -// visible to `it.skipIf` at definition time. -function trySetupCert(): - | { ok: true; key: Buffer; cert: Buffer; dir: string } - | { ok: false; reason: string } { - try { - execSync("openssl version", { stdio: "pipe" }); - } catch (err) { - return { ok: false, reason: `openssl missing: ${(err as Error).message}` }; - } - try { - const dir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-proxy-fix-e2e-")); - // 7-day expiry. Cert is generated at module load (once per test run); - // 1-day was tight enough that long `vitest --watch` sessions could - // outrun it. Still ephemeral — `afterAll` rms the dir. - execSync( - `openssl req -x509 -newkey rsa:2048 -keyout "${dir}/key.pem" -out "${dir}/cert.pem" ` + - `-days 7 -nodes -subj "/CN=localhost" ` + - `-addext "subjectAltName=DNS:localhost,IP:127.0.0.1"`, - { stdio: "pipe" }, - ); - return { - ok: true, - dir, - key: fs.readFileSync(path.join(dir, "key.pem")), - cert: fs.readFileSync(path.join(dir, "cert.pem")), - }; - } catch (err) { - return { ok: false, reason: `cert generation failed: ${(err as Error).message}` }; - } -} - -const certSetup = trySetupCert(); -const opensslAvailable = certSetup.ok; -if (!certSetup.ok) { - if (process.env.CI === "true") { - // CI runners (ubuntu-latest, macos-latest) ship openssl. A skip here - // would be a silent green; fail loud instead so missing infra is - // visible. - throw new Error( - `[http-proxy-fix-e2e] CI=true but openssl unavailable: ${certSetup.reason}. ` + - `This test must not silently skip in CI — install openssl on the runner.`, - ); - } - console.warn(`[http-proxy-fix-e2e] skipping locally: ${certSetup.reason}`); -} -const key = certSetup.ok ? certSetup.key : Buffer.alloc(0); -const cert = certSetup.ok ? certSetup.cert : Buffer.alloc(0); -const certDir = certSetup.ok ? certSetup.dir : ""; - -afterAll(() => { - if (certDir) { - fs.rmSync(certDir, { recursive: true, force: true }); - } -}); - -type CapturedRequest = { - method?: string; - url?: string; - headers: http.IncomingHttpHeaders; - body: string; -}; - -function loadWrapper() { - delete require.cache[FIX_PATH]; - require(FIX_PATH); -} - -function startMock(): Promise<{ - port: number; - close: () => Promise; - received: CapturedRequest[]; -}> { - return new Promise((resolve, reject) => { - const received: CapturedRequest[] = []; - const server = https.createServer({ key, cert }, (req, res) => { - const chunks: Buffer[] = []; - req.on("data", (c) => chunks.push(c)); - req.on("end", () => { - received.push({ - method: req.method, - url: req.url, - headers: req.headers, - body: Buffer.concat(chunks).toString("utf-8"), - }); - res.writeHead(200, { "Content-Type": "application/json" }); - res.end( - JSON.stringify({ - id: "chatcmpl-test", - choices: [{ index: 0, message: { role: "assistant", content: "PONG" } }], - }), - ); - }); - }); - server.on("error", reject); - server.listen(0, "127.0.0.1", () => { - const addr = server.address(); - if (!addr || typeof addr === "string") { - reject(new Error("mock server address unavailable")); - return; - } - resolve({ - port: addr.port, - received, - close: () => - new Promise((r) => { - server.close(() => r()); - }), - }); - }); - }); -} - -type ResponseSnapshot = { - status?: number; - body: string; -}; - -function sendForwardModeRequest(opts: { - port: number; - body: string; - authorizationHeader: string; -}): Promise { - return new Promise((resolve, reject) => { - // Mimic exactly what axios + HTTPS_PROXY produces: an http.request to - // the proxy with a full https:// URL in `path`, an http.Agent set up - // for the proxy hop, basic-auth meant for the proxy hop, and a Host - // header pointing at the proxy. - const proxyAgent = new http.Agent({ keepAlive: false }); - const req = http.request( - { - hostname: PROXY_HOST, - port: 3128, - path: `https://localhost:${opts.port}/v1/openai/chat/completions`, - method: "POST", - agent: proxyAgent, - auth: "proxyuser:proxypass", - headers: { - Host: `${PROXY_HOST}:3128`, - "Proxy-Authorization": "Basic dXNlcjpwYXNz", - "Proxy-Connection": "keep-alive", - Authorization: opts.authorizationHeader, - "Content-Type": "application/json", - "Content-Length": String(Buffer.byteLength(opts.body)), - }, - // Self-signed cert - mock is local. Only honored if the wrapper - // forwards this option through to https.request. - [TLS_VALIDATION_OPTION]: false, - } as http.RequestOptions & { rejectUnauthorized?: boolean }, - (res) => { - const chunks: Buffer[] = []; - res.on("data", (c) => chunks.push(c)); - res.on("end", () => - resolve({ status: res.statusCode, body: Buffer.concat(chunks).toString("utf-8") }), - ); - }, - ); - req.on("error", reject); - req.setTimeout(5000, () => req.destroy(new Error("request timeout"))); - req.write(opts.body); - req.end(); - }); -} - -describe("http-proxy-fix end-to-end against a local OpenAI-compatible mock", () => { - let mock: Awaited>; - // The wrapper monkey-patches `http.request` at module load. Save the - // original here so we can restore it in afterEach — leaving the patched - // function in place would chain wrappers across this file's tests and - // (because vitest `cli` runs many test files in one worker) into other - // suites in the same process. - let origHttpRequest: typeof http.request; - - beforeEach(async () => { - if (!opensslAvailable) return; - mock = await startMock(); - // Use vi.stubEnv consistently with the rewrite suite. Raw process.env - // mutation here would skip the unstub-on-fail behavior `vi` provides - // and could leak `NODE_USE_ENV_PROXY=1` into adjacent test files if - // an assertion threw before afterEach. - vi.stubEnv("NODE_USE_ENV_PROXY", "1"); - vi.stubEnv("HTTPS_PROXY", `http://${PROXY_HOST}:3128`); - vi.stubEnv("https_proxy", ""); - vi.stubEnv("HTTP_PROXY", ""); - vi.stubEnv("http_proxy", ""); - origHttpRequest = http.request; - loadWrapper(); - }); - - afterEach(async () => { - if (!opensslAvailable) return; - http.request = origHttpRequest; - await mock.close(); - vi.unstubAllEnvs(); - }); - - it.skipIf(!opensslAvailable)( - "completes a chat-completions POST against a custom upstream through the FORWARD→CONNECT-equivalent rewrite", - async () => { - const requestBody = JSON.stringify({ - model: "deepseek-ai/DeepSeek-V4-Flash", - messages: [{ role: "user", content: "What is 6 multiplied by 7?" }], - }); - - const response = await sendForwardModeRequest({ - port: mock.port, - body: requestBody, - authorizationHeader: "Bearer real-deepinfra-token", - }); - - // Positive: the request actually reached the upstream and we got - // the mock's reply. Without the fix, the forward-proxy http.Agent - // rides along into https.request and the TLS handshake fails — no - // response, request errors out. - expect(response.status).toBe(200); - const parsed = JSON.parse(response.body); - expect(parsed?.choices?.[0]?.message?.content).toBe("PONG"); - - // Server-side proof of the strips: - expect(mock.received).toHaveLength(1); - const captured = mock.received[0]!; - expect(captured.method).toBe("POST"); - expect(captured.url).toBe("/v1/openai/chat/completions"); - // Caller-intent auth survived the rewrite. - expect(captured.headers.authorization).toBe("Bearer real-deepinfra-token"); - // Host points at the actual target now, not the proxy. - expect(captured.headers.host).toBe(`localhost:${mock.port}`); - // Proxy-hop headers did not leak through to the upstream. - expect(captured.headers["proxy-authorization"]).toBeUndefined(); - expect(captured.headers["proxy-connection"]).toBeUndefined(); - // Body survived the rewrite intact. - expect(captured.body).toBe(requestBody); - }, - ); -}); diff --git a/test/networking/http-proxy-fix-rewrite.test.ts b/test/networking/http-proxy-fix-rewrite.test.ts deleted file mode 100644 index baa04c058b0..00000000000 --- a/test/networking/http-proxy-fix-rewrite.test.ts +++ /dev/null @@ -1,334 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 -// -// Behavioural tests for the FORWARD-mode → CONNECT-tunnel rewrite in -// nemoclaw-blueprint/scripts/http-proxy-fix.js. -// -// The wrapper is a NODE_OPTIONS=--require preload installed at sandbox boot. -// In-process we exercise it by clearing the require cache, setting the env -// the wrapper inspects, requiring the file (its IIFE patches http.request), -// then calling http.request and asserting what the rewritten https.request -// receives. https.request is stubbed via vi.spyOn — http.request inside the -// wrapper grabs https with a fresh require('https') so the spy takes effect. -// -// These tests pin the regression deepinfra users hit on 0.0.24: the wrapper -// shallow-copied options, dragging the forward-proxy http.Agent and proxy -// basic-auth into the rewritten https.request and surfacing as -// "LLM request failed: network connection error" against non-NVIDIA -// upstreams. See the canonical wrapper for the per-field rationale. - -import http from "node:http"; -import https from "node:https"; -import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -const FIX_PATH = path.resolve( - import.meta.dirname, - "../..", - "nemoclaw-blueprint", - "scripts", - "http-proxy-fix.js", -); - -const PROXY_URL = "http://10.200.0.1:3128"; -const PROXY_HOST = "10.200.0.1"; -const TLS_VALIDATION_OPTION = ["reject", "Unauthorized"].join("") as "rejectUnauthorized"; - -type RewrittenOptions = http.RequestOptions & { - protocol?: string; - servername?: string; - checkServerIdentity?: unknown; - socketPath?: string; - localAddress?: string; - lookup?: unknown; - family?: number; - hints?: number; -}; - -function loadWrapper() { - // Clear cached copies so the IIFE re-runs and reads our test env. - delete require.cache[FIX_PATH]; - require(FIX_PATH); -} - -describe("http-proxy-fix rewrite for a deepinfra-style failure (#2344)", () => { - let origHttpRequest: typeof http.request; - let httpsSpy: ReturnType; - let captured: RewrittenOptions | null; - - beforeEach(() => { - origHttpRequest = http.request; - captured = null; - vi.stubEnv("NODE_USE_ENV_PROXY", "1"); - vi.stubEnv("HTTPS_PROXY", PROXY_URL); - vi.stubEnv("https_proxy", ""); - vi.stubEnv("HTTP_PROXY", ""); - vi.stubEnv("http_proxy", ""); - loadWrapper(); - // Wrapper grabs `https` via a fresh require inside the rewrite branch, - // so spying on https.request after the wrapper installs is fine. - httpsSpy = vi - .spyOn(https, "request") - // @ts-expect-error stubbed return — the wrapper just hands it back. - .mockImplementation((options: RewrittenOptions) => { - captured = options; - return { on: () => undefined, end: () => undefined } as unknown as http.ClientRequest; - }); - }); - - afterEach(() => { - httpsSpy.mockRestore(); - http.request = origHttpRequest; - vi.unstubAllEnvs(); - }); - - it("rewrites FORWARD-mode http.request to https.request against the target", () => { - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/openai/chat/completions", - method: "POST", - headers: { "Content-Type": "application/json" }, - }); - - expect(captured).not.toBeNull(); - expect(captured?.hostname).toBe("api.deepinfra.com"); - expect(captured?.host).toBe("api.deepinfra.com"); - expect(captured?.port).toBe(443); - expect(captured?.path).toBe("/v1/openai/chat/completions"); - expect(captured?.protocol).toBe("https:"); - expect(captured?.method).toBe("POST"); - }); - - it("strips a forward-proxy http.Agent that cannot speak TLS (root cause of deepinfra 'Connection error')", () => { - const proxyAgent = new http.Agent({ keepAlive: true }); - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/foo", - agent: proxyAgent, - headers: {}, - }); - - expect(captured).not.toBeNull(); - expect("agent" in (captured ?? {})).toBe(false); - }); - - it("strips proxy-hop basic auth so it is not Basic-auth'd to the target", () => { - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/foo", - auth: "proxyuser:proxypass", - headers: {}, - }); - - expect(captured).not.toBeNull(); - expect("auth" in (captured ?? {})).toBe(false); - }); - - it.each([ - "Host", - "host", - "Proxy-Authorization", - "Proxy-Connection", - "Proxy-Authenticate", - "Connection", - "Keep-Alive", - "TE", - "Trailer", - "Transfer-Encoding", - "Upgrade", - ])( - "strips Host / Proxy-* / RFC-7230-§6.1 hop-by-hop headers; preserves target-intent headers [%s]", - (k) => { - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/foo", - headers: { - Host: `${PROXY_HOST}:3128`, - "Proxy-Authorization": "Basic dXNlcjpwYXNz", - "Proxy-Connection": "keep-alive", - "Proxy-Authenticate": "Basic realm=p", - Connection: "close", - "Keep-Alive": "timeout=5", - TE: "trailers", - Trailer: "Expires", - "Transfer-Encoding": "chunked", - Upgrade: "h2c", - Authorization: "Bearer real-target-token", - "Content-Type": "application/json", - }, - }); - - expect(captured).not.toBeNull(); - const headers = (captured?.headers ?? {}) as Record; - // RFC 7230 §6.1 hop-by-hop set + proxy-pointing Host all stripped. - - expect(headers[k]).toBeUndefined(); - - // Target intent (caller's Authorization to the upstream and content - // negotiation) must survive. - expect(headers.Authorization).toBe("Bearer real-target-token"); - expect(headers["Content-Type"]).toBe("application/json"); - }, - ); - - it("strips tokens named in the Connection header (RFC 7230 §6.1 transitive hop-by-hop)", () => { - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/foo", - headers: { - Connection: "close, X-Hop-Token, X-Other-Hop", - "X-Hop-Token": "leaks-without-strip", - "X-Other-Hop": "also-leaks", - "X-Keep-Me": "survives", - }, - }); - - expect(captured).not.toBeNull(); - const headers = (captured?.headers ?? {}) as Record; - expect(headers.Connection).toBeUndefined(); - expect(headers["X-Hop-Token"]).toBeUndefined(); - expect(headers["X-Other-Hop"]).toBeUndefined(); - expect(headers["X-Keep-Me"]).toBe("survives"); - }); - - it("preserves signal, timeout, and TLS material the caller supplied", () => { - const ac = new AbortController(); - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/foo", - signal: ac.signal, - timeout: 12345, - [TLS_VALIDATION_OPTION]: false, - headers: {}, - } as http.RequestOptions); - - expect(captured).not.toBeNull(); - expect(captured?.signal).toBe(ac.signal); - expect(captured?.timeout).toBe(12345); - expect((captured as { rejectUnauthorized?: boolean })?.[TLS_VALIDATION_OPTION]).toBe(false); - }); - - it("strips proxy-hop TLS identity fields (servername, checkServerIdentity)", () => { - const customCheck = (): undefined => undefined; - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/foo", - servername: PROXY_HOST, - checkServerIdentity: customCheck, - headers: {}, - } as http.RequestOptions & { servername?: string; checkServerIdentity?: unknown }); - - expect(captured).not.toBeNull(); - expect("servername" in (captured ?? {})).toBe(false); - expect("checkServerIdentity" in (captured ?? {})).toBe(false); - }); - - it.each(["socketPath", "localAddress", "lookup", "family", "hints"])( - "strips proxy-hop transport hints (socketPath, localAddress, lookup, family, hints) [%s]", - (k) => { - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://api.deepinfra.com/v1/foo", - socketPath: "/var/run/cntlm.sock", - localAddress: "10.0.0.42", - lookup: () => undefined, - family: 4, - hints: 0, - headers: {}, - } as http.RequestOptions & { - socketPath?: string; - localAddress?: string; - lookup?: unknown; - family?: number; - hints?: number; - }); - - expect(captured).not.toBeNull(); - - expect(k in (captured ?? {})).toBe(false); - }, - ); - - it("uses the explicit target port when one is present in the URL", () => { - http.request({ - hostname: PROXY_HOST, - port: 3128, - path: "https://internal.example.com:8443/v1/x", - headers: {}, - }); - - expect(captured).not.toBeNull(); - expect(captured?.port).toBe("8443"); - expect(captured?.hostname).toBe("internal.example.com"); - }); - - it("passes plain non-FORWARD requests through untouched", () => { - // Abort immediately so the test does not attempt a real socket - // connection to a port nothing is listening on. - const ac = new AbortController(); - ac.abort(); - const req = http.request({ - hostname: "127.0.0.1", - port: 4242, - path: "/health", - headers: {}, - signal: ac.signal, - } as http.RequestOptions); - req.on("error", () => undefined); - req.destroy(); - - expect(httpsSpy).not.toHaveBeenCalled(); - }); -}); - -describe("http-proxy-fix bisect: control case for the bug class", () => { - // Pins the regression independent of the wrapper itself. Constructs the - // exact rewrite shape the *broken* pre-fix wrapper produced (no agent / - // auth strip, hop-by-hop headers preserved) and asserts that - // https.request rejects it. If a future maintainer reverts the strip, - // this test still fails — the bug class doesn't depend on the wrapper. - // Combined with the rewrite tests above (which prove the wrapper does - // strip), this gives a two-sided proof of correctness without storing - // a copy of the broken wrapper in the repo. - it("https.request throws TypeError when a forward-proxy http.Agent rides into rewritten options (Node 22 surface)", () => { - const proxyAgent = new http.Agent({ keepAlive: false }); - const callerOptions = { - hostname: PROXY_HOST, - port: 3128, - path: "https://example.invalid/v1/x", - method: "POST", - agent: proxyAgent, - auth: "proxyuser:proxypass", - headers: { - Host: `${PROXY_HOST}:3128`, - "Proxy-Authorization": "Basic x", - }, - }; - - const target = new URL(callerOptions.path); - // Reproduce the broken pre-fix wrapper's rewrite verbatim: shallow - // Object.assign with no field strips and no header sanitization. - const broken: http.RequestOptions = Object.assign({}, callerOptions, { - method: callerOptions.method || "GET", - hostname: target.hostname, - host: target.hostname, - port: Number.parseInt(target.port, 10) || 443, - path: target.pathname + target.search, - protocol: "https:", - } as http.RequestOptions); - - // Node 22's _http_agent.js validates `agent.protocol` and throws - // synchronously. Older Node falls through and fails the TLS handshake - // instead — same root cause, different surface error. The wrapper's - // job is to make sure neither path is reachable. - expect(() => https.request(broken)).toThrow(/Protocol "https:" not supported/); - }); -}); diff --git a/test/networking/http-proxy-fix-sync.test.ts b/test/networking/http-proxy-fix-sync.test.ts deleted file mode 100644 index d76dedc933c..00000000000 --- a/test/networking/http-proxy-fix-sync.test.ts +++ /dev/null @@ -1,82 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -import { spawnSync } from "node:child_process"; -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; -import { describe, expect, it } from "vitest"; - -const ROOT = path.join(import.meta.dirname, "../.."); -const CANONICAL_FIX = path.join(ROOT, "nemoclaw-blueprint", "scripts", "http-proxy-fix.js"); -const START_SCRIPT = path.join(ROOT, "scripts", "nemoclaw-start.sh"); - -function extractShellFunction(source: string, name: string): string { - const header = `${name}() {`; - const start = source.indexOf(header); - expect(start, `expected ${name} in nemoclaw-start.sh`).not.toBe(-1); - const body = source.slice(start + header.length); - const closing = body.match(/^}$/m); - expect(closing, `expected closing brace for ${name}`).not.toBeNull(); - return `${name}() {${body.slice(0, closing?.index ?? 0)}\n}`; -} - -describe("http-proxy-fix preload sync (#2109)", () => { - it("entrypoint emits the proxy fix preload and registers it in NODE_OPTIONS", () => { - const startScript = fs.readFileSync(START_SCRIPT, "utf-8"); - const start = startScript.indexOf('_PROXY_FIX_SCRIPT="/tmp/nemoclaw-http-proxy-fix.js"'); - const end = startScript.indexOf( - "# NVIDIA endpoint model-specific inference parameter injection", - start, - ); - if (start === -1 || end === -1 || end <= start) { - throw new Error("Expected HTTP proxy fix entrypoint block in scripts/nemoclaw-start.sh"); - } - - const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-http-proxy-fix-")); - const fixPath = path.join(tempDir, "http-proxy-fix.js"); - const block = startScript - .slice(start, end) - .replace( - '_PROXY_FIX_SCRIPT="/tmp/nemoclaw-http-proxy-fix.js"', - `_PROXY_FIX_SCRIPT=${JSON.stringify(fixPath)}`, - ) - .replace( - '_PROXY_FIX_SOURCE="/usr/local/lib/nemoclaw/preloads/http-proxy-fix.js"', - `_PROXY_FIX_SOURCE=${JSON.stringify(CANONICAL_FIX)}`, - ); - const wrapper = [ - "#!/usr/bin/env bash", - "set -euo pipefail", - 'emit_sandbox_sourced_file() { local target="$1"; cat > "$target"; chmod 444 "$target"; }', - "NODE_USE_ENV_PROXY=1", - "NODE_OPTIONS='--require /already-loaded.js'", - extractShellFunction(startScript, "node_options_has_require"), - extractShellFunction(startScript, "append_node_require_once"), - block, - `_SANDBOX_SAFETY_NET=${JSON.stringify(path.join(tempDir, "safety-net.js"))}`, - `_SANDBOX_SAFETY_NET_SOURCE=${JSON.stringify(CANONICAL_FIX)}`, - `_NEMOTRON_FIX_SCRIPT=${JSON.stringify(path.join(tempDir, "nemotron-fix.js"))}`, - `_NEMOTRON_FIX_SOURCE=${JSON.stringify(CANONICAL_FIX)}`, - extractShellFunction(startScript, "install_core_runtime_preloads"), - "install_core_runtime_preloads", - "printf 'NODE_OPTIONS=%s\\n' \"$NODE_OPTIONS\"", - "printf 'SCRIPT=%s\\n' \"$_PROXY_FIX_SCRIPT\"", - ].join("\n"); - const wrapperPath = path.join(tempDir, "run.sh"); - - try { - fs.writeFileSync(wrapperPath, wrapper, { mode: 0o700 }); - const result = spawnSync("bash", [wrapperPath], { encoding: "utf-8", timeout: 5000 }); - expect(result.status).toBe(0); - expect(result.stdout).toContain(`SCRIPT=${fixPath}`); - expect(result.stdout).toContain("--require /already-loaded.js"); - expect(result.stdout).toContain(`--require ${fixPath}`); - const generated = fs.readFileSync(fixPath, "utf-8"); - expect(generated).not.toContain("axios-proxy-fix.js"); - expect((fs.statSync(fixPath).mode & 0o777).toString(8)).toBe("444"); - } finally { - fs.rmSync(tempDir, { recursive: true, force: true }); - } - }); -}); diff --git a/test/onboarding/blueprint-runtime-identity-lifecycle.test.ts b/test/onboarding/blueprint-runtime-identity-lifecycle.test.ts index 24fed76a2d1..05f378e13e2 100644 --- a/test/onboarding/blueprint-runtime-identity-lifecycle.test.ts +++ b/test/onboarding/blueprint-runtime-identity-lifecycle.test.ts @@ -85,11 +85,10 @@ function save(exitCode, stdout = "", stderr = "") { process.exit(exitCode); } -if (args.join(" ") === "settings get --global --json") { +if (args.join(" ") === "policy list --global --page-size 1 --output json") { save(0, JSON.stringify({ - scope: "global", - settings_revision: 1, - settings: { providers_v2_enabled: "true" }, + revisions: [], + next_page_token: "", })); } @@ -296,3 +295,18 @@ describe("blueprint runtime identity lifecycle integration", () => { expect(state.profiles).toEqual(["okta-runtime-v1"]); }); }); + +it("loads the host native inference observer through the TSX CommonJS boundary", async () => { + const root = path.resolve(import.meta.dirname, "../.."); + const result = await execa( + process.execPath, + [ + path.join(root, "node_modules/tsx/dist/cli.mjs"), + "--input-type=module", + "--eval", + `const { default: hostRunner } = await import(${JSON.stringify(path.join(root, "src/lib/blueprint-runner.ts"))}); const { observeBlueprintNativeInference } = hostRunner; if (typeof observeBlueprintNativeInference !== "function") throw new Error("Host native observer missing"); process.stdout.write("observer-loaded");`, + ], + { cwd: root, timeout: 30_000 }, + ); + expect(result.stdout).toBe("observer-loaded"); +}); diff --git a/test/onboarding/config-set.test.ts b/test/onboarding/config-set.test.ts index 559430a53a4..ccc68332383 100644 --- a/test/onboarding/config-set.test.ts +++ b/test/onboarding/config-set.test.ts @@ -88,10 +88,10 @@ describe("selectDirectSandboxContainer", () => { describe("config set helpers", () => { describe("buildConfigSetRestartGuidance", () => { - it("names Hermes in the Hermes restart note (#8614)", () => { + it("reports the verified Hermes reload and optional additional restart (#8614)", () => { const output = buildConfigSetRestartGuidance("alpha").join("\n"); - expect(output).toContain("Hermes may restart"); + expect(output).toContain("Hermes verified its gateway reload"); expect(output).toContain("--restart"); expect(output).toContain("nemoclaw 'alpha' gateway restart"); }); @@ -134,19 +134,28 @@ describe("config set helpers", () => { expect(calls).toEqual(["alpha"]); }); - it("fails with a written-but-not-applied message and a retry hint when the restart fails", async () => { - let thrown: unknown; - try { - await restartSandboxAgentAfterConfigSet("alpha", "openclaw", async () => ({ ok: false })); - } catch (error) { - thrown = error; - } - - const message = thrown instanceof Error ? thrown.message : String(thrown); - expect(message).toContain("written to disk but NOT applied to the running agent"); - expect(message).toContain("openclaw gateway restart did not complete for 'alpha'"); - expect(message).toContain("nemoclaw 'alpha' gateway restart"); - }); + it.each([ + ["openclaw", "written to disk but NOT applied to the running agent"], + [ + "hermes", + "configuration was applied and its reload verified. The additional restart was not confirmed.", + ], + ])( + "reports the %s application state and a retry hint when the restart fails", + async (agentName, applicationState) => { + let thrown: unknown; + try { + await restartSandboxAgentAfterConfigSet("alpha", agentName, async () => ({ ok: false })); + } catch (error) { + thrown = error; + } + + const message = thrown instanceof Error ? thrown.message : String(thrown); + expect(message).toContain(applicationState); + expect(message).toContain(`${agentName} gateway restart did not complete for 'alpha'`); + expect(message).toContain("nemoclaw 'alpha' gateway restart"); + }, + ); }); describe("extractDotpath", () => { diff --git a/test/onboarding/onboard-build-recreate-credential-reuse.test.ts b/test/onboarding/onboard-build-recreate-credential-reuse.test.ts index 251ed4de5a7..34ff61d0fd6 100644 --- a/test/onboarding/onboard-build-recreate-credential-reuse.test.ts +++ b/test/onboarding/onboard-build-recreate-credential-reuse.test.ts @@ -44,8 +44,8 @@ describe("onboard build recreate credential reuse (#5441)", () => { fs.writeFileSync( path.join(fakeBin, "openshell"), `#!/usr/bin/env bash -if [ "$1" = "settings" ] && [ "$2" = "get" ]; then - printf '%s\\n' '{"scope":"global","settings":{"providers_v2_enabled":"true"}}' +if [ "$1" = "policy" ] && [ "$2" = "list" ]; then + printf '%s\\n' '{"revisions":[],"next_page_token":""}' exit 0 fi if [ "$1" = "inference" ] && [ "$2" = "get" ]; then @@ -61,7 +61,7 @@ EOF fi if [ "$1" = "provider" ] && [ "$2" = "profile" ] && [ "$3" = "-g" ] && [ "$4" = "nemoclaw" ] && [ "$5" = "export" ]; then cat <<'EOF' -{"id":"nemoclaw-nvidia-inference-v1","credentials":[{"name":"api_key","env_vars":["NVIDIA_INFERENCE_API_KEY"],"required":true,"auth_style":"bearer","header_name":"authorization","query_param":""}],"endpoints":[{"host":"integrate.api.nvidia.com","port":443,"protocol":"rest","enforcement":"enforce","rules":[{"allow":{"method":"GET","path":"/v1/models"}},{"allow":{"method":"POST","path":"/v1/chat/completions"}}]}],"binaries":["/usr/local/bin/node","/usr/bin/node","/opt/hermes/.venv/bin/python","/opt/hermes/.venv/bin/python3","/opt/venv/bin/python3","/usr/local/bin/curl","/usr/bin/curl"],"inference_capable":true} +{"id":"nemoclaw-nvidia-inference-v1","credentials":[{"name":"api_key","env_vars":["NVIDIA_INFERENCE_API_KEY"],"required":true,"auth_style":"bearer","header_name":"authorization","query_param":""}],"endpoints":[{"host":"integrate.api.nvidia.com","port":443,"protocol":"rest","enforcement":"enforce","rules":[{"allow":{"method":"GET","path":"/v1/models"}},{"allow":{"method":"POST","path":"/v1/chat/completions"}}]}],"binaries":["/usr/local/bin/node","/usr/bin/node","/opt/hermes/.venv/bin/python","/opt/hermes/.venv/bin/python3","/usr/bin/python3.13","/opt/venv/bin/python3","/usr/local/bin/curl","/usr/bin/curl"],"inference_capable":true} EOF exit 0 fi diff --git a/test/onboarding/onboard-failed-backup-route.test.ts b/test/onboarding/onboard-failed-backup-route.test.ts index 17e66c79abd..cd09ebd6fd8 100644 --- a/test/onboarding/onboard-failed-backup-route.test.ts +++ b/test/onboarding/onboard-failed-backup-route.test.ts @@ -63,7 +63,7 @@ runner.run = (command, options = {}) => { if (cmd.includes("provider update") && cmd.includes("nvidia-prod")) { providerConfig.credential = options.env?.NVIDIA_INFERENCE_API_KEY ?? ""; } - const profile = mocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profile = mocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profile !== null) return profile; if (cmd.includes("sandbox delete")) throw new Error("unexpected sandbox delete"); return live.run(command) ?? { status: 0 }; @@ -229,7 +229,7 @@ runner.run = (command) => { const cmd = commandText(command); events.push({ kind: "run", cmd }); if (cmd.includes("sandbox delete")) throw new Error("unexpected sandbox delete"); - const profile = mocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profile = mocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profile !== null) return profile; return live.run(command) ?? { status: 0 }; }; diff --git a/test/onboarding/onboard-fresh-create-identity.test.ts b/test/onboarding/onboard-fresh-create-identity.test.ts index 3a86f8d2699..c3a6b6e4f1c 100644 --- a/test/onboarding/onboard-fresh-create-identity.test.ts +++ b/test/onboarding/onboard-fresh-create-identity.test.ts @@ -388,7 +388,7 @@ runner.run = (command, opts = {}) => { const cmd = _n(command); _deleted = _deleted || cmd.includes("sandbox delete"); commands.push({ command: cmd, env: opts.env || null }); - const profileResult = require(${onboardScriptMocksPath}).mockEndpointlessProviderProfileRun(command, "nemoclaw-mcp-v1", false); + const profileResult = require(${onboardScriptMocksPath}).mockEndpointlessProviderProfileRun(command, "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; const providerResult = require(${onboardScriptMocksPath}).mockNvidiaProviderGetRun(command, ${JSON.stringify(gatewayName)}); if (providerResult !== null) return providerResult; diff --git a/test/onboarding/onboard-fsm-live-slices.test.ts b/test/onboarding/onboard-fsm-live-slices.test.ts index 0aed8a6c665..5049f3ef2be 100644 --- a/test/onboarding/onboard-fsm-live-slices.test.ts +++ b/test/onboarding/onboard-fsm-live-slices.test.ts @@ -146,7 +146,7 @@ function writeSuccessfulOpenShell(tmpDir: string): string { `#!${process.execPath} const args = process.argv.slice(2); if (args[0] === "policy" && args[1] === "list" && args.includes("--global")) process.stderr.write("No global policy history found\\n"); -if (args[0] === "-V" || args[0] === "--version") process.stdout.write("openshell 0.0.116\\n"); +if (args[0] === "-V" || args[0] === "--version") process.stdout.write("openshell 0.1.7\\n"); if (args[0] === "status") { process.stderr.write("No active gateway\\n"); process.exit(1); } if (args[0] === "gateway" && args[1] === "info") { process.stderr.write("No gateway metadata found\\n"); process.exit(1); } if (args[0] === "gateway" && args[1] === "list") process.stdout.write("[]\\n"); @@ -158,10 +158,12 @@ process.exit(0); } function probeEnvironment(tmpDir: string): NodeJS.ProcessEnv { + // Slice probes must not inspect the host's installed Homebrew formula. + fs.writeFileSync(path.join(tmpDir, "brew"), "#!/bin/sh\nexit 1\n", { mode: 0o755 }); const env: NodeJS.ProcessEnv = { HOME: tmpDir, TMPDIR: tmpDir, - PATH: process.env.PATH || "/usr/bin:/bin", + PATH: `${tmpDir}${path.delimiter}${process.env.PATH || "/usr/bin:/bin"}`, NEMOCLAW_OPENSHELL_BIN: writeSuccessfulOpenShell(tmpDir), NODE_ENV: "test", NEMOCLAW_NON_INTERACTIVE: "1", diff --git a/test/onboarding/onboard-gateway-port-conflict-fast-fail.test.ts b/test/onboarding/onboard-gateway-port-conflict-fast-fail.test.ts index 2a184755c7d..a24703fa842 100644 --- a/test/onboarding/onboard-gateway-port-conflict-fast-fail.test.ts +++ b/test/onboarding/onboard-gateway-port-conflict-fast-fail.test.ts @@ -45,7 +45,7 @@ describe("onboard gateway port conflict readiness (#6752)", () => { "#!/usr/bin/env bash", "# openshell capabilities: request-body-credential-rewrite websocket-credential-rewrite allow_all_known_mcp_methods", 'case "$*" in', - ' --version|-V) printf "%s 0.0.116\\n" "${0##*/}"; exit 0;;', + ' --version|-V) printf "%s 0.1.7\\n" "${0##*/}"; exit 0;;', ' status) printf "No active gateway\\n"; exit 1;;', ' "gateway list -o json") printf "[]"; exit 0;;', "esac", @@ -130,7 +130,7 @@ describe("onboard gateway port conflict readiness (#6752)", () => { /The gateway port is held by an incompatible or ambiguous owner|OpenShell gateway needs this port/, ); expect(combined).not.toMatch(/occupied by unknown/); - expect(combined).toMatch(/\(PID \d+\)/); + expect(combined).toMatch(/PID \d+/); expect(combined).toContain(`sudo lsof -i :${String(gatewayPort)} -sTCP:LISTEN -P -n`); expect(combined).toContain("signal only the matching PID from that fresh result"); expect(combined).not.toMatch(/sudo kill \d+/); @@ -232,7 +232,7 @@ Module._load = function(request, parent, isMain) { "# openshell capabilities: request-body-credential-rewrite websocket-credential-rewrite allow_all_known_mcp_methods", `printf '%s\\n' "$*" >>${JSON.stringify(commandLog)}`, 'case "$*" in', - ' --version|-V) printf "%s 0.0.116\\n" "${0##*/}"; exit 0;;', + ' --version|-V) printf "%s 0.1.7\\n" "${0##*/}"; exit 0;;', ` status|"status -g ${gatewayName}") printf ${JSON.stringify(gatewayStatus)}; exit 0;;`, ` "gateway list -o json") printf ${JSON.stringify(gatewayList)}; exit 0;;`, ` "gateway select ${gatewayName}") exit 0;;`, @@ -317,7 +317,7 @@ Module._load = function(request, parent, isMain) { "#!/usr/bin/env bash", "# openshell capabilities: request-body-credential-rewrite websocket-credential-rewrite allow_all_known_mcp_methods", 'case "$*" in', - ' --version|-V) printf "%s 0.0.116\\n" "${0##*/}"; exit 0;;', + ' --version|-V) printf "%s 0.1.7\\n" "${0##*/}"; exit 0;;', ` status|"status -g ${gatewayName}") printf ${JSON.stringify(gatewayStatus)}; exit 0;;`, ` "gateway list -o json") printf ${JSON.stringify(gatewayList)}; exit 0;;`, "esac", @@ -347,7 +347,7 @@ Module._load = function(request, parent, isMain) { ' case "$3" in', ' "{{.State.Running}}") printf "true\\n";;', ` "{{json .NetworkSettings.Ports}}") printf '%s\\n' ${JSON.stringify(portBindings)};;`, - ' "{{.Config.Image}}") printf "nvcr.io/nvidia/openshell/cluster:0.0.116\\n";;', + ' "{{.Config.Image}}") printf "nvcr.io/nvidia/openshell/cluster:0.1.7\\n";;', " *) exit 1;;", " esac", " exit 0", diff --git a/test/onboarding/onboard-gateway-runtime.test.ts b/test/onboarding/onboard-gateway-runtime.test.ts index bbbd680287f..71c0b01ff56 100644 --- a/test/onboarding/onboard-gateway-runtime.test.ts +++ b/test/onboarding/onboard-gateway-runtime.test.ts @@ -74,7 +74,7 @@ describe("onboard gateway runtime helpers", () => { expect(isLinuxDockerDriverGatewayEnabled("darwin", "arm64")).toBe(true); expect(isLinuxDockerDriverGatewayEnabled("darwin", "x64")).toBe(false); expect(isLinuxDockerDriverGatewayEnabled("win32")).toBe(false); - const linuxEnv = getDockerDriverGatewayEnv("openshell 0.0.116", "linux"); + const linuxEnv = getDockerDriverGatewayEnv("openshell 0.1.7", "linux"); expect(linuxEnv.OPENSHELL_DRIVERS).toBe("docker"); expect(linuxEnv.OPENSHELL_BIND_ADDRESS).toBe("127.0.0.1"); expect(linuxEnv.OPENSHELL_GRPC_ENDPOINT).toBe("https://127.0.0.1:8080"); @@ -84,7 +84,7 @@ describe("onboard gateway runtime helpers", () => { expect(linuxEnv.OPENSHELL_SSH_GATEWAY_HOST).toBe("127.0.0.1"); expect(linuxEnv.OPENSHELL_CLUSTER_IMAGE).toBeUndefined(); expect(linuxEnv.OPENSHELL_DOCKER_SUPERVISOR_IMAGE).toBe( - "ghcr.io/nvidia/openshell/supervisor@sha256:c8c42aef16c200063e32cbf72e553e4ead027085427b555efafd95063ecead42", + "ghcr.io/nvidia/openshell/supervisor@sha256:a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06", ); const originalOverlayFix = process.env.NEMOCLAW_DISABLE_OVERLAY_FIX; @@ -195,7 +195,7 @@ describe("onboard gateway runtime helpers", () => { }); it("detects stale Docker-driver gateway runtime state before reuse", () => { - const desiredEnv = getDockerDriverGatewayEnv("openshell 0.0.116", "linux"); + const desiredEnv = getDockerDriverGatewayEnv("openshell 0.1.7", "linux"); const gatewayBin = process.execPath; expect( @@ -251,7 +251,7 @@ describe("onboard gateway runtime helpers", () => { }); it("reuses a healthy containerized-compat gateway whose parent is /usr/bin/docker (#4520)", () => { - const desiredEnv = getDockerDriverGatewayEnv("openshell 0.0.116", "linux"); + const desiredEnv = getDockerDriverGatewayEnv("openshell 0.1.7", "linux"); // The compat gateway is a `docker run ... /opt/nemoclaw/openshell-gateway` // parent, so /proc//exe is /usr/bin/docker. The runtime identity sets diff --git a/test/onboarding/onboard-host-local-inference-routing.test.ts b/test/onboarding/onboard-host-local-inference-routing.test.ts index 1793f6446c0..c28d60a9cb0 100644 --- a/test/onboarding/onboard-host-local-inference-routing.test.ts +++ b/test/onboarding/onboard-host-local-inference-routing.test.ts @@ -458,42 +458,6 @@ describe("onboard host-local inference routing", () => { }, ); - it("explicitly clears stale host-local ownership for a remote route", async () => { - const harness = createHarness({ - overrides: { - isRoutedInferenceProvider: () => true, - reconcileModelRouter: vi.fn(async () => undefined), - routedInference: { - upsertRoutedProvider: vi.fn(() => ({ - ok: true, - endpointUrl: "https://api.example.test/v1", - result: { message: "configured" }, - })), - }, - }, - }); - await expect( - harness.setupInference( - "sandbox-a", - "remote-model", - "nvidia-router", - "https://api.example.test/v1", - "REMOTE_API_KEY", - null, - [], - ), - ).resolves.toEqual({ ok: true }); - - expect(harness.updateSandbox).toHaveBeenCalledWith( - "sandbox-a", - expect.objectContaining({ - provider: "nvidia-router", - model: "remote-model", - hostLocalInferenceReceipt: null, - }), - ); - }); - it.each(SHARED_APPLICATIONS)( "routes explicit shared %s inference through inference.local without legacy host probes", async (application) => { diff --git a/test/onboarding/onboard-inference-failure-paths.test.ts b/test/onboarding/onboard-inference-failure-paths.test.ts index 3f935bb1ee9..87a1dc0a07e 100644 --- a/test/onboarding/onboard-inference-failure-paths.test.ts +++ b/test/onboarding/onboard-inference-failure-paths.test.ts @@ -919,6 +919,9 @@ describe("setupInference dependency failures", () => { "nvidia-router", "http://host.openshell.internal:4000/v1", "NVIDIA_INFERENCE_API_KEY", + null, + [], + { agentName: "openclaw" }, ), ).rejects.toThrow("EXIT_CALLED:1"); @@ -929,181 +932,4 @@ describe("setupInference dependency failures", () => { expect(harness.errors).toEqual([" ✗ Failed to start model router: router unavailable"]); expectNoPostFailureSideEffects(harness); }); - - it("preserves a routed-provider upsert status through the injected exit boundary", async () => { - const exitProcess = createInjectedExit(); - const reconcileModelRouter = vi.fn(async () => {}); - const upsertProvider = vi.fn(async () => ({ ok: true })); - const hydrateCredentialEnv = vi.fn(() => "unused-secret"); - const upsertRoutedProvider = vi.fn< - SetupInferenceDeps["routedInference"]["upsertRoutedProvider"] - >(() => ({ - ok: false, - result: { status: 29, message: "routed provider registration rejected" }, - })); - const harness = createDirectSetupInferenceHarness({ - overrides: { - isRoutedInferenceProvider: (provider) => provider === "nvidia-router", - exitProcess, - reconcileModelRouter, - upsertProvider, - hydrateCredentialEnv, - routedInference: { upsertRoutedProvider }, - }, - }); - - await expect( - harness.setupInference( - "test-box", - "router/model", - "nvidia-router", - "http://host.openshell.internal:4000/v1", - "NVIDIA_INFERENCE_API_KEY", - ), - ).rejects.toThrow("EXIT_CALLED:29"); - - expect(reconcileModelRouter).toHaveBeenCalledOnce(); - expect(upsertRoutedProvider).toHaveBeenCalledOnce(); - expect(upsertRoutedProvider).toHaveBeenCalledWith( - "nvidia-router", - "http://host.openshell.internal:4000/v1", - "NVIDIA_INFERENCE_API_KEY", - { - upsertProvider: expect.any(Function), - hydrateCredentialEnv, - }, - ); - const routedUpsertProvider = upsertRoutedProvider.mock.calls[0][3].upsertProvider; - expect(routedUpsertProvider).not.toBe(upsertProvider); - expect(upsertProvider).not.toHaveBeenCalled(); - expect(hydrateCredentialEnv).not.toHaveBeenCalled(); - routedUpsertProvider( - "nvidia-router", - "openai", - "NVIDIA_INFERENCE_API_KEY", - "http://host.openshell.internal:4000/v1", - { NVIDIA_INFERENCE_API_KEY: "test-secret" }, - ); - expect(upsertProvider).toHaveBeenCalledWith( - "nvidia-router", - "openai", - "NVIDIA_INFERENCE_API_KEY", - "http://host.openshell.internal:4000/v1", - { NVIDIA_INFERENCE_API_KEY: "test-secret" }, - "nemoclaw", - { revalidateSandboxIdentity: expect.any(Function) }, - ); - expect(exitProcess).toHaveBeenCalledOnce(); - expect(exitProcess).toHaveBeenCalledWith(29); - expect(harness.errors).toEqual([" routed provider registration rejected"]); - expectNoPostFailureSideEffects(harness); - }); - - it("redacts a routed inference-set failure and preserves its status at the exit boundary", async () => { - const exitProcess = createInjectedExit(); - const reconcileModelRouter = vi.fn(async () => {}); - const upsertRoutedProvider = vi.fn(() => ({ ok: true, result: {} })); - const commandRouter = createDirectCommandRouter([ - { - name: "routed-inference-set", - matches: (command) => command.startsWith("inference set"), - results: [ - { status: 41, stdout: "", stderr: `routed apply failed ${NVIDIA_REDACTION_CANARY}` }, - ], - }, - ]); - const harness = createDirectSetupInferenceHarness({ - runOpenshell: commandRouter.runOpenshell, - overrides: { - isRoutedInferenceProvider: (provider) => provider === "nvidia-router", - exitProcess, - reconcileModelRouter, - routedInference: { upsertRoutedProvider }, - }, - }); - - await expect( - harness.setupInference( - "test-box", - "router/model", - "nvidia-router", - "http://host.openshell.internal:4000/v1", - "NVIDIA_INFERENCE_API_KEY", - ), - ).rejects.toThrow("EXIT_CALLED:41"); - - expect(reconcileModelRouter).toHaveBeenCalledOnce(); - expect(upsertRoutedProvider).toHaveBeenCalledOnce(); - expect(commandRouter.callCount("routed-inference-set")).toBe(1); - expect(harness.commands.at(-1)).toMatchObject({ ignoreError: true }); - expect(exitProcess).toHaveBeenCalledOnce(); - expect(exitProcess).toHaveBeenCalledWith(41); - expect(harness.errors.join("\n")).toContain("routed apply failed"); - expect(harness.errors.join("\n")).not.toContain(NVIDIA_REDACTION_CANARY); - expectNoPostFailureSideEffects(harness, [ - "inference set -g nemoclaw --no-verify --provider nvidia-router --model router/model", - ]); - }); - - it("runs shared finalization after routed inference setup succeeds", async () => { - const exitProcess = createInjectedExit(); - const reconcileModelRouter = vi.fn(async () => {}); - const upsertRoutedProvider = vi.fn(() => ({ ok: true, result: {} })); - const harness = createDirectSetupInferenceHarness({ - overrides: { - isRoutedInferenceProvider: (provider) => provider === "nvidia-router", - exitProcess, - reconcileModelRouter, - routedInference: { upsertRoutedProvider }, - }, - }); - - await expect( - harness.setupInference( - "test-box", - "router/model", - "nvidia-router", - "http://host.openshell.internal:4000/v1", - "NVIDIA_INFERENCE_API_KEY", - ), - ).resolves.toEqual({ ok: true }); - - expect(reconcileModelRouter).toHaveBeenCalledOnce(); - expect(upsertRoutedProvider).toHaveBeenCalledOnce(); - expect(harness.commands).toEqual([ - { - command: - "inference set -g nemoclaw --no-verify --provider nvidia-router --model router/model", - ignoreError: true, - env: undefined, - }, - ]); - expect(harness.verifyInferenceRoute).toHaveBeenCalledOnce(); - expect(harness.verifyInferenceRoute).toHaveBeenCalledWith( - "nemoclaw", - "nvidia-router", - "router/model", - ); - expect(harness.verifyOnboardInferenceSmoke).toHaveBeenCalledOnce(); - expect(harness.verifyOnboardInferenceSmoke).toHaveBeenCalledWith({ - provider: "nvidia-router", - model: "router/model", - endpointUrl: "http://host.openshell.internal:4000/v1", - credentialEnv: "NVIDIA_INFERENCE_API_KEY", - }); - expect(harness.updateSandbox).toHaveBeenCalledOnce(); - expect(harness.updateSandbox).toHaveBeenCalledWith("test-box", { - model: "router/model", - provider: "nvidia-router", - endpointUrl: "http://host.openshell.internal:4000/v1", - endpointSource: "onboard", - credentialEnv: "NVIDIA_INFERENCE_API_KEY", - preferredInferenceApi: null, - gatewayName: "nemoclaw", - hostLocalInferenceReceipt: null, - }); - expect(harness.logs).toEqual([" ✓ Inference route set: nvidia-router / router/model"]); - expect(harness.errors).toEqual([]); - expect(exitProcess).not.toHaveBeenCalled(); - }); }); diff --git a/test/onboarding/onboard-inference-reconciliation.test.ts b/test/onboarding/onboard-inference-reconciliation.test.ts index f05c9e41aec..e40e43c3bc7 100644 --- a/test/onboarding/onboard-inference-reconciliation.test.ts +++ b/test/onboarding/onboard-inference-reconciliation.test.ts @@ -8,6 +8,10 @@ import os from "node:os"; import path from "node:path"; import { describe, expect, it, vi } from "vitest"; +import { + HERMES_NATIVE_RECEIPT, + createHermesNativeHarness, +} from "../support/native-hermes-setup-harness"; import { clearPersistedOllamaHostIfUnused, loadPersistedOllamaHost, @@ -29,22 +33,6 @@ import { withProcessEnv, } from "../support/setup-inference-test-harness.js"; -const HERMES_OAUTH_PROVIDER_METADATA = [ - "Name: hermes-provider", - "Type: openai", - "Credential keys: OPENAI_API_KEY", - "Config keys: OPENAI_BASE_URL", - "", -].join("\n"); - -const HERMES_API_KEY_PROVIDER_METADATA = [ - "Name: hermes-provider", - "Type: openai", - "Credential keys: NOUS_API_KEY", - "Config keys: OPENAI_BASE_URL", - "", -].join("\n"); - const COMPATIBLE_PROVIDER_METADATA = [ "Name: compatible-endpoint", "Type: openai", @@ -61,13 +49,7 @@ describe("onboard helpers", () => { OPENAI_API_KEY: "openai-host-secret", }, async () => { - const harness = createDirectSetupInferenceHarness({ - runOpenshell: (args) => - args.join(" ") === "provider get -g nemoclaw hermes-provider" - ? { status: 0, stdout: HERMES_OAUTH_PROVIDER_METADATA, stderr: "" } - : undefined, - overrides: { isNonInteractive: () => true }, - }); + const harness = createHermesNativeHarness(); await harness.setupInference( "test-box", @@ -79,13 +61,20 @@ describe("onboard helpers", () => { ); const commands = harness.commands; - assert.equal(commands.length, 3); + assert.equal(commands.length, 1); assert.equal(commands[0].command, "provider list -g nemoclaw"); - assert.equal(commands[1].command, "provider get -g nemoclaw hermes-provider"); - assert.match( - commands[2].command, - /inference set -g nemoclaw --no-verify --provider hermes-provider/, + expect(harness.providerAdapter.getProvider).toHaveBeenCalledWith({ + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: HERMES_NATIVE_RECEIPT.providerName, + }); + expect(harness.providerAdapter.updateProvider).not.toHaveBeenCalled(); + expect(harness.providerAdapter.createProvider).not.toHaveBeenCalled(); + expect(harness.writeAuthority).toHaveBeenCalledWith( + "nemoclaw", + "hermes-provider", + HERMES_NATIVE_RECEIPT, ); + assert.ok(!commands.some((entry) => entry.command.startsWith("inference "))); assert.ok(!commands.some((entry) => entry.command.startsWith("gateway select"))); assert.ok(!commands.some((entry) => /provider (create|update)/.test(entry.command))); assert.ok(!commands.some((entry) => entry.env?.NOUS_API_KEY || entry.env?.OPENAI_API_KEY)); @@ -96,6 +85,37 @@ describe("onboard helpers", () => { }, ); }); + it.each(["wrong identity", "missing credentials"])( + "refuses native Hermes resume with %s before sandbox mutation", + async (failure) => { + await withProcessEnv({ NOUS_API_KEY: undefined, OPENAI_API_KEY: undefined }, async () => { + const harness = createHermesNativeHarness({ + getNativeHostedProviderAuthority: () => + failure === "wrong identity" + ? { ...HERMES_NATIVE_RECEIPT, providerId: "different-owner" } + : undefined, + }); + await expect( + harness.setupInference( + "test-box", + "moonshotai/kimi-k2.6", + "hermes-provider", + "https://8.8.8.8/v1", + "OPENAI_API_KEY", + "oauth", + ), + ).rejects.toThrow(); + expect(harness.providerAdapter.createProvider).not.toHaveBeenCalled(); + expect(harness.providerAdapter.updateProvider).not.toHaveBeenCalled(); + expect(harness.writeAuthority).not.toHaveBeenCalled(); + expect(harness.updateSandbox).not.toHaveBeenCalled(); + expect( + harness.commands.every((entry) => entry.command === "provider list -g nemoclaw"), + ).toBe(true); + }); + }, + ); + it("routes Bedrock Runtime custom Anthropic endpoints through the hidden OpenAI adapter", async () => { await withProcessEnv({ COMPATIBLE_ANTHROPIC_API_KEY: "bedrock-bearer" }, async () => { const updateSandbox = vi.fn(() => true); @@ -202,7 +222,6 @@ describe("onboard helpers", () => { const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-onboard-hermes-resume-")); const fakeBin = path.join(tmpDir, "bin"); const scriptPath = path.join(tmpDir, "hermes-resume-sandbox-name-check.js"); - const inferenceCommandLogPath = path.join(tmpDir, "inference-commands.log"); const openshellPath = JSON.stringify(path.join(fakeBin, "openshell")); const onboardPath = JSON.stringify(path.join(repoRoot, "src", "lib", "onboard.ts")); const runnerPath = JSON.stringify(path.join(repoRoot, "src", "lib", "runner.ts")); @@ -242,14 +261,7 @@ describe("onboard helpers", () => { ); fs.mkdirSync(fakeBin, { recursive: true }); - writeOkOpenshell(fakeBin, { - inferenceRoute: { - gatewayName: "nemoclaw", - provider: "hermes-provider", - model: "moonshotai/kimi-k2.6", - commandLogPath: inferenceCommandLogPath, - }, - }); + writeOkOpenshell(fakeBin); fs.writeFileSync(path.join(fakeBin, "brew"), "#!/bin/sh\nexit 1\n", { mode: 0o755 }); const script = String.raw` @@ -316,7 +328,31 @@ const prompts = []; const registryUpdates = []; const done = new Error("INFERENCE_STEP_DONE"); let inferenceSessionSnapshot = null; -const hermesApiKeyProviderMetadata = ${JSON.stringify(HERMES_API_KEY_PROVIDER_METADATA)}; +const nativeReceipt = ${JSON.stringify(HERMES_NATIVE_RECEIPT)}; +const nativeProviderCalls = []; +const providers = require(${JSON.stringify(path.join(repoRoot, "src/lib/onboard/providers.ts"))}); +const productionProviderDeps = providers.setupInferenceProviderDeps; +providers.setupInferenceProviderDeps = (run) => ({ + ...productionProviderDeps(run), + getNativeHostedProviderAuthority: () => nativeReceipt, + setNativeHostedProviderAuthority: (gateway, provider, receipt) => { + nativeProviderCalls.push({ operation: "persist", gateway, provider, receipt }); + }, + providerAdapter: { + importProviderProfile: async () => ({ ok: true }), + ensureProviderPolicyComposition: async () => ({ ok: true, value: undefined }), + getProvider: async (request) => { + nativeProviderCalls.push({ operation: "get", ...request }); + return { ok: true, value: { + name: nativeReceipt.providerName, type: nativeReceipt.profileId, + credentialKeys: ["OPENAI_API_KEY"], configKeys: [], + revision: { id: nativeReceipt.providerId, resourceVersion: 1 }, + } }; + }, + createProvider: async () => { throw new Error("Resume must reuse the owned provider"); }, + updateProvider: async () => { throw new Error("Resume must not collect a new credential"); }, + }, +}); delete process.env.NEMOCLAW_NON_INTERACTIVE; delete process.env.NEMOCLAW_SANDBOX_NAME; @@ -340,26 +376,9 @@ try { runner.run = (command, opts = {}) => { const normalized = _n(command); commands.push({ command: normalized, env: opts.env || null }); - const providerGet = "provider get -g nemoclaw hermes-provider"; - if (normalized === providerGet || normalized.endsWith(" " + providerGet)) { - return { status: 0, stdout: hermesApiKeyProviderMetadata, stderr: "" }; - } return { status: 0, stdout: "", stderr: "" }; }; -runner.runCapture = (command) => { - const normalized = _n(command); - if (normalized.includes("inference get")) { - return [ - "Gateway inference:", - "", - " Route: inference.local", - " Provider: hermes-provider", - " Model: moonshotai/kimi-k2.6", - " Version: 1", - ].join("\n"); - } - return ""; -}; +runner.runCapture = () => ""; registry.getSandbox = (name) => name === "hermes-resume" @@ -369,6 +388,8 @@ registry.getSandbox = (name) => provider: "hermes-provider", model: "moonshotai/kimi-k2.6", hermesToolGateways: [], + endpointUrl: nativeReceipt.endpointUrl, + nativeHostedProviderAttachment: nativeReceipt, } : null; registry.reserveSandboxInferenceRoute = (name, updates) => { @@ -381,6 +402,7 @@ registry.removeSandbox = () => true; credentials.prompt = async (question) => { prompts.push(String(question)); if (String(question).includes("Sandbox name")) return "hermes-resume"; + if (String(question).includes("Choose [1]")) return "1"; return "yes"; }; @@ -477,6 +499,7 @@ const { onboard } = require(${onboardPath}); if (error === done || error?.message === done.message) { console.log(JSON.stringify({ commands, + nativeProviderCalls, prompts, registryUpdates, inferenceSessionSandboxName: inferenceSessionSnapshot?.sandboxName ?? null, @@ -507,28 +530,36 @@ const { onboard } = require(${onboardPath}); }); assert.equal(result.status, 0, result.stderr); - const inferenceCommands = fs.readFileSync(inferenceCommandLogPath, "utf8").trim().split("\n"); - assert.ok( - inferenceCommands.includes("inference get -g nemoclaw"), - `expected a scoped inference read, received ${JSON.stringify(inferenceCommands)}`, - ); - assert.ok( - inferenceCommands.some((command) => - /inference set -g nemoclaw --no-verify --provider hermes-provider/.test(command), - ), - `expected a scoped asynchronous inference set, received ${JSON.stringify(inferenceCommands)}`, - ); assert.doesNotMatch( `${result.stderr}\n${result.stdout}`, /Hermes Provider requires a sandbox name/, ); const payload = parseStdoutJson<{ commands: CommandEntry[]; + nativeProviderCalls: Array<{ + operation: string; + target?: { kind: string; gatewayName: string }; + providerName?: string; + gateway?: string; + receipt?: typeof HERMES_NATIVE_RECEIPT; + }>; prompts: string[]; registryUpdates: Array<{ name: string; updates: Record }>; inferenceSessionSandboxName: string | null; }>(result.stdout); + expect(payload.nativeProviderCalls).toContainEqual({ + operation: "get", + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: HERMES_NATIVE_RECEIPT.providerName, + }); + expect(payload.nativeProviderCalls).toContainEqual({ + operation: "persist", + gateway: "nemoclaw", + provider: "hermes-provider", + receipt: HERMES_NATIVE_RECEIPT, + }); + assert.ok(!payload.commands.some((entry) => /inference (get|set)/.test(entry.command))); assert.ok( payload.prompts.some((question) => question.includes("Sandbox name")), "resume should prompt for the missing sandbox name before Hermes inference reconciliation", @@ -554,7 +585,7 @@ const { onboard } = require(${onboardPath}); endpointUrl: "https://8.8.8.8/v1", }, ], - "a non-staged Hermes route must keep the registry aligned with the immediate gateway update", + "native Hermes resume keeps the registry aligned with its owned provider", ); }, ); @@ -566,13 +597,7 @@ const { onboard } = require(${onboardPath}); OPENAI_API_KEY: undefined, }, async () => { - const harness = createDirectSetupInferenceHarness({ - runOpenshell: (args) => - args.join(" ") === "provider get -g nemoclaw hermes-provider" - ? { status: 0, stdout: HERMES_OAUTH_PROVIDER_METADATA, stderr: "" } - : undefined, - overrides: { isNonInteractive: () => true }, - }); + const harness = createHermesNativeHarness(); await harness.setupInference( "test-box", @@ -583,19 +608,22 @@ const { onboard } = require(${onboardPath}); "api_key", ); - const update = harness.commands.find((entry) => - /provider update -g nemoclaw hermes-provider/.test(entry.command), - ); - assert.ok(update); - assert.match(update.command, /--credential NOUS_API_KEY/); - assert.equal(update.env?.NOUS_API_KEY, "nous-host-secret"); + expect(harness.providerAdapter.updateProvider).toHaveBeenCalledExactlyOnceWith({ + target: { kind: "named", gatewayName: "nemoclaw" }, + providerName: HERMES_NATIVE_RECEIPT.providerName, + credentials: [{ name: "OPENAI_API_KEY", value: "nous-host-secret" }], + config: [], + }); + expect(harness.providerAdapter.createProvider).not.toHaveBeenCalled(); assert.ok( !harness.commands.some((entry) => /nous-host-secret/.test(entry.command)), "shell credential value must not appear in argv", ); - assert.match( - harness.commands.at(-1)?.command || "", - /inference set -g nemoclaw --no-verify --provider hermes-provider/, + assert.ok(!harness.commands.some((entry) => entry.command.startsWith("inference "))); + expect(harness.writeAuthority).toHaveBeenCalledWith( + "nemoclaw", + "hermes-provider", + HERMES_NATIVE_RECEIPT, ); }, ); diff --git a/test/onboarding/onboard-messaging.test.ts b/test/onboarding/onboard-messaging.test.ts index 90e7e2d3df1..ccb6fb8f751 100644 --- a/test/onboarding/onboard-messaging.test.ts +++ b/test/onboarding/onboard-messaging.test.ts @@ -522,15 +522,18 @@ const fixtureMocks = require(${onboardScriptMocksPath}); const _n = (c) => (Array.isArray(c) ? c.join(" ") : String(c)).replace(/'/g, ""); const childProcess = require("node:child_process"), { EventEmitter } = require("node:events"); const commands = [], credentialKeys = ${JSON.stringify(providerCredentialKeys)}; let registered = null; +const { buildCredentialFamilyProfile } = require(${JSON.stringify(path.join(repoRoot, "src/lib/messaging/applier/credential-family-profile.ts"))}); +const telegramProfile = buildCredentialFamilyProfile(require("node:fs").readFileSync(${JSON.stringify(path.join(repoRoot, "nemoclaw-blueprint/provider-profiles/nemoclaw-messaging-v1.yaml"))}, "utf8"), "TELEGRAM_BOT_TOKEN", credentialKeys["my-assistant-telegram-bridge"]); const createdSandbox = fixtureMocks.createCreatedSandboxFixture({ sandboxName: "my-assistant" }); createdSandbox.installRuntimeObservation(); const providers = Object.keys(credentialKeys), revisions = new Map(providers.map((name) => [name, 1])), providerGetCounts = new Map(); const rawGatewayCredential = ${JSON.stringify(rawGatewayCredential)}, gatewaySecrets = new Map(providers.map((name) => [name, rawGatewayCredential])); registry.registerSandbox({ name: "my-assistant", messaging: { schemaVersion: 1, plan: ${messagingPlanLiteral(["slack", "telegram", "whatsapp"])} } }); runner.run = (command, opts = {}) => { const normalized = _n(command); commands.push({ command: normalized, env: opts.env || null }); - const profileResult = require(${onboardScriptMocksPath}).mockEndpointlessProviderProfileRun(command, "nemoclaw-mcp-v1", false); if (profileResult !== null) return profileResult; + if (normalized.includes("profile") && normalized.includes("export " + telegramProfile.profileType + " ")) return { status: 0, stdout: telegramProfile.source }; + const profileResult = require(${onboardScriptMocksPath}).mockEndpointlessProviderProfileRun(command, "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; const providerGet = normalized.match(/provider get -g nemoclaw ([^ ]+)$/)?.[1]; if (providerGet) providerGetCounts.set(providerGet, (providerGetCounts.get(providerGet) || 0) + 1); if (providerGet === process.env.NEMOCLAW_TEST_FAIL_PROVIDER && providerGetCounts.get(providerGet) >= 4) return { status: 2, stderr: "transport unavailable" }; - if (providerGet && revisions.has(providerGet)) return { status: 0, stdout: "Name: " + providerGet + "\nType: " + (providerGet === "compatible-endpoint" ? "openai" : "nemoclaw-mcp-v1") + "\nCredential keys: " + credentialKeys[providerGet] + "\nConfig keys: " + (providerGet === "compatible-endpoint" ? "OPENAI_BASE_URL" : "") + "\n" }; + if (providerGet && revisions.has(providerGet)) return { status: 0, stdout: "Name: " + providerGet + "\nType: " + (providerGet === "compatible-endpoint" ? "openai" : providerGet === "my-assistant-telegram-bridge" ? telegramProfile.profileType : "nemoclaw-messaging-v1") + "\nCredential keys: " + credentialKeys[providerGet] + "\nConfig keys: " + (providerGet === "compatible-endpoint" ? "OPENAI_BASE_URL" : "") + "\n" }; const refresh = normalized.match(/provider update -g nemoclaw ([^ ]+)$/)?.[1]; if (refresh && gatewaySecrets.has(refresh)) { if (refresh === process.env.NEMOCLAW_TEST_FAIL_PROVIDER) return { status: 1 }; revisions.set(refresh, revisions.get(refresh) + 1); return { status: 0 }; } if (normalized.includes("provider get")) return { status: 1 }; @@ -605,7 +608,7 @@ const { createSandbox } = require(${onboardPath}); .map(({ entry }: { entry: CommandEntry }) => providerName(entry.command)) .sort(); const denied = await runScenario("my-assistant-telegram-bridge"); - assert.equal(denied.status, 1); + assert.equal(denied.status, 1, denied.stdout); const deniedPayload = parseStdoutJson(denied.stdout); const deniedCommands = (deniedPayload.commands as CommandEntry[]).map( ({ command }) => command, @@ -712,7 +715,7 @@ registry.registerSandbox({ runner.run = (command, opts = {}) => { const normalized = _n(command); commands.push({ command: normalized, env: opts.env || null }); - if (normalized.includes("provider get -g nemoclaw my-assistant-telegram-bridge")) return { status: 0, stdout: "Name: my-assistant-telegram-bridge\nType: nemoclaw-mcp-v1\nCredential keys: TELEGRAM_BOT_TOKEN\nConfig keys: \n" }; + if (normalized.includes("provider get -g nemoclaw my-assistant-telegram-bridge")) return { status: 0, stdout: "Name: my-assistant-telegram-bridge\nType: nemoclaw-messaging-v1\nCredential keys: TELEGRAM_BOT_TOKEN\nConfig keys: \n" }; const providerGetResult = fixtureMocks.mockNvidiaOrMissingProviderGetRun(command, "nemoclaw"); if (providerGetResult !== null) return providerGetResult; return createdSandbox.run(command) ?? { status: 0 }; }; @@ -1258,6 +1261,7 @@ const runner = require(${runnerPath}); const _n = (c) => (Array.isArray(c) ? c.join(" ") : String(c)).replace(/'/g, ""); const registry = require(${registryPath}); const fixtureMocks = require(${onboardScriptMocksPath}); +fixtureMocks.mockStandaloneGatewayTeardownAuthority(); const commands = []; const existingSandbox = fixtureMocks.createCreatedSandboxFixture({ lifecycleState: "created" }); existingSandbox.installRuntimeObservation(); @@ -1265,16 +1269,15 @@ const messagingProviderRunner = require(${onboardScriptMocksPath}).createStatefu commands, createdSandbox: existingSandbox, initialProviders: [ - ["my-assistant-discord-bridge", "nemoclaw-mcp-v1", "DISCORD_BOT_TOKEN"], - ["my-assistant-slack-bridge", "nemoclaw-mcp-v1", "SLACK_BOT_TOKEN"], - ["my-assistant-slack-app", "nemoclaw-mcp-v1", "SLACK_APP_TOKEN"], + ["my-assistant-discord-bridge", "nemoclaw-messaging-v1", "DISCORD_BOT_TOKEN"], + ["my-assistant-slack-bridge", "nemoclaw-messaging-v1", "SLACK_BOT_TOKEN"], + ["my-assistant-slack-app", "nemoclaw-messaging-v1", "SLACK_APP_TOKEN"], ], }); runner.run = messagingProviderRunner; runner.runCapture = (command) => { const sandboxCapture = existingSandbox.capture(command); if (sandboxCapture !== null) return sandboxCapture; - // All messaging providers already exist in gateway if (_n(command).includes("provider get")) return "Provider: exists"; if (_n(command).includes("forward list")) return "SANDBOX BIND PORT PID STATUS"; return ""; @@ -1443,7 +1446,6 @@ const { createSandbox } = require(${onboardPath}); assert.equal(result.status, 0, result.stderr); const payload = parseStdoutJson(result.stdout); - // Only telegram provider should be created const providerCommands = payload.commands.filter((e: CommandEntry) => e.command.includes("provider create"), ); @@ -1452,7 +1454,6 @@ const { createSandbox } = require(${onboardPath}); ); assert.ok(telegramProvider, "expected telegram provider to be created"); - // Discord and slack providers should NOT be created const discordProvider = providerCommands.find((e: CommandEntry) => e.command.includes("my-assistant-discord-bridge"), ); @@ -1463,7 +1464,6 @@ const { createSandbox } = require(${onboardPath}); ); assert.ok(!slackProvider, "slack provider should be filtered out"); - // Sandbox create should only have the telegram --provider flag const createCommand = payload.commands.find((e: CommandEntry) => e.command.includes("sandbox create"), ); diff --git a/test/onboarding/onboard-model-router.test.ts b/test/onboarding/onboard-model-router.test.ts index f7ad148a7e3..dfe8781d150 100644 --- a/test/onboarding/onboard-model-router.test.ts +++ b/test/onboarding/onboard-model-router.test.ts @@ -6,9 +6,8 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, describe, it, vi } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; -import { getSandboxInferenceConfig } from "../../src/lib/inference/config"; import { createProductionModelRouterCommandProvisioner, isManagedModelRouterCurrent, @@ -29,7 +28,6 @@ import { } from "../support/model-router-process-test-helpers.js"; import { createDirectSetupInferenceHarnessFactory, - type DirectCommandEntry, withProcessEnv, } from "../support/setup-inference-test-harness.js"; @@ -165,55 +163,31 @@ function createCommandHarness(options: CommandHarnessOptions = {}) { }; } -function findCommand(commands: DirectCommandEntry[], pattern: RegExp): DirectCommandEntry { - const command = commands.find((entry) => pattern.test(entry.command)); - assert.ok(command, JSON.stringify(commands)); - return command; -} - describe("onboard Model Router setup", () => { - it("configures Model Router as a host provider while sandboxes keep inference.local", async () => { + it("rejects an unowned Model Router endpoint before provider or shared-route mutation", async () => { await withProcessEnv({ NVIDIA_INFERENCE_API_KEY: NVIDIA_TEST_CREDENTIAL }, async () => { const reconcileModelRouter = vi.fn(async () => undefined); const harness = createDirectSetupInferenceHarness({ - runOpenshell: (args) => - args[0] === "provider" && args[1] === "get" ? { status: 1 } : undefined, overrides: { isRoutedInferenceProvider: (provider: string) => provider === "nvidia-router", reconcileModelRouter, }, }); const routerPort = 44000 + (process.pid % 10000); - - await harness.setupInference( - "router-box", - "nvidia-routed", - "nvidia-router", - `http://host.openshell.internal:${routerPort}/v1`, - "NVIDIA_INFERENCE_API_KEY", - ); - - assert.equal(reconcileModelRouter.mock.calls.length, 1); - const providerCommand = findCommand(harness.commands, /provider create/); - assert.match(providerCommand.command, /--name nvidia-router/); - assert.match(providerCommand.command, /--credential NVIDIA_INFERENCE_API_KEY/); - assert.match( - providerCommand.command, - new RegExp(`OPENAI_BASE_URL=http:\\/\\/host\\.openshell\\.internal:${routerPort}\\/v1`), + await expect( + harness.setupInference( + "router-box", + "nvidia-routed", + "nvidia-router", + `http://host.openshell.internal:${routerPort}/v1`, + "NVIDIA_INFERENCE_API_KEY", + ), + ).rejects.toThrow( + "Model Router requires a native provider for the selected agent and configured endpoint", ); - assert.doesNotMatch(providerCommand.command, new RegExp(NVIDIA_TEST_CREDENTIAL)); - assert.equal(providerCommand.env?.NVIDIA_INFERENCE_API_KEY, NVIDIA_TEST_CREDENTIAL); - - const inferenceCommand = findCommand(harness.commands, /inference set/); - assert.match(inferenceCommand.command, /--provider nvidia-router/); - assert.match(inferenceCommand.command, /--model nvidia-routed/); - assert.deepEqual(getSandboxInferenceConfig("nvidia-routed", "nvidia-router"), { - providerKey: "inference", - primaryModelRef: "inference/nvidia-routed", - inferenceBaseUrl: "https://inference.local/v1", - inferenceApi: "openai-completions", - inferenceCompat: null, - }); + expect(reconcileModelRouter).not.toHaveBeenCalled(); + expect(harness.commands).toEqual([]); + expect(harness.updateSandbox).not.toHaveBeenCalled(); }); }); @@ -695,8 +669,9 @@ describe("onboard Model Router setup", () => { assert.equal(poolTargetsOnlyNvidiaEndpoints(shippedPool), true); }); - it("appends the last router health error and log path to the startup timeout error (#8962)", async () => { + it("retains the last completed health error when the final router request times out (#8962)", async () => { const pid = 12_345; + let healthCalls = 0; const unhealthyBody = JSON.stringify({ healthy_endpoints: [], unhealthy_endpoints: [ @@ -722,7 +697,10 @@ describe("onboard Model Router setup", () => { resolveProviderCredential: () => null, buildSubprocessEnv: () => ({}), isRouterResponsive: async () => false, - getRouterHealthSnapshot: async () => healthSnapshot(false, unhealthyBody), + getRouterHealthSnapshot: async () => + ++healthCalls === 1 + ? healthSnapshot(true, unhealthyBody) + : healthSnapshot(false, null, { outcome: "timeout" }), openRouterLog: () => ({ fd: 99, startOffset: 0 }), closeRouterLog: () => undefined, readRouterLogTail: () => "", diff --git a/test/onboarding/onboard-native-provider-reservation.test.ts b/test/onboarding/onboard-native-provider-reservation.test.ts index bdc4354c706..ece2cc9fa9a 100644 --- a/test/onboarding/onboard-native-provider-reservation.test.ts +++ b/test/onboarding/onboard-native-provider-reservation.test.ts @@ -11,6 +11,8 @@ import { } from "../../src/lib/onboard/machine/handlers/provider-inference.test-support"; import { nativeLocalSetupReceipt } from "../support/native-local-setup-harness"; import { createDirectSetupInferenceHarness } from "../helpers/onboard-split-context"; +import type { SetupInferenceDeps } from "../../src/lib/onboard/setup-inference"; +import { serializedHostLocalInferenceReceipt } from "../helpers/host-local-inference-receipt"; // Abandoned-reservation recovery has its own state tests. Keep this harness from // reading the developer's registry through the CommonJS onboarding boundary. @@ -84,6 +86,65 @@ describe("native provider ownership before route reservation", () => { expect(harness.native.providers.size).toBe(0); expect(harness.native.authorities.size).toBe(0); }); + it("clears stale host-local ownership when selecting native NVIDIA inference", async () => { + const providerAdapter = { + importProviderProfile: vi.fn(async () => ({ ok: true as const })), + ensureProviderPolicyComposition: vi.fn(async () => ({ ok: true as const, value: undefined })), + getProvider: vi + .fn() + .mockResolvedValueOnce({ + ok: false, + error: { kind: "command", reason: "not_found", message: "absent" }, + }) + .mockResolvedValue({ + ok: true, + value: { + name: "nemoclaw-nvidia-prod-v1", + type: "nemoclaw-nvidia-inference-v1", + credentialKeys: ["NVIDIA_INFERENCE_API_KEY"], + configKeys: [], + revision: { id: "native-identity", resourceVersion: 1 }, + }, + }), + createProvider: vi.fn(async () => ({ ok: true as const })), + } as unknown as NonNullable; + const harness = createDirectSetupInferenceHarness({ + overrides: { + providerAdapter, + hydrateCredentialEnv: () => "host-secret", + getNativeNvidiaProviderAuthority: () => undefined, + setNativeNvidiaProviderAuthority: vi.fn(), + getSandbox: () => ({ + name: "sandbox-a", + provider: "vllm-local", + model: "qwen3.5-9b", + hostLocalInferenceReceipt: serializedHostLocalInferenceReceipt(), + }), + }, + }); + await expect( + harness.setupInference( + "sandbox-a", + "remote-model", + "nvidia-prod", + null, + "NVIDIA_API_KEY", + null, + [], + ), + ).resolves.toEqual({ ok: true }); + + expect(harness.updateSandbox).toHaveBeenCalledWith( + "sandbox-a", + expect.objectContaining({ + provider: "nvidia-prod", + model: "remote-model", + hostLocalInferenceReceipt: null, + nativeNvidiaProviderAttachment: expect.objectContaining({ providerId: "native-identity" }), + }), + ); + expect(harness.commands.some(({ command }) => command.startsWith("inference set"))).toBe(false); + }); }); describe("native provider cleanup uncertainty", () => { @@ -310,3 +371,230 @@ it("stages explicit legacy Hermes recreation with a Hermes-scoped receipt", asyn expect(row).toEqual(before); expect(harness.commands.some(({ command }) => command.startsWith("inference set"))).toBe(false); }); + +it.each(["openai-completions", "openai-responses"] as const)( + "creates a sandbox-owned OpenClaw compatible %s provider before publishing its reservation", + async (api) => { + const harness = createDirectSetupInferenceHarness({ + overrides: { + hydrateCredentialEnv: () => "fixture-only-host-key", + isNonInteractive: () => true, + }, + }); + await expect( + harness.setupInference( + "test-box", + "served-alias", + "compatible-endpoint", + "http://127.0.0.1:8000/v1", + "HOST_COMPATIBLE_KEY", + null, + [], + { agentName: "openclaw", preferredInferenceApi: api }, + ), + ).resolves.toEqual({ ok: true }); + expect([...harness.native.authorities.values()]).toEqual([ + expect.objectContaining({ + agent: "openclaw", + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:8000/v1", + authMode: "authenticated", + }), + ]); + expect(harness.native.profiles[0]).toMatchObject({ + endpoints: [ + { + rules: [ + { allow: { method: "GET", path: "/v1/models" } }, + { + allow: { + method: "POST", + path: api === "openai-responses" ? "/v1/responses" : "/v1/chat/completions", + }, + }, + ], + }, + ], + }); + expect(harness.commands.some(({ command }) => command.startsWith("inference "))).toBe(false); + }, +); + +it.each([ + { agent: "openclaw", api: "openai-responses", expectedAgent: "openclaw" }, + { agent: "hermes", api: "openai-completions", expectedAgent: "hermes" }, + { agent: "hermes", api: "openai-responses", expectedAgent: "hermes" }, + { agent: "langchain-deepagents-code", api: "openai-completions", expectedAgent: undefined }, +])( + "registers $agent compatible inference on an accepted reserved-alias port", + async ({ agent, api, expectedAgent }) => { + const harness = createDirectSetupInferenceHarness({ + overrides: { + hydrateCredentialEnv: () => "fixture-only-host-key", + isNonInteractive: () => true, + }, + }); + await expect( + harness.setupInference( + "test-box", + "served-alias", + "compatible-endpoint", + "http://host.openshell.internal:42103/v1", + "HOST_COMPATIBLE_KEY", + null, + [], + { agentName: agent, preferredInferenceApi: api }, + ), + ).resolves.toEqual({ ok: true }); + expect([...harness.native.authorities.values()]).toEqual([ + expect.objectContaining({ + provider: "compatible-endpoint", + endpointUrl: "http://host.openshell.internal:42103/v1", + authMode: "authenticated", + }), + ]); + expect([...harness.native.authorities.values()][0]?.agent).toBe(expectedAgent); + expect(harness.native.profiles[0]).toMatchObject({ + endpoints: [ + { + port: 42103, + rules: [ + { allow: { method: "GET", path: "/v1/models" } }, + { + allow: { + method: "POST", + path: api === "openai-responses" ? "/v1/responses" : "/v1/chat/completions", + }, + }, + ], + }, + ], + }); + expect(harness.commands.some(({ command }) => command.startsWith("inference "))).toBe(false); + }, +); + +it.each(["openclaw", "hermes", "pi", "langchain-deepagents-code"])( + "registers Local NIM as a sentinel native provider for %s", + async (agent) => { + const nim = + require("../../src/lib/inference/nim.ts") as typeof import("../../src/lib/inference/nim"); + vi.spyOn(nim, "getServedModelId").mockReturnValue("local/nim-model"); + const getManagedVllmProviderBinding = vi.fn(() => ({ + baseUrl: "http://host.openshell.internal:18000/v1", + apiKey: "managed-vllm-secret", + })); + const harness = createDirectSetupInferenceHarness({ + overrides: { applyLocalInferenceRoute: undefined, getManagedVllmProviderBinding }, + }); + await expect( + harness.setupInference( + "test-box", + "local/nim-model", + "vllm-local", + nim.getNimInferenceEndpoints().baseUrl, + null, + null, + [], + { agentName: agent, localNimSelected: true }, + ), + ).resolves.toEqual({ ok: true }); + const attachment = [...harness.native.authorities.values()][0]; + expect(attachment).toMatchObject({ + provider: "vllm-local", + authMode: "sentinel", + endpointUrl: nim.getNimInferenceEndpoints().baseUrl, + sandboxName: "test-box", + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + }); + expect(attachment.agent).toBe(agent === "langchain-deepagents-code" ? undefined : agent); + expect(harness.native.adapter.createProvider.mock.calls[0][0].credentials).toEqual([ + { name: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", value: "dummy" }, + ]); + expect(getManagedVllmProviderBinding).not.toHaveBeenCalled(); + expect(harness.commands.some(({ command }) => command.startsWith("inference set"))).toBe(false); + }, +); + +describe("published native route backup boundary", () => { + it.each([ + { + agent: "openclaw", + model: "old-model", + staged: false, + key: "fixture-only-host-key", + reuse: false, + }, + { + agent: "openclaw", + model: "new-model", + staged: true, + key: "fixture-only-host-key", + reuse: false, + }, + { agent: "openclaw", model: "old-model", staged: false, key: null, reuse: true }, + { agent: "openclaw", model: "new-model", staged: true, key: null, reuse: true }, + { agent: "hermes", model: "old-model", staged: false, key: null, reuse: true }, + { agent: "hermes", model: "new-model", staged: true, key: null, reuse: true }, + ] as const)( + "preserves the source and provider credential when selecting $model", + async ({ agent, model, staged, key, reuse }) => { + const attachment = nativeLocalSetupReceipt({ + agent, + provider: "compatible-endpoint", + sandboxName: "test-box", + gatewayName: "nemoclaw", + endpointUrl: "http://host.openshell.internal:8000/v1", + authMode: "authenticated", + }); + const row = { + name: "test-box", + agent, + provider: "compatible-endpoint", + model: "old-model", + gatewayName: "nemoclaw", + endpointUrl: "http://127.0.0.1:8000/v1", + credentialEnv: "HOST_COMPATIBLE_KEY", + preferredInferenceApi: "openai-completions", + nativeLocalProviderAttachment: attachment, + }; + const before = structuredClone(row); + const harness = createDirectSetupInferenceHarness({ + overrides: { + getSandbox: () => row, + hydrateCredentialEnv: () => key, + isNonInteractive: () => true, + }, + }); + harness.native.seed(attachment); + const providers = structuredClone(harness.native.providers); + const result = await harness.setupInference( + "test-box", + model, + "compatible-endpoint", + row.endpointUrl, + row.credentialEnv, + null, + [], + { + agentName: agent, + preferredInferenceApi: "openai-completions", + deferPublishedRouteReservation: true, + reservationSessionId: "session", + reuseGatewayCredentialWithoutLocalKey: reuse, + skipHostInferenceSmoke: reuse, + }, + ); + expect(result).toEqual( + staged + ? { ok: true, stagedNativeLocalRecreation: { source: before, attachment } } + : { ok: true }, + ); + expect(row).toEqual(before); + expect(harness.updateSandbox).not.toHaveBeenCalled(); + expect(harness.native.adapter.updateProvider).not.toHaveBeenCalled(); + expect(harness.native.adapter.createProvider).not.toHaveBeenCalled(); + expect(harness.native.providers).toEqual(providers); + }, + ); +}); diff --git a/test/onboarding/onboard-openshell-install-stream.test.ts b/test/onboarding/onboard-openshell-install-stream.test.ts index 71d7b0efc08..dfbdd76fdf8 100644 --- a/test/onboarding/onboard-openshell-install-stream.test.ts +++ b/test/onboarding/onboard-openshell-install-stream.test.ts @@ -40,6 +40,10 @@ describe("runOpenshellInstall progress streaming (#4431)", () => { spawnSyncMock.mockReturnValue({ status: 0 }); runOpenshellInstall(makeDeps()); expect(spawnSyncMock).toHaveBeenCalledTimes(1); + expect(spawnSyncMock.mock.calls[0].slice(0, 2)).toEqual([ + "bash", + ["/fake/scripts/install-openshell.sh"], + ]); const options = spawnSyncMock.mock.calls[0][2]; expect(options.stdio).toEqual(["ignore", "inherit", "inherit"]); }); diff --git a/test/onboarding/onboard-openshell-version.test.ts b/test/onboarding/onboard-openshell-version.test.ts index ab0135b411a..43e62139da0 100644 --- a/test/onboarding/onboard-openshell-version.test.ts +++ b/test/onboarding/onboard-openshell-version.test.ts @@ -73,6 +73,9 @@ describe("OpenShell version helpers", () => { ["0.1.4", false], ["0.1.5", true], ["0.1.6", false], + ["0.1.7", true], + ["0.1.7-dev", false], + ["0.1.8", false], ["0.1.5-dev", false], ] as const)("classifies native projection for OpenShell %s as %s", (version, expected) => { expect(usesOpenShellNativeProjection(version)).toBe(expected); @@ -438,7 +441,7 @@ describe("computeOpenshellInstallEnv", () => { }, ); expect(result.env).toBe(null); - expect(errors.join("\n")).toContain("requires exact stable OpenShell 0.0.116"); + expect(errors.join("\n")).toContain("requires exact stable OpenShell 0.1.7"); }); it("overlays MIN/MAX/PIN env vars from blueprint when latest exceeds max", () => { diff --git a/test/onboarding/onboard-reservation-recreate.test.ts b/test/onboarding/onboard-reservation-recreate.test.ts index d6e597d8aff..8925667d9fb 100644 --- a/test/onboarding/onboard-reservation-recreate.test.ts +++ b/test/onboarding/onboard-reservation-recreate.test.ts @@ -522,7 +522,7 @@ const forwardService = fixtureMocks.installForwardServiceReachabilityFixture(); let createChild = null; runner.run = (command) => { const cmd = normalize(command); - const profile = fixtureMocks.mockEndpointlessProviderProfileRun(command, "nemoclaw-mcp-v1", false); + const profile = fixtureMocks.mockEndpointlessProviderProfileRun(command, "nemoclaw-messaging-v1", false); if (profile !== null) return profile; return createdSandbox.run(command) ?? { status: 0 }; }; diff --git a/test/onboarding/onboard-sandbox-create-failure.test.ts b/test/onboarding/onboard-sandbox-create-failure.test.ts index 2d4ae92f752..130d8cbd032 100644 --- a/test/onboarding/onboard-sandbox-create-failure.test.ts +++ b/test/onboarding/onboard-sandbox-create-failure.test.ts @@ -183,16 +183,19 @@ describe("sandbox create failure diagnostics", () => { ); }); - // OpenShell 0.0.116 prints supervisor startup failures only to the sandbox - // container's stderr; neither `openshell logs` nor sandbox status carry the - // reason (#13178). Remove once the OpenShell pin reaches 0.1.0. + // Preserve startup output when create fails before sandbox status is readable + // and rollback removes the container (#13178). it("captures, saves, and prints redacted sandbox container output through Docker (#13178)", () => { const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-create-failure-container-")); const homeDir = path.join(tmp, "home"); const containerId = "f".repeat(64); const keyedSecret = "nvapi-" + "A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6"; const opaqueSecret = "opaque-value-9f8e7d6c5b4a"; - docker.dockerRun.mockReturnValue({ status: 0, stdout: `${containerId}\n`, stderr: "" }); + docker.dockerRun.mockReturnValue({ + status: 0, + stdout: `${containerId}\topenshell\tdefault\tsandbox-id\tnamespace\t\n`, + stderr: "", + }); docker.dockerCapture.mockImplementation((args: readonly string[]) => args.includes("--format") ? "" diff --git a/test/onboarding/onboard-sandbox-recreation.test.ts b/test/onboarding/onboard-sandbox-recreation.test.ts index 25bea6f7c38..043f8a3ec02 100644 --- a/test/onboarding/onboard-sandbox-recreation.test.ts +++ b/test/onboarding/onboard-sandbox-recreation.test.ts @@ -169,7 +169,7 @@ const commands = []; let registeredSandbox = null; }, { sandboxId: sourceSandboxId }); runner.run = (command, opts = {}) => { const cmd = _n(command); - const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; if (cmd.includes("sandbox delete") && createdSandbox.state.lifecycleState === "created") createdSandbox.delete(); commands.push({ command: cmd, env: opts.env || null }); @@ -369,7 +369,7 @@ processRecovery.abortUnregisteredOpenClawPostRestoreDoctor = async () => ({ ok: const createdSandbox = fixtureMocks.createCreatedSandboxFixture({ lifecycleState: "created" }); runner.run = (command) => { const cmd = _n(command); - const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; if (cmd.includes("sandbox delete") && createdSandbox.state.lifecycleState === "created") createdSandbox.delete(); events.push({ kind: "run", cmd }); @@ -572,7 +572,7 @@ const events = []; const createdSandbox = fixtureMocks.createCreatedSandboxFixture({ lifecycleState: "created" }); runner.run = (command) => { const cmd = _n(command); - const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; if (cmd.includes("sandbox delete") && createdSandbox.state.lifecycleState === "created") createdSandbox.delete(); events.push({ kind: "run", cmd }); @@ -733,7 +733,7 @@ const createdSandbox = fixtureMocks.createCreatedSandboxFixture({ }); runner.run = (command) => { const cmd = _n(command); - const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; if (cmd.includes("sandbox delete") && createdSandbox.state.lifecycleState === "created") createdSandbox.delete(); events.push({ kind: "run", cmd }); @@ -919,7 +919,7 @@ const commands = []; const createdSandbox = fixtureMocks.createCreatedSandboxFixture({ lifecycleState: "created" }); runner.run = (command, opts = {}) => { const cmd = _n(command); - const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; if (cmd.includes("sandbox delete") && createdSandbox.state.lifecycleState === "created") createdSandbox.delete(); const commandString = Array.isArray(command) ? command.join(" ") : String(command); @@ -1139,7 +1139,7 @@ let reconciliationRecorded = null; process.on("exit", () => console.log(JSON.stringify({ commands, prompts, reconciliationRecorded, sourceSandboxId, sandboxId: createdSandbox.state.sandboxId }))); runner.run = (command, opts = {}) => { const cmd = _n(command); - const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; if (cmd.includes("sandbox delete") && ${journalOnly}) { reconciliationRecorded = require(${JSON.stringify(path.join(repoRoot, "src/lib/state/onboard-session.ts"))}).loadSession()?.checkpoint?.sandboxRecreate?.reconcileOpenClawInference === true; @@ -1347,7 +1347,7 @@ const createdSandbox = fixtureMocks.createCreatedSandboxFixture({ }); runner.run = (command, opts = {}) => { const cmd = _n(command); - const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockProviderPreparationRun(command, "nemoclaw", "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; if (cmd.includes("sandbox delete") && createdSandbox.state.lifecycleState === "created") createdSandbox.delete(); commands.push({ command: cmd, env: opts.env || null }); diff --git a/test/onboarding/onboard-terminal-dashboard.test.ts b/test/onboarding/onboard-terminal-dashboard.test.ts index 00ade5e9a3d..77df6e11ab4 100644 --- a/test/onboarding/onboard-terminal-dashboard.test.ts +++ b/test/onboarding/onboard-terminal-dashboard.test.ts @@ -93,7 +93,7 @@ const managedProviderResult = (normalized) => { status: 0, stdout: [ "Name: " + providerName, - "Type: nemoclaw-mcp-v1", + "Type: nemoclaw-messaging-v1", "Credential keys: " + credential, "Config keys: ", ].join("\n"), diff --git a/test/onboarding/onboard.test.ts b/test/onboarding/onboard.test.ts index 1c634135b7e..7512d719ba3 100644 --- a/test/onboarding/onboard.test.ts +++ b/test/onboarding/onboard.test.ts @@ -738,7 +738,7 @@ runner.run = (command, opts = {}) => { fs.writeFileSync(path.join(command.at(-1), "config.json"), JSON.stringify({ provider: "nvidia-prod", model: "gpt-5.4" })); return { status: 0 }; } - const profileResult = fixtureMocks.mockEndpointlessProviderProfileRun(command, "nemoclaw-mcp-v1", false); + const profileResult = fixtureMocks.mockEndpointlessProviderProfileRun(command, "nemoclaw-messaging-v1", false); if (profileResult !== null) return profileResult; return existingSandbox.run(sandboxCommand(command)) ?? { status: 0 }; }; diff --git a/test/onboarding/openshell-0.0.101-migration-review.test.ts b/test/onboarding/openshell-0.0.101-migration-review.test.ts index 0c7e189286e..347ba1ea386 100644 --- a/test/onboarding/openshell-0.0.101-migration-review.test.ts +++ b/test/onboarding/openshell-0.0.101-migration-review.test.ts @@ -63,9 +63,9 @@ describe("OpenShell 0.0.101 executable contracts", () => { }), ); - expect(dockerToml).toContain('compute_drivers = ["docker"]'); + expect(dockerToml).toContain('compute_driver = "docker"'); expect(dockerToml).toContain("[openshell.drivers.docker]"); - expect(podmanToml).toContain('compute_drivers = ["podman"]'); + expect(podmanToml).toContain('compute_driver = "podman"'); expect(podmanToml).toContain("[openshell.drivers.podman]"); expect(podmanToml).toContain('socket_path = "/run/user/1001/podman/podman.sock"'); const toml = ({ Docker: dockerToml, Podman: podmanToml } as const)[scenario]!; diff --git a/test/onboarding/openshell-0.0.85-migration-review.test.ts b/test/onboarding/openshell-0.0.85-migration-review.test.ts index 809c7353936..a7536a5fe01 100644 --- a/test/onboarding/openshell-0.0.85-migration-review.test.ts +++ b/test/onboarding/openshell-0.0.85-migration-review.test.ts @@ -7,16 +7,11 @@ import { describe, expect, it } from "vitest"; const repoRoot = path.resolve(import.meta.dirname, "../.."); describe("OpenShell migration executable contracts", () => { - it("keeps exact-main runtime proofs separate from upstream-only fault injection", () => { + it("wires native runtime and MCP privacy proofs without upstream fault injection", () => { const helper = fs.readFileSync( path.join(repoRoot, "test", "e2e", "live", "openshell-exact-main-runtime-contracts.ts"), "utf8", ); - expect(helper).toContain('chain.policy !== "accept"'); - expect(helper).toContain("expected exactly one ${family} ${protocol} reject rule"); - expect(helper).toContain("DIRECT_BYPASS_PROBE_CODE"); - expect(helper).toContain('sha256sum \\"/proc/$old_pid/exe\\"'); - expect(helper).toContain('[ "$new_status" = 403 ]'); expect(helper).toContain("line.includes(`tools=${options.expectedTool}`)"); expect(helper).toContain("not.toMatch(/\\barguments\\b[\"']?\\s*[:=]/iu)"); expect(helper).not.toContain("iptables -F"); @@ -26,7 +21,7 @@ describe("OpenShell migration executable contracts", () => { path.join(repoRoot, "test", "e2e", "live", "openshell-exact-main-mcp-proof.ts"), "utf8", ); - expect(mcpProof).toContain("assertExactMainPolicyNftAndIdentityContracts({"); + expect(mcpProof).toContain("assertExactMainPolicyAndIdentityContracts({"); expect(mcpProof).toContain("assertExactMainMcpLogPrivacy({"); }); @@ -36,8 +31,9 @@ describe("OpenShell migration executable contracts", () => { "utf8", ); expect(helper).toContain("parse as parseToml"); - expect(helper).toContain('expect(gateway.compute_drivers).toEqual(["docker"])'); - expect(helper).toContain("Object.keys(drivers)"); + expect(helper).toContain('expect(gateway.compute_driver).toBe("docker")'); + expect(helper).toContain("expect(openshell.version).toBe(2)"); + expect(helper).toContain('requireRecord(drivers.docker, "openshell.drivers.docker")'); expect(helper).toContain("fs.realpathSync(`/proc/${gatewayPid}/exe`)"); expect(helper).toContain('["-H", "-ltnp"]'); expect(helper).toContain('"{{json .HostConfig.Binds}}"'); @@ -54,9 +50,13 @@ describe("OpenShell migration executable contracts", () => { expect(mcpProof).toContain("driverConfig.assertAfterRebuild()"); expect(mcpProof).toContain("from deepagents_code import _nemoclaw_managed as managed"); expect(mcpProof).not.toContain("/opt/nemoclaw-deepagents-code/managed-dcode-runtime.py"); - expect(mcpProof).toContain("def reject_primary_tmpfile(path, flags, *args, **kwargs):"); - expect(mcpProof).toContain("if directory == managed._MCP_ANONYMOUS_DIRECTORY:"); - expect(mcpProof).toContain("managed._MCP_PRIVATE_ANONYMOUS_DIRECTORY,"); + expect(mcpProof).toContain("managed._validate_private_managed_mcp_tmpfs()"); + expect(mcpProof).toContain( + "managed._anonymous_managed_mcp_snapshot_at(\n payload, managed._MCP_PRIVATE_ANONYMOUS_DIRECTORY", + ); + expect(mcpProof).toContain( + "managed._read_bound_managed_mcp_descriptor(descriptor, binding) == payload", + ); }); it("keeps current stable selectors aligned", () => { const blueprint = fs.readFileSync( @@ -71,15 +71,15 @@ describe("OpenShell migration executable contracts", () => { "lib", "actions", "sandbox", - "openshell-child-visible-credentials.v0.0.116.json", + "openshell-child-visible-credentials.v0.1.7.json", ), "utf8", ), ) as { openshellVersion: string }; - expect(blueprint).toContain('min_openshell_version: "0.0.116"'); - expect(blueprint).toContain('max_openshell_version: "0.0.116"'); - expect(manifest.openshellVersion).toBe("0.0.116"); + expect(blueprint).toContain('min_openshell_version: "0.1.7"'); + expect(blueprint).toContain('max_openshell_version: "0.1.7"'); + expect(manifest.openshellVersion).toBe("0.1.7"); }); it("does not reintroduce newline-only code transports at migrated consumers", () => { const migratedConsumers = [ diff --git a/test/onboarding/openshell-sdk-export-reads.test.ts b/test/onboarding/openshell-sdk-export-reads.test.ts index 140d89d13b8..6861818a861 100644 --- a/test/onboarding/openshell-sdk-export-reads.test.ts +++ b/test/onboarding/openshell-sdk-export-reads.test.ts @@ -26,6 +26,61 @@ function hasSdkArtifact(): boolean { } describe("released OpenShell SDK export reads", () => { + it.skipIf(!hasSdkArtifact()).each([ + [0, 0, 0, {}], + [1, 0, 0, { tls: "skip" }], + [0, 1, 1, { enforcement: "enforce", access: "read-only" }], + [0, 2, 2, { enforcement: "audit", access: "read-write" }], + [0, 1, 3, { enforcement: "enforce", access: "full" }], + ] as const)( + "exports SDK policy modes %i/%i/%i without runtime ownership markers", + async (tls, enforcement, access, expected) => { + const sdkPackage = "@nvidia/openshell-sdk/raw"; + const protobufPackage = "@bufbuild/protobuf"; + const [raw, { create }] = await Promise.all([import(sdkPackage), import(protobufPackage)]); + const policy = create(raw.SandboxPolicySchema, { + version: 1, + networkPolicies: { + api: { + endpoints: [ + { + host: "api.example", + port: 443, + tls, + enforcement, + access, + advisorProposed: true, + providerCredentialed: true, + tokenGrantOwner: "provider-owned-endpoint", + }, + ], + }, + }, + }); + const document = YAML.parse(await serializeSdkPolicy(policy)); + expect(document.network_policies.api.endpoints).toEqual([ + { host: "api.example", port: 443, ...expected }, + ]); + }, + ); + + it.skipIf(!hasSdkArtifact()).each([ + ["tls", 2], + ["tls", 3], + ["tls", 99], + ["enforcement", 99], + ["access", 99], + ] as const)("rejects unsupported SDK policy mode %s=%i", async (field, value) => { + const sdkPackage = "@nvidia/openshell-sdk/raw"; + const protobufPackage = "@bufbuild/protobuf"; + const [raw, { create }] = await Promise.all([import(sdkPackage), import(protobufPackage)]); + const policy = create(raw.SandboxPolicySchema, { + version: 1, + networkPolicies: { api: { endpoints: [{ host: "api.example", [field]: value }] } }, + }); + await expect(serializeSdkPolicy(policy)).rejects.toMatchObject({ kind: "schema" }); + }); + it.skipIf(!hasSdkArtifact()).each(["brave"] as const)( "qualifies the checked-in %s profile through generated SDK responses (#10904)", async (profileId) => { @@ -47,7 +102,12 @@ describe("released OpenShell SDK export reads", () => { scope: "workspace", resourceVersion: "4", credentials: checkedIn.credentials, - endpoints: checkedIn.endpoints, + endpoints: checkedIn.endpoints.map((endpoint: Record) => ({ + ...endpoint, + access: + raw.NetworkAccessPreset[String(endpoint.access).replaceAll("-", "_").toUpperCase()], + enforcement: raw.NetworkEnforcementMode[String(endpoint.enforcement).toUpperCase()], + })), binaries: checkedIn.binaries.map((path: string) => ({ path })), inference_capable: checkedIn.inference_capable, }); @@ -78,14 +138,16 @@ describe("released OpenShell SDK export reads", () => { }, }, }; - const result = await createProviders(async () => client).get({ - target: { kind: "named", gatewayName: "nemoclaw" }, + const request = { + target: { kind: "named", gatewayName: "nemoclaw" } as const, workspace: "default", name: "alpha", configKeys: [], profileContract: profileId, signal: new AbortController().signal, - }); + }; + const providers = createProviders(async () => client); + const result = await providers.get(request); expect(result?.managedProfile).toEqual({ id: profileId, source: "user", @@ -93,6 +155,9 @@ describe("released OpenShell SDK export reads", () => { resourceVersion: "4", }); expect(result?.profileWorkspace).toBe("default"); + // Decoding an invalid access preset must not qualify the shipped profile. + profile.endpoints[0].access = 0; + await expect(providers.get(request)).rejects.toMatchObject({ kind: "schema" }); }, ); @@ -105,7 +170,7 @@ describe("released OpenShell SDK export reads", () => { async (providerType, profileWorkspace) => { const sdkPackage = "@nvidia/openshell-sdk/raw"; const protobufPackage = "@bufbuild/protobuf"; - const [raw, { create, toBinary, fromBinary }] = await Promise.all([ + const [raw, { create, toBinary, fromBinary, toJson }] = await Promise.all([ import(sdkPackage), import(protobufPackage), ]); @@ -121,7 +186,17 @@ describe("released OpenShell SDK export reads", () => { const client: OpenShellReadClient = { raw: { getProviderProfile: async (request) => { - expect(request).toEqual({ id: "nvidia", workspace: profileWorkspace }); + expect( + toJson( + raw.OpenShell.method.getProviderProfile.input, + roundTrip(raw.OpenShell.method.getProviderProfile.input, request), + ), + ).toEqual({ + id: "nvidia", + ...(profileWorkspace === "" + ? {} + : { workspaceScope: { workspace: profileWorkspace } }), + }); return roundTrip(raw.OpenShell.method.getProviderProfile.output, { profile: { id: "nvidia", @@ -131,8 +206,17 @@ describe("released OpenShell SDK export reads", () => { }, }); }, - getProvider: async () => - roundTrip(raw.OpenShell.method.getProvider.output, { + getProvider: async (input) => { + expect( + toJson( + raw.OpenShell.method.getProvider.input, + roundTrip(raw.OpenShell.method.getProvider.input, input), + ), + ).toEqual({ + name: "alpha", + workspaceScope: { workspace: "default" }, + }); + return roundTrip(raw.OpenShell.method.getProvider.output, { provider: { metadata, type: providerType, @@ -141,17 +225,34 @@ describe("released OpenShell SDK export reads", () => { config: providerType === "nvidia" ? {} : { OPENAI_BASE_URL: "https://api.example/v1" }, }, - }), - getSandbox: async () => - roundTrip(raw.OpenShell.method.getSandbox.output, { + }); + }, + getSandbox: async (input) => { + expect( + toJson( + raw.OpenShell.method.getSandbox.input, + roundTrip(raw.OpenShell.method.getSandbox.input, input), + ), + ).toEqual({ + name: "alpha", + workspaceScope: { workspace: "default" }, + }); + return roundTrip(raw.OpenShell.method.getSandbox.output, { sandbox: { metadata, spec: { template: { image: "image@sha256:" + "a".repeat(64) } }, status: { currentPolicyVersion: 3 }, }, - }), - getSandboxConfig: async () => - roundTrip(raw.GetSandboxConfigResponseSchema, { + }); + }, + getSandboxConfig: async (input) => { + expect( + toJson( + raw.GetSandboxConfigRequestSchema, + roundTrip(raw.GetSandboxConfigRequestSchema, input), + ), + ).toEqual({ name: "alpha", workspaceScope: { workspace: "default" } }); + return roundTrip(raw.GetSandboxConfigResponseSchema, { policy: { version: 1, filesystem: { readOnly: ["/usr"] }, @@ -169,7 +270,8 @@ describe("released OpenShell SDK export reads", () => { policySource: 1, configRevision: 9007199254740993n, providerEnvRevision: 18446744073709551615n, - }), + }); + }, }, }; const connect = async () => client; @@ -579,7 +681,18 @@ describe.skipIf(!hasSdkArtifact())("released OpenShell policy document conversio raw: { getProvider: async () => undefined, getProviderProfile: async () => undefined, - getSandbox: async () => undefined, + getSandbox: async () => ({ + sandbox: { + metadata: { + id: "verified-id", + name: "alpha", + workspace: "default", + resourceVersion: 1n, + }, + status: { currentPolicyVersion: 3 }, + spec: { template: { image: "fixture-image" }, providers: [] }, + }, + }), getSandboxConfig: async () => ({ policy, workspace: "default", @@ -594,6 +707,7 @@ describe.skipIf(!hasSdkArtifact())("released OpenShell policy document conversio })); const config = await reader.get({ sandboxId: "verified-id", + name: "alpha", target: { kind: "named", gatewayName: "nemoclaw" }, workspace: "default", signal: new AbortController().signal, diff --git a/test/onboarding/portable-ollama-native-route.test.ts b/test/onboarding/portable-ollama-native-route.test.ts new file mode 100644 index 00000000000..5c21a7ede0c --- /dev/null +++ b/test/onboarding/portable-ollama-native-route.test.ts @@ -0,0 +1,170 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import { createDirectSetupInferenceHarnessFactory } from "../support/setup-inference-test-harness"; +import { createHermesPortableInferenceFixture } from "../helpers/hermes-portable-onboarding-fixture"; +import type { SetupInferenceDeps, SetupInference } from "../../src/lib/onboard/setup-inference"; +import { prepareHermesPortableOllamaProviderRetirement } from "../../src/lib/onboard/experimental/hermes-portable-ollama-gateway-transaction"; +import type { NativeLocalProviderAttachment } from "../../src/lib/inference/native-local/profile"; +import { nativeLocalIdentity } from "../../src/lib/inference/native-local/contract"; + +const onboard = require("../../src/lib/onboard") as { + createSetupInference: (overrides?: Partial) => SetupInference; +}; +const createHarness = createDirectSetupInferenceHarnessFactory(onboard.createSetupInference); +const input = { + gatewayName: "nemoclaw", + sandboxName: "portable-hermes", + provider: "ollama-local", + model: "qwen3-vl:4b", + providerBaseUrl: "http://host.openshell.internal:11434/v1", +} as const; +function journalFile(fixture: ReturnType) { + const root = path.join(fixture.homeDir, "state", "portable-inference"); + return path.join(root, fs.readdirSync(root)[0]!, "portable-gateway-provider.json"); +} + +describe("Portable Ollama native publication", () => { + it("publishes the transaction-owned native attachment through complete inference setup", async () => { + const fixture = createHermesPortableInferenceFixture(); + const selection = fixture.resolve()!; + const updateSandbox = vi.fn(() => true); + const harness = createHarness({ overrides: { updateSandbox } }); + await expect( + harness.setupInference(input.sandboxName, input.model, input.provider, null, null, null, [], { + agentName: "hermes", + hostLocalInference: selection, + }), + ).resolves.toEqual({ ok: true }); + const journal = JSON.parse(fs.readFileSync(journalFile(fixture), "utf8")); + const row = updateSandbox.mock.calls.at(-1)![1]; + const attachment = row.nativeLocalProviderAttachment!; + expect(attachment).toMatchObject({ + ...nativeLocalIdentity(attachment), + transactionId: journal.intent.transactionId, + providerId: journal.providerAuthority.id, + credentialEnv: "NEMOCLAW_LOCAL_INFERENCE_TOKEN", + endpointUrl: input.providerBaseUrl, + agent: "hermes", + authMode: "sentinel", + }); + expect(journal.phase).toBe("committed"); + expect(harness.verifyInferenceRoute).not.toHaveBeenCalled(); + expect(harness.commands.some(({ command }) => command.includes("inference set"))).toBe(false); + expect( + fixture.events.some((event) => event.includes("--publish 169.254.2.2:11434:11434")), + ).toBe(true); + }); + + it("rolls back a failed setup through its journal without generic native retirement", async () => { + const fixture = createHermesPortableInferenceFixture(); + const selection = fixture.resolve()!; + const genericRetirement = vi.fn(() => { + throw new Error("unowned generic cleanup"); + }); + const harness = createHarness({ + overrides: { clearNativeLocalProviderAuthority: genericRetirement }, + }); + const ownedSelection = { + ...selection, + prepareGatewayMutation: async ( + value: Parameters[0], + ) => { + const mutation = await selection.prepareGatewayMutation(value); + return { + ...mutation, + commit() { + throw new Error("injected prepublication failure"); + }, + }; + }, + }; + await expect( + harness.setupInference(input.sandboxName, input.model, input.provider, null, null, null, [], { + agentName: "hermes", + hostLocalInference: ownedSelection, + }), + ).rejects.toThrow("EXIT_CALLED:1"); + expect(genericRetirement).not.toHaveBeenCalled(); + expect(fixture.gatewayProvider.isPresent()).toBe(false); + expect(fixture.harness.container()).toBeNull(); + expect(JSON.parse(fs.readFileSync(journalFile(fixture), "utf8")).phase).toBe("rolled-back"); + }); + + it.each([ + { + change: "profile", + mutate: ( + fixture: ReturnType, + _attachment: NativeLocalProviderAttachment, + ) => fixture.gatewayProvider.setProfileState("incompatible"), + }, + { + change: "id", + mutate: ( + fixture: ReturnType, + attachment: NativeLocalProviderAttachment, + ) => + fixture.gatewayProvider.setIdentity( + attachment.providerName, + attachment.profileId, + "replacement", + ), + }, + { + change: "name", + mutate: ( + fixture: ReturnType, + attachment: NativeLocalProviderAttachment, + ) => fixture.gatewayProvider.setIdentity("foreign-provider", attachment.profileId), + }, + { + change: "revision", + mutate: ( + fixture: ReturnType, + _attachment: NativeLocalProviderAttachment, + ) => fixture.gatewayProvider.bumpResourceVersion(), + }, + ])("preserves a changed native $change during rollback", async ({ mutate }) => { + const fixture = createHermesPortableInferenceFixture(); + const mutation = await fixture.resolve()!.prepareGatewayMutation(input); + const attachment = await mutation.registerNativeProvider!(); + mutate(fixture, attachment); + await expect(mutation.rollback()).rejects.toThrow(); + expect(fixture.gatewayProvider.isPresent()).toBe(true); + expect(fixture.events.some((event) => event.includes("provider delete"))).toBe(false); + }); + + it("retains legacy credential-key proof for uninstall and refuses native adoption", async () => { + const fixture = createHermesPortableInferenceFixture(); + const mutation = await fixture.resolve()!.prepareGatewayMutation(input); + await mutation.registerNativeProvider!(); + const file = journalFile(fixture); + const journal = JSON.parse(fs.readFileSync(file, "utf8")); + journal.phase = "committed"; + journal.intent.type = "openai"; + journal.intent.providerCredentialEnv = `${journal.intent.credentialEnv}_${journal.intent.transactionId.toUpperCase()}`; + fs.writeFileSync(file, `${JSON.stringify(journal)}\n`); + fixture.gatewayProvider.setIdentity("ollama-local", "openai"); + fixture.gatewayProvider.setCredentialEnv(journal.intent.providerCredentialEnv); + expect(() => fixture.resolve()).toThrow(); + const options = { + directory: path.dirname(file), + ...input, + transactionId: journal.intent.transactionId, + targetSha256: journal.intent.targetSha256, + credentialEnv: journal.intent.credentialEnv, + runGatewayOpenshell: fixture.gatewayProvider.run, + }; + fixture.gatewayProvider.setCredentialEnv("FOREIGN_LEGACY_TOKEN"); + expect(() => prepareHermesPortableOllamaProviderRetirement(options)).toThrow(); + fixture.gatewayProvider.setCredentialEnv(journal.intent.providerCredentialEnv); + const retirement = prepareHermesPortableOllamaProviderRetirement(options); + await retirement.removeAndVerify(); + retirement.verifyAbsent(); + expect(fixture.gatewayProvider.isPresent()).toBe(false); + }); +}); diff --git a/test/onboarding/validate-config-schemas.test.ts b/test/onboarding/validate-config-schemas.test.ts index d4bf2cd41cd..312ff91ca0b 100644 --- a/test/onboarding/validate-config-schemas.test.ts +++ b/test/onboarding/validate-config-schemas.test.ts @@ -516,8 +516,15 @@ describe("sandbox-policy.schema.json", () => { }, }; - it("accepts a minimal sandbox policy", () => { - expectValid(validate, validSandboxPolicy, "minimal sandbox policy"); + it.each([ + ["minimal", {}, true], + ["hard_requirement", { landlock: { compatibility: "hard_requirement" } }, true], + ["best_effort", { landlock: { compatibility: "best_effort" } }, true], + ["strict", { landlock: { compatibility: "strict" } }, false], + ["unknown", { landlock: { compatibility: "unknown" } }, false], + ] as const)("validates the %s sandbox policy variant", (_label, properties, accepted) => { + const result = validate({ ...validSandboxPolicy, ...properties }); + expect(result, JSON.stringify(validate.errors)).toBe(accepted); }); it("rejects policy with missing network_policies", () => { diff --git a/test/onboarding/vllm-export-docker-format.test.ts b/test/onboarding/vllm-export-docker-format.test.ts index c62a651c0e6..695eee978e0 100644 --- a/test/onboarding/vllm-export-docker-format.test.ts +++ b/test/onboarding/vllm-export-docker-format.test.ts @@ -45,6 +45,14 @@ describe.skipIf(!dockerClientAvailable)("managed vLLM Docker format boundary", ( }); it.each([ + [ + "extra published bridge binding", + (f: Fixture) => + f.objects.container.NetworkSettings.Ports["8000/tcp"].push({ + HostIp: "172.18.0.1", + HostPort: "18000", + }), + ], [ "additional capability", (f: Fixture) => Object.assign(f.objects.container.HostConfig, { CapAdd: ["SYS_ADMIN"] }), diff --git a/test/onboarding/vllm-export-format-fixture.ts b/test/onboarding/vllm-export-format-fixture.ts index fb7b6d642d8..338660e724f 100644 --- a/test/onboarding/vllm-export-format-fixture.ts +++ b/test/onboarding/vllm-export-format-fixture.ts @@ -118,10 +118,7 @@ export function vllmExportFormatFixture(directory: string) { ], NetworkSettings: { Ports: { - "8000/tcp": [ - { HostIp: "127.0.0.1", HostPort: "18000" }, - { HostIp: "172.18.0.1", HostPort: "18000" }, - ], + "8000/tcp": [{ HostIp: "127.0.0.1", HostPort: "18000" }], }, }, }, diff --git a/test/package-contract/blueprint-external-target-plan.test.ts b/test/package-contract/blueprint-external-target-plan.test.ts index 98f06cf667e..7c9f4b9492c 100644 --- a/test/package-contract/blueprint-external-target-plan.test.ts +++ b/test/package-contract/blueprint-external-target-plan.test.ts @@ -300,12 +300,12 @@ describe.sequential("packaged Blueprint Runner external target", () => { blueprintFile = path.join(blueprintRoot, "blueprint.yaml"); validBlueprintDocument = { version: "1.0.0", - min_openshell_version: "0.0.116", - max_openshell_version: "0.0.116", + min_openshell_version: "0.1.7", + max_openshell_version: "0.1.7", openshell_target: { endpoint: "https://192.0.2.1:8443", workspace: "default", - expected_release: "0.0.116", + expected_release: "0.1.7", lifecycle: "external", trust: { ca_file: privateCaPath }, authentication: { credential_file: privateAuthenticationPath }, @@ -466,7 +466,7 @@ describe.sequential("packaged Blueprint Runner external target", () => { // Result expect(execution.result.status, execution.safeDiagnostics).toBe(1); expect(execution.result.stderr).toContain( - "external OpenShell target expected_release must be 0.0.116", + "external OpenShell target expected_release must be 0.1.7", ); expect(execution.result.stdout).not.toContain("openshell_target"); expect(execution.evidence).toEqual({ effects: [] }); @@ -490,7 +490,7 @@ describe.sequential("packaged Blueprint Runner external target", () => { openshell_target: { endpoint: "https://192.0.2.1:8443", workspace: "default", - expected_release: "0.0.116", + expected_release: "0.1.7", lifecycle: "external", authentication_source: "file", ca_fingerprint: expectedFingerprint, diff --git a/test/package-contract/blueprint-external-target-timeout.test.ts b/test/package-contract/blueprint-external-target-timeout.test.ts index daf089f0c2c..8cf07b854e3 100644 --- a/test/package-contract/blueprint-external-target-timeout.test.ts +++ b/test/package-contract/blueprint-external-target-timeout.test.ts @@ -178,12 +178,12 @@ function writeExternalBlueprint( path.join(blueprintRoot, "blueprint.yaml"), YAML.stringify({ version: "1.0.0", - min_openshell_version: "0.0.116", - max_openshell_version: "0.0.116", + min_openshell_version: "0.1.7", + max_openshell_version: "0.1.7", openshell_target: { endpoint: `https://${address}:${String(port)}`, workspace: "default", - expected_release: "0.0.116", + expected_release: "0.1.7", lifecycle: "external", trust: { ca_file: caPath }, authentication: { credential_file: authenticationPath }, diff --git a/test/package-contract/lifecycle-api.test.ts b/test/package-contract/lifecycle-api.test.ts index 49166ba6148..b83160afdf0 100644 --- a/test/package-contract/lifecycle-api.test.ts +++ b/test/package-contract/lifecycle-api.test.ts @@ -196,7 +196,7 @@ void (async () => { target: { gatewayIdentity: digest("1"), workspace: "hermes-workspace", - openshellVersion: "0.0.116", + openshellVersion: "0.1.5", }, sandbox: { name: "hermes-agent", @@ -305,7 +305,7 @@ const observer: OpenShellHermesAgentObserver = { target: { gatewayIdentity: digest("1"), workspace: "hermes-workspace", - openshellVersion: "0.0.116", + openshellVersion: "0.1.5", }, sandbox: { name: "hermes-agent", diff --git a/test/package-contract/messaging-provider-profile-path.test.ts b/test/package-contract/messaging-provider-profile-path.test.ts index b917c93c558..e4e74d86ee7 100644 --- a/test/package-contract/messaging-provider-profile-path.test.ts +++ b/test/package-contract/messaging-provider-profile-path.test.ts @@ -22,7 +22,12 @@ describe("compiled messaging credential profile path", () => { expect(repositoryRoot.REPOSITORY_ROOT).toBe(REPOSITORY_ROOT); expect(profile.messagingCredentialProviderProfilePath(repositoryRoot.REPOSITORY_ROOT)).toBe( - path.join(REPOSITORY_ROOT, "nemoclaw-blueprint", "provider-profiles", "nemoclaw-mcp-v1.yaml"), + path.join( + REPOSITORY_ROOT, + "nemoclaw-blueprint", + "provider-profiles", + "nemoclaw-messaging-v1.yaml", + ), ); }); }); diff --git a/test/package-contract/openshell-policy-boundary.test.ts b/test/package-contract/openshell-policy-boundary.test.ts index c0e4849e73c..0e4764ef4f7 100644 --- a/test/package-contract/openshell-policy-boundary.test.ts +++ b/test/package-contract/openshell-policy-boundary.test.ts @@ -149,9 +149,8 @@ describe("OpenShell policy boundary package contract", () => { const globalMetadata = JSON.stringify({ scope: "global", status: "loaded", - policy_source: "global", hash: "sha256:global", - active_version: 1, + version: 1, policy: { version: 1, network_policies: {} }, }); const expectedGlobalMetadata = { @@ -424,7 +423,7 @@ process.stdout.write("validated"); "openshell-policy-boundary.cjs", ); expect(auditOpenShellPolicyBoundaryDependencies(fs.readFileSync(boundaryPath, "utf8"))).toEqual( - ["node:util", "yaml"], + ["node:net", "node:util", "yaml"], ); expect(() => diff --git a/test/package-contract/openshell-sdk-install.test.ts b/test/package-contract/openshell-sdk-install.test.ts index 774f9e6e524..52f5fb9b6c7 100644 --- a/test/package-contract/openshell-sdk-install.test.ts +++ b/test/package-contract/openshell-sdk-install.test.ts @@ -45,6 +45,7 @@ function fixture() { const env = { PATH: process.env.PATH, HOME: root, + TMPDIR: root, NPM_CONFIG_CACHE: path.join(root, "cache"), NPM_CONFIG_USERCONFIG: path.join(root, "empty.npmrc"), NPM_CONFIG_GLOBALCONFIG: path.join(root, "empty-global.npmrc"), @@ -125,10 +126,18 @@ describe("required OpenShell SDK installation", () => { }; const packed = spawnSync( "npm", - ["pack", "--ignore-scripts", "--silent", "--json", "--pack-destination", root], + ["pack", "--ignore-scripts", "--silent", "--pack-destination", root], { cwd: repositoryRoot, env, encoding: "utf8", timeout: 30_000 }, ); - expect(packed.status, packed.stderr).toBe(0); + expect( + packed.status, + JSON.stringify({ + stdout: packed.stdout, + stderr: packed.stderr, + error: packed.error?.message, + signal: packed.signal, + }), + ).toBe(0); const archives = readdirSync(root).filter((entry) => entry.endsWith(".tgz")); expect(archives).toHaveLength(1); const archivePath = path.join(root, archives[0]!); @@ -245,10 +254,7 @@ console.log(JSON.stringify([typeof sdk.OpenShellClient.connect, raw.SandboxPolic it("rejects a changed archive before creating an npm cache", () => { const { root, probe } = fixture(); - const archive = path.join( - root, - "scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.0.116.tgz", - ); + const archive = path.join(root, "scripts/vendor/openshell-sdk/nvidia-openshell-sdk-0.1.7.tgz"); writeFileSync(archive, "corrupted archive"); const result = probe("prepare"); expect(result.status).toBe(1); @@ -260,7 +266,7 @@ console.log(JSON.stringify([typeof sdk.OpenShellClient.connect, raw.SandboxPolic const { root, probe } = fixture(); const configPath = path.join(root, "ci", "reviewed-npm-audit.json"); const config = JSON.parse(readFileSync(configPath, "utf8")); - config.sourceRegistryPackage.integrity = "sha512-reviewed-identity-drift"; + config.sourceRegistryPackageReplacement.integrity = "sha512-reviewed-identity-drift"; writeFileSync(configPath, JSON.stringify(config)); const result = probe("prepare"); diff --git a/test/repository/cli-coverage-sequencer.test.ts b/test/repository/cli-coverage-sequencer.test.ts index b432bb1b3b5..30aba1be1a0 100644 --- a/test/repository/cli-coverage-sequencer.test.ts +++ b/test/repository/cli-coverage-sequencer.test.ts @@ -7,10 +7,6 @@ import path from "node:path"; import { describe, expect, it } from "vitest"; import type { TestSpecification, Vitest } from "vitest/node"; -import { - discoverVitestCandidates, - expectedProjectForTestPath, -} from "../../scripts/checks/vitest-project-overlap.mts"; import { assignStableShards, CliCoverageSequencer, @@ -48,22 +44,6 @@ function sequencer(index: number, count: number): CliCoverageSequencer { } as unknown as Vitest); } -function currentCliCoverageEntries(): WeightedShardEntry[] { - const coverageProjects = new Set(["cli", "integration", "e2e-support"]); - return [...discoverVitestCandidates()].flatMap((file) => { - const projectName = expectedProjectForTestPath(file); - return projectName && coverageProjects.has(projectName) - ? [ - { - key: `${projectName}:${file}`, - weightMs: timingWeightForPath(file), - value: file, - }, - ] - : []; - }); -} - describe("stable CLI coverage sharding", () => { it("assigns every file exactly once and independently of discovery order", () => { const entries = [ @@ -131,18 +111,6 @@ describe("stable CLI coverage sharding", () => { // arbitrary roster additions. Use retained CI timings to assess performance; // exact-once assignment and roster-independent ownership are tested above. - it("balances the serialized integration lane across the twelve CI shards (#6237)", () => { - const integrationEntries = currentCliCoverageEntries().filter((entry) => - entry.key.startsWith("integration:"), - ); - expect(integrationEntries.length).toBeGreaterThan(0); - - const weights = assignStableShards(integrationEntries, 12).map((shard) => shard.totalWeightMs); - const averageWeight = weights.reduce((total, weight) => total + weight, 0) / weights.length; - - expect(Math.max(...weights)).toBeLessThanOrEqual(averageWeight * 1.1); - }); - it("uses stable sharding only for projects owned by the CLI coverage matrix", () => { expect(shouldUseCliCoverageSharding(["cli", "integration", "e2e-support"])).toBe(true); expect(shouldUseCliCoverageSharding(["integration"])).toBe(true); diff --git a/test/repository/dependency-pins-check.test.ts b/test/repository/dependency-pins-check.test.ts index ca4bd290bce..cdb673aff47 100644 --- a/test/repository/dependency-pins-check.test.ts +++ b/test/repository/dependency-pins-check.test.ts @@ -231,13 +231,16 @@ describe("dependency pin drift check", () => { ); }); - it("accepts 0.1.3 from the registry without an inline release record", () => { - withFixture( - "nemoclaw-dependency-pins-registry-", - { openshellMin: "0.1.3", openshellMax: "0.1.3" }, - (root) => expect(verifyDependencyPins(root)).toEqual([]), - ); - }); + it.each(["0.1.5", "0.1.7"])( + "accepts %s from the registry without an inline release record", + (version) => { + withFixture( + "nemoclaw-dependency-pins-registry-", + { openshellMin: version, openshellMax: version }, + (root) => expect(verifyDependencyPins(root)).toEqual([]), + ); + }, + ); it("retains legacy release identities when the registry is empty", () => { withFixture( diff --git a/test/repository/package-openshell-sdk-for-pr.test.ts b/test/repository/package-openshell-sdk-for-pr.test.ts index 46d93b9460a..fbf24d7dda6 100644 --- a/test/repository/package-openshell-sdk-for-pr.test.ts +++ b/test/repository/package-openshell-sdk-for-pr.test.ts @@ -75,7 +75,7 @@ describe("reviewed OpenShell SDK transition packaging", () => { const key = parsed.pathname + parsed.search; return new Response(JSON.stringify(responses[key] ?? {}), { status: key in responses ? 200 : 404 }); }; - process.argv[1] = ${JSON.stringify(path.join(root, "scripts/checks/prepare-ci-npm-install.mts"))}; + process.argv[1] = ${JSON.stringify(fs.realpathSync(path.join(root, "scripts/checks/prepare-ci-npm-install.mts")))}; await import('./scripts/checks/prepare-ci-npm-install.mts'); `, ], @@ -179,17 +179,17 @@ describe("reviewed OpenShell SDK transition packaging", () => { expect(artifactDirectory).toBe(source.output); expect(fs.readdirSync(artifactDirectory).sort()).toEqual([ "nvidia-openshell-sdk-0.0.116.tgz", - "nvidia-openshell-sdk-0.1.5.tgz", + "nvidia-openshell-sdk-0.1.7.tgz", ]); expect(source.requests.map(({ packageSpec }) => packageSpec)).toEqual([ "@nvidia/openshell-sdk@0.0.116", - "@nvidia/openshell-sdk@0.1.5", + "@nvidia/openshell-sdk@0.1.7", ]); expect(source.requests[1]).toMatchObject({ expectedIntegrity: - "sha512-pqeumlEkIoFko9kL8EmemULuEPGf7dbFz1DXac9mB5H5e0+vGs24T1S+jT/UOtjhYEdmBSR3WjMojBBAyh6lSQ==", + "sha512-38Zs30o/lvAxHFH+7eUINbXG+QrfFOFUY3rIxoPNXFbURSdWGoo+SNwzCHtP9ulFs9Q0odVsmRr50AImXDM5hA==", tarballUrl: - "https://npm.pkg.github.com/download/@nvidia/openshell-sdk/0.1.5/141f111b92761575225e0ae977e1cf86d2c6515e", + "https://npm.pkg.github.com/download/@nvidia/openshell-sdk/0.1.7/e34e85f18bf581cce2e99a74edc3424644cf4ed6", }); expect(source.remove).toHaveBeenCalledTimes(2); }); @@ -201,7 +201,7 @@ describe("reviewed OpenShell SDK transition packaging", () => { expect(artifactDirectory).toBe(source.output); expect(source.requests.map(({ packageSpec }) => packageSpec)).toEqual([ "@nvidia/openshell-sdk@0.0.116", - "@nvidia/openshell-sdk@0.1.5", + "@nvidia/openshell-sdk@0.1.7", ]); }); diff --git a/test/repository/source-shape-scanner.test.ts b/test/repository/source-shape-scanner.test.ts index 740eb4710c8..90f23b09b32 100644 --- a/test/repository/source-shape-scanner.test.ts +++ b/test/repository/source-shape-scanner.test.ts @@ -24,6 +24,18 @@ function detectedCaseNames(source: string): string[] { } describe("source-shape scanner", () => { + it("excludes private E2E runtime state but retains E2E source tests", () => { + const repoRoot = path.resolve("."); + expect(isSourceShapePathSkipped(path.join(repoRoot, ".e2e"))).toBe(true); + expect(isSourceShapePathSkipped(path.join(repoRoot, ".e2e", "runtime.test.ts"))).toBe(true); + expect(isSourceShapePathSkipped(path.join(repoRoot, ".e2e-source", "example.test.ts"))).toBe( + false, + ); + expect(isSourceShapePathSkipped(path.join(repoRoot, "test", "e2e", "example.test.ts"))).toBe( + false, + ); + }); + it("skips the local nested worktree checkout container", () => { const repoRoot = path.resolve("."); diff --git a/test/repository/vitest-watch-triggers.test.ts b/test/repository/vitest-watch-triggers.test.ts index fa415565708..68d07ad1a3c 100644 --- a/test/repository/vitest-watch-triggers.test.ts +++ b/test/repository/vitest-watch-triggers.test.ts @@ -114,6 +114,15 @@ function triggeredBy(relativePath: string): string[] { } describe("Vitest opaque-input watch triggers", () => { + it("maps the installer cutover boundary to its process tests", () => { + expect(triggeredBy("scripts/install-openshell.sh")).toEqual([ + "test/installer-integration/install-openshell-version-check.test.ts", + "test/installer-integration/install-openshell-retained-state-size.test.ts", + "test/installer-integration/install-openshell-version-pin.test.ts", + "test/installer-integration/install-openshell-directory-safety.test.ts", + "test/install/installer-directory-bsd-stat.test.ts", + ]); + }); it.each(["test/helpers/onboard-fixture-contract.json", "test/helpers/onboard-script-mocks.cjs"])( "maps %s to every sandbox identity consumer (#10463)", (fixturePath) => { @@ -272,6 +281,8 @@ describe("Vitest opaque-input watch triggers", () => { ]); expect(triggeredBy("agents/hermes/runtime-config-guard.py")).toEqual([ "src/lib/actions/sandbox/gateway-restart-hermes-drift.test.ts", + "test/agents/hermes/hermes-runtime-config-guard.test.ts", + "test/agents/hermes/hermes-capability-free-supervisor.test.ts", "test/agents/hermes/hermes-restart-config-seal-write-lock.test.ts", "test/agents/hermes/hermes-nvidia-restore-validation.test.ts", ]); diff --git a/test/runtime/gateway/gateway-health-honest.test.ts b/test/runtime/gateway/gateway-health-honest.test.ts index 6e9787b1c7d..f4058c2b810 100644 --- a/test/runtime/gateway/gateway-health-honest.test.ts +++ b/test/runtime/gateway/gateway-health-honest.test.ts @@ -58,7 +58,7 @@ exit 127 writeExecutable( openshellBin, `#!/usr/bin/env bash -if [ "\${1:-}" = "--version" ]; then printf '%s\n' 'openshell 0.0.116'; fi +if [ "\${1:-}" = "--version" ]; then printf '%s\n' 'openshell 0.1.7'; fi exit 0 `, ); diff --git a/test/runtime/gateway/gateway-start-wait.test.ts b/test/runtime/gateway/gateway-start-wait.test.ts index 0414436fd2f..5e3389f5858 100644 --- a/test/runtime/gateway/gateway-start-wait.test.ts +++ b/test/runtime/gateway/gateway-start-wait.test.ts @@ -133,7 +133,7 @@ describe("gateway bootstrap secret repair", () => { loadOnboard(); expect(getGatewayLocalEndpoint()).toBe("https://127.0.0.1:9443"); - expect(getDockerDriverGatewayEnv("openshell 0.0.116", "linux")).toMatchObject({ + expect(getDockerDriverGatewayEnv("openshell 0.1.7", "linux")).toMatchObject({ OPENSHELL_BIND_ADDRESS: "0.0.0.0", OPENSHELL_GRPC_ENDPOINT: "https://127.0.0.1:9443", OPENSHELL_SSH_GATEWAY_HOST: "127.0.0.1", diff --git a/test/runtime/gateway/gateway-state-reconcile-2276.test.ts b/test/runtime/gateway/gateway-state-reconcile-2276.test.ts index 9a8bdbf7fda..d157e53014d 100644 --- a/test/runtime/gateway/gateway-state-reconcile-2276.test.ts +++ b/test/runtime/gateway/gateway-state-reconcile-2276.test.ts @@ -22,7 +22,7 @@ import { testTimeout } from "../../helpers/timeouts"; const TIMEOUT_MS = testTimeout(60_000); const SANDBOX_NAME = "my-assistant"; -const OPENSHELL_FIXTURE_VERSION = "0.0.116"; +const OPENSHELL_FIXTURE_VERSION = "0.1.7"; // Output fixtures that mirror real OpenShell CLI output. const gatewayListNemoclaw = (gatewayName: string, port: number) => @@ -191,7 +191,7 @@ if (args[0] === "provider" && args[1] === "get") { } if (args[0] === "provider" && args[1] === "list") { - process.stdout.write('[{"name":"nvidia-prod","credential_keys":["NVIDIA_INFERENCE_API_KEY"]}]\\n'); + process.stdout.write('{"providers":[{"name":"nvidia-prod","credential_keys":["NVIDIA_INFERENCE_API_KEY"]}],"next_page_token":""}\\n'); process.exit(0); } @@ -294,7 +294,7 @@ if (a[0] === "inspect") { const responses = new Map([ ["{{.State.Running}}", "true\\n"], ["{{json .NetworkSettings.Ports}}", JSON.stringify({[gatewayPort + "/tcp"]:[{HostPort:gatewayPort}]}) + "\\n"], - ["{{.Config.Image}}", "nvcr.io/nvidia/openshell/cluster:0.0.116\\n"], + ["{{.Config.Image}}", "nvcr.io/nvidia/openshell/cluster:0.1.7\\n"], ]); if (target !== expectedTarget || !responses.has(format)) process.exit(64); process.stdout.write(responses.get(format)); diff --git a/test/runtime/gateway/service-env.test.ts b/test/runtime/gateway/service-env.test.ts index 116a16a66fe..9f7a9e97e7a 100644 --- a/test/runtime/gateway/service-env.test.ts +++ b/test/runtime/gateway/service-env.test.ts @@ -18,7 +18,7 @@ import { } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { beforeAll, describe, expect, it } from "vitest"; +import { describe, expect, it } from "vitest"; const NEMOCLAW_START_SCRIPT = join(import.meta.dirname, "..", "..", "../scripts/nemoclaw-start.sh"); const ENTRYPOINT_ENV_WRAPPER = join( @@ -83,13 +83,13 @@ function extractToolRedirectsSnippet() { function extractProxyVarsSnippet() { const src = readFileSync(NEMOCLAW_START_SCRIPT, "utf-8"); - const start = src.indexOf("PROXY_HOST="); - const endMarker = 'export no_proxy="$_NO_PROXY_VAL"'; + const start = src.indexOf("unset HTTP_PROXY HTTPS_PROXY NO_PROXY"); + const endMarker = "unset NODE_USE_ENV_PROXY NEMOCLAW_PROXY_HOST NEMOCLAW_PROXY_PORT"; const end = src.indexOf(endMarker, start); if (start === -1 || end === -1 || end <= start) { throw new Error( "Failed to extract proxy configuration from scripts/nemoclaw-start.sh — " + - "the PROXY_HOST..no_proxy block may have been moved or renamed", + "the native egress block may have been moved or renamed", ); } return src.slice(start, end + endMarker.length); @@ -198,7 +198,7 @@ describe("service environment", () => { const scriptPath = join(import.meta.dirname, "..", "..", "../scripts/nemoclaw-start.sh"); const src = readFileSync(scriptPath, "utf-8"); const start = src.indexOf("# Git TLS CA bundle fix"); - const end = src.indexOf("# HTTP library + NODE_USE_ENV_PROXY", start); + const end = src.indexOf("# Node runtime preload modules", start); if (start === -1 || end === -1 || end <= start) { throw new Error("Failed to extract SSL_CERT_FILE handling block"); } @@ -432,93 +432,42 @@ describe("service environment", () => { // shared library. Wrappers that execute the extracted block must source it. const sandboxInitSource = `source ${JSON.stringify(join(import.meta.dirname, "..", "..", "../scripts/lib/sandbox-init.sh"))}`; - function extractProxyVars(env: Record = {}) { - const proxyBlock = extractProxyVarsSnippet(); - const wrapper = [ - "#!/usr/bin/env bash", - proxyBlock.trimEnd(), - 'echo "HTTP_PROXY=${HTTP_PROXY}"', - 'echo "HTTPS_PROXY=${HTTPS_PROXY}"', - 'echo "NO_PROXY=${NO_PROXY}"', - 'echo "http_proxy=${http_proxy}"', - 'echo "https_proxy=${https_proxy}"', - 'echo "no_proxy=${no_proxy}"', - ].join("\n"); - const tmpFile = join(tmpdir(), `nemoclaw-proxy-test-${process.pid}.sh`); - try { - writeFileSync(tmpFile, wrapper, { mode: 0o700 }); - const out = execFileSync("bash", [tmpFile], { - encoding: "utf-8", - env: { ...process.env, ...env }, - }).trim(); - return Object.fromEntries( - out.split("\n").map((l) => { - const idx = l.indexOf("="); - return [l.slice(0, idx), l.slice(idx + 1)]; - }), - ); - } finally { - try { - unlinkSync(tmpFile); - } catch { - /* ignore */ - } - } - } - - let defaultProxyVars: Record; - let hostOverrideProxyVars: Record; - let portOverrideProxyVars: Record; - - beforeAll(() => { - defaultProxyVars = extractProxyVars(); - hostOverrideProxyVars = extractProxyVars({ NEMOCLAW_PROXY_HOST: "192.168.64.1" }); - portOverrideProxyVars = extractProxyVars({ NEMOCLAW_PROXY_PORT: "8080" }); - }); - - it("sets HTTP_PROXY to default gateway address", () => { - const vars = defaultProxyVars; - expect(vars.HTTP_PROXY).toBe("http://10.200.0.1:3128"); - }); - - it("sets HTTPS_PROXY to default gateway address", () => { - const vars = defaultProxyVars; - expect(vars.HTTPS_PROXY).toBe("http://10.200.0.1:3128"); - }); - - it("NEMOCLAW_PROXY_HOST overrides default gateway IP", () => { - const vars = hostOverrideProxyVars; - expect(vars.HTTP_PROXY).toBe("http://192.168.64.1:3128"); - expect(vars.HTTPS_PROXY).toBe("http://192.168.64.1:3128"); - }); - - it("NEMOCLAW_PROXY_PORT overrides default proxy port", () => { - const vars = portOverrideProxyVars; - expect(vars.HTTP_PROXY).toBe("http://10.200.0.1:8080"); - expect(vars.HTTPS_PROXY).toBe("http://10.200.0.1:8080"); - }); - - it("NO_PROXY includes loopback only, not inference.local", () => { - const vars = defaultProxyVars; - const noProxy = vars.NO_PROXY.split(","); - expect(noProxy).toContain("localhost"); - expect(noProxy).toContain("127.0.0.1"); - expect(noProxy).toContain("::1"); - expect(noProxy).not.toContain("inference.local"); - }); - - it("NO_PROXY includes OpenShell gateway IP", () => { - const vars = defaultProxyVars; - expect(vars.NO_PROXY).toContain("10.200.0.1"); - }); - - it("exports lowercase proxy variants for undici/gRPC compatibility", () => { - const vars = defaultProxyVars; - expect(vars.http_proxy).toBe("http://10.200.0.1:3128"); - expect(vars.https_proxy).toBe("http://10.200.0.1:3128"); - const noProxy = vars.no_proxy.split(","); - expect(noProxy).not.toContain("inference.local"); - expect(noProxy).toContain("10.200.0.1"); + const proxyNames = [ + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "http_proxy", + "https_proxy", + "no_proxy", + "ALL_PROXY", + "all_proxy", + "OPENAI_PROXY", + "grpc_proxy", + "GRPC_PROXY", + "NODE_USE_ENV_PROXY", + "NEMOCLAW_PROXY_HOST", + "NEMOCLAW_PROXY_PORT", + ]; + + it("clears all obsolete proxy inputs before startup and preserves CA settings", () => { + const seed = Object.fromEntries(proxyNames.map((name) => [name, "obsolete-proxy-value"])); + const out = execFileSync( + "bash", + [ + "-c", + [ + "set -eu", + extractProxyVarsSnippet(), + ...proxyNames.map((name) => `printf '%s=%s\\n' '${name}' "\${${name}+set}"`), + 'printf "CA=%s\\n" "$SSL_CERT_FILE"', + ].join("\n"), + ], + { + encoding: "utf8", + env: { ...process.env, ...seed, SSL_CERT_FILE: "/trusted/ca.pem" }, + }, + ); + expect(out).toBe(proxyNames.map((name) => `${name}=\n`).join("") + "CA=/trusted/ca.pem\n"); }); it.each([ @@ -585,11 +534,8 @@ describe("service environment", () => { ).toBe("native@example.invalid"); const envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8"); - expect(envFile).toContain('export HTTP_PROXY="http://10.200.0.1:3128"'); - expect(envFile).toContain('export HTTPS_PROXY="http://10.200.0.1:3128"'); - expect(envFile).toContain("export NO_PROXY="); - expect(envFile).not.toContain("inference.local"); - expect(envFile).toContain("10.200.0.1"); + expect(envFile).toContain("unset HTTP_PROXY HTTPS_PROXY NO_PROXY"); + expect(envFile).not.toContain("10.200.0.1"); expect(envFile).toContain('export AWS_EC2_METADATA_DISABLED="true"'); expect(envFile).toContain("export OPENCLAW_GATEWAY_TOKEN"); @@ -700,8 +646,8 @@ describe("service environment", () => { const envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8"); // cat > overwrites the file each time, so there should be exactly one - // HTTP_PROXY line — no duplication from repeated runs. - const httpProxyCount = (envFile.match(/export HTTP_PROXY=/g) || []).length; + // proxy cleanup line — no duplication from repeated runs. + const httpProxyCount = (envFile.match(/unset HTTP_PROXY /g) || []).length; expect(httpProxyCount).toBe(1); const metadataCount = (envFile.match(/export AWS_EC2_METADATA_DISABLED=/g) || []).length; expect(metadataCount).toBe(1); @@ -725,7 +671,7 @@ describe("service environment", () => { } }); - it("entrypoint replaces stale proxy values on restart", () => { + it("entrypoint clears stale proxy values on restart", () => { const fakeDataDir = join(tmpdir(), `nemoclaw-replace-test-${process.pid}`); mkdirSync(fakeDataDir, { recursive: true }); const tmpFile = join(tmpdir(), `nemoclaw-replace-write-test-${process.pid}.sh`); @@ -749,12 +695,12 @@ describe("service environment", () => { writeFileSync(tmpFile, makeWrapper("10.200.0.1"), { mode: 0o700 }); execFileSync("bash", [tmpFile], { encoding: "utf-8" }); let envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8"); - expect(envFile).toContain("10.200.0.1"); + expect(envFile).not.toContain("10.200.0.1"); writeFileSync(tmpFile, makeWrapper("192.168.1.99"), { mode: 0o700 }); execFileSync("bash", [tmpFile], { encoding: "utf-8" }); envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8"); - expect(envFile).toContain("192.168.1.99"); + expect(envFile).not.toContain("192.168.1.99"); expect(envFile).not.toContain("10.200.0.1"); } finally { try { @@ -809,92 +755,71 @@ describe("service environment", () => { } }); - it("overrides narrow NO_PROXY and no_proxy while sourcing proxy-env.sh in simulation", () => { - const fakeDataDir = mkdtempSync(join(tmpdir(), "nemoclaw-bashi-test-")); - try { - const envContent = [ - 'export HTTP_PROXY="http://10.200.0.1:3128"', - 'export HTTPS_PROXY="http://10.200.0.1:3128"', - 'export NO_PROXY="localhost,127.0.0.1,::1,10.200.0.1"', - 'export http_proxy="http://10.200.0.1:3128"', - 'export https_proxy="http://10.200.0.1:3128"', - 'export no_proxy="localhost,127.0.0.1,::1,10.200.0.1"', - ].join("\n"); - writeFileSync(join(fakeDataDir, "proxy-env.sh"), envContent); - - const out = execFileSync( - "bash", - [ - "--norc", - "-c", - [ - 'export NO_PROXY="127.0.0.1,localhost,::1"', - 'export no_proxy="127.0.0.1,localhost,::1"', - `source ${JSON.stringify(join(fakeDataDir, "proxy-env.sh"))}`, - 'echo "NO_PROXY=$NO_PROXY"', - 'echo "no_proxy=$no_proxy"', - ].join("; "), - ], - { encoding: "utf-8" }, - ).trim(); - - expect(out).toContain("NO_PROXY=localhost,127.0.0.1,::1,10.200.0.1"); - expect(out).toContain("no_proxy=localhost,127.0.0.1,::1,10.200.0.1"); - } finally { + it.each(["sh", "bash"])( + "clears proxy inputs in %s connect shells even when NODE_USE_ENV_PROXY=1", + (shell) => { + const fakeDataDir = join(tmpdir(), `nemoclaw-http-fix-test-${process.pid}`); + mkdirSync(fakeDataDir, { recursive: true }); + const tmpFile = join(tmpdir(), `nemoclaw-http-fix-env-${process.pid}.sh`); + const fakeFixPath = "/tmp/nemoclaw-http-proxy-fix.js"; try { - rmSync(fakeDataDir, { recursive: true, force: true }); - } catch { - /* ignore */ - } - } - }); - - it("includes NODE_OPTIONS --require in proxy-env.sh when NODE_USE_ENV_PROXY=1 (#2109)", () => { - const fakeDataDir = join(tmpdir(), `nemoclaw-http-fix-test-${process.pid}`); - mkdirSync(fakeDataDir, { recursive: true }); - const tmpFile = join(tmpdir(), `nemoclaw-http-fix-env-${process.pid}.sh`); - const fakeFixPath = "/tmp/nemoclaw-http-proxy-fix.js"; - try { - const persistBlock = extractRuntimeShellEnvSnippet(); - const wrapper = [ - "#!/usr/bin/env bash", - "set -euo pipefail", - sandboxInitSource, - 'PROXY_HOST="10.200.0.1"', - 'PROXY_PORT="3128"', - '_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}"', - '_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}"', - "NODE_USE_ENV_PROXY=1", - "_TOOL_REDIRECTS=()", - `_PROXY_FIX_SCRIPT="${fakeFixPath}"`, - `_NEMOTRON_FIX_SCRIPT="/tmp/nemoclaw-nemotron-inference-fix.js"`, - "set +u # array expansion safe on macOS bash", - persistBlock - .trimEnd() - .replaceAll("/tmp/nemoclaw-proxy-env.sh", `${fakeDataDir}/proxy-env.sh`), - ].join("\n"); - writeFileSync(tmpFile, wrapper, { mode: 0o700 }); - execFileSync("bash", [tmpFile], { encoding: "utf-8" }); + const persistBlock = extractRuntimeShellEnvSnippet(); + const wrapper = [ + "#!/usr/bin/env bash", + "set -euo pipefail", + sandboxInitSource, + 'PROXY_HOST="10.200.0.1"', + 'PROXY_PORT="3128"', + '_PROXY_URL="http://${PROXY_HOST}:${PROXY_PORT}"', + '_NO_PROXY_VAL="localhost,127.0.0.1,::1,${PROXY_HOST}"', + "NODE_USE_ENV_PROXY=1", + "_TOOL_REDIRECTS=()", + `_PROXY_FIX_SCRIPT="${fakeFixPath}"`, + `_NEMOTRON_FIX_SCRIPT="/tmp/nemoclaw-nemotron-inference-fix.js"`, + "set +u # array expansion safe on macOS bash", + persistBlock + .trimEnd() + .replaceAll("/tmp/nemoclaw-proxy-env.sh", `${fakeDataDir}/proxy-env.sh`), + ].join("\n"); + writeFileSync(tmpFile, wrapper, { mode: 0o700 }); + execFileSync("bash", [tmpFile], { encoding: "utf-8" }); - const envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8"); - expect(envFile).toContain("NODE_OPTIONS"); - expect(envFile).toContain("--require"); - // Preload target is the in-sandbox /tmp path; no dependency on an - // external /opt path (see axios-proxy-fix Bug 1 — scripts/ never - // made it into the optimized build context). The JS is embedded in - // nemoclaw-start.sh and written to /tmp at boot. - expect(envFile).toContain(fakeFixPath); - } finally { - try { - rmSync(fakeDataDir, { recursive: true, force: true }); - rmSync(tmpFile, { force: true }); - } catch { - /* ignore */ + const envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8"); + expect(envFile).toContain("NODE_OPTIONS"); + expect(envFile).toContain("--require"); + expect(envFile).not.toContain(fakeFixPath); + expect(envFile).toContain("nemotron-inference-fix"); + const out = execFileSync( + shell, + [ + "-c", + [ + `. "${join(fakeDataDir, "proxy-env.sh")}"`, + ...proxyNames.map((name) => `printf '%s=%s\\n' '${name}' "\${${name}+set}"`), + ].join("\n"), + ], + { + encoding: "utf8", + env: { + ...process.env, + ...Object.fromEntries(proxyNames.map((name) => [name, "obsolete-proxy-value"])), + NODE_OPTIONS: "", + }, + }, + ); + expect(out).toBe(proxyNames.map((name) => `${name}=\n`).join("")); + } finally { + try { + rmSync(fakeDataDir, { recursive: true, force: true }); + rmSync(tmpFile, { force: true }); + } catch { + /* ignore */ + } } - } - }); + }, + ); - it("omits NODE_OPTIONS from proxy-env.sh when NODE_USE_ENV_PROXY is unset (#2109)", () => { + it("keeps the inference preload when NODE_USE_ENV_PROXY is unset", () => { const fakeDataDir = join(tmpdir(), `nemoclaw-http-noop-test-${process.pid}`); mkdirSync(fakeDataDir, { recursive: true }); const tmpFile = join(tmpdir(), `nemoclaw-http-noop-env-${process.pid}.sh`); @@ -921,9 +846,7 @@ describe("service environment", () => { execFileSync("bash", [tmpFile], { encoding: "utf-8" }); const envFile = readFileSync(join(fakeDataDir, "proxy-env.sh"), "utf-8"); - // Proxy preloads should NOT be injected when NODE_USE_ENV_PROXY - // is not 1. The Nemotron inference fix is - // unconditional (always needed regardless of proxy config). + // Native egress does not need the old double-proxy preload. expect(envFile).not.toContain("http-proxy-fix"); expect(envFile).toContain("nemotron-inference-fix"); } finally { diff --git a/test/runtime/gateway/startup-process-identity.test.ts b/test/runtime/gateway/startup-process-identity.test.ts index 1532688266f..d79f8f0f160 100644 --- a/test/runtime/gateway/startup-process-identity.test.ts +++ b/test/runtime/gateway/startup-process-identity.test.ts @@ -349,3 +349,274 @@ describe.each(GUARDS)("%s exact startup argv", (name, guardPath) => { expect(proof.openshell_empty_argument_spoof).toBe(false); }); }); + +const CAPABILITY_FREE_IDENTITY_HARNESS = String.raw` +import importlib.util +import json +import os +import sys +import tempfile +from types import SimpleNamespace + +spec = importlib.util.spec_from_file_location("guard", sys.argv[1]) +guard = importlib.util.module_from_spec(spec) +sys.modules[spec.name] = guard +spec.loader.exec_module(guard) +case = sys.argv[2] +driver, separator, variant = case.partition(":") +if separator: case = variant +uid, gid = 998, 999 +guard.pwd.getpwnam = lambda _name: SimpleNamespace(pw_uid=uid, pw_gid=gid) +kernel_metadata = None +if case in ("linux-nondumpable", "linux-dumpable"): + import ctypes + # This subprocess changes only its own credentials/dumpability, never the + # test runner or host configuration. Root CI containers use nobody here. + if os.geteuid() == 0: + os.setgroups([]) + os.setgid(65534) + os.setuid(65534) + assert os.geteuid() > 0 and os.getegid() > 0 + libc = ctypes.CDLL(None, use_errno=True) + dumpable = 0 if case == "linux-nondumpable" else 1 + assert libc.prctl(4, dumpable, 0, 0, 0) == 0 + directory = os.stat("/proc/self") + kernel_metadata = os.stat("/proc/self/status") + assert (directory.st_uid, directory.st_gid) == (os.geteuid(), os.getegid()) + expected_owner = (0, 0) if dumpable == 0 else (os.geteuid(), os.getegid()) + assert (kernel_metadata.st_uid, kernel_metadata.st_gid) == expected_owner + +with tempfile.TemporaryDirectory() as root: + guard.PROC_ROOT = root + namespace = os.path.join(root, "namespace") + with open(namespace, "wb") as stream: + stream.write(b"fixture") + argv = [b"/opt/openshell/bin/openshell-sandbox", b"launch-capability-free", + b"998", b"999", b"/.openshell/channel/sandbox/bootstrap.json", b"/sandbox"] + if case == "foreign-executable": argv[0] = b"/tmp/openshell-sandbox" + if case == "foreign-mode": argv[1] = b"--bootstrap" + if case == "wrong-argv-uid": argv[2] = b"999" + if case == "wrong-argv-gid": argv[3] = b"998" + if case == "foreign-bootstrap": argv[4] = b"/tmp/bootstrap.json" + if case == "foreign-workspace": argv[5] = b"/tmp" + if case == "extra-argv": argv.append(b"nemoclaw-start") + if driver in ("docker", "rootful-podman"): + executable = (b"/.openshell/runtime/openshell-sandbox" if driver == "docker" + else b"/opt/openshell/bin/openshell-sandbox") + argv = [executable, b"--bootstrap", b"/.openshell/channel/sandbox/bootstrap.json"] + if case == "foreign-bootstrap": argv[2] = b"/tmp/bootstrap.json" + if case == "extra-argv": argv.append(b"nemoclaw-start") + if case == "foreign-executable": argv[0] = b"/tmp/openshell-sandbox" + status = { + "Uid": "998 998 998 998", "Gid": "999 999 999 999", "Groups": "", + "NSpid": "1", "NoNewPrivs": "1", + "CapInh": "0000000000000000", "CapPrm": "0000000000000000", + "CapEff": "0000000000000000", "CapBnd": "0000000000000000", + "CapAmb": "0000000000000000", + } + if case == "saved-root": status["Uid"] = "998 998 0 998" + if case == "wrong-uid": status["Uid"] = "999 999 999 999" + if case == "wrong-gid": status["Gid"] = "998 998 998 998" + if case == "supplementary-groups": status["Groups"] = "0" + if case == "can-gain-privileges": status["NoNewPrivs"] = "0" + if case.startswith("retained-"): status[case.removeprefix("retained-")] = "1" + if case == "missing-capability": del status["CapEff"] + if case == "non-init": status["NSpid"] = "2" + + def write_process(pid, parent, command, process_status): + directory = os.path.join(root, str(pid)) + os.makedirs(os.path.join(directory, "ns")) + fields = ["S", str(parent)] + ["0"] * 17 + [str(111111 + pid)] + with open(os.path.join(directory, "stat"), "w") as stream: + stream.write(str(pid) + " (fixture) " + " ".join(fields)) + with open(os.path.join(directory, "cmdline"), "wb") as stream: + stream.write(b"\0".join(command) + b"\0") + with open(os.path.join(directory, "status"), "w") as stream: + stream.write("".join(key + ":\t" + value + "\n" for key, value in process_status.items())) + if pid == 1 and case == "duplicate-field": stream.write("CapEff:\t0\n") + os.link(namespace, os.path.join(directory, "ns", "pid")) + write_process(1, 2 if case == "foreign-parent" else 0, argv, status) + child_status = {"Uid": "998 998 998 998", "NSpid": "412"} + if case == "root-child": child_status["Uid"] = "0 0 0 0" + child_argv = [b"bash", b"/usr/local/bin/nemoclaw-start"] + if case == "spoofed-child": child_argv = [b"python3", b"/usr/local/bin/nemoclaw-start"] + write_process(412, 77 if case == "indirect-child" else 1, child_argv, child_status) + if case == "duplicate-child": + write_process(413, 1, child_argv, {"Uid": "998 998 998 998", "NSpid": "413"}) + if case == "missing-child": os.unlink(os.path.join(root, "412", "cmdline")) + # Model Linux's distinct directory/status ownership on macOS. Linux cases + # use actual kernel status metadata after toggling this process's dumpability. + # Parse fixture bytes and exercise identity/uniqueness/parent checks unchanged. + original_fstat = os.fstat + original_stat = os.stat + init_stat = os.stat(os.path.join(root, "1")) + def fixture_fstat(fd): + value = original_fstat(fd) + if (value.st_dev, value.st_ino) == (init_stat.st_dev, init_stat.st_ino): + return SimpleNamespace(st_dev=value.st_dev, st_ino=value.st_ino, + st_uid=uid, st_gid=gid) + return value + metadata_reads = 0 + def fixture_stat(name, *args, **kwargs): + global metadata_reads + value = original_stat(name, *args, **kwargs) + fd = kwargs.get("dir_fd") + if name == "status" and fd is not None and original_fstat(fd).st_ino == init_stat.st_ino: + assert kwargs.get("follow_symlinks") is False + metadata_reads += 1 + dumpable = case == "dumpable-owner" or (case == "raced-owner" and metadata_reads >= 4) + owner = kernel_metadata or SimpleNamespace(st_uid=uid if dumpable else 0, + st_gid=gid if dumpable else 0) + return SimpleNamespace(st_dev=value.st_dev, st_ino=value.st_ino, + st_mode=value.st_mode, st_uid=owner.st_uid, st_gid=owner.st_gid) + return value + guard.os.fstat = fixture_fstat + guard.os.stat = fixture_stat + if case == "namespace-inaccessible": guard._proc_pid_namespace_inode = lambda _fd: None + original_read = guard._read_proc_pid_file + reads = 0 + def read_process(fd, name, display): + global reads + value = original_read(fd, name, display) + if display == root + "/1/status": + reads += 1 + if ((case == "raced-status" and reads >= 4) + or (case == "raced-final-status" and reads >= 5)): + return value.replace(b"NoNewPrivs:\t1", b"NoNewPrivs:\t0") + return value + guard._read_proc_pid_file = read_process + try: + if case.startswith(("startup-", "host-", "reconciliation-")): + guard.__file__ = guard.INSTALLED_RUNTIME_CONFIG_GUARD + guard.HERMES_STARTUP_READY_FILE = os.path.join(root, "ready") + guard.os.getppid = lambda: 413 if case == "startup-wrong-parent" else 412 + if case.endswith("stale-marker"): + with open(guard.HERMES_STARTUP_READY_FILE, "w") as stream: + stream.write("invalid\n") + if case.startswith("reconciliation-"): + accepted = guard._managed_nonroot_reconciliation_is_allowed() + else: + try: + guard._validate_action_readiness( + "ensure-api-key" if case.startswith("startup-") else "seal-restart", + case.startswith("startup-") and case != "startup-missing-owner", + ) + accepted = True + except guard.UnsafePathError: + accepted = False + print(json.dumps(accepted)) + else: + print(json.dumps(guard._openshell_supervised_nonroot_start_is_live( + 0, uid, 413 if case == "wrong-required-parent" else 412))) + finally: + guard.os.fstat = original_fstat + guard.os.stat = original_stat +`; + +describe("Hermes capability-free OpenShell startup identity", () => { + it.each([ + "valid", + "namespace-inaccessible", + "startup-authorized", + "host-authorized", + "reconciliation-authorized", + "docker:valid", + "docker:startup-authorized", + "docker:host-authorized", + "docker:reconciliation-authorized", + "rootful-podman:valid", + ])("accepts the pinned boundary: %s", (scenario) => { + const result = spawnSync( + "python3", + [ + "-c", + CAPABILITY_FREE_IDENTITY_HARNESS, + path.resolve("agents/hermes/runtime-config-guard.py"), + scenario, + ], + { encoding: "utf-8", timeout: 5000 }, + ); + expect(result.status, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toBe(true); + }); + + it.each([ + "foreign-executable", + "foreign-mode", + "wrong-argv-uid", + "wrong-argv-gid", + "foreign-bootstrap", + "foreign-workspace", + "extra-argv", + "saved-root", + "wrong-uid", + "wrong-gid", + "supplementary-groups", + "can-gain-privileges", + "retained-CapInh", + "retained-CapPrm", + "retained-CapEff", + "retained-CapBnd", + "retained-CapAmb", + "missing-capability", + "duplicate-field", + "non-init", + "foreign-parent", + "dumpable-owner", + "raced-owner", + "root-child", + "spoofed-child", + "indirect-child", + "duplicate-child", + "missing-child", + "wrong-required-parent", + "raced-status", + "raced-final-status", + "startup-missing-owner", + "startup-wrong-parent", + "startup-stale-marker", + "host-stale-marker", + "reconciliation-stale-marker", + "docker:foreign-executable", + "docker:foreign-bootstrap", + "docker:extra-argv", + "docker:saved-root", + "docker:retained-CapEff", + "docker:dumpable-owner", + "docker:startup-wrong-parent", + "rootful-podman:extra-argv", + "rootful-podman:saved-root", + "rootful-podman:can-gain-privileges", + ])("rejects an untrusted boundary without mutation: %s", (scenario) => { + const result = spawnSync( + "python3", + [ + "-c", + CAPABILITY_FREE_IDENTITY_HARNESS, + path.resolve("agents/hermes/runtime-config-guard.py"), + scenario, + ], + { encoding: "utf-8", timeout: 5000 }, + ); + expect(result.status, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toBe(false); + }); + + it.runIf(process.platform === "linux").each([ + ["linux-nondumpable", true], + ["linux-dumpable", false], + ] as const)("uses actual Linux proc ownership: %s", (scenario, accepted) => { + const result = spawnSync( + "python3", + [ + "-c", + CAPABILITY_FREE_IDENTITY_HARNESS, + path.resolve("agents/hermes/runtime-config-guard.py"), + scenario, + ], + { encoding: "utf-8", timeout: 5000 }, + ); + expect(result.status, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toBe(accepted); + }); +}); diff --git a/test/runtime/messaging/discord-template-resolver-proxy.test.ts b/test/runtime/messaging/discord-template-resolver-proxy.test.ts index d791a8e1199..bdbd3bcad75 100644 --- a/test/runtime/messaging/discord-template-resolver-proxy.test.ts +++ b/test/runtime/messaging/discord-template-resolver-proxy.test.ts @@ -5,13 +5,8 @@ import { describe, expect, it } from "vitest"; import { resolveDiscordTemplateReference } from "../../../src/lib/messaging/channels/discord/template-resolver.js"; -// OpenClaw's Discord plugin validates channels.discord.accounts.default.proxy -// and rejects any non-loopback host (validateDiscordProxyUrl: "Proxy URL must -// target a loopback host"). The sandbox egress proxy (10.200.0.1:3128) is not -// loopback, so a per-account proxy must NOT be emitted; Discord gateway/REST -// egress is carried by the top-level managed proxy (proxy.loopbackMode: -// "gateway-only") instead. discordProxyUrl therefore resolves to undefined so -// the proxy field is dropped from the rendered config (#5544; reverts #5248). +// Discord uses OpenShell native egress. Obsolete proxy inputs must not create +// an account-level proxy configuration. const ctx = (env: Record) => ({ inputs: [], env }); describe("discord template-resolver: discordProxyUrl", () => { diff --git a/test/runtime/messaging/repro-2201.test.ts b/test/runtime/messaging/repro-2201.test.ts index 03fe3a029e7..c5c3e85ce7b 100644 --- a/test/runtime/messaging/repro-2201.test.ts +++ b/test/runtime/messaging/repro-2201.test.ts @@ -25,7 +25,11 @@ describe("rebuild uses the registry target instead of stale session state", () = it("uses the registry Hermes target after OpenClaw was onboarded last (#2201)", async () => { const harness = createRebuildFlowHarness({ - sandboxEntry: { agent: "hermes", provider: "nvidia-router" }, + sandboxEntry: { + agent: "hermes", + provider: "nvidia-prod", + credentialEnv: "NVIDIA_INFERENCE_API_KEY", + }, sessionSandboxName: "openclaw", }); harness.session.agent = null; diff --git a/test/runtime/policy/repro-5978-policy-denial-hint.test.ts b/test/runtime/policy/repro-5978-policy-denial-hint.test.ts index 24022d1e296..251ced70d3d 100644 --- a/test/runtime/policy/repro-5978-policy-denial-hint.test.ts +++ b/test/runtime/policy/repro-5978-policy-denial-hint.test.ts @@ -165,7 +165,8 @@ describe("sandbox policy-denial logs breadcrumb (#5978)", () => { // observable outcomes through the boundary rather than internal shape. const gate = (env: NodeJS.ProcessEnv) => runInPty([stanza, "SHLVL=1; _nemoclaw_maybe_policy_denial_hint"].join("\n"), { - HTTPS_PROXY: "http://127.0.0.1:3128", + HTTPS_PROXY: "", + https_proxy: "", ...env, }); @@ -175,7 +176,7 @@ describe("sandbox policy-denial logs breadcrumb (#5978)", () => { expect(stdout).toContain("nemoclaw qa-5978 logs --tail 50"); }); - it.each(["", "1", "true"])( + it.each(["1", "true"])( "falls back to when OPENSHELL_SANDBOX is unusable (older OpenShell) [case %#]", (value) => { const { stdout, status } = gate({ OPENSHELL_SANDBOX: value }); @@ -287,10 +288,10 @@ describe("sandbox policy-denial logs breadcrumb (#5978)", () => { }, ); - it("emits a tool-agnostic breadcrumb naming the 403 signature and `logs --tail 50`", () => { + it("points sandbox users to network denial logs without proxy settings", () => { const { stdout, status } = gate({ OPENSHELL_SANDBOX: "qa-5978" }); expect(status).toBe(0); - expect(stdout).toContain("CONNECT tunnel failed, response 403"); + expect(stdout).toContain("OpenShell blocks requests that the network policy denies."); expect(stdout).toContain("nemoclaw qa-5978 logs --tail 50"); }); @@ -305,10 +306,10 @@ describe("sandbox policy-denial logs breadcrumb (#5978)", () => { expect(stdout).not.toContain("logs --tail 50"); }); - it("stays silent when no proxy is configured (nothing to deny)", () => { + it("stays silent outside an OpenShell sandbox", () => { const snippet = [stanza, "SHLVL=1; _nemoclaw_maybe_policy_denial_hint"].join("\n"); const { stdout, status } = runInPty(snippet, { - OPENSHELL_SANDBOX: "qa-5978", + OPENSHELL_SANDBOX: "", HTTPS_PROXY: "", https_proxy: "", }); diff --git a/test/runtime/sandbox/sandbox-build-context.test.ts b/test/runtime/sandbox/sandbox-build-context.test.ts index 43ccb918dc3..8d0f1efc548 100644 --- a/test/runtime/sandbox/sandbox-build-context.test.ts +++ b/test/runtime/sandbox/sandbox-build-context.test.ts @@ -266,7 +266,7 @@ describe("sandbox build context staging", () => { ); writeFixture(path.join("src", "lib", "tool-disclosure.ts")); writeFixture(path.join("src", "lib", "providerless-inference.ts")); - for (const fileName of ["agent-config.ts", "hosted.ts", "contract.ts"]) { + for (const fileName of ["agent-config.ts", "catalog.ts"]) { writeFixture(path.join("src", "lib", "inference", "native-provider", fileName)); } for (const relativePath of [ @@ -284,6 +284,7 @@ describe("sandbox build context staging", () => { writeFixture(path.join("src", "lib", relativePath)); } writeFixture(path.join("scripts", "patch-openclaw-tool-catalog.mts")); + writeFixture(path.join("scripts", "lib", "patch-openclaw-native-network.mts")); writeFixture(path.join("scripts", "lib", "patch-openclaw-npm12-pack-json.mts")); writeFixture(path.join("scripts", "lib", "patch-openclaw-dependency-overrides.mts")); writeFixture(path.join("scripts", "lib", "patch-openclaw-container-restart.mts")); diff --git a/test/runtime/sandbox/sandbox-rlimit-hooks.test.ts b/test/runtime/sandbox/sandbox-rlimit-hooks.test.ts index efab358c953..0f9326a42f3 100644 --- a/test/runtime/sandbox/sandbox-rlimit-hooks.test.ts +++ b/test/runtime/sandbox/sandbox-rlimit-hooks.test.ts @@ -611,7 +611,7 @@ describe("sandbox rlimit system hooks (#2173)", () => { const mcpTransaction = path.join(localLib, "hermes-mcp-config-transaction.py"); const mcpCredentialBoundary = path.join( localLib, - "openshell-child-visible-credentials.v0.0.116.json", + "openshell-child-visible-credentials.v0.1.7.json", ); const preloadDir = path.join(localLib, "preloads"); const safetyNet = path.join(preloadDir, "sandbox-safety-net.js"); @@ -692,7 +692,7 @@ describe("sandbox rlimit system hooks (#2173)", () => { .replaceAll("/usr/local/lib/nemoclaw/finalize-tirith-marker.py", tirithMarkerFinalizer) .replaceAll("/usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py", mcpTransaction) .replaceAll( - "/usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.0.116.json", + "/usr/local/lib/nemoclaw/openshell-child-visible-credentials.v0.1.7.json", mcpCredentialBoundary, ) .replaceAll("/usr/local/lib/nemoclaw/preloads/sandbox-safety-net.js", safetyNet) diff --git a/test/security/admin-approval-helper.test.ts b/test/security/admin-approval-helper.test.ts index 13c6125dbda..990f4592a17 100644 --- a/test/security/admin-approval-helper.test.ts +++ b/test/security/admin-approval-helper.test.ts @@ -130,9 +130,9 @@ function omitLocalIdentity(root: string, _identity: Record): st return path.join(root, "state"); } -function writePairingStateHelper(root: string): string { +function writePairingStateHelper(root: string, body = PAIRING_STATE_HELPER_PY): string { const helperPath = path.join(root, "openclaw_pairing_state.py"); - fs.writeFileSync(helperPath, PAIRING_STATE_HELPER_PY); + fs.writeFileSync(helperPath, body); return helperPath; } @@ -185,6 +185,7 @@ function runAdminApprovalScript( omitPreparedWrapper?: boolean; pendingRequestIds?: readonly string[]; selectExpectedRequest?: boolean; + pairingHelper?: string; } = {}, ): { commands: string[]; @@ -215,7 +216,7 @@ exec "$FAKE_REAL_PYTHON" "$@" { mode: 0o755 }, ); const stateRoot = writeLocalIdentity(root); - const helperPath = writePairingStateHelper(root); + const helperPath = writePairingStateHelper(root, options.pairingHelper); const terminalProbe = path.join(root, "terminal-probe.py"); fs.writeFileSync(terminalProbe, ADMIN_APPROVAL_TEST_PTY_PY); const terminalRc = path.join(root, "connect.bashrc"); @@ -817,6 +818,10 @@ describe("prepared connect-shell administrative approval", () => { it.each([ ["LLM request timed out.", "message-llm-timeout"], + ["cron isolated agent run aborted", "message-agent-aborted"], + ["cron: job execution timed out", "message-cron-deadline"], + ["cron isolated agent run aborted PRIVATE_SUFFIX", "unavailable"], + ["PRIVATE_PREFIX cron: job execution timed out", "unavailable"], ["LLM request failed.", "message-llm-failed"], ["LLM request failed with an unknown error.", "message-llm-unknown"], ["LLM request failed: provider returned an internal error.", "message-llm-provider-internal"], @@ -844,6 +849,18 @@ describe("prepared connect-shell administrative approval", () => { ["HTTP 599: PRIVATE_BODY", "error-text-http-599"], ["HTTP 408: PRIVATE_BODY", "error-text-http-408"], ["HTTP 429: PRIVATE_BODY", "error-text-http-429"], + [ + "\u26a0\ufe0f inference/PRIVATE_MODEL request failed (provider internal error, HTTP 500). This is usually temporary — try again shortly.", + "error-text-http-500", + ], + [ + "PRIVATE_PREFIX \u26a0\ufe0f inference/PRIVATE_MODEL request failed (provider internal error, HTTP 500). This is usually temporary — try again shortly.", + "unavailable", + ], + [ + "\u26a0\ufe0f inference/PRIVATE_MODEL request failed (provider internal error, HTTP 500). This is usually temporary — try again shortly. PRIVATE_SUFFIX", + "unavailable", + ], ["PRIVATE_PREFIX LLM request timed out.", "unavailable"], ["LLM request timed out. PRIVATE_SUFFIX", "unavailable"], ["PRIVATE_BODY contains 503", "unavailable"], @@ -890,6 +907,38 @@ describe("prepared connect-shell administrative approval", () => { expect(stagedScriptRetained).toBe(false); }); + it.each([ + [0, "ADMIN_CRON_DURATION_MS=0\n"], + [120001, "ADMIN_CRON_DURATION_MS=120001\n"], + [86400000, "ADMIN_CRON_DURATION_MS=86400000\n"], + [-1, ""], + [86400001, ""], + [1.5, ""], + [true, ""], + [null, ""], + ["PRIVATE_DURATION test-gateway-token", ""], + [{ value: 120001 }, ""], + ])("retains only bounded terminal cron duration %j", (durationMs, expected) => { + const response = { + ...FAILED_CRON_RUN, + run: { ...FAILED_CRON_RUN.run, durationMs, error: "PRIVATE_ERROR test-gateway-token" }, + }; + const { result, commands, privateCapturesRemaining, stagedScriptRetained } = + runAdminApprovalScript(undefined, 0, EXPECTED_REQUEST_ID, { + cronRunOutput: JSON.stringify(response), + cronRunExitCode: 1, + removeCaptures: true, + }); + expect(result.status).toBe(29); + expect(result.stderr.match(/^ADMIN_CRON_DURATION_MS=.*\n/gm)?.join("") ?? "").toBe(expected); + expect(`${result.stdout}\n${result.stderr}`).not.toMatch( + /PRIVATE_|test-gateway-token|cron-1|run-1/, + ); + expect(commands.filter((command) => command.startsWith("cron run "))).toHaveLength(1); + expect(privateCapturesRemaining).toEqual([]); + expect(stagedScriptRetained).toBe(false); + }); + it("ignores cron signatures outside the bound error-message fields", () => { const message = "HTTP 503: PRIVATE_BODY"; const response = { @@ -1065,6 +1114,93 @@ describe("prepared connect-shell administrative approval", () => { }, ); + it.each([true, false])( + "rechecks the local identity after a pairing-state read settles (identity matches: %s)", + (identityMatches) => { + const identity = identityMatches + ? EXPECTED_IDENTITY + : { + deviceId: OTHER_DEVICE_ID, + publicKey: OTHER_PUBLIC_KEY, + }; + const pairingHelper = `${PAIRING_STATE_HELPER_PY} +class OpenClawPairingStateRetryableError(OSError): + pass +read_stable_state = read_openclaw_pairing_state +attempts = 0 +def read_openclaw_pairing_state(*args, **kwargs): + global attempts + attempts += 1 + if attempts < 3: + raise OpenClawPairingStateRetryableError("canonical pairing-state WAL changed while reading") + records, metadata = read_stable_state(*args, **kwargs) + records["identity"] = ${JSON.stringify(identity)} + return records, metadata +`; + const { commands, result, stagedScriptRetained } = runAdminApprovalScript( + undefined, + 0, + EXPECTED_REQUEST_ID, + { + verifyCronConsumer: false, + selectExpectedRequest: true, + expectedRequestId: EXPECTED_REQUEST_ID, + pairingHelper, + }, + ); + expect(result.status).toBe(identityMatches ? 0 : 26); + expect(commands.some((command) => command.startsWith("devices approve "))).toBe( + identityMatches, + ); + expect(result.stdout.includes("ISSUE_5324_ADMIN_APPROVAL_OK")).toBe(identityMatches); + expect(stagedScriptRetained).toBe(false); + }, + ); + + it.each([ + ["OpenClawPairingStateRetryableError", "canonical pairing-state database is busy", "busy"], + [ + "OpenClawPairingStateRetryableError", + "canonical pairing-state WAL changed while reading", + "changed", + ], + [ + "OpenClawPairingStateRetryableError", + "canonical pairing-state WAL is missing its shared-memory sidecar", + "wal-incomplete", + ], + [ + "OpenClawPairingStateRetryableError", + "SQLite reopened an unvalidated pairing-state file identity", + "unvalidated", + ], + ["OSError", "unsafe canonical pairing-state SQLite entry", "unsafe"], + ["FileNotFoundError", "missing database", "missing"], + ["PermissionError", "cannot read database", "permission"], + ["ValueError", "unsupported canonical device state schema", "schema"], + ["ValueError", "canonical primary device identity is unavailable", "read"], + ])( + "preserves refusal and redacts the %s pairing-state failure (%s)", + (errorType, message, category) => { + const { commands, result, stagedScriptRetained, privateCapturesRemaining } = + runAdminApprovalScript(undefined, 0, EXPECTED_REQUEST_ID, { + verifyCronConsumer: false, + selectExpectedRequest: true, + expectedRequestId: EXPECTED_REQUEST_ID, + removeCaptures: true, + pairingHelper: `class OpenClawPairingStateRetryableError(OSError):\n pass\ndef read_openclaw_pairing_state(*args, **kwargs):\n raise ${errorType}(${JSON.stringify(message + " PRIVATE_PAIRING_DATA")})\n`, + }); + expect(result.status).toBe(26); + expect(commands).toEqual(["devices list --json"]); + expect(result.stderr).toContain(`ADMIN_DIAGNOSTIC=selection-state-${category}`); + expect(result.stdout + result.stderr).not.toMatch( + /PRIVATE_PAIRING_DATA|test-gateway-token|Traceback/, + ); + expect(stagedScriptRetained).toBe(false); + expect(privateCapturesRemaining).toEqual([]); + }, + ); + it("rejects malformed cron output and removes its staged script (#13342)", () => { const { result, stagedScriptRetained } = runAdminApprovalScript( undefined, diff --git a/test/security/config-rotate-token.test.ts b/test/security/config-rotate-token.test.ts index 527d9c800ba..93ae611b2e9 100644 --- a/test/security/config-rotate-token.test.ts +++ b/test/security/config-rotate-token.test.ts @@ -13,10 +13,10 @@ describe("config rotate-token", () => { it("rotates an OpenAI provider without a compatibility-profile mutation (#11229)", async () => { const appendAuditEntry = vi.fn(); const captureOpenshellCommand = vi.fn(() => ({ - output: "openshell 0.0.116\n", + output: "openshell 0.1.7\n", status: 0, stderr: "", - stdout: "openshell 0.0.116\n", + stdout: "openshell 0.1.7\n", })); const runOpenshellCommand = vi.fn( (): ReturnType => @@ -100,7 +100,7 @@ describe("config rotate-token", () => { } satisfies RotateTokenDeps; await expect(rotateSandboxToken("rotate-profile-test", {}, deps)).rejects.toThrow( - "expected 0.0.116, actual 0.0.106", + "expected 0.1.7, actual 0.0.106", ); expect(captureOpenshellCommand).toHaveBeenCalledOnce(); @@ -170,10 +170,10 @@ describe("config rotate-token", () => { () => ({ status: 0 }) as ReturnType, ); const captureOpenshellCommand = vi.fn(() => ({ - output: "openshell 0.0.116\n", + output: "openshell 0.1.7\n", status: 0, stderr: "", - stdout: "openshell 0.0.116\n", + stdout: "openshell 0.1.7\n", })); const deps = { appendAuditEntry: vi.fn(), @@ -269,10 +269,10 @@ describe("config rotate-token", () => { const deps = { appendAuditEntry: vi.fn(), captureOpenshellCommand: vi.fn(() => ({ - output: "openshell 0.0.116\n", + output: "openshell 0.1.7\n", status: 0, stderr: "", - stdout: "openshell 0.0.116\n", + stdout: "openshell 0.1.7\n", })), fail: (lines: string | readonly string[]): never => { throw new Error(typeof lines === "string" ? lines : lines.join("\n")); @@ -351,10 +351,10 @@ describe("config rotate-token", () => { const deps = { appendAuditEntry, captureOpenshellCommand: vi.fn(() => ({ - output: "openshell 0.0.116\n", + output: "openshell 0.1.7\n", status: 0, stderr: "", - stdout: "openshell 0.0.116\n", + stdout: "openshell 0.1.7\n", })), fail: (lines: string | readonly string[]): never => { throw new Error(typeof lines === "string" ? lines : lines.join("\n")); diff --git a/test/security/fetch-guard-patch-regression.test.ts b/test/security/fetch-guard-patch-regression.test.ts index f3347048597..08239535000 100644 --- a/test/security/fetch-guard-patch-regression.test.ts +++ b/test/security/fetch-guard-patch-regression.test.ts @@ -8,6 +8,7 @@ import path from "node:path"; import { describe, expect, it } from "vitest"; import { CURRENT_REVIEWED_OPENCLAW_PATCH_CLASSIFIER_VERSION, + LEGACY_OPENCLAW_PATCH_FIXTURE_VERSION, dockerRunCommandBetween, runDockerfilePatchBlock, runFetchGuardPatchBlock, @@ -149,7 +150,7 @@ function runOpenClawUpgradeBlock(currentVersion: string) { fs.writeFileSync(reviewedArchive, "fake reviewed OpenClaw archive"); const command = dockerRunCommandBetween( "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ) .replaceAll( "bash /scripts/lib/verify-mcporter-audit.sh", @@ -295,7 +296,7 @@ function webGuardedFetchFixtureSource(): string { ].join("\n"); } -describe("fetch-guard patch regression guard", () => { +describe("OpenClaw installation and legacy proxy fixtures", () => { it("anchors web_fetch proxy mode to the reviewed OpenClaw 2026.7.1 contract", () => { expect(REVIEWED_OPENCLAW_2026_7_1_WEB_FETCH_SHAPE).toContain( "function fetchWithWebToolsNetworkGuard(params)", @@ -406,12 +407,12 @@ describe("fetch-guard patch regression guard", () => { expect( dockerRunCommandBetween( "# OPENCLAW_VERSION is the NemoClaw runtime build target", - "# Patch OpenClaw media fetch", + "# Patch OpenClaw network boundaries", ), ).toContain("rm -rf /usr/local/lib/node_modules/mcporter /usr/local/bin/mcporter"); }); - it("applies the Dockerfile OpenClaw compatibility patch block to executable fixtures", async () => { + it("applies legacy proxy and shared install patches to executable fixtures", async () => { const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openclaw-patches-")); const dist = path.join(tmp, "dist"); fs.mkdirSync(dist, { recursive: true }); @@ -483,7 +484,7 @@ describe("fetch-guard patch regression guard", () => { dist, tmp, "# Native OpenClaw restart must reload", - CURRENT_REVIEWED_OPENCLAW_PATCH_CLASSIFIER_VERSION, + LEGACY_OPENCLAW_PATCH_FIXTURE_VERSION, ); expect(patch.status, `${patch.stdout}${patch.stderr}`).toBe(0); expect(patch.stdout).toContain("Patch 1 applied"); @@ -606,11 +607,7 @@ describe("fetch-guard patch regression guard", () => { fs.writeFileSync(webGuardPath, webGuardedFetchFixtureSource()); try { - const patch = runFetchGuardPatchBlock( - dist, - tmp, - CURRENT_REVIEWED_OPENCLAW_PATCH_CLASSIFIER_VERSION, - ); + const patch = runFetchGuardPatchBlock(dist, tmp, LEGACY_OPENCLAW_PATCH_FIXTURE_VERSION); expect(patch.status, `${patch.stdout}${patch.stderr}`).toBe(0); expect(patch.stdout).toContain("Patch 1 applied"); expect(patch.stdout).toContain("Patch 2 applied"); @@ -1120,11 +1117,7 @@ if (!blocked) throw new Error('private IP literal was not blocked');`, ); try { - const patch = runFetchGuardPatchBlock( - dist, - tmp, - CURRENT_REVIEWED_OPENCLAW_PATCH_CLASSIFIER_VERSION, - ); + const patch = runFetchGuardPatchBlock(dist, tmp, LEGACY_OPENCLAW_PATCH_FIXTURE_VERSION); expect(patch.status, `${patch.stdout}${patch.stderr}`).toBe(0); expect(patch.stdout).toContain("Patch 4 applied"); const patched = fs.readFileSync(modulePath, "utf-8"); @@ -1316,7 +1309,7 @@ if (!blocked) throw new Error('private IP literal was not blocked');`, const patch = runFetchGuardPatchBlock(dist, tmp); expect(patch.status, `${patch.stdout}${patch.stderr}`).toBe(0); expect(patch.stdout).toContain( - "Patch 6 applied to OpenClaw 2026.9.2 cron preflight trusted env-proxy", + "Patch 6 applied to OpenClaw 2026.7.1 cron preflight trusted env-proxy", ); const patched = fs.readFileSync(preflightPath, "utf-8"); expect( @@ -1372,7 +1365,7 @@ if (!blocked) throw new Error('private IP literal was not blocked');`, const patch = runFetchGuardPatchBlock(dist, tmp); expect(patch.status, `${patch.stdout}${patch.stderr}`).toBe(0); expect(patch.stdout).toContain( - "OpenClaw 2026.9.2 has no cron model-provider preflight; Patch 6 not needed", + "OpenClaw 2026.7.1 has no cron model-provider preflight; Patch 6 not needed", ); } finally { fs.rmSync(tmp, { recursive: true, force: true }); diff --git a/test/security/openclaw-native-network.test.ts b/test/security/openclaw-native-network.test.ts new file mode 100644 index 00000000000..7469d047ae2 --- /dev/null +++ b/test/security/openclaw-native-network.test.ts @@ -0,0 +1,213 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import { isIP } from "node:net"; +import os from "node:os"; +import path from "node:path"; +import vm from "node:vm"; +import { describe, expect, it } from "vitest"; +import { runDockerfilePatchBlock } from "../helpers/fetch-guard-patch-harness"; +import { + patchNativeNetworkText, + patchOpenClawNativeNetwork, +} from "../../scripts/lib/patch-openclaw-native-network.mts"; + +// Exercise the reviewed loop against special-use DNS answers. +// The complete pinned classifier is checked by the package probe. +const DNS_BOUNDARY = [ + "function assertAllowedResolvedAddressesOrThrow(results, policy) {", + "\tfor (const entry of results) if (isBlockedHostnameOrIp(entry.address, policy)) throw new SsrFBlockedError(BLOCKED_RESOLVED_IP_MESSAGE);", + "}", +].join("\n"); + +function checkSpecialUseAnswers(addresses: string[], insideOpenShell: boolean): void { + const source = patchNativeNetworkText(DNS_BOUNDARY, "dns").replace( + 'import { isIP as nemoclawIsIP } from "node:net";', + "", + ); + vm.runInNewContext(`${source}\nassertAllowedResolvedAddressesOrThrow(results, {});`, { + nemoclawIsIP: isIP, + process: { env: { OPENSHELL_SANDBOX: insideOpenShell ? "1" : undefined } }, + // All supplied fixtures are special-use addresses. Upstream's real IP + // classifier is exercised separately against the integrity-pinned package. + isBlockedHostnameOrIp: () => true, + SsrFBlockedError: Error, + BLOCKED_RESOLVED_IP_MESSAGE: "blocked DNS answer", + results: addresses.map((address) => ({ address })), + }); +} + +describe("OpenClaw native DNS boundary", () => { + it.each([ + "198.18.0.2", + "198.19.255.255", + "fd23:6f70:656e::2", + "FD23:6F70:656E:0000:0000:0000:0000:0002", + ])("accepts mapped DNS answer %s only inside OpenShell", (address) => { + expect(() => checkSpecialUseAnswers([address], true)).not.toThrow(); + expect(() => checkSpecialUseAnswers([address], false)).toThrow("blocked DNS answer"); + }); + + it.each([ + "198.18.0.1", // OpenShell's sandbox-local API is not an external DNS mapping. + "198.17.255.255", + "198.20.0.0", + "10.0.0.1", + "127.0.0.1", + "169.254.169.254", + "0.0.0.0", + "fd23:6f70:656f::2", + "fd00::2", + "::1", + "::ffff:198.18.0.2", + "fd23:6f70:656e::2%eth0", + "198.18.0.256", + ])("retains rejection of unmapped or malformed DNS answer %s", (address) => { + expect(() => checkSpecialUseAnswers([address], true)).toThrow("blocked DNS answer"); + }); + + it("rejects mixed mapped and private DNS answers", () => { + expect(() => checkSpecialUseAnswers(["198.18.0.2", "10.0.0.1"], true)).toThrow( + "blocked DNS answer", + ); + }); + + it.each(["", `${DNS_BOUNDARY}\n${DNS_BOUNDARY}`])( + "refuses missing or duplicated DNS boundaries", + (source) => { + expect(() => patchNativeNetworkText(source, "dns")).toThrow("expected one reviewed target"); + }, + ); +}); + +const WEB_BOUNDARY = [ + "function run(resolved, useEnvProxy) {", + "return fetchWithSsrFGuard(useEnvProxy ? withTrustedEnvProxyGuardedFetchMode(resolved) : withStrictGuardedFetchMode(resolved));", + "}", + "run(input, false);", +].join("\n"); + +describe("OpenClaw host-gateway fetch policy", () => { + it.each([ + [true, "host.openshell.internal", ["existing.example", "host.openshell.internal"]], + [false, "host.openshell.internal", ["existing.example"]], + [true, "other.internal", ["existing.example"]], + ])("keeps caller restrictions with sandbox=%s and host=%s", (inside, hostname, allowed) => { + const result = vm.runInNewContext(patchNativeNetworkText(WEB_BOUNDARY, "web-fetch"), { + URL, + process: { env: { OPENSHELL_SANDBOX: inside ? "1" : undefined } }, + input: { + url: `https://${hostname}/`, + policy: { + allowedHostnames: ["existing.example"], + hostnameAllowlist: ["permitted.example"], + blockedHostnames: ["blocked.example"], + }, + }, + fetchWithSsrFGuard: (params: unknown) => params, + withTrustedEnvProxyGuardedFetchMode: () => { + throw new Error("unexpected proxy path"); + }, + withStrictGuardedFetchMode: (params: object) => ({ ...params, mode: "strict" }), + }); + expect(result).toEqual({ + url: `https://${hostname}/`, + mode: "strict", + policy: { + allowedHostnames: allowed, + hostnameAllowlist: ["permitted.example"], + blockedHostnames: ["blocked.example"], + }, + }); + }); +}); + +it("rejects an unreviewed module before modifying either target", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-native-network-")); + try { + const dist = path.join(directory, "dist"); + fs.mkdirSync(dist); + fs.writeFileSync( + path.join(directory, "package.json"), + JSON.stringify({ name: "openclaw", version: "2026.9.2" }), + ); + const files = ["ssrf-0QyXWOVG.js", "web-guarded-fetch-DfTFSIPT.js"]; + fs.writeFileSync(path.join(dist, files[0]), "unreviewed source\n"); + fs.writeFileSync(path.join(dist, files[1]), "unreviewed source\n"); + expect(() => patchOpenClawNativeNetwork(dist)).toThrow("Unreviewed OpenClaw network module"); + expect(files.map((file) => fs.readFileSync(path.join(dist, file), "utf8"))).toEqual([ + "unreviewed source\n", + "unreviewed source\n", + ]); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +describe("OpenClaw image network selection", () => { + it("refuses a symbolic link module without changing its target", () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-network-symlink-")); + try { + const dist = path.join(directory, "dist"); + fs.mkdirSync(dist); + fs.writeFileSync( + path.join(directory, "package.json"), + JSON.stringify({ name: "openclaw", version: "2026.9.2" }), + ); + const victim = path.join(directory, "outside-module.js"); + fs.writeFileSync(victim, "keep this file unchanged\n"); + fs.symlinkSync(victim, path.join(dist, "ssrf-0QyXWOVG.js")); + expect(() => patchOpenClawNativeNetwork(dist)).toThrow(); + expect(fs.readFileSync(victim, "utf8")).toBe("keep this file unchanged\n"); + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } + }); + + it.each([false, true])( + "rejects modified current modules even with legacy fixture=%s", + (legacyFixture) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-image-network-")); + try { + const dist = path.join(root, "dist"); + fs.mkdirSync(dist); + fs.writeFileSync( + path.join(root, "package.json"), + JSON.stringify({ name: "openclaw", version: "2026.9.2" }), + ); + const modulePath = path.join(dist, "ssrf-0QyXWOVG.js"); + fs.writeFileSync(modulePath, "unreviewed input"); + const result = runDockerfilePatchBlock( + dist, + root, + "# --- Patch 3: follow symlinks", + "2026.9.2", + legacyFixture, + ); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("Unreviewed OpenClaw network module"); + expect(fs.readFileSync(modulePath, "utf8")).toBe("unreviewed input"); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, + ); + + it("refuses legacy network patches without the E2E fixture selector", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-image-network-")); + try { + const result = runDockerfilePatchBlock( + root, + root, + "# --- Patch 3: follow symlinks", + "2026.4.24", + false, + ); + expect(result.status).not.toBe(0); + expect(result.stderr).toContain("Native network patch requires the reviewed current version"); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/test/state/native-quiescence.test.ts b/test/state/native-quiescence.test.ts index 93a1a12a089..3225d63c736 100644 --- a/test/state/native-quiescence.test.ts +++ b/test/state/native-quiescence.test.ts @@ -62,19 +62,24 @@ function assertNativeStateResumed(): void { } it.each([ - ["exited leader with stopped sibling", "Z", "Z", "T"], - ["exited thread group", "Z", "Z", "Z"], -] as const)("captures quiescent native state for %s", (_case, leader, first, second) => { - writeNativeQuiescenceStates(procRoot, leader, first, second); - const backup = sandboxState.backupSandboxState("alpha"); - expect(backup.success, backup.error).toBe(true); - assertNativeStateResumed(); - expect(backup.manifest?.backupComplete).toBe(true); - const archive = path.join(backup.manifest!.backupPath, "native-home.tar"); - expect(spawnSync("tar", ["-xOf", archive, "./payload.txt"], { encoding: "utf8" }).stdout).toBe( - "native payload", - ); -}); + ["exited leader with stopped sibling", "Z", "Z", "T", false], + ["exited thread group", "Z", "Z", "Z", false], + ["native init ancestor", "T", "T", "T", true], +] as const)( + "captures quiescent native state for %s", + (_case, leader, first, second, initAncestor) => { + vi.stubEnv("NEMOCLAW_TEST_NATIVE_INIT_ANCESTOR", String(Number(initAncestor))); + writeNativeQuiescenceStates(procRoot, leader, first, second); + const backup = sandboxState.backupSandboxState("alpha"); + expect(backup.success, backup.error).toBe(true); + assertNativeStateResumed(); + expect(backup.manifest?.backupComplete).toBe(true); + const archive = path.join(backup.manifest!.backupPath, "native-home.tar"); + expect(spawnSync("tar", ["-xOf", archive, "./payload.txt"], { encoding: "utf8" }).stdout).toBe( + "native payload", + ); + }, +); it("classifies a churning live capture without retaining a failed backup (#12867)", () => { vi.stubEnv("NEMOCLAW_TEST_NATIVE_CHURN_MARKER", churnMarker); diff --git a/test/state/snapshot-managed-restore-authority.test.ts b/test/state/snapshot-managed-restore-authority.test.ts index 26236a79b93..ed8b1418cfe 100644 --- a/test/state/snapshot-managed-restore-authority.test.ts +++ b/test/state/snapshot-managed-restore-authority.test.ts @@ -22,6 +22,9 @@ const ORIGINAL_OPENSHELL = process.env.NEMOCLAW_OPENSHELL_BIN; const BIN_DIR = path.join(TMP_HOME, "bin"); fs.mkdirSync(BIN_DIR, { recursive: true }); +fs.writeFileSync(path.join(BIN_DIR, "sftp"), "#!/usr/bin/env node\nprocess.stdin.resume();\n", { + mode: 0o755, +}); fs.writeFileSync( path.join(BIN_DIR, "openshell"), `#!/bin/sh @@ -41,6 +44,10 @@ fs.writeFileSync( path.join(BIN_DIR, "ssh"), `#!/usr/bin/env node const command = process.argv.at(-1) || ""; +if (command.includes('mkdir -m 700 -- "$transfer"')) { + process.stdout.write("1:1\\n"); + process.exit(0); +} if (command.includes("printf") && command.includes("$HOME")) { process.stdout.write(Buffer.from("/sandbox\\0/sandbox\\0")); process.exit(0); @@ -180,6 +187,10 @@ describe("managed rebuild restore authority", () => { const fs = require("node:fs"); const command = process.argv.at(-1) || ""; if (${exitCode} !== 0) { process.stderr.write("operation failed"); process.exit(${exitCode}); } +if (command.includes('mkdir -m 700 -- "$transfer"')) { + process.stdout.write("1:1\\n"); + process.exit(0); +} if (command.includes("printf") && command.includes("$HOME")) { process.stdout.write(Buffer.from("/sandbox\\0/sandbox\\0")); process.exit(0); @@ -283,7 +294,7 @@ process.stdin.on("end", () => process.exit(0)); }, ); expect(restored.success, restored.error).toBe(true); - expect(validateBeforeMutation).toHaveBeenCalledOnce(); + expect(validateBeforeMutation).toHaveBeenCalledTimes(2); }, ); }); diff --git a/test/state/snapshot-native-backslash.test.ts b/test/state/snapshot-native-backslash.test.ts index 8848d027846..08cabcc5e31 100644 --- a/test/state/snapshot-native-backslash.test.ts +++ b/test/state/snapshot-native-backslash.test.ts @@ -396,7 +396,9 @@ describe("native backup of backslash paths across tar quoting modes", () => { ' fs.writeFileSync(listFile, Buffer.from(patterns.map((pattern) => pattern.replaceAll("\\\\\\\\", "\\\\")).join("\\0") + "\\0"));', " }", "}", - `process.exit(spawnSync(${JSON.stringify(systemTar)}, args, { stdio: "inherit" }).status ?? 90);`, + // The restore path passes an unlinked archive through descriptor 3. + 'const stdio = args.includes("/dev/fd/3") || args.includes("/proc/self/fd/3") ? ["inherit", "inherit", "inherit", 3] : "inherit";', + `process.exit(spawnSync(${JSON.stringify(systemTar)}, args, { stdio }).status ?? 90);`, ].join("\n"), ); }, diff --git a/test/state/snapshot-native-inventory.test.ts b/test/state/snapshot-native-inventory.test.ts index 4038b87df26..c9642b0e1d8 100644 --- a/test/state/snapshot-native-inventory.test.ts +++ b/test/state/snapshot-native-inventory.test.ts @@ -524,3 +524,10 @@ process.exit(result.status ?? 1); } }, ); + +it("reserves capacity for every concurrent stopped-state copy", () => { + expect(sandboxState.nativeStateCaptureMaxBytes(testHome, 1024, 2)).toBe(512); + expect(() => sandboxState.nativeStateCaptureMaxBytes(testHome, 1024, 0)).toThrow( + "concurrent-copy count must be a positive integer", + ); +}); diff --git a/test/state/snapshot-native-transfer.test.ts b/test/state/snapshot-native-transfer.test.ts new file mode 100644 index 00000000000..38ec9aaaf6a --- /dev/null +++ b/test/state/snapshot-native-transfer.test.ts @@ -0,0 +1,170 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import childProcess from "node:child_process"; +import { syncBuiltinESMExports } from "node:module"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { + writeFakeOpenshell, + writeFakeSsh, + writeSnapshotRegistry, +} from "../support/native-snapshot"; + +const originalHome = process.env.HOME; +const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-native-transfer-")); +process.env.HOME = home; +const state = await import("../../src/lib/state/sandbox"); +let nativeRoot: string; +let bin: string; +let backupPath: string; +const payload = Buffer.alloc(5 * 1024 * 1024 + 17, 97); + +beforeEach(() => { + const fixture = fs.mkdtempSync(path.join(home, "case-")); + nativeRoot = path.join(fixture, "native"); + bin = path.join(fixture, "bin"); + fs.mkdirSync(nativeRoot); + fs.mkdirSync(bin); + writeFakeOpenshell(bin); + writeFakeSsh(bin); + writeSnapshotRegistry(home, "alpha", "hermes"); + vi.stubEnv("PATH", `${bin}:${process.env.PATH}`); + vi.stubEnv("NEMOCLAW_OPENSHELL_BIN", path.join(bin, "openshell")); + vi.stubEnv("NEMOCLAW_TEST_NATIVE_ROOT", nativeRoot); + vi.stubEnv("NEMOCLAW_TEST_NATIVE_HOME", nativeRoot); + vi.stubEnv("NEMOCLAW_TEST_NATIVE_WORKSPACE", nativeRoot); + fs.writeFileSync(path.join(nativeRoot, "payload"), payload); + const backup = state.backupSandboxState("alpha", { + nativeStateSource: { root: nativeRoot, directory: nativeRoot, assertCurrent: vi.fn() }, + }); + expect(backup.success, backup.error).toBe(true); + backupPath = backup.manifest!.backupPath; + fs.rmSync(path.join(nativeRoot, "payload")); + fs.writeFileSync(path.join(nativeRoot, "replacement"), "unchanged"); +}); + +afterAll(() => { + originalHome === undefined + ? Reflect.deleteProperty(process.env, "HOME") + : Reflect.set(process.env, "HOME", originalHome); + fs.rmSync(home, { recursive: true, force: true }); +}); + +afterEach(() => { + vi.restoreAllMocks(); + syncBuiltinESMExports(); +}); + +describe("native restore transfer", () => { + it("uploads the validated descriptor even when the original archive is replaced", async () => { + const sftp = path.join(bin, "sftp"); + fs.writeFileSync( + sftp, + fs + .readFileSync(sftp, "utf8") + .replace( + "const batch =", + `fs.writeFileSync(${JSON.stringify(path.join(backupPath, "native-home.tar"))}, "replaced after validation");\nconst batch =`, + ), + ); + + const restored = await state.restoreSandboxState("alpha", backupPath); + + expect(restored.success, restored.error).toBe(true); + expect(fs.readFileSync(path.join(backupPath, "native-home.tar"), "utf8")).toBe( + "replaced after validation", + ); + expect(fs.readFileSync(path.join(nativeRoot, "payload")).equals(payload)).toBe(true); + expect( + fs.readdirSync(nativeRoot).filter((name) => name.startsWith(".nemoclaw-native-restore")), + ).toEqual([]); + expect(fs.existsSync(path.join(nativeRoot, "replacement"))).toBe(false); + }); + + it.each([ + ["checksum mismatch", 'fs.writeFileSync(target, "tampered");'], + ["exit 42", "process.exit(42);"], + ["exit 1", 'fs.linkSync(target, path.join(root, "retained-hardlink"));'], + ])("preserves replacement state and the backup after %s", async (error, injection) => { + fs.appendFileSync(path.join(bin, "sftp"), injection); + const originalArchive = fs.readFileSync(path.join(backupPath, "native-home.tar")); + + const restored = await state.restoreSandboxState("alpha", backupPath); + + expect(restored).toMatchObject({ success: false, restoredDirs: [] }); + expect(restored.error).toContain(error); + expect(fs.readFileSync(path.join(nativeRoot, "replacement"), "utf8")).toBe("unchanged"); + expect(fs.existsSync(path.join(nativeRoot, "payload"))).toBe(false); + expect( + fs.readdirSync(nativeRoot).filter((name) => name.startsWith(".nemoclaw-native-restore")), + ).toEqual([]); + expect(fs.readFileSync(path.join(backupPath, "native-home.tar")).equals(originalArchive)).toBe( + true, + ); + }); + + it("rechecks authority after upload and removes staging when authority changes", async () => { + const validateBeforeMutation = vi + .fn() + .mockResolvedValueOnce(undefined) + .mockRejectedValueOnce(new Error("replacement generation changed")); + + const restored = await state.restoreSandboxState("alpha", backupPath, { + validateBeforeMutation, + }); + + expect(validateBeforeMutation).toHaveBeenCalledTimes(2); + expect(restored).toMatchObject({ success: false, restoredDirs: [] }); + expect(restored.error).toContain("replacement generation changed"); + expect(fs.readdirSync(nativeRoot)).toEqual(["replacement"]); + }); + + it("reports cleanup failure without deleting a substituted staging directory", async () => { + const sftp = path.join(bin, "sftp"); + fs.appendFileSync( + sftp, + ` +const transfer = path.dirname(target); +fs.renameSync(transfer, transfer + ".retained"); +fs.mkdirSync(transfer); +fs.writeFileSync(path.join(transfer, "unrelated"), "preserve"); +process.exit(42); +`, + ); + + const restored = await state.restoreSandboxState("alpha", backupPath); + + expect(restored.success).toBe(false); + expect(restored.error).toContain("upload failed: exit 42"); + expect(restored.error).toContain("cleanup failed"); + const substituted = fs + .readdirSync(nativeRoot) + .find((name) => name.startsWith(".nemoclaw-native-restore") && !name.endsWith(".retained"))!; + expect(fs.readFileSync(path.join(nativeRoot, substituted, "unrelated"), "utf8")).toBe( + "preserve", + ); + expect(fs.readFileSync(path.join(nativeRoot, "replacement"), "utf8")).toBe("unchanged"); + }); + + it("rejects an unavailable SFTP client before obtaining target access", async () => { + const calls = vi.spyOn(childProcess, "spawnSync").mockReturnValue({ + pid: 0, + output: [], + stdout: Buffer.alloc(0), + stderr: Buffer.alloc(0), + status: null, + signal: null, + error: new Error("ENOENT"), + }); + syncBuiltinESMExports(); + + const restored = await state.restoreSandboxState("alpha", backupPath); + + expect(restored.error).toContain("Install the OpenSSH client tools"); + expect(calls.mock.calls.map(([command]) => command)).toEqual(["sftp"]); + expect(fs.readdirSync(nativeRoot)).toEqual(["replacement"]); + }); +}); diff --git a/test/state/snapshot.test.ts b/test/state/snapshot.test.ts index dd1b81591e8..24f6d796858 100644 --- a/test/state/snapshot.test.ts +++ b/test/state/snapshot.test.ts @@ -935,12 +935,6 @@ describe("complete native home persistence", () => { } }); - it("reserves capacity for every concurrent stopped-state copy", () => { - expect(sandboxState.nativeStateCaptureMaxBytes(TMP_HOME, 1024, 2)).toBe(512); - expect(() => sandboxState.nativeStateCaptureMaxBytes(TMP_HOME, 1024, 0)).toThrow( - "concurrent-copy count must be a positive integer", - ); - }); it.each([ ["a short assignment in an ordinary note", "workspace/notes.txt", "password=abc"], [ @@ -1168,7 +1162,9 @@ describe("complete native home persistence", () => { path.join(binDir, "stat"), `#!/usr/bin/env node const fs = require("node:fs"); -if (process.argv[2] !== "-c" || process.argv[3] !== "%u") process.exit(95); +if (process.argv[2] !== "-c" || process.argv[3] !== "%u") { + process.exit(require("node:child_process").spawnSync("native-stat", process.argv.slice(2), { stdio: process.argv.at(-1).startsWith("/dev/fd/") ? ["inherit", "inherit", "inherit", 3] : "inherit" }).status ?? 95); +} process.stdout.write(String(fs.lstatSync(process.argv.at(-1)).uid) + "\\n"); `, ); @@ -1205,7 +1201,7 @@ process.stdout.write(String(fs.lstatSync(process.argv.at(-1)).uid) + "\\n"); expect(commands).toContain('2>/dev/null < "$task/status"'); expect(commands).not.toContain('< "$proc/status" 2>/dev/null'); expect(commands).toContain("-links +1"); - expect(commands).toContain('mktemp -d "$root/.nemoclaw-native-restore.XXXXXX"'); + expect(commands).toContain('mkdir -m 700 -- "$transfer"'); expect(commands).toContain('owner="$(stat -c %u -- "$target_item")"'); expect(commands).toContain('[ "$owner" = "$uid" ] && [ -w "$target_dir" ]'); expect(commands).toContain('restore_dir "$source_item" "$target_item"'); @@ -1257,7 +1253,9 @@ process.stdout.write(String(fs.lstatSync(process.argv.at(-1)).uid) + "\\n"); const fs = require("node:fs"); const path = require("node:path"); const target = process.argv.at(-1); -if (process.argv[2] !== "-c" || process.argv[3] !== "%u" || !target) process.exit(95); +if (process.argv[2] !== "-c" || process.argv[3] !== "%u" || !target) { + process.exit(require("node:child_process").spawnSync("native-stat", process.argv.slice(2), { stdio: process.argv.at(-1).startsWith("/dev/fd/") ? ["inherit", "inherit", "inherit", 3] : "inherit" }).status ?? 95); +} const uid = path.basename(target) === "image-owned" && (!fs.lstatSync(target).isFile() || fs.readFileSync(target, "utf8") !== "must be removed") ? (process.getuid?.() ?? 1000) + 1 : fs.lstatSync(target).uid; process.stdout.write(String(uid) + "\\n"); `, diff --git a/test/support/dcode-start-script-fixture.ts b/test/support/dcode-start-script-fixture.ts index 5443fc0692c..a4a2a92bc0a 100644 --- a/test/support/dcode-start-script-fixture.ts +++ b/test/support/dcode-start-script-fixture.ts @@ -9,19 +9,13 @@ const repoRoot = path.join(import.meta.dirname, "..", ".."); const START_SCRIPT = path.join(repoRoot, "agents", "langchain-deepagents-code", "start.sh"); const ENTRYPOINT_ENV_WRAPPER = path.join(repoRoot, "scripts", "lib", "entrypoint-env-wrapper.sh"); -export type ManagedProxyEndpoint = { host: string; port: string }; - -export const DEFAULT_MANAGED_PROXY: ManagedProxyEndpoint = { host: "10.200.0.1", port: "3128" }; - -export type ManagedProxyScriptOptions = { +export type ManagedScriptOptions = { installRlimitHelper?: (rlimitLib: string) => void; - managedProxy?: ManagedProxyEndpoint; + caFile?: string; }; -export type StartScriptFixtureOptions = ManagedProxyScriptOptions & { +export type StartScriptFixtureOptions = ManagedScriptOptions & { envDir?: string; - fallbackCaFile?: string; - liveCaFile?: string; markerDir?: string; }; @@ -43,41 +37,23 @@ function writeReadOnlyFile(file: string, contents: string): void { fs.chmodSync(file, 0o444); } -export function prepareManagedProxyFixture( +export function prepareManagedScriptFixture( source: string, tempDir: string, - options: ManagedProxyScriptOptions = {}, + options: ManagedScriptOptions = {}, ): string { - const managedProxy = options.managedProxy ?? DEFAULT_MANAGED_PROXY; const installRlimitHelper = options.installRlimitHelper ?? writeRlimitStub; const rlimitLib = path.join(tempDir, "sandbox-rlimits.sh"); - const hostFile = path.join(tempDir, "trusted-proxy-host"); - const portFile = path.join(tempDir, "trusted-proxy-port"); - const caFile = path.join(tempDir, "trusted-ca-bundle.pem"); - writeReadOnlyFile(hostFile, `${managedProxy.host}\n`); - writeReadOnlyFile(portFile, `${managedProxy.port}\n`); - writeReadOnlyFile(caFile, "trusted CA bundle\n"); + const caFile = options.caFile ?? path.join(tempDir, "trusted-ca-bundle.pem"); + if (options.caFile === undefined) writeReadOnlyFile(caFile, "trusted CA bundle\n"); installRlimitHelper(rlimitLib); return source .replace("/usr/local/lib/nemoclaw/entrypoint-env-wrapper.sh", ENTRYPOINT_ENV_WRAPPER) .replace("/usr/local/lib/nemoclaw/sandbox-rlimits.sh", rlimitLib) .replace("../../scripts/lib/sandbox-rlimits.sh", "missing-dev-sandbox-rlimits.sh") - .replaceAll("/tmp/nemoclaw-managed-startup-ca-bundle.pem", caFile) - .replace( - 'readonly MANAGED_PROXY_HOST_FILE="/usr/local/share/nemoclaw/dcode-proxy-host"', - `readonly MANAGED_PROXY_HOST_FILE="${hostFile}"`, - ) .replace( - 'readonly MANAGED_PROXY_PORT_FILE="/usr/local/share/nemoclaw/dcode-proxy-port"', - `readonly MANAGED_PROXY_PORT_FILE="${portFile}"`, - ) - .replace( - 'readonly MANAGED_FETCH_CA_BUNDLE_FILE="/etc/openshell-tls/ca-bundle.pem"', + 'readonly MANAGED_FETCH_CA_BUNDLE_FILE="/run/openshell-supervisor-ca/material/ca-bundle.crt"', `readonly MANAGED_FETCH_CA_BUNDLE_FILE="${caFile}"`, - ) - .replace( - "readonly MANAGED_PROXY_OWNER_UID=0", - `readonly MANAGED_PROXY_OWNER_UID=${process.getuid?.() ?? 0}`, ); } @@ -92,21 +68,12 @@ export function makeStartScriptFixture( const envFile = path.join(envDir, "proxy-env.sh"); const scriptPath = path.join(tempDir, "start.sh"); const markerDir = options.markerDir; - const original = fs - .readFileSync(START_SCRIPT, "utf8") - .replaceAll( - "/etc/openshell-tls/ca-bundle.pem", - options.liveCaFile ?? "/etc/openshell-tls/ca-bundle.pem", - ) - .replaceAll( - "/tmp/nemoclaw-managed-startup-ca-bundle.pem", - options.fallbackCaFile ?? "/tmp/nemoclaw-managed-startup-ca-bundle.pem", - ); + const original = fs.readFileSync(START_SCRIPT, "utf8"); assert.ok(original.includes("local target=/tmp/nemoclaw-proxy-env.sh")); assert.ok(original.includes('tmp="$(mktemp /tmp/nemoclaw-proxy-env.XXXXXX)"')); assert.ok(original.includes("local marker_dir=/sandbox/.deepagents")); fs.mkdirSync(envDir, { recursive: true }); - const envRedirected = prepareManagedProxyFixture(original, tempDir, options) + const envRedirected = prepareManagedScriptFixture(original, tempDir, options) .replace("local target=/tmp/nemoclaw-proxy-env.sh", `local target="${envFile}"`) .replace( 'tmp="$(mktemp /tmp/nemoclaw-proxy-env.XXXXXX)"', diff --git a/test/support/docker-driver-gateway-env-test-support.ts b/test/support/docker-driver-gateway-env-test-support.ts index 359bacfcb41..e28a7fa867d 100644 --- a/test/support/docker-driver-gateway-env-test-support.ts +++ b/test/support/docker-driver-gateway-env-test-support.ts @@ -4,14 +4,14 @@ import fs from "node:fs"; import path from "node:path"; -import { DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS } from "../../src/lib/onboard/docker-driver-gateway-config"; - export function writeSafeGatewayAuthConfig(dir: string): string { const configPath = path.join(dir, "openshell-gateway.toml"); const jwtDir = path.join(dir, "jwt"); const signingKeyPath = path.join(jwtDir, "signing.pem"); const publicKeyPath = path.join(jwtDir, "public.pem"); const kidPath = path.join(jwtDir, "kid"); + const caPath = path.join(dir, "test-ca.crt"); + fs.writeFileSync(caPath, "test certificate fixture\n", { mode: 0o600 }); fs.mkdirSync(jwtDir, { recursive: true, mode: 0o700 }); for (const [filePath, value] of [ [signingKeyPath, "test signing key\n"], @@ -23,11 +23,16 @@ export function writeSafeGatewayAuthConfig(dir: string): string { fs.writeFileSync( configPath, [ + "[openshell]", + "version = 2", + "", "[openshell.gateway]", "disable_tls = false", "", "[openshell.gateway.tls]", - "require_client_auth = true", + `cert_path = ${JSON.stringify(caPath)}`, + `key_path = ${JSON.stringify(signingKeyPath)}`, + `client_ca_path = ${JSON.stringify(caPath)}`, "", "[openshell.gateway.mtls_auth]", "enabled = true", @@ -37,7 +42,6 @@ export function writeSafeGatewayAuthConfig(dir: string): string { `public_key_path = ${JSON.stringify(publicKeyPath)}`, `kid_path = ${JSON.stringify(kidPath)}`, 'gateway_id = "nemoclaw-test"', - `ttl_secs = ${DOCKER_DRIVER_GATEWAY_JWT_TTL_SECS}`, "", "[openshell.gateway.auth]", "allow_unauthenticated_users = false", diff --git a/test/support/native-hermes-setup-harness.ts b/test/support/native-hermes-setup-harness.ts new file mode 100644 index 00000000000..09c4ae42bd5 --- /dev/null +++ b/test/support/native-hermes-setup-harness.ts @@ -0,0 +1,51 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { vi } from "vitest"; +import { hostedNativeProvider } from "../../src/lib/inference/native-provider/hosted"; +import type { OpenShellProviderAdapter } from "../../src/lib/adapters/openshell/provider-adapter"; +import type { SetupInferenceDeps } from "../../src/lib/onboard/setup-inference"; +import { createDirectSetupInferenceHarness } from "../helpers/onboard-split-context"; + +const HERMES_NATIVE_DEFINITION = { + ...hostedNativeProvider("hermes-provider", "https://8.8.8.8/v1")!, + allowedIps: ["8.8.8.8"], +}; +export const HERMES_NATIVE_RECEIPT = { + schemaVersion: 1 as const, + profileId: HERMES_NATIVE_DEFINITION.profileId, + providerName: HERMES_NATIVE_DEFINITION.providerName, + providerId: "hermes-owned-provider", + endpointUrl: HERMES_NATIVE_DEFINITION.endpoint, + allowedIps: HERMES_NATIVE_DEFINITION.allowedIps, +}; + +export function createHermesNativeHarness(overrides: Partial = {}) { + const providerAdapter = { + importProviderProfile: vi.fn(async () => ({ ok: true })), + ensureProviderPolicyComposition: vi.fn(async () => ({ ok: true, value: undefined })), + getProvider: vi.fn(async () => ({ + ok: true, + value: { + name: HERMES_NATIVE_RECEIPT.providerName, + type: HERMES_NATIVE_RECEIPT.profileId, + credentialKeys: ["OPENAI_API_KEY"], + configKeys: [], + revision: { id: HERMES_NATIVE_RECEIPT.providerId, resourceVersion: 1 }, + }, + })), + createProvider: vi.fn(async () => ({ ok: true })), + updateProvider: vi.fn(async () => ({ ok: true })), + } as unknown as OpenShellProviderAdapter; + const writeAuthority = vi.fn(); + const harness = createDirectSetupInferenceHarness({ + overrides: { + isNonInteractive: () => true, + providerAdapter, + getNativeHostedProviderAuthority: () => HERMES_NATIVE_RECEIPT, + setNativeHostedProviderAuthority: writeAuthority, + ...overrides, + }, + }); + return { ...harness, providerAdapter, writeAuthority }; +} diff --git a/test/support/native-local-setup-harness.ts b/test/support/native-local-setup-harness.ts index ebb19f4c440..5464a715299 100644 --- a/test/support/native-local-setup-harness.ts +++ b/test/support/native-local-setup-harness.ts @@ -132,10 +132,8 @@ export async function withNativePolicyFixture(operation: () => Promise): P executable, `#!/bin/sh set -eu -if [ "$#" = 6 ] && [ "$1" = settings ] && [ "$2" = get ] && [ "$3" = -g ] && [ "$5" = --global ] && [ "$6" = --json ]; then - printf '%s\\n' '{"scope":"global","settings":{"providers_v2_enabled":"true"}}' -elif [ "$#" = 7 ] && [ "$1" = policy ] && [ "$2" = list ] && [ "$3" = -g ] && [ "$5" = --global ] && [ "$6" = --limit ] && [ "$7" = 1 ]; then - printf '%s\\n' 'No global policy history found' >&2 +if [ "$#" = 9 ] && [ "$1" = policy ] && [ "$2" = list ] && [ "$3" = -g ] && [ "$5" = --global ] && [ "$6" = --page-size ] && [ "$7" = 1 ] && [ "$8" = --output ] && [ "$9" = json ]; then + printf '%s\\n' '{"revisions":[],"next_page_token":""}' else echo 'Unexpected policy fixture command' >&2 exit 1 diff --git a/test/support/native-snapshot.ts b/test/support/native-snapshot.ts index c02d9b499e8..fa951f1ae0b 100644 --- a/test/support/native-snapshot.ts +++ b/test/support/native-snapshot.ts @@ -48,6 +48,50 @@ process.exit(0); } export function writeFakeSsh(binDir: string): void { + writeExecutable( + path.join(binDir, "sftp"), + `#!/usr/bin/env node +const fs = require("node:fs"); +const path = require("node:path"); +const args = process.argv.slice(2); +if (args[0] === "-h") process.exit(1); +const batch = fs.readFileSync(args[args.indexOf("-b") + 1], "utf8"); +const match = /^put [/]dev[/]stdin "(.+)"\\n$/.exec(batch); +if (!match || args[args.indexOf("-R") + 1] !== "16" || args[args.indexOf("-B") + 1] !== "32768") process.exit(97); +const remote = match[1].replace(/\\\\([\\\\"])/g, "$1"); +const root = process.env.NEMOCLAW_TEST_NATIVE_ROOT; +const remoteRoot = process.env.NEMOCLAW_TEST_NATIVE_HOME || "/sandbox"; +const target = path.join(root, path.relative(remoteRoot, remote)); +fs.writeFileSync(target, fs.readFileSync(0), { mode: 0o600, flag: "wx" }); +`, + ); + writeExecutable( + path.join(binDir, "native-stat"), + `#!/usr/bin/env node +const fs = require("node:fs"); +const args = process.argv.slice(2); +const target = args.at(-1); +const fd = /^[/](?:proc[/]self[/]fd|dev[/]fd)[/](\\d+)$/.exec(target); +const s = fd ? fs.fstatSync(Number(fd[1])) : fs.lstatSync(target); +process.stdout.write(args[1].replace(/%[udih]/g, (key) => String({ "%u": s.uid, "%d": s.dev, "%i": s.ino, "%h": s.nlink }[key])) + "\\n"); +`, + ); + writeExecutable(path.join(binDir, "stat"), '#!/bin/sh\nexec native-stat "$@"\n'); + writeExecutable( + path.join(binDir, "sha256sum"), + `#!/usr/bin/env node +const fs = require("node:fs"); +const crypto = require("node:crypto"); +const fd = Number(process.argv.at(-1).split("/").at(-1)); +const hash = crypto.createHash("sha256"); +const block = Buffer.alloc(65536); +let offset = 0, size; +while ((size = fs.readSync(fd, block, 0, block.length, offset))) { + hash.update(block.subarray(0, size)); offset += size; +} +process.stdout.write(hash.digest("hex") + " " + process.argv.at(-1) + "\\n"); +`, + ); writeExecutable( path.join(binDir, "ssh"), `#!/usr/bin/env node @@ -90,6 +134,15 @@ if (command.includes("tar -C")) { } const procRoot = process.env.NEMOCLAW_TEST_NATIVE_PROC; if (procRoot) { + if (process.env.NEMOCLAW_TEST_NATIVE_INIT_ANCESTOR === "1") { + const parentRoot = path.join(procRoot, String(process.pid)); + const initRoot = path.join(procRoot, "1"); + fs.mkdirSync(parentRoot, { recursive: true }); + fs.mkdirSync(path.join(initRoot, "task", "1"), { recursive: true }); + fs.writeFileSync(path.join(parentRoot, "status"), "PPid:\\t1\\n"); + fs.writeFileSync(path.join(initRoot, "status"), "PPid:\\t0\\nUid:\\t" + process.getuid() + "\\n"); + fs.writeFileSync(path.join(initRoot, "task", "1", "status"), "State:\\tS\\n"); + } const prelude = 'kill() { printf "%s\\\\n" "$*" >> "$NEMOCLAW_TEST_SIGNAL_LOG"; }; sleep() { :; }; tar() { command tar -C "$NEMOCLAW_TEST_NATIVE_ROOT" -cf - .; }; '; const capture = spawnSync("sh", ["-c", prelude + command.replaceAll("/proc/", procRoot + "/")], { stdio: ["ignore", "inherit", "inherit"], @@ -128,36 +181,10 @@ if (command.includes("tar -C")) { } } if (command.includes("nemoclaw-native-restore")) { - if (process.env.NEMOCLAW_TEST_EXECUTE_RESTORE_SCRIPT === "1") { - const restored = spawnSync("sh", ["-c", command], { - stdio: ["inherit", "inherit", "inherit"], - }); - process.exit(restored.status ?? 94); - } - const stage = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-native-restore-test-")); - try { - const extracted = spawnSync("tar", ["--no-same-owner", "-xf", "-", "-C", stage], { stdio: ["inherit", "inherit", "inherit"] }); - if (extracted.status !== 0) process.exit(extracted.status ?? 92); - const walk = (current) => { - for (const name of fs.readdirSync(current)) { - const full = path.join(current, name); - const stat = fs.lstatSync(full); - if (stat.isDirectory()) { - walk(full); - } else if (stat.isFile() && stat.nlink > 1) { - process.exit(21); - } - } - }; - walk(stage); - for (const entry of fs.readdirSync(root)) fs.rmSync(path.join(root, entry), { recursive: true, force: true }); - for (const entry of fs.readdirSync(stage)) { - copyTree(path.join(stage, entry), path.join(root, entry)); - } - process.exit(0); - } finally { - fs.rmSync(stage, { recursive: true, force: true }); - } + const restored = spawnSync("sh", ["-c", command.replaceAll(remoteRoot, root).replaceAll("/proc/self/fd/", "/dev/fd/")], { + stdio: ["inherit", "inherit", "inherit"], + }); + process.exit(restored.status ?? 94); } process.exit(93); `, diff --git a/test/support/openshell-arm64-download.ts b/test/support/openshell-arm64-download.ts index c8f9318e6ef..67aa6dd9626 100644 --- a/test/support/openshell-arm64-download.ts +++ b/test/support/openshell-arm64-download.ts @@ -5,13 +5,15 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import credentialBoundaryManifest from "../../src/lib/actions/sandbox/openshell-child-visible-credentials.v0.0.116.json"; +import credentialBoundaryManifest from "../../src/lib/actions/sandbox/openshell-child-visible-credentials.v0.1.7.json"; import { OPENSHELL_FEATURE_MARKERS, OPENSHELL_MCP_FEATURE_MARKER, PINNED_OPEN_SHELL_SHA256, } from "../helpers/openshell-release-fixtures"; +import { prepareInstallerBin } from "./openshell-installed-version"; + const REQUIRED_OPENSHELL_VERSION = credentialBoundaryManifest.openshellVersion; function writeExecutable(target: string, contents: string) { @@ -25,7 +27,7 @@ export function createLinuxArm64DownloadFixture() { const fakeBin = path.join(tmp, "bin"); const downloadLog = path.join(tmp, "downloads.log"); const checksumLog = path.join(tmp, "checksums.log"); - fs.mkdirSync(fakeBin); + prepareInstallerBin(fakeBin); writeExecutable( path.join(fakeBin, "uname"), diff --git a/test/support/openshell-gateway-config-helpers.ts b/test/support/openshell-gateway-config-helpers.ts index b136c041fa6..c1ede0920a4 100644 --- a/test/support/openshell-gateway-config-helpers.ts +++ b/test/support/openshell-gateway-config-helpers.ts @@ -70,12 +70,6 @@ export function parseTomlString(toml: string, key: string): string { return match?.[1] ?? ""; } -export function parseTomlInteger(toml: string, key: string): number { - const match = toml.match(new RegExp(`^${key} = (\\d+)$`, "m")); - expect(match, `missing TOML integer key ${key}`).not.toBeNull(); - return Number(match?.[1] ?? "0"); -} - export function jwtBundlePaths(stateDir: string): JwtBundlePaths { return { signingKeyPath: path.join(stateDir, "jwt", "signing.pem"), diff --git a/test/support/openshell-installed-version.ts b/test/support/openshell-installed-version.ts new file mode 100644 index 00000000000..6a8134a871b --- /dev/null +++ b/test/support/openshell-installed-version.ts @@ -0,0 +1,394 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { shellQuote } from "../../src/lib/core/shell-quote"; +import { + OPENSHELL_FEATURE_MARKERS, + OPENSHELL_MCP_FEATURE_MARKER, + OPENSHELL_REWRITE_FEATURE_MARKERS, + PINNED_OPEN_SHELL_SHA256, +} from "../helpers/openshell-release-fixtures"; + +const SCRIPTS = path.join(import.meta.dirname, "../..", "scripts"); +type OpenShellFeaturePlacement = "openshell" | "gateway" | "split-mcp-gateway" | "none"; + +export function writeExecutable(target: string, contents: string) { + fs.writeFileSync(target, contents, { mode: 0o755 }); +} +export function prepareInstallerBin(dir: string) { + fs.mkdirSync(dir); + fs.symlinkSync(process.execPath, path.join(dir, "node")); + writeExecutable( + path.join(dir, "mktemp"), + `#!/usr/bin/env bash +if [ "$*" = "-d" ]; then exec /usr/bin/mktemp -d ${shellQuote(path.join(path.dirname(dir), "install.XXXXXX"))}; fi +exec /usr/bin/mktemp "$@"`, + ); +} +export function runCommandWithInstalledVersion( + version: string, + extraEnv: NodeJS.ProcessEnv = {}, + options: { + entrypoint?: "direct" | "feature-check"; + capability?: boolean; + curlExitCode?: number; + featurePlacement?: OpenShellFeaturePlacement; + driverBins?: boolean | "gateway" | "gateway-vm"; + driverLocation?: "path" | "explicit" | "symlink"; + driverVersion?: string; + sandboxVersion?: string; + sandboxMarkers?: string; + sandboxVersionExit?: number; + sandboxBinaryDigest?: string; + driverVersionExit?: number; + driverReadable?: boolean; + homebrewAvailable?: boolean; + homebrewFormulaDownload?: boolean; + homebrewFormulaDigest?: string; + homebrewOperationLog?: string; + legacyGateway?: + | true + | "empty" + | "partial" + | "invalid" + | "unreachable" + | "stderr" + | "retired" + | "metadata-link" + | "metadata-directory" + | "corrupt" + | "fixed-system" + | "system"; + legacyInventory?: unknown; + legacyMetadata?: string; + legacyRegistrationFile?: "user" | "system" | "fixed-system"; + retainedRegistry?: string; + registryGatewayPort?: string; + missingCli?: boolean; + missingNode?: boolean; + systemGatewayDirectory?: string; + versionExit?: number; + os?: string; + arch?: string; + } = {}, +) { + const capability = options.capability ?? true; + const featurePlacement: OpenShellFeaturePlacement = capability + ? (options.featurePlacement ?? "openshell") + : "none"; + const openshellMarkers = + featurePlacement === "openshell" + ? OPENSHELL_FEATURE_MARKERS + : featurePlacement === "split-mcp-gateway" + ? OPENSHELL_REWRITE_FEATURE_MARKERS + : ""; + const gatewayMarkers = + featurePlacement === "gateway" + ? OPENSHELL_FEATURE_MARKERS + : featurePlacement === "split-mcp-gateway" + ? OPENSHELL_MCP_FEATURE_MARKER + : ""; + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-openshell-ver-")); + try { + const downloads = path.join(tmp, "downloads"); + const fakeBin = path.join(tmp, "bin"); + const driverBin = options.driverLocation ? path.join(tmp, "driver-bin") : fakeBin; + prepareInstallerBin(fakeBin); + if (options.missingNode) { + fs.unlinkSync(path.join(fakeBin, "node")); + // Exclude system PATH entries that can contain Node.js on CI hosts. + for (const name of ["bash", "awk", "cat", "chmod", "dirname", "grep", "mkdir", "rm", "sed"]) { + const binary = ["/usr/bin", "/bin"] + .map((root) => path.join(root, name)) + .find(fs.existsSync); + if (!binary) throw new Error(`Missing installer fixture tool: ${name}`); + fs.symlinkSync(binary, path.join(fakeBin, name)); + } + } + fs.mkdirSync(driverBin, { recursive: true }); + const configHome = path.join(tmp, "config"); + const retainedFiles = new Map(); + if (options.legacyRegistrationFile) { + const root = + options.legacyRegistrationFile === "user" + ? path.join(configHome, "openshell") + : path.join(tmp, `${options.legacyRegistrationFile}-openshell`); + const registration = path.join(root, "gateways", "nemoclaw"); + fs.mkdirSync(path.dirname(registration), { recursive: true }); + const contents = "synthetic retained gateway registration"; + fs.writeFileSync(registration, contents); + retainedFiles.set(registration, contents); + } + if (options.retainedRegistry !== undefined) { + const registry = path.join( + tmp, + ".nemoclaw", + ...(options.registryGatewayPort ? ["gateways", options.registryGatewayPort] : []), + "sandboxes.json", + ); + fs.mkdirSync(path.dirname(registry), { recursive: true }); + fs.writeFileSync(registry, options.retainedRegistry); + retainedFiles.set(registry, options.retainedRegistry); + } + if (options.legacyGateway) { + const gatewayRoot = + options.legacyGateway === "fixed-system" + ? path.join(tmp, "fixed-system-openshell") + : options.legacyGateway === "system" + ? path.join(tmp, "system-openshell") + : path.join(configHome, "openshell"); + const gatewayDir = path.join(gatewayRoot, "gateways", "nemoclaw"); + fs.mkdirSync(gatewayDir, { recursive: true }); + if (!["retired", "metadata-directory"].includes(String(options.legacyGateway))) { + retainedFiles.set( + path.join( + gatewayDir, + options.legacyGateway === "metadata-link" ? "metadata-target.json" : "metadata.json", + ), + options.legacyGateway === "corrupt" + ? "invalid metadata" + : (options.legacyMetadata ?? + JSON.stringify({ name: "nemoclaw", gateway_endpoint: "https://127.0.0.1:8080" })), + ); + } + retainedFiles.set(path.join(gatewayDir, "client.key"), "synthetic retained mTLS sentinel"); + for (const [file, contents] of retainedFiles) fs.writeFileSync(file, contents); + if (options.legacyGateway === "metadata-link") + fs.symlinkSync("metadata-target.json", path.join(gatewayDir, "metadata.json")); + if (options.legacyGateway === "metadata-directory") + fs.mkdirSync(path.join(gatewayDir, "metadata.json")); + } + + writeExecutable( + path.join(fakeBin, "uname"), + `#!/usr/bin/env bash +if [ "\${1:-}" = "-m" ]; then echo "${options.arch ?? "x86_64"}"; else echo "${options.os ?? "Linux"}"; fi`, + ); + + // Fake openshell that reports the given version + writeExecutable( + path.join(fakeBin, "openshell"), + `#!/usr/bin/env bash +if [ "\${1:-}" = "--version" ]; then echo "openshell ${version}"; exit ${options.versionExit ?? 0}; fi +if [ "$*" = "gateway list -o json" ]; then + printf '%s\\n' ${shellQuote(JSON.stringify(options.legacyGateway ? [{ name: "nemoclaw", endpoint: "https://127.0.0.1:8080", active: true }] : []))} + exit 0 +fi +if [ "$*" = "sandbox list -g nemoclaw -o json" ]; then + printf '%s\\n' ${shellQuote(options.legacyInventory !== undefined ? JSON.stringify(options.legacyInventory) : options.legacyGateway === "empty" || options.legacyGateway === "stderr" ? "[]" : options.legacyGateway === "partial" ? '{"sandboxes":[],"next_page_token":"remaining"}' : options.legacyGateway === "invalid" ? "invalid JSON" : '[{"id":"11111111-1111-4111-8111-111111111111","name":"retained-box","phase":"Stopped","labels":{},"resource_version":1,"created_at":"2026-10-13T00:00:00Z","current_policy_version":1}]')} + ${options.legacyGateway === "stderr" ? "echo 'inventory unavailable' >&2" : ":"} + exit ${options.legacyGateway === "unreachable" ? 1 : 0} +fi +${openshellMarkers ? `# ${openshellMarkers}` : ""} +exit 99`, + ); + if (options.missingCli) fs.unlinkSync(path.join(fakeBin, "openshell")); + else + retainedFiles.set( + path.join(fakeBin, "openshell"), + fs.readFileSync(path.join(fakeBin, "openshell"), "utf8"), + ); + + const driverFixtures: Array<{ name: string; markers: string }> = + options.driverBins === false + ? [] + : [ + { name: "openshell-gateway", markers: gatewayMarkers }, + ...(options.driverBins === "gateway" + ? [] + : [ + { + name: "openshell-sandbox", + markers: options.sandboxMarkers ?? OPENSHELL_MCP_FEATURE_MARKER, + }, + ]), + ...(options.driverBins === "gateway-vm" + ? [ + { + name: "openshell-driver-vm", + markers: OPENSHELL_MCP_FEATURE_MARKER, + }, + ] + : []), + ]; + for (const fixture of driverFixtures) { + writeExecutable( + path.join(driverBin, fixture.name), + `#!/usr/bin/env bash +if [ "\${1:-}" = "--version" ]; then echo "${fixture.name} ${fixture.name === "openshell-sandbox" ? (options.sandboxVersion ?? options.driverVersion ?? version) : (options.driverVersion ?? version)}"; exit ${fixture.name === "openshell-sandbox" ? (options.sandboxVersionExit ?? options.driverVersionExit ?? 0) : (options.driverVersionExit ?? 0)}; fi +# ${fixture.markers} +exit 0`, + ); + if (options.driverReadable === false) fs.chmodSync(path.join(driverBin, fixture.name), 0o111); + if (options.driverLocation === "symlink") { + fs.symlinkSync(path.join(driverBin, fixture.name), path.join(fakeBin, fixture.name)); + } + } + + switch (options.sandboxBinaryDigest) { + case undefined: + break; + default: + writeExecutable( + path.join(fakeBin, "sha256sum"), + `#!/usr/bin/env bash +case "\${1:-}" in + */openshell-sandbox) + printf '%s %s\\n' '${options.sandboxBinaryDigest}' "$1" + exit 0 + ;; +esac +exit 1`, + ); + } + + writeExecutable( + path.join(fakeBin, "curl"), + options.homebrewFormulaDownload + ? `#!/usr/bin/env bash +printf '%s\n' "$*" >&2; out="" +while [ "$#" -gt 0 ]; do + if [ "$1" = "-o" ]; then shift; out="$1"; fi + shift || true +done +[ -n "$out" ] || exit 1 +printf '%s\n' '# downloaded OpenShell formula' 'class Openshell < Formula' > "$out" +exit 0` + : `#!/usr/bin/env bash +printf 'download\\n' >> ${shellQuote(downloads)} +echo "curl stub: $*" >&2 +exit ${options.curlExitCode ?? 1}`, + ); + + writeExecutable( + path.join(fakeBin, "gh"), + `#!/usr/bin/env bash +exit 1`, + ); + + if ((options.os ?? "Linux") === "Darwin") { + const tap = path.join(tmp, "tap"); + const formula = path.join(tap, "Formula", "openshell.rb"); + fs.mkdirSync(path.dirname(formula), { recursive: true }); + fs.writeFileSync(formula, "class Openshell < Formula\nend\n"); + writeExecutable( + path.join(fakeBin, "sha256sum"), + `#!/usr/bin/env bash +case "\${1:-}" in + ${JSON.stringify(formula)}) + if grep -q '^# downloaded OpenShell formula$' "$1"; then + printf '%s %s\\n' '${PINNED_OPEN_SHELL_SHA256.formula}' "$1" + else + printf '%s %s\\n' '${options.homebrewFormulaDigest ?? PINNED_OPEN_SHELL_SHA256.formula}' "$1" + fi + ;; + */openshell.rb) printf '%s %s\\n' '${PINNED_OPEN_SHELL_SHA256.formula}' "$1" ;; + *) /usr/bin/sha256sum "$@" ;; +esac`, + ); + writeExecutable( + path.join(fakeBin, "codesign"), + `#!/usr/bin/env bash +state="\${NEMOCLAW_FAKE_CODESIGN_STATE:-}" +if [ "\${1:-}" = "-d" ]; then + if [ "\${NEMOCLAW_FAKE_CODESIGN_HAS_ENTITLEMENT:-1}" = "1" ] || { [ -n "$state" ] && [ -f "$state" ]; }; then + printf '%s\\n' 'com.apple.security.hypervisor' + fi + exit 0 +fi +if [ -n "\${NEMOCLAW_FAKE_CODESIGN_LOG:-}" ]; then + printf '%s\\n' "$*" >> "$NEMOCLAW_FAKE_CODESIGN_LOG" +fi +if [ -n "$state" ]; then + : > "$state" +fi +exit 0`, + ); + switch (options.homebrewAvailable ?? true) { + case true: + writeExecutable( + path.join(fakeBin, "brew"), + `#!/usr/bin/env bash +${options.homebrewOperationLog ? `printf '%s\\n' "$*" >> ${JSON.stringify(options.homebrewOperationLog)}` : ""} +case "$*" in + "--repository nvidia/openshell") printf '%s\\n' ${JSON.stringify(tap)}; exit 0 ;; + "list --formula openshell") + exit 0 + ;; + "info --json=v2 openshell") + printf '%s\n' '{"formulae":[{"name":"openshell","tap":"nvidia/openshell"}]}' + exit 0 + ;; + "services restart openshell") + exit 0 + ;; + "install --formula nvidia/openshell/openshell"|"reinstall --formula nvidia/openshell/openshell") + exit "\${NEMOCLAW_TEST_BREW_INSTALL_STATUS:-0}" + ;; + "--prefix") + printf '%s\\n' ${JSON.stringify(tmp)} + exit 0 + ;; +esac +exit 0`, + ); + break; + } + } + + const explicitDriverEnv = + options.driverLocation === "explicit" + ? { + NEMOCLAW_OPENSHELL_GATEWAY_BIN: path.join(driverBin, "openshell-gateway"), + NEMOCLAW_OPENSHELL_SANDBOX_BIN: path.join(driverBin, "openshell-sandbox"), + } + : {}; + // Protected execution projects the installer without adjacent candidate helpers. + const isolatedInstaller = path.join(tmp, "install-openshell.sh"); + // Map the fixed system path into the fixture; never read or mutate host registrations. + fs.writeFileSync( + isolatedInstaller, + fs + .readFileSync(path.join(SCRIPTS, "install-openshell.sh"), "utf8") + .replaceAll("/etc/openshell", path.join(tmp, "fixed-system-openshell")), + ); + if (options.entrypoint === "feature-check") { + // Exercise the feature gate independently of the selected install version. + const source = fs.readFileSync(isolatedInstaller, "utf8"); + const selectionStart = source.indexOf('\nACTIVE_OPENSHELL_BIN=""\n'); + if (selectionStart === -1) throw new Error("Missing installer selection boundary"); + fs.writeFileSync( + isolatedInstaller, + `${source.slice(0, selectionStart)}\nrequire_openshell_messaging_features "$(command -v openshell)"\n`, + ); + } + const result = spawnSync("bash", [isolatedInstaller], { + env: { + ...process.env, + NEMOCLAW_OPENSHELL_CHANNEL: "stable", + ...explicitDriverEnv, + ...extraEnv, + HOME: tmp, + TMPDIR: tmp, + XDG_CONFIG_HOME: configHome, + OPENSHELL_SYSTEM_GATEWAY_DIR: + options.systemGatewayDirectory ?? path.join(tmp, "system-openshell"), + PATH: `${fakeBin}:${driverBin}${options.missingNode ? "" : ":/usr/bin:/bin"}`, + }, + encoding: "utf8", + }); + return { + ...result, + downloads: fs.existsSync(downloads) ? fs.readFileSync(downloads, "utf8") : "", + retainedFilesUnchanged: [...retainedFiles].every( + ([file, contents]) => fs.readFileSync(file, "utf8") === contents, + ), + }; + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } +} diff --git a/test/support/protected-openshell-workspace.ts b/test/support/protected-openshell-workspace.ts index 6d3d3d09bd2..89f2be6745a 100644 --- a/test/support/protected-openshell-workspace.ts +++ b/test/support/protected-openshell-workspace.ts @@ -107,6 +107,8 @@ export function approvedFixture() { ) as Record; const selected = validateInstallerSources(candidate); const release = structuredClone(selected.installerReleases[0]!); + // Revocations belong to the registry; resolved releases carry a derived copy. + delete release.revokedTemplateSha256; candidate.installerSource += "\n# Reviewed installer fixture change.\n"; release.installerTemplateSha256 = [ ...release.installerTemplateSha256, diff --git a/test/support/rebuild-post-restore-fixtures.ts b/test/support/rebuild-post-restore-fixtures.ts new file mode 100644 index 00000000000..3893099faf7 --- /dev/null +++ b/test/support/rebuild-post-restore-fixtures.ts @@ -0,0 +1,79 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { vi } from "vitest"; +import { + createBuiltInChannelManifestRegistry, + createBuiltInRenderTemplateResolver, +} from "../../src/lib/messaging/channels"; +import { MessagingWorkflowPlanner } from "../../src/lib/messaging/compiler"; + +export function rebuildPostRestoreInput(agentName: string) { + return { + sandboxName: "alpha", + targetAgentName: agentName, + messagingPlan: null, + backupManifest: null, + mcpEntries: [], + restoreSucceeded: true, + failedPresets: [], + finalBuiltinPresets: [], + failedPresetRemovals: [], + policyPresetReconciliationVerified: true, + preparedBackupRecovery: false, + versionCheck: { expectedVersion: null } as never, + log: vi.fn(), + bail: vi.fn() as never, + }; +} + +export async function hermesTeamsPlan(disabled = false) { + vi.stubEnv("MSTEAMS_APP_ID", "test-teams-app"); + vi.stubEnv("MSTEAMS_TENANT_ID", "test-teams-tenant"); + vi.stubEnv("TEAMS_ALLOWED_USERS", "00000000-0000-0000-0000-000000000001"); + vi.stubEnv("MSTEAMS_PORT", "3979"); + return new MessagingWorkflowPlanner( + createBuiltInChannelManifestRegistry(), + undefined, + createBuiltInRenderTemplateResolver(), + ).buildPlan({ + sandboxName: "alpha", + agent: "hermes", + workflow: "rebuild", + isInteractive: false, + configuredChannels: ["teams"], + disabledChannels: disabled ? ["teams"] : [], + credentialAvailability: { MSTEAMS_APP_PASSWORD: true }, + }); +} + +export const runtimeKindByAgent = { + openclaw: "gateway", + hermes: "gateway", + "langchain-deepagents-code": "terminal", + pi: "terminal", +} as const; + +export const hermesRebuildVerificationCases = [ + { + cronGate: false, + preparedBackupRecovery: false, + restoreSucceeded: true, + gatewayDetail: "nemoclaw alpha gateway restart", + recoveryDetail: "If gateway health is still unverified", + }, + { + cronGate: true, + preparedBackupRecovery: false, + restoreSucceeded: true, + gatewayDetail: "Hermes gateway health was not verified after state restore.", + recoveryDetail: "Hermes cron dispatch remains drained", + }, + { + cronGate: false, + preparedBackupRecovery: true, + restoreSucceeded: false, + gatewayDetail: "Correct the reported restore problem", + recoveryDetail: "nemoclaw alpha recover", + }, +]; diff --git a/test/support/setup-inference-test-harness.ts b/test/support/setup-inference-test-harness.ts index d572b279d53..7448d26949f 100644 --- a/test/support/setup-inference-test-harness.ts +++ b/test/support/setup-inference-test-harness.ts @@ -197,8 +197,8 @@ if (nativeProfile) { if (argv[0] === "provider" && argv[1] === "profile" && argv.includes("export")) { process.stdout.write(JSON.stringify(nativeProfile)); process.exit(0); } - if (argv[0] === "settings" && argv[1] === "get") { - process.stdout.write(JSON.stringify({ scope: "global", settings: { providers_v2_enabled: "true" } })); process.exit(0); + if (argv[0] === "policy" && argv[1] === "list") { + process.stdout.write(JSON.stringify({ revisions: [], next_page_token: "" })); process.exit(0); } if (argv[0] === "provider" && argv[1] === "create") { fs.writeFileSync(created, "created"); process.exit(0); diff --git a/test/support/status-flow-test-harness.ts b/test/support/status-flow-test-harness.ts index c51f05a25a9..42bedd511b2 100644 --- a/test/support/status-flow-test-harness.ts +++ b/test/support/status-flow-test-harness.ts @@ -38,6 +38,7 @@ delete require.cache[requireDist.resolve(statusModulePath)]; export type StatusFlowHarness = { checkAgentVersionSpy: MockInstance; + classifyGatewayFailureSpy: MockInstance; collectSandboxStatusSnapshotSpy: MockInstance; getActiveSandboxSessionsSpy: MockInstance; getSandboxDockerRuntimeSpy: MockInstance; @@ -140,6 +141,16 @@ export function createStatusFlowHarness(options: StatusFlowHarnessOptions = {}): vi.spyOn(console, "error").mockImplementation(() => undefined); const statusPreflight = requireDist("../../src/lib/actions/sandbox/status-preflight.js"); + const failureClassifier = requireDist( + "../../src/lib/actions/sandbox/gateway-failure-classifier.js", + ); + // Status rendering consumes the classification; classifier tests own real probe behavior. + const classifyGatewayFailureSpy = vi + .spyOn(failureClassifier, "classifyGatewayFailure") + .mockResolvedValue({ + layer: "gateway_unreachable", + detail: "The selected gateway is unreachable.", + }); const statusSnapshot = requireDist("../../src/lib/actions/sandbox/status-snapshot.js"); const dockerHealth = requireDist("../../src/lib/actions/sandbox/docker-health.js"); const statusProcessRecovery = requireDist( @@ -337,6 +348,7 @@ export function createStatusFlowHarness(options: StatusFlowHarnessOptions = {}): return { checkAgentVersionSpy, + classifyGatewayFailureSpy, collectSandboxStatusSnapshotSpy, getActiveSandboxSessionsSpy, getSandboxDockerRuntimeSpy, diff --git a/test/support/uninstall-state-volume.ts b/test/support/uninstall-state-volume.ts new file mode 100644 index 00000000000..182505da7db --- /dev/null +++ b/test/support/uninstall-state-volume.ts @@ -0,0 +1,209 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { vi } from "vitest"; +import { createDockerRuntimeProviderBundle } from "../../src/lib/onboard/runtime-provider/docker"; +import { createRuntimeProviderBundleRegistry } from "../../src/lib/onboard/runtime-provider/registry"; +import { managedStartupStateRoots } from "../../src/lib/onboard/managed-startup/state-roots"; +import { withSuccessfulPreUninstallBackup } from "./uninstall-managed-gateway-test-support"; +import { + type RunResult, + runUninstallPlanProduction as runUninstallPlanBase, + type UninstallRunDeps, + type UninstallRunOptions, +} from "../../src/lib/actions/uninstall/run-plan"; + +function ok(stdout = ""): RunResult { + return { status: 0, stdout, stderr: "" }; +} + +async function runUninstallPlan(options: UninstallRunOptions, deps: UninstallRunDeps) { + return await runUninstallPlanBase( + options, + withSuccessfulPreUninstallBackup({ + resolveGatewayTeardownAuthority: ({ gatewayName, gatewayPort }) => ({ + gatewayName, + gatewayPort, + mode: "nemoclaw-managed", + source: gatewayPort === 8080 ? "packaged-service" : "standalone", + endpoint: null, + stateDir: null, + supervisor: null, + requiredCapabilities: [], + }), + ...deps, + }), + ); +} + +export async function runManagedHermesVolumeUninstall( + mode: "foreign" | "owned" | "remove-fails" | "remove-fails-once", + destroyUserData: boolean, + containerMode: "absent" | "foreign" | "owned" = "absent", + agent: "hermes" | "openclaw" = "hermes", + workspace = "default", + registryAgent: "hermes" | "openclaw" | null = agent, +) { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-uninstall-hermes-volume-")); + const stateDir = path.join(home, ".nemoclaw"); + const registryFile = path.join(stateDir, "sandboxes.json"); + const root = managedStartupStateRoots({ + agent, + sandboxName: "hermes", + agentIdentity: { uid: 1000, gid: 1000 }, + })[0]!; + const volumeName = root.resourceIdentity; + fs.mkdirSync(stateDir, { recursive: true }); + fs.writeFileSync( + registryFile, + JSON.stringify({ + defaultSandbox: "hermes", + sandboxes: { + hermes: { + agent: registryAgent, + gatewayName: "nemoclaw", + gatewayPort: 8080, + name: "hermes", + openshellDriver: "docker", + workload: { kind: "managed-image" }, + }, + }, + }), + ); + + const dockerCalls: string[][] = []; + const errors: string[] = []; + const events: string[] = []; + const logs: string[] = []; + const containerId = "hermes-sandbox-container"; + let containerPresent = containerMode !== "absent"; + let gatewayRegistered = true; + let volumeRemovalAttempts = 0; + let volumePresent = true; + const run = vi.fn((command: string, args: string[]) => { + events.push(`${command} ${args.join(" ")}`); + switch (`${command} ${args.join(" ")}`) { + case "openshell gateway list": + case "openshell gateway list -o json": + return ok(JSON.stringify(gatewayRegistered ? [{ name: "nemoclaw", active: false }] : [])); + case "openshell gateway remove nemoclaw": + gatewayRegistered = false; + return ok(); + case "openshell sandbox delete --all": + return gatewayRegistered + ? ok() + : { status: 1, stdout: "", stderr: "Unknown gateway 'nemoclaw'" }; + default: + return ok(); + } + }); + const runDocker = vi.fn((args: string[]) => { + dockerCalls.push(args); + events.push(`docker ${args.join(" ")}`); + switch (args.join(" ")) { + case "ps -a --format {{.ID}} {{.Image}} {{.Names}}": + switch (containerPresent ? containerMode : "absent") { + case "owned": + return ok( + `${containerId} ghcr.io/nvidia/nemoclaw/hermes-sandbox:latest openshell-default--hermes-runtime-id\n`, + ); + case "foreign": + return ok(`${containerId} redis:7 foreign-service\n`); + default: + return ok(); + } + case `rm -f ${containerId}`: + containerPresent = false; + return ok(`${containerId}\n`); + case `volume inspect --format {{json .}} ${volumeName}`: + return volumePresent + ? ok( + `${JSON.stringify({ + Labels: + mode === "foreign" ? { "com.example.owner": "foreign" } : root.ownershipLabels, + Name: volumeName, + })}\n`, + ) + : { status: 1, stdout: "", stderr: `Error: no such volume: ${volumeName}` }; + case `volume rm ${volumeName}`: + switch (containerPresent ? "attached" : mode) { + case "attached": + case "remove-fails": + return { status: 1, stdout: "", stderr: "volume is still in use" }; + case "remove-fails-once": + volumeRemovalAttempts += 1; + switch (volumeRemovalAttempts) { + case 1: + return { status: 1, stdout: "", stderr: "volume is still in use" }; + default: + volumePresent = false; + return ok(`${volumeName}\n`); + } + default: + volumePresent = false; + return ok(`${volumeName}\n`); + } + case "volume inspect openshell-cluster-nemoclaw": + return { status: 1, stdout: "", stderr: "Error: no such volume" }; + default: + return ok(); + } + }); + + const execute = () => + runUninstallPlan( + { assumeYes: true, deleteModels: false, destroyUserData, keepOpenShell: false }, + { + commandExists: () => true, + env: { + HOME: home, + OPENSHELL_WORKSPACE: workspace === "default" ? undefined : workspace, + } as NodeJS.ProcessEnv, + error: (line) => errors.push(line), + existsSync: (target) => target.startsWith(home) && fs.existsSync(target), + hasPortableRuntimeCleanup: () => false, + isTty: false, + kill: () => true, + log: (line) => logs.push(line), + rmSync: fs.rmSync, + run, + runDocker, + runtimeProviders: createRuntimeProviderBundleRegistry([ + [ + "docker", + createDockerRuntimeProviderBundle({ + captureHostCommand: (_command, args) => { + const result = runDocker(args); + return { + status: result.status ?? 1, + stdout: result.stdout, + stderr: result.stderr, + }; + }, + }), + ], + ]), + }, + ); + const result = await execute(); + + return { + cleanup: () => fs.rmSync(home, { force: true, recursive: true }), + stateDir, + containerId, + containerPresent: () => containerPresent, + dockerCalls, + errors, + events, + gatewayRegistered: () => gatewayRegistered, + logs, + registryFile, + result, + retry: execute, + volumeName, + volumePresent: () => volumePresent, + }; +} diff --git a/tools/e2e/candidate-openshell-version.mts b/tools/e2e/candidate-openshell-version.mts index b1c07adeee3..b8274097e84 100644 --- a/tools/e2e/candidate-openshell-version.mts +++ b/tools/e2e/candidate-openshell-version.mts @@ -81,6 +81,13 @@ export function reviewedOpenShellMcpRuntime(version: string) { gateway = "c8fa78fd49f245978f64d038207ce0205d98fcdd48bf115bae8a0b9f4f4d5003"; standaloneSandbox = "65aa2a2afb96e7d1351c1d197ded7a966e20bcad68e18a58a94621a8f23769bc"; break; + case "0.1.7": + sourceSha = "5d25211b25a8fc887e8d5c606d57d6ab22f1d76c"; + supervisorDigest = "a07864bd0c546716039c2000a919f50ce8f7f1f67ba37dda350cf64d3a7cfb06"; + cli = "718f40674738206a84bc3a967966225ff47dfed9ba5738099d56777faa4af793"; + gateway = "0b7cb54fd8558bfad4c57f556f7fc1a20ef8cd94ebb11f4f40a9a90b74b4ad17"; + standaloneSandbox = "d13966b032b07d75c917d3fbb7471e608edfdf1f25e9d922e5aa7c779cf6ad21"; + break; default: throw new Error("The selected OpenShell release has no reviewed MCP runtime provenance"); } diff --git a/tools/e2e/target-catalogue.mts b/tools/e2e/target-catalogue.mts index 1a874b5e6cf..97b4cc6fecb 100644 --- a/tools/e2e/target-catalogue.mts +++ b/tools/e2e/target-catalogue.mts @@ -322,6 +322,7 @@ const PI_IMAGE_SOURCE_OWNING_PATHS = [ "src/lib/core/endpoint-url-safety.ts", "src/lib/inference/native-local/agent-config.ts", "src/lib/inference/native-provider/agent-config.ts", + "src/lib/inference/native-provider/catalog.ts", "src/lib/inference/native-nvidia/contract.ts", "src/lib/inference/native-provider/contract.ts", "src/lib/inference/native-provider/hosted.ts", @@ -399,7 +400,7 @@ const GATEWAY_UPGRADE_OWNING_PATHS = Object.freeze([ const GATEWAY_UPGRADE_TARGET = dockerOnlyTarget("openshell-gateway-upgrade-v0-0-89-x86-64", { targetId: "openshell-gateway-upgrade", - displayName: `Upgrade: preserves a ${REVIEWED_GATEWAY_UPGRADE_FIXTURE.nemoclawRef} sandbox on x86-64`, + displayName: `Cutover: preserves a ${REVIEWED_GATEWAY_UPGRADE_FIXTURE.nemoclawRef} gateway with sandboxes on x86-64`, agentRuntime: "openclaw", environmentOrInferenceEndpoint: "x86-64 Ubuntu; GitHub release artifacts; host-local compatible inference endpoint", @@ -421,8 +422,8 @@ const GATEWAY_UPGRADE_TARGET = dockerOnlyTarget("openshell-gateway-upgrade-v0-0- NEMOCLAW_OLD_SANDBOX_BASE_IMAGE_REF: REVIEWED_GATEWAY_UPGRADE_FIXTURE.sandboxBaseImageRef, NEMOCLAW_OLD_OPENSHELL_VERSION: REVIEWED_GATEWAY_UPGRADE_FIXTURE.openShellVersion, NEMOCLAW_OLD_OPENCLAW_VERSION: REVIEWED_GATEWAY_UPGRADE_FIXTURE.openclawVersion, - // This target upgrades only the host gateway. Its survivor intentionally - // keeps the reviewed historical OpenClaw image and state format. + // The attempted cutover must leave this historical gateway and its + // reviewed OpenClaw sandbox state intact. OPENSHELL_GATEWAY: "nemoclaw", }, }); @@ -431,7 +432,7 @@ const GATEWAY_REGISTRATION_UPGRADE_TARGET = dockerOnlyTarget( "openshell-gateway-upgrade-v0-0-123-x86-64", { targetId: "openshell-gateway-upgrade", - displayName: `Upgrade: restores ${REVIEWED_GATEWAY_REGISTRATION_UPGRADE_FIXTURE.nemoclawRef} gateway registration and sandboxes on x86-64`, + displayName: `Cutover: preserves ${REVIEWED_GATEWAY_REGISTRATION_UPGRADE_FIXTURE.nemoclawRef} gateway registration and sandboxes on x86-64`, agentRuntime: "openclaw", environmentOrInferenceEndpoint: "x86-64 Ubuntu; GitHub release artifacts; host-local compatible inference endpoint", @@ -1003,6 +1004,7 @@ export const E2E_TARGET_CATALOGUE: readonly E2eCatalogueTarget[] = [ NEMOCLAW_SWITCH_MODEL: "mock-anthropic-model", NEMOCLAW_SWITCH_INFERENCE_API: "anthropic-messages", NEMOCLAW_SWITCH_MOCK_ANTHROPIC: "1", + NEMOCLAW_SWITCH_NATIVE_CUSTOM: "1", OPENSHELL_GATEWAY: "nemoclaw", }, }), diff --git a/tools/mcp-tool-discovery-runtime/managed-startup-direct-entry.ts b/tools/mcp-tool-discovery-runtime/managed-startup-direct-entry.ts index 31535df2f2f..a857b23fb14 100644 --- a/tools/mcp-tool-discovery-runtime/managed-startup-direct-entry.ts +++ b/tools/mcp-tool-discovery-runtime/managed-startup-direct-entry.ts @@ -7,7 +7,7 @@ export { NVIDIA_HOSTED_NATIVE_ENDPOINT, NVIDIA_HOSTED_CREDENTIAL_ENV, } from "../../src/lib/inference/native-nvidia/contract.ts"; -export { hostedNativeProvider } from "../../src/lib/inference/native-provider/agent-config.ts"; +export { fixedHostedNativeProvider } from "../../src/lib/inference/native-provider/catalog.ts"; export * from "../../src/lib/onboard/managed-startup/image-runtime"; diff --git a/tools/mcp-tool-discovery-runtime/reviewed-runtime-bundle/managed-startup-direct-image-runtime.bundle b/tools/mcp-tool-discovery-runtime/reviewed-runtime-bundle/managed-startup-direct-image-runtime.bundle index 6b80096cd3f..9e6ce17a9b3 100644 --- a/tools/mcp-tool-discovery-runtime/reviewed-runtime-bundle/managed-startup-direct-image-runtime.bundle +++ b/tools/mcp-tool-discovery-runtime/reviewed-runtime-bundle/managed-startup-direct-image-runtime.bundle @@ -1,14 +1,18 @@ -"use strict";var __create=Object.create;var __defProp=Object.defineProperty;var __getOwnPropDesc=Object.getOwnPropertyDescriptor;var __getOwnPropNames=Object.getOwnPropertyNames;var __getProtoOf=Object.getPrototypeOf;var __hasOwnProp=Object.prototype.hasOwnProperty;var __export=(target2,all)=>{for(var name in all)__defProp(target2,name,{get:all[name],enumerable:true})};var __copyProps=(to,from,except,desc)=>{if(from&&typeof from==="object"||typeof from==="function"){for(let key of __getOwnPropNames(from))if(!__hasOwnProp.call(to,key)&&key!==except)__defProp(to,key,{get:()=>from[key],enumerable:!(desc=__getOwnPropDesc(from,key))||desc.enumerable})}return to};var __toESM=(mod,isNodeMode,target2)=>(target2=mod!=null?__create(__getProtoOf(mod)):{},__copyProps(isNodeMode||!mod||!mod.__esModule?__defProp(target2,"default",{value:mod,enumerable:true}):target2,mod));var __toCommonJS=mod=>__copyProps(__defProp({},"__esModule",{value:true}),mod);var managed_startup_direct_entry_exports={};__export(managed_startup_direct_entry_exports,{MANAGED_STARTUP_CA_ENV:()=>MANAGED_STARTUP_CA_ENV,MANAGED_STARTUP_COMPLETION_FILE:()=>MANAGED_STARTUP_COMPLETION_FILE,MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION:()=>MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,MANAGED_STARTUP_MERGED_CA_FILE:()=>MANAGED_STARTUP_MERGED_CA_FILE,MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES:()=>MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES,MANAGED_STARTUP_PROFILE_ENV:()=>MANAGED_STARTUP_PROFILE_ENV,MANAGED_STARTUP_RUNTIME_ENV_FILE:()=>MANAGED_STARTUP_RUNTIME_ENV_FILE,MANAGED_STARTUP_RUNTIME_EXECUTABLE:()=>MANAGED_STARTUP_RUNTIME_EXECUTABLE,ManagedStartupImageActionPlanError:()=>ManagedStartupImageActionPlanError,ManagedStartupImageRuntimeError:()=>ManagedStartupImageRuntimeError,NVIDIA_HOSTED_CREDENTIAL_ENV:()=>NVIDIA_HOSTED_CREDENTIAL_ENV,NVIDIA_HOSTED_NATIVE_ENDPOINT:()=>NVIDIA_HOSTED_NATIVE_ENDPOINT,applyManagedStartupCommandEnvironmentPlan:()=>applyManagedStartupCommandEnvironmentPlan,applyManagedStartupImageProfile:()=>applyManagedStartupImageProfile,applyManagedStartupRootRequest:()=>applyManagedStartupRootRequest,atomicWriteRootFile:()=>atomicWriteRootFile,buildManagedStartupImageActionPlan:()=>buildManagedStartupImageActionPlan,hostedNativeProvider:()=>hostedNativeProvider,installCorporateCaSystemAnchors:()=>installCorporateCaSystemAnchors,installHermesManagedPolicy:()=>installHermesManagedPolicy,installHermesSwitchyardRoutingFiles:()=>installHermesSwitchyardRoutingFiles,main:()=>main,managedStartupSandboxPrefix:()=>managedStartupSandboxPrefix,normalizeHermesManagedConfigDescriptor:()=>normalizeHermesManagedConfigDescriptor,normalizeManagedStartupWorkspaceRoot:()=>normalizeManagedStartupWorkspaceRoot,publishManagedStartupCompletionAfterCommit:()=>publishManagedStartupCompletionAfterCommit,readStableRegularFile:()=>readStableRegularFile,readStableRegularFileSnapshot:()=>readStableRegularFileSnapshot,serializeManagedStartupCompletionMarker:()=>serializeManagedStartupCompletionMarker,serializeManagedStartupRuntimeEnvironment:()=>serializeManagedStartupRuntimeEnvironment,validateManagedStartupApplicationRuntimePlan:()=>validateManagedStartupApplicationRuntimePlan,verifyHermesSwitchyardRoutingFiles:()=>verifyHermesSwitchyardRoutingFiles,verifyManagedStartupImageCompletion:()=>verifyManagedStartupImageCompletion,waitForManagedStartupImageCompletion:()=>waitForManagedStartupImageCompletion});module.exports=__toCommonJS(managed_startup_direct_entry_exports);var NVIDIA_HOSTED_LOGICAL_PROVIDER="nvidia-prod";var NVIDIA_HOSTED_NATIVE_ENDPOINT="https://integrate.api.nvidia.com/v1";var NVIDIA_HOSTED_CREDENTIAL_ENV="NVIDIA_INFERENCE_API_KEY";var NATIVE_NVIDIA_AUTH_HEADER_SCRIPT=['case "${NVIDIA_INFERENCE_API_KEY:-}" in *[!a-zA-Z0-9:_]*) exit 2 ;; esac',`printf '%s' "\${NVIDIA_INFERENCE_API_KEY:-}" | LC_ALL=C grep -Eq '^openshell:resolve:env:((v[0-9]{1,20}|s[a-f0-9]{64})_)?NVIDIA_INFERENCE_API_KEY$' || exit 2`,'AUTH_HEADER="Authorization: Bearer ${NVIDIA_INFERENCE_API_KEY}"'].join("; ");var import_node_crypto=require("node:crypto");function isCustomHostedProvider(provider2){return provider2==="compatible-endpoint"||provider2==="compatible-anthropic-endpoint"}function customNativeCredentialEnv(provider2){if(provider2==="compatible-endpoint")return"COMPATIBLE_API_KEY";if(provider2==="compatible-anthropic-endpoint")return"COMPATIBLE_ANTHROPIC_API_KEY";throw new Error("Unsupported custom hosted provider")}var HOSTED_NATIVE_PROVIDERS=[{logicalProvider:"openai-api",label:"OpenAI",profileId:"nemoclaw-openai-inference-v1",providerName:"nemoclaw-openai-api-v1",credentialEnv:"OPENAI_API_KEY",endpoint:"https://api.openai.com/v1",api:"openai-completions"},{logicalProvider:"anthropic-prod",label:"Anthropic",profileId:"nemoclaw-anthropic-inference-v1",providerName:"nemoclaw-anthropic-prod-v1",credentialEnv:"ANTHROPIC_API_KEY",endpoint:"https://api.anthropic.com",api:"anthropic-messages"},{logicalProvider:"gemini-api",label:"Gemini",profileId:"nemoclaw-gemini-inference-v1",providerName:"nemoclaw-gemini-api-v1",credentialEnv:"GEMINI_API_KEY",endpoint:"https://generativelanguage.googleapis.com/v1beta/openai/",api:"openai-completions"},{logicalProvider:"openrouter-api",label:"OpenRouter",profileId:"nemoclaw-openrouter-inference-v1",providerName:"nemoclaw-openrouter-api-v1",credentialEnv:"OPENROUTER_API_KEY",endpoint:"https://openrouter.ai/api/v1",api:"openai-completions"},{logicalProvider:"hermes-provider",label:"Hermes Provider",profileId:"nemoclaw-hermes-inference-v1",providerName:"nemoclaw-hermes-provider-v1",credentialEnv:"OPENAI_API_KEY",endpoint:"https://inference-api.nousresearch.com/v1",api:"openai-completions"}];function canonicalHermesNativeEndpoint(value){const endpoint=new URL(value);if(endpoint.hostname==="inference.local"||endpoint.hostname==="localhost"||endpoint.hostname.endsWith(".internal")||endpoint.protocol!=="https:"||endpoint.username||endpoint.password||endpoint.search||endpoint.hash||/[\\\s]/u.test(value)||/%2f|%5c/iu.test(endpoint.pathname)){throw new Error("Hermes inference requires an HTTPS endpoint without credentials, query, fragment, or encoded separators")}return`${endpoint.origin}${endpoint.pathname.replace(/\/+$/,"")}`}function hostedNativeProvider(provider2,endpointUrl){const definition=HOSTED_NATIVE_PROVIDERS.find(entry=>entry.logicalProvider===provider2?.trim());if(!definition||definition.logicalProvider!=="hermes-provider"||!endpointUrl)return definition;const endpoint=canonicalHermesNativeEndpoint(endpointUrl);if(endpoint===definition.endpoint)return definition;const suffix=(0,import_node_crypto.createHash)("sha256").update(endpoint).digest("hex").slice(0,20);return{...definition,endpoint,endpointUrl:endpoint,profileId:`nemoclaw-hermes-inference-${suffix}-v1`,providerName:`nemoclaw-hermes-${suffix}-v1`}}function nativeHostedAgentConfig(provider2,baseUrl){if(isCustomHostedProvider(provider2)&&new URL(baseUrl).hostname!=="inference.local"){const endpoint2=new URL(baseUrl);if(!["https:","http:"].includes(endpoint2.protocol)||endpoint2.username||endpoint2.password||endpoint2.search||endpoint2.hash)throw new Error("Invalid native custom inference endpoint");if(["localhost","127.0.0.1","[::1]"].includes(endpoint2.hostname))return void 0;const credentialEnv=customNativeCredentialEnv(provider2);return{credentialEnv,apiKey:`openshell:resolve:env:${credentialEnv}`}}if(!hostedNativeProvider(provider2))return void 0;if(new URL(baseUrl).hostname==="inference.local")return void 0;const definition=hostedNativeProvider(provider2,provider2==="hermes-provider"?baseUrl:void 0);if(!definition)return void 0;const endpoint=new URL(baseUrl);const expected=new URL(definition.endpoint);if(endpoint.username||endpoint.password||endpoint.search||endpoint.hash||endpoint.origin!==expected.origin||endpoint.pathname.replace(/\/+$/,"")!==expected.pathname.replace(/\/+$/,"")){throw new Error(`Native ${definition.label} configuration does not match its attached provider endpoint`)}return{credentialEnv:definition.credentialEnv,apiKey:`openshell:resolve:env:${definition.credentialEnv}`,...provider2==="openrouter-api"?{headers:{"HTTP-Referer":"https://www.nvidia.com/nemoclaw/","X-OpenRouter-Title":"NVIDIA NemoClaw"}}:{}}}var import_node_child_process=require("node:child_process");var import_node_crypto7=require("node:crypto");var import_node_fs3=__toESM(require("node:fs"));var import_node_path3=__toESM(require("node:path"));var MAX_CORPORATE_CA_BYTES=128*1024;var PEM_CERTIFICATE_RE_GLOBAL=/-----BEGIN CERTIFICATE-----[\s\S]+?-----END CERTIFICATE-----/g;var deep_agents_code_runtime_identity_default={uid:999,gid:999};var DEEP_AGENTS_CODE_RUNTIME_IDENTITY=Object.freeze(deep_agents_code_runtime_identity_default);var DEEP_AGENTS_CODE_SANDBOX_USER=`${DEEP_AGENTS_CODE_RUNTIME_IDENTITY.uid}:${DEEP_AGENTS_CODE_RUNTIME_IDENTITY.gid}`;var SHIPPED_MANAGED_IMAGE_AGENTS=["openclaw","hermes","langchain-deepagents-code"];var CANDIDATE_MANAGED_IMAGE_AGENTS=["pi"];var MANAGED_IMAGE_AGENTS=[...SHIPPED_MANAGED_IMAGE_AGENTS,...CANDIDATE_MANAGED_IMAGE_AGENTS];var MANAGED_IMAGE_RUNTIME_IDENTITIES=Object.freeze({openclaw:Object.freeze({uid:998,gid:998,workdir:"/sandbox"}),hermes:Object.freeze({uid:998,gid:999,workdir:"/sandbox"}),"langchain-deepagents-code":Object.freeze({...DEEP_AGENTS_CODE_RUNTIME_IDENTITY,workdir:"/sandbox"}),pi:Object.freeze({uid:999,gid:999,workdir:"/sandbox"})});function managedImageRuntimeIdentity(agent){return MANAGED_IMAGE_RUNTIME_IDENTITIES[agent]}var import_node_buffer=require("node:buffer");var HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION=1;var HERMES_SWITCHYARD_ROUTE_ID="switchyard";var HERMES_SWITCHYARD_ROUTING_ENV="NEMOCLAW_HERMES_SWITCHYARD_ROUTING_B64";var HERMES_RELAY_PLUGINS_FILE="/usr/local/share/nemoclaw/hermes-relay-plugins.toml";var HERMES_RELAY_PLUGIN_STATE_FILE="/usr/local/share/nemoclaw/.dynamic-plugins.json";var HERMES_SWITCHYARD_DEPLOYMENT_FILE="/usr/local/share/nemoclaw/hermes-switchyard-v1.toml";var HERMES_SWITCHYARD_PLUGIN_MANIFEST="/opt/nemo-relay/plugins/switchyard/relay-plugin.toml";var HERMES_RELAY_PLUGINS_STAGING_FILE="hermes-relay-plugins.toml";var HERMES_RELAY_PLUGIN_STATE_STAGING_FILE="hermes-relay-plugin-state.json";var HERMES_SWITCHYARD_DEPLOYMENT_STAGING_FILE="hermes-switchyard-v1.toml";var MAX_MODEL_BYTES=512;var MAX_ROUTING_BYTES=8*1024;var STANDARD_BASE64_RE=/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u;var MODEL_PATTERN=/^[A-Za-z0-9][A-Za-z0-9._:+/-]*$/u;var CREDENTIAL_ENV_PATTERN=/^[A-Z_][A-Z0-9_]*$/u;var HEADER_NAME_PATTERN=/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/u;var SECRET_VALUE_PATTERNS=[/(?:nvapi|nvcf|sk-(?:proj-|ant-)?|ghp_|github_pat_|hf_|glpat-|gsk_|pypi-|tvly-)/u,/Bearer\s+/iu,/-----BEGIN (?:[A-Z0-9]+ )?PRIVATE KEY-----/u];var HOSTED_SWITCHYARD_PROVIDER_DEFINITIONS=HOSTED_NATIVE_PROVIDERS.filter(definition=>definition.logicalProvider!=="hermes-provider");var HERMES_SWITCHYARD_PROVIDERS=[NVIDIA_HOSTED_LOGICAL_PROVIDER,...HOSTED_SWITCHYARD_PROVIDER_DEFINITIONS.map(definition=>definition.logicalProvider)];var HermesSwitchyardRoutingError=class extends Error{constructor(message){super(`Invalid Hermes Switchyard routing: ${message}`);this.name="HermesSwitchyardRoutingError"}};function fail(message){throw new HermesSwitchyardRoutingError(message)}function record(value,where){if(typeof value!=="object"||value===null||Array.isArray(value)){return fail(`${where} must be an object`)}return value}function rejectUnknownKeys(value,allowed,where){const unknown=Object.keys(value).filter(key=>!allowed.has(key));if(unknown.length>0)fail(`${where} contains unknown fields`)}function provider(value,where){if(typeof value!=="string"||!HERMES_SWITCHYARD_PROVIDERS.includes(value)){return fail(`${where} must name an approved fixed hosted provider profile`)}return value}function model(value,where){if(typeof value!=="string"||import_node_buffer.Buffer.byteLength(value,"utf8")===0||import_node_buffer.Buffer.byteLength(value,"utf8")>MAX_MODEL_BYTES||!MODEL_PATTERN.test(value)||value.includes("://")||value.includes("//")||value.includes("..")||value.includes("${")||value.toLowerCase().includes("openshell:resolve")||SECRET_VALUE_PATTERNS.some(pattern=>pattern.test(value))){return fail(`${where} must be a bounded provider model ID without credentials or endpoints`)}return value}function target(value,where){const selection=record(value,where);rejectUnknownKeys(selection,new Set(["provider","model"]),where);return{provider:provider(selection.provider,`${where}.provider`),model:model(selection.model,`${where}.model`)}}function validateHermesSwitchyardRoutingIntent(value){const routing=record(value,"routing");rejectUnknownKeys(routing,new Set(["schemaVersion","routeId","efficient","capable","judge"]),"routing");if(routing.schemaVersion!==HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION){fail(`schemaVersion must be ${String(HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION)}`)}if(routing.routeId!==HERMES_SWITCHYARD_ROUTE_ID){fail(`routeId must be ${HERMES_SWITCHYARD_ROUTE_ID}`)}const efficient=target(routing.efficient,"routing.efficient");const capable=target(routing.capable,"routing.capable");if(efficient.model===capable.model){fail("efficient and capable must select different model IDs")}const judge=record(routing.judge,"routing.judge");rejectUnknownKeys(judge,new Set(["mode","target"]),"routing.judge");if(judge.mode!=="reuse-capable"||judge.target!=="capable"){fail("judge must explicitly reuse the capable target")}return{schemaVersion:HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION,routeId:HERMES_SWITCHYARD_ROUTE_ID,efficient,capable,judge:{mode:"reuse-capable",target:"capable"}}}function encodeHermesSwitchyardRoutingIntent(value){const routing=validateHermesSwitchyardRoutingIntent(value);return import_node_buffer.Buffer.from(JSON.stringify(routing),"utf8").toString("base64")}function decodeHermesSwitchyardRoutingIntent(encoded){if(encoded.length===0||encoded.length>Math.ceil(MAX_ROUTING_BYTES/3)*4||!STANDARD_BASE64_RE.test(encoded)){return fail("encoded routing transport is invalid")}const decoded=import_node_buffer.Buffer.from(encoded,"base64");if(decoded.length>MAX_ROUTING_BYTES||decoded.toString("base64")!==encoded){return fail("encoded routing transport is invalid")}let parsed;try{parsed=JSON.parse(decoded.toString("utf8"))}catch{return fail("encoded routing transport must contain JSON")}return validateHermesSwitchyardRoutingIntent(parsed)}function readHermesSwitchyardRoutingEnvironment(env){const encoded=env[HERMES_SWITCHYARD_ROUTING_ENV];return encoded===void 0||encoded===""?null:decodeHermesSwitchyardRoutingIntent(encoded)}function validateClientProfile(profile){let endpoint;try{endpoint=new URL(profile.baseUrl)}catch{return fail("provider profile has an invalid endpoint")}if(endpoint.protocol!=="https:"||endpoint.username||endpoint.password||endpoint.search||endpoint.hash||endpoint.hostname==="inference.local"||/[\\\s]/u.test(profile.baseUrl)||!CREDENTIAL_ENV_PATTERN.test(profile.apiKeyEnv)){return fail("provider profile is not safe for Switchyard onboarding")}for(const[name,value]of Object.entries(profile.extraHeaders)){if(!HEADER_NAME_PATTERN.test(name)||/[\r\n]/u.test(value)||["authorization","proxy-authorization","x-api-key"].includes(name.toLowerCase())){return fail("provider profile contains an unsafe Switchyard header")}}return profile}function clientProfile(providerId){if(providerId===NVIDIA_HOSTED_LOGICAL_PROVIDER){return validateClientProfile({baseUrl:NVIDIA_HOSTED_NATIVE_ENDPOINT,apiKeyEnv:NVIDIA_HOSTED_CREDENTIAL_ENV,format:"openai_chat",extraHeaders:{}})}const definition=hostedNativeProvider(providerId);if(!definition||definition.logicalProvider==="hermes-provider"){return fail("provider profile is not supported by Switchyard onboarding")}const agentConfig=nativeHostedAgentConfig(providerId,definition.endpoint);if(!agentConfig)return fail("provider profile has no managed credential binding");return validateClientProfile({baseUrl:definition.endpoint,apiKeyEnv:definition.credentialEnv,format:definition.api==="anthropic-messages"?"anthropic_messages":"openai_chat",extraHeaders:agentConfig.headers??{}})}function tomlString(value){return JSON.stringify(value)}function tomlStringMap(value){const fields=Object.entries(value).sort(([left],[right])=>left.localeCompare(right)).map(([name,headerValue])=>`${tomlString(name)} = ${tomlString(headerValue)}`);return`{ ${fields.join(", ")} }`}function clientAndTargetToml(role,selection){const client=clientProfile(selection.provider);return[`[llm_clients.${role}]`,`format = ${tomlString(client.format)}`,`base_url = ${tomlString(client.baseUrl)}`,`api_key_env = ${tomlString(client.apiKeyEnv)}`,"forward_auth = false",...Object.keys(client.extraHeaders).length>0?[`extra_headers = ${tomlStringMap(client.extraHeaders)}`]:[],"",`[targets.${role}]`,`id = ${tomlString(selection.model)}`,`llm_client = ${tomlString(role)}`,""]}function serializeHermesSwitchyardRouting(value){const routing=validateHermesSwitchyardRoutingIntent(value);const relayPluginsToml=["version = 1","","[[plugins.dynamic]]",`manifest = ${tomlString(HERMES_SWITCHYARD_PLUGIN_MANIFEST)}`,"","[plugins.dynamic.config]","priority = 0",`switchyard_config_path = ${tomlString(HERMES_SWITCHYARD_DEPLOYMENT_FILE)}`,"",'[plugins.policy.overrides."nvidia.switchyard"]','attestation = "integrity_only"',""].join("\n");const switchyardDeploymentToml=["schema_version = 1","",...clientAndTargetToml("efficient",routing.efficient),...clientAndTargetToml("capable",routing.capable),`[routes.${routing.routeId}]`,`id = ${tomlString(routing.routeId)}`,'type = "llm_classifier"','mode = "capability"','classifier_target = "capable"','strong_target = "capable"','weak_target = "efficient"',"base_threshold = 0.5",'classify_trigger = "new_session"',"message_hash_fallback = true",""].join("\n");const relayPluginStateJson=`${JSON.stringify({schema_version:1,records:[{metadata:{id:"nvidia.switchyard",kind:"rust_dynamic",generation:1},source:{manifest_ref:HERMES_SWITCHYARD_PLUGIN_MANIFEST},spec:{present:true,enabled:true},compatibility:{rust_dynamic:{relay:">=0.8.0, <1.0.0",native_api:"1"}},load:{rust_dynamic:{library:"libswitchyard_nemo_relay_plugin.so",symbol:"nemo_relay_register_plugin"}},status:{}}]},null,2)} -`;return{relayPluginsToml,relayPluginStateJson,switchyardDeploymentToml}}var import_node_buffer3=require("node:buffer");var PROVIDERLESS_INFERENCE_ENV=Object.freeze({NEMOCLAW_MODEL:"",NEMOCLAW_INFERENCE_PROVIDER_ID:"",NEMOCLAW_UPSTREAM_PROVIDER:"",NEMOCLAW_INFERENCE_BASE_URL:"",NEMOCLAW_INFERENCE_API:""});function isObjectRecord(value){return typeof value==="object"&&value!==null&&!Array.isArray(value)}var ChannelManifestRegistry=class{manifests=new Map;constructor(manifests=[]){for(const manifest of manifests){this.register(manifest)}}register(manifest){if(this.manifests.has(manifest.id)){throw new Error(`Duplicate channel manifest id '${manifest.id}'`)}this.manifests.set(manifest.id,manifest);return this}get(channelId){return this.manifests.get(channelId)}list(){return Array.from(this.manifests.values())}listAvailable(ctx={}){const supportedChannelIds=Array.isArray(ctx.supportedChannelIds)?new Set(ctx.supportedChannelIds):null;return this.list().filter(manifest=>{if(ctx.agent&&!manifest.supportedAgents.includes(ctx.agent)){return false}if(supportedChannelIds&&!supportedChannelIds.has(manifest.id)){return false}return true})}};function createChannelManifestRegistry(manifests=[]){return new ChannelManifestRegistry(manifests)}var discordManifest={schemaVersion:1,id:"discord",displayName:"Discord",description:"Discord bot messaging",supportedAgents:["openclaw","hermes"],state:{hermes:["platforms/discord"]},auth:{mode:"token-paste"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"DISCORD_BOT_TOKEN",formatPattern:"^(?!$)\\S+$",formatHint:"Replace the documentation placeholder with your real Discord bot token.",prompt:{label:"Discord Bot Token",help:"Discord Developer Portal \u2192 Applications \u2192 Bot \u2192 Reset/Copy Token."}},{id:"serverId",kind:"config",required:false,envKey:"DISCORD_SERVER_ID",statePath:"discordGuilds.serverId",prompt:{label:"Discord Server ID (for guild workspace access)",help:"Enable Developer Mode in Discord, then right-click your server and copy the Server ID.",emptyValueMessage:"guild channels stay disabled"}},{id:"requireMention",kind:"config",required:false,envKey:"DISCORD_REQUIRE_MENTION",statePath:"discordGuilds.requireMention",promptWhenInput:"serverId",validValues:["0","1"],defaultValue:"1",prompt:{label:"Discord mention mode",help:"Choose whether the bot should reply only when @mentioned or to all messages in this server."}},{id:"userId",kind:"config",required:false,envKey:"DISCORD_USER_ID",statePath:"discordGuilds.userIds",promptWhenInput:"serverId",prompt:{label:"Discord User ID (optional guild allowlist)",help:"Optional: enable Developer Mode in Discord, then right-click your user/avatar and copy the User ID. Leave blank to allow any member of the configured server to message the bot.",emptyValueMessage:"any member in the configured server can message the bot"}}],credentials:[{id:"discordBotToken",sourceInput:"botToken",providerName:"{sandboxName}-discord-bridge",providerEnvKey:"DISCORD_BOT_TOKEN",placeholder:"openshell:resolve:env:DISCORD_BOT_TOKEN"}],policyPresets:[{name:"discord",requiredAtCreate:true,validationWarningLines:["For Discord preset validation, do not use curl as the success signal:","curl is not in the preset binary allowlist, so curl probes can fail even","when the policy is working. Validate the configured messaging bridge/gateway path.",'DNS-only checks such as dns.resolve("gateway.discord.gg")',"can also be inconclusive behind a proxy.","The agent-specific gateway probe prints an HTTP status when it reaches Discord.","Any HTTP response confirms reachability. A transport error or OpenShell policy","denial means validation failed."],validationWarningLinesByAgent:{openclaw:["OpenClaw validation uses its Node runtime:",`node -e "require('node:https').get('https://discord.com/api/v10/gateway',r=>console.log(r.statusCode)).on('error',e=>{console.error(e.message);process.exitCode=1})"`],hermes:["Hermes validation uses its virtual-environment Python runtime:",`nemohermes exec -- /opt/hermes/.venv/bin/python -c "import urllib.error, urllib.request; u='https://discord.com/api/v10/gateway'; +"use strict";var __create=Object.create;var __defProp=Object.defineProperty;var __getOwnPropDesc=Object.getOwnPropertyDescriptor;var __getOwnPropNames=Object.getOwnPropertyNames;var __getProtoOf=Object.getPrototypeOf;var __hasOwnProp=Object.prototype.hasOwnProperty;var __export=(target2,all)=>{for(var name in all)__defProp(target2,name,{get:all[name],enumerable:true})};var __copyProps=(to,from,except,desc)=>{if(from&&typeof from==="object"||typeof from==="function"){for(let key of __getOwnPropNames(from))if(!__hasOwnProp.call(to,key)&&key!==except)__defProp(to,key,{get:()=>from[key],enumerable:!(desc=__getOwnPropDesc(from,key))||desc.enumerable})}return to};var __toESM=(mod,isNodeMode,target2)=>(target2=mod!=null?__create(__getProtoOf(mod)):{},__copyProps(isNodeMode||!mod||!mod.__esModule?__defProp(target2,"default",{value:mod,enumerable:true}):target2,mod));var __toCommonJS=mod=>__copyProps(__defProp({},"__esModule",{value:true}),mod);var managed_startup_direct_entry_exports={};__export(managed_startup_direct_entry_exports,{MANAGED_STARTUP_CA_ENV:()=>MANAGED_STARTUP_CA_ENV,MANAGED_STARTUP_COMPLETION_FILE:()=>MANAGED_STARTUP_COMPLETION_FILE,MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION:()=>MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,MANAGED_STARTUP_MERGED_CA_FILE:()=>MANAGED_STARTUP_MERGED_CA_FILE,MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES:()=>MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES,MANAGED_STARTUP_PROFILE_ENV:()=>MANAGED_STARTUP_PROFILE_ENV,MANAGED_STARTUP_RUNTIME_ENV_FILE:()=>MANAGED_STARTUP_RUNTIME_ENV_FILE,MANAGED_STARTUP_RUNTIME_EXECUTABLE:()=>MANAGED_STARTUP_RUNTIME_EXECUTABLE,ManagedStartupImageActionPlanError:()=>ManagedStartupImageActionPlanError,ManagedStartupImageRuntimeError:()=>ManagedStartupImageRuntimeError,NVIDIA_HOSTED_CREDENTIAL_ENV:()=>NVIDIA_HOSTED_CREDENTIAL_ENV,NVIDIA_HOSTED_NATIVE_ENDPOINT:()=>NVIDIA_HOSTED_NATIVE_ENDPOINT,applyManagedStartupCommandEnvironmentPlan:()=>applyManagedStartupCommandEnvironmentPlan,applyManagedStartupImageProfile:()=>applyManagedStartupImageProfile,applyManagedStartupOwnerRequest:()=>applyManagedStartupOwnerRequest,applyManagedStartupRootRequest:()=>applyManagedStartupRootRequest,atomicWriteRootFile:()=>atomicWriteRootFile,buildManagedStartupImageActionPlan:()=>buildManagedStartupImageActionPlan,finishManagedStartupDelegatedRequest:()=>finishManagedStartupDelegatedRequest,fixedHostedNativeProvider:()=>fixedHostedNativeProvider,installCorporateCaSystemAnchors:()=>installCorporateCaSystemAnchors,installHermesManagedPolicy:()=>installHermesManagedPolicy,installHermesSwitchyardRoutingFiles:()=>installHermesSwitchyardRoutingFiles,main:()=>main,managedStartupSandboxPrefix:()=>managedStartupSandboxPrefix,normalizeHermesManagedConfigDescriptor:()=>normalizeHermesManagedConfigDescriptor,normalizeManagedStartupWorkspaceRoot:()=>normalizeManagedStartupWorkspaceRoot,prepareManagedStartupDelegatedRequest:()=>prepareManagedStartupDelegatedRequest,publishManagedStartupCompletionAfterCommit:()=>publishManagedStartupCompletionAfterCommit,readStableRegularFile:()=>readStableRegularFile,readStableRegularFileSnapshot:()=>readStableRegularFileSnapshot,serializeManagedStartupCompletionMarker:()=>serializeManagedStartupCompletionMarker,serializeManagedStartupRuntimeEnvironment:()=>serializeManagedStartupRuntimeEnvironment,validateManagedStartupApplicationRuntimePlan:()=>validateManagedStartupApplicationRuntimePlan,verifyHermesSwitchyardRoutingFiles:()=>verifyHermesSwitchyardRoutingFiles,verifyManagedStartupImageCompletion:()=>verifyManagedStartupImageCompletion,waitForManagedStartupImageCompletion:()=>waitForManagedStartupImageCompletion});module.exports=__toCommonJS(managed_startup_direct_entry_exports);var NVIDIA_HOSTED_LOGICAL_PROVIDER="nvidia-prod";var NVIDIA_HOSTED_NATIVE_ENDPOINT="https://integrate.api.nvidia.com/v1";var NVIDIA_HOSTED_CREDENTIAL_ENV="NVIDIA_INFERENCE_API_KEY";var NATIVE_NVIDIA_AUTH_HEADER_SCRIPT=['case "${NVIDIA_INFERENCE_API_KEY:-}" in *[!a-zA-Z0-9:_]*) exit 2 ;; esac',`printf '%s' "\${NVIDIA_INFERENCE_API_KEY:-}" | LC_ALL=C grep -Eq '^openshell:resolve:env:((v[0-9]{1,20}|s[a-f0-9]{64})_)?NVIDIA_INFERENCE_API_KEY$' || exit 2`,'AUTH_HEADER="Authorization: Bearer ${NVIDIA_INFERENCE_API_KEY}"'].join("; ");function endpointUrlHasUserinfoQueryOrFragment(value){const raw=String(value||"").trim();if(!raw)return false;try{const url=new URL(raw);if(!["http:","https:"].includes(url.protocol)){return/[?#@]/.test(raw)}return Boolean(url.username||url.password)||/^https?:[\\/]*[^/?#\\]*@/i.test(raw)||/[?#]/.test(raw)}catch{return/[?#@]/.test(raw)}}var ENDPOINT_URL_ALLOWED_CHARACTERS=/^[A-Za-z0-9_./:=,@%+\-[\]~]+$/u;var CONTROL_OR_FORMAT_CHARACTER=/[\p{Cc}\p{Cf}]/u;function trimEndpointUrlAsciiSpaces(value){return value.replace(/^ +/u,"").replace(/ +$/u,"")}function unsafeEndpointUrlViolation(value){const input=String(value||"");const raw=trimEndpointUrlAsciiSpaces(input);if(!raw)return null;if(CONTROL_OR_FORMAT_CHARACTER.test(input)){return{kind:"control-characters",reason:"must not contain control characters."}}if(endpointUrlHasUserinfoQueryOrFragment(raw)){return{kind:"userinfo-query-fragment",reason:"must not contain userinfo, query, or fragment components."}}let decoded;try{decoded=decodeURIComponent(raw)}catch{return{kind:"invalid-url",reason:"must be a valid HTTP or HTTPS URL."}}if(CONTROL_OR_FORMAT_CHARACTER.test(decoded)){return{kind:"encoded-control-characters",reason:"must not contain percent-encoded control characters."}}if(!ENDPOINT_URL_ALLOWED_CHARACTERS.test(raw)){return{kind:"unsupported-characters",reason:"must contain only URL-safe ASCII characters."}}try{const url=new URL(raw);if(!["http:","https:"].includes(url.protocol)){return{kind:"unsupported-protocol",reason:"must use HTTP or HTTPS."}}}catch{return{kind:"invalid-url",reason:"must be a valid HTTP or HTTPS URL."}}return null}function isCustomHostedProvider(provider2){return provider2==="compatible-endpoint"||provider2==="compatible-anthropic-endpoint"}function customNativeCredentialEnv(provider2){if(provider2==="compatible-endpoint")return"COMPATIBLE_API_KEY";if(provider2==="compatible-anthropic-endpoint")return"COMPATIBLE_ANTHROPIC_API_KEY";throw new Error("Unsupported custom hosted provider")}var HOSTED_NATIVE_PROVIDERS=[{logicalProvider:"openai-api",label:"OpenAI",profileId:"nemoclaw-openai-inference-v1",providerName:"nemoclaw-openai-api-v1",credentialEnv:"OPENAI_API_KEY",endpoint:"https://api.openai.com/v1",api:"openai-completions"},{logicalProvider:"anthropic-prod",label:"Anthropic",profileId:"nemoclaw-anthropic-inference-v1",providerName:"nemoclaw-anthropic-prod-v1",credentialEnv:"ANTHROPIC_API_KEY",endpoint:"https://api.anthropic.com",api:"anthropic-messages"},{logicalProvider:"gemini-api",label:"Gemini",profileId:"nemoclaw-gemini-inference-v1",providerName:"nemoclaw-gemini-api-v1",credentialEnv:"GEMINI_API_KEY",endpoint:"https://generativelanguage.googleapis.com/v1beta/openai/",api:"openai-completions"},{logicalProvider:"openrouter-api",label:"OpenRouter",profileId:"nemoclaw-openrouter-inference-v1",providerName:"nemoclaw-openrouter-api-v1",credentialEnv:"OPENROUTER_API_KEY",endpoint:"https://openrouter.ai/api/v1",api:"openai-completions"},{logicalProvider:"hermes-provider",label:"Hermes Provider",profileId:"nemoclaw-hermes-inference-v1",providerName:"nemoclaw-hermes-provider-v1",credentialEnv:"OPENAI_API_KEY",endpoint:"https://inference-api.nousresearch.com/v1",api:"openai-completions"}];function recordedHermesEndpointUrl(value){const endpoint=new URL(value);if(!["http:","https:"].includes(endpoint.protocol)||endpoint.username||endpoint.password||endpoint.search||endpoint.hash||unsafeEndpointUrlViolation(value)||/%2f|%5c/iu.test(endpoint.pathname))throw new Error("Recorded Hermes inference endpoint is not an admitted HTTP(S) destination");return endpoint}function fixedHostedNativeProvider(provider2){return HOSTED_NATIVE_PROVIDERS.find(entry=>entry.logicalProvider===provider2?.trim())}var import_node_child_process=require("node:child_process");var import_node_crypto6=require("node:crypto");var import_node_fs3=__toESM(require("node:fs"));var import_node_path3=__toESM(require("node:path"));var MAX_CORPORATE_CA_BYTES=128*1024;var PEM_CERTIFICATE_RE_GLOBAL=/-----BEGIN CERTIFICATE-----[\s\S]+?-----END CERTIFICATE-----/g;var deep_agents_code_runtime_identity_default={uid:999,gid:999};var DEEP_AGENTS_CODE_RUNTIME_IDENTITY=Object.freeze(deep_agents_code_runtime_identity_default);var DEEP_AGENTS_CODE_SANDBOX_USER=`${DEEP_AGENTS_CODE_RUNTIME_IDENTITY.uid}:${DEEP_AGENTS_CODE_RUNTIME_IDENTITY.gid}`;var SHIPPED_MANAGED_IMAGE_AGENTS=["openclaw","hermes","langchain-deepagents-code"];var CANDIDATE_MANAGED_IMAGE_AGENTS=["pi"];var MANAGED_IMAGE_AGENTS=[...SHIPPED_MANAGED_IMAGE_AGENTS,...CANDIDATE_MANAGED_IMAGE_AGENTS];var MANAGED_IMAGE_RUNTIME_IDENTITIES=Object.freeze({openclaw:Object.freeze({uid:998,gid:998,workdir:"/sandbox"}),hermes:Object.freeze({uid:998,gid:999,workdir:"/sandbox"}),"langchain-deepagents-code":Object.freeze({...DEEP_AGENTS_CODE_RUNTIME_IDENTITY,workdir:"/sandbox"}),pi:Object.freeze({uid:999,gid:999,workdir:"/sandbox"})});function managedImageRuntimeIdentity(agent){return MANAGED_IMAGE_RUNTIME_IDENTITIES[agent]}var import_node_buffer3=require("node:buffer");var import_node_crypto2=require("node:crypto");var import_node_fs=__toESM(require("node:fs"));var import_node_path=__toESM(require("node:path"));var import_node_util2=require("node:util");var import_node_buffer2=require("node:buffer");var import_node_crypto=require("node:crypto");var import_node_util=require("node:util");function isLoopbackHostname(hostname=""){const normalized=String(hostname||"").trim().toLowerCase().replace(/^\[|\]$/g,"");return normalized==="localhost"||normalized==="::1"||/^127(?:\.\d{1,3}){3}$/.test(normalized)}function isLoopbackDashboardUrl(value){return isLoopbackHostname(new URL(value).hostname)}var import_node_buffer=require("node:buffer");function nativeHostedAgentConfig(provider2,baseUrl){if(isCustomHostedProvider(provider2)&&new URL(baseUrl).hostname!=="inference.local"){const endpoint2=new URL(baseUrl);if(!["https:","http:"].includes(endpoint2.protocol)||endpoint2.username||endpoint2.password||endpoint2.search||endpoint2.hash)throw new Error("Invalid native custom inference endpoint");if(["localhost","127.0.0.1","[::1]"].includes(endpoint2.hostname))return void 0;const credentialEnv=customNativeCredentialEnv(provider2);return{credentialEnv,apiKey:`openshell:resolve:env:${credentialEnv}`}}const definition=fixedHostedNativeProvider(provider2);if(!definition)return void 0;if(new URL(baseUrl).hostname==="inference.local")return void 0;const endpoint=provider2==="hermes-provider"?recordedHermesEndpointUrl(baseUrl):new URL(baseUrl);const expected=provider2==="hermes-provider"?endpoint:new URL(definition.endpoint);if(endpoint.username||endpoint.password||endpoint.search||endpoint.hash||endpoint.origin!==expected.origin||endpoint.pathname.replace(/\/+$/,"")!==expected.pathname.replace(/\/+$/,"")){throw new Error(`Native ${definition.label} configuration does not match its attached provider endpoint`)}return{credentialEnv:definition.credentialEnv,apiKey:`openshell:resolve:env:${definition.credentialEnv}`,...provider2==="openrouter-api"?{headers:{"HTTP-Referer":"https://www.nvidia.com/nemoclaw/","X-OpenRouter-Title":"NVIDIA NemoClaw"}}:{}}}var HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION=1;var HERMES_SWITCHYARD_ROUTE_ID="switchyard";var HERMES_SWITCHYARD_ROUTING_ENV="NEMOCLAW_HERMES_SWITCHYARD_ROUTING_B64";var HERMES_RELAY_PLUGINS_FILE="/usr/local/share/nemoclaw/hermes-relay-plugins.toml";var HERMES_RELAY_PLUGIN_STATE_FILE="/usr/local/share/nemoclaw/.dynamic-plugins.json";var HERMES_SWITCHYARD_DEPLOYMENT_FILE="/usr/local/share/nemoclaw/hermes-switchyard-v1.toml";var HERMES_SWITCHYARD_PLUGIN_MANIFEST="/opt/nemo-relay/plugins/switchyard/relay-plugin.toml";var HERMES_RELAY_PLUGINS_STAGING_FILE="hermes-relay-plugins.toml";var HERMES_RELAY_PLUGIN_STATE_STAGING_FILE="hermes-relay-plugin-state.json";var HERMES_SWITCHYARD_DEPLOYMENT_STAGING_FILE="hermes-switchyard-v1.toml";var MAX_MODEL_BYTES=512;var MAX_ROUTING_BYTES=8*1024;var STANDARD_BASE64_RE=/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u;var MODEL_PATTERN=/^[A-Za-z0-9][A-Za-z0-9._:+/-]*$/u;var CREDENTIAL_ENV_PATTERN=/^[A-Z_][A-Z0-9_]*$/u;var HEADER_NAME_PATTERN=/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/u;var SECRET_VALUE_PATTERNS=[/(?:nvapi|nvcf|sk-(?:proj-|ant-)?|ghp_|github_pat_|hf_|glpat-|gsk_|pypi-|tvly-)/u,/Bearer\s+/iu,/-----BEGIN (?:[A-Z0-9]+ )?PRIVATE KEY-----/u];var HOSTED_SWITCHYARD_PROVIDER_DEFINITIONS=HOSTED_NATIVE_PROVIDERS.filter(definition=>definition.logicalProvider!=="hermes-provider");var HERMES_SWITCHYARD_PROVIDERS=[NVIDIA_HOSTED_LOGICAL_PROVIDER,...HOSTED_SWITCHYARD_PROVIDER_DEFINITIONS.map(definition=>definition.logicalProvider)];var HermesSwitchyardRoutingError=class extends Error{constructor(message){super(`Invalid Hermes Switchyard routing: ${message}`);this.name="HermesSwitchyardRoutingError"}};function fail(message){throw new HermesSwitchyardRoutingError(message)}function record(value,where){if(typeof value!=="object"||value===null||Array.isArray(value)){return fail(`${where} must be an object`)}return value}function rejectUnknownKeys(value,allowed,where){const unknown=Object.keys(value).filter(key=>!allowed.has(key));if(unknown.length>0)fail(`${where} contains unknown fields`)}function provider(value,where){if(typeof value!=="string"||!HERMES_SWITCHYARD_PROVIDERS.includes(value)){return fail(`${where} must name an approved fixed hosted provider profile`)}return value}function model(value,where){if(typeof value!=="string"||import_node_buffer.Buffer.byteLength(value,"utf8")===0||import_node_buffer.Buffer.byteLength(value,"utf8")>MAX_MODEL_BYTES||!MODEL_PATTERN.test(value)||value.includes("://")||value.includes("//")||value.includes("..")||value.includes("${")||value.toLowerCase().includes("openshell:resolve")||SECRET_VALUE_PATTERNS.some(pattern=>pattern.test(value))){return fail(`${where} must be a bounded provider model ID without credentials or endpoints`)}return value}function target(value,where){const selection=record(value,where);rejectUnknownKeys(selection,new Set(["provider","model"]),where);return{provider:provider(selection.provider,`${where}.provider`),model:model(selection.model,`${where}.model`)}}function validateHermesSwitchyardRoutingIntent(value){const routing=record(value,"routing");rejectUnknownKeys(routing,new Set(["schemaVersion","routeId","efficient","capable","judge"]),"routing");if(routing.schemaVersion!==HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION){fail(`schemaVersion must be ${String(HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION)}`)}if(routing.routeId!==HERMES_SWITCHYARD_ROUTE_ID){fail(`routeId must be ${HERMES_SWITCHYARD_ROUTE_ID}`)}const efficient=target(routing.efficient,"routing.efficient");const capable=target(routing.capable,"routing.capable");if(efficient.model===capable.model){fail("efficient and capable must select different model IDs")}const judge=record(routing.judge,"routing.judge");rejectUnknownKeys(judge,new Set(["mode","target"]),"routing.judge");if(judge.mode!=="reuse-capable"||judge.target!=="capable"){fail("judge must explicitly reuse the capable target")}return{schemaVersion:HERMES_SWITCHYARD_ROUTING_SCHEMA_VERSION,routeId:HERMES_SWITCHYARD_ROUTE_ID,efficient,capable,judge:{mode:"reuse-capable",target:"capable"}}}function encodeHermesSwitchyardRoutingIntent(value){const routing=validateHermesSwitchyardRoutingIntent(value);return import_node_buffer.Buffer.from(JSON.stringify(routing),"utf8").toString("base64")}function decodeHermesSwitchyardRoutingIntent(encoded){if(encoded.length===0||encoded.length>Math.ceil(MAX_ROUTING_BYTES/3)*4||!STANDARD_BASE64_RE.test(encoded)){return fail("encoded routing transport is invalid")}const decoded=import_node_buffer.Buffer.from(encoded,"base64");if(decoded.length>MAX_ROUTING_BYTES||decoded.toString("base64")!==encoded){return fail("encoded routing transport is invalid")}let parsed;try{parsed=JSON.parse(decoded.toString("utf8"))}catch{return fail("encoded routing transport must contain JSON")}return validateHermesSwitchyardRoutingIntent(parsed)}function readHermesSwitchyardRoutingEnvironment(env){const encoded=env[HERMES_SWITCHYARD_ROUTING_ENV];return encoded===void 0||encoded===""?null:decodeHermesSwitchyardRoutingIntent(encoded)}function validateClientProfile(profile){let endpoint;try{endpoint=new URL(profile.baseUrl)}catch{return fail("provider profile has an invalid endpoint")}if(endpoint.protocol!=="https:"||endpoint.username||endpoint.password||endpoint.search||endpoint.hash||endpoint.hostname==="inference.local"||/[\\\s]/u.test(profile.baseUrl)||!CREDENTIAL_ENV_PATTERN.test(profile.apiKeyEnv)){return fail("provider profile is not safe for Switchyard onboarding")}for(const[name,value]of Object.entries(profile.extraHeaders)){if(!HEADER_NAME_PATTERN.test(name)||/[\r\n]/u.test(value)||["authorization","proxy-authorization","x-api-key"].includes(name.toLowerCase())){return fail("provider profile contains an unsafe Switchyard header")}}return profile}function clientProfile(providerId){if(providerId===NVIDIA_HOSTED_LOGICAL_PROVIDER){return validateClientProfile({baseUrl:NVIDIA_HOSTED_NATIVE_ENDPOINT,apiKeyEnv:NVIDIA_HOSTED_CREDENTIAL_ENV,format:"openai_chat",extraHeaders:{}})}const definition=fixedHostedNativeProvider(providerId);if(!definition||definition.logicalProvider==="hermes-provider"){return fail("provider profile is not supported by Switchyard onboarding")}const agentConfig=nativeHostedAgentConfig(providerId,definition.endpoint);if(!agentConfig)return fail("provider profile has no managed credential binding");return validateClientProfile({baseUrl:definition.endpoint,apiKeyEnv:definition.credentialEnv,format:definition.api==="anthropic-messages"?"anthropic_messages":"openai_chat",extraHeaders:agentConfig.headers??{}})}function tomlString(value){return JSON.stringify(value)}function tomlStringMap(value){const fields=Object.entries(value).sort(([left],[right])=>left.localeCompare(right)).map(([name,headerValue])=>`${tomlString(name)} = ${tomlString(headerValue)}`);return`{ ${fields.join(", ")} }`}function clientAndTargetToml(role,selection){const client=clientProfile(selection.provider);return[`[llm_clients.${role}]`,`format = ${tomlString(client.format)}`,`base_url = ${tomlString(client.baseUrl)}`,`api_key_env = ${tomlString(client.apiKeyEnv)}`,"forward_auth = false",...Object.keys(client.extraHeaders).length>0?[`extra_headers = ${tomlStringMap(client.extraHeaders)}`]:[],"",`[targets.${role}]`,`id = ${tomlString(selection.model)}`,`llm_client = ${tomlString(role)}`,""]}function serializeHermesSwitchyardRouting(value){const routing=validateHermesSwitchyardRoutingIntent(value);const relayPluginsToml=["version = 1","","[[plugins.dynamic]]",`manifest = ${tomlString(HERMES_SWITCHYARD_PLUGIN_MANIFEST)}`,"","[plugins.dynamic.config]","priority = 0",`switchyard_config_path = ${tomlString(HERMES_SWITCHYARD_DEPLOYMENT_FILE)}`,"",'[plugins.policy.overrides."nvidia.switchyard"]','attestation = "integrity_only"',""].join("\n");const switchyardDeploymentToml=["schema_version = 1","",...clientAndTargetToml("efficient",routing.efficient),...clientAndTargetToml("capable",routing.capable),`[routes.${routing.routeId}]`,`id = ${tomlString(routing.routeId)}`,'type = "llm_classifier"','mode = "capability"','classifier_target = "capable"','strong_target = "capable"','weak_target = "efficient"',"base_threshold = 0.5",'classify_trigger = "new_session"',"message_hash_fallback = true",""].join("\n");const relayPluginStateJson=`${JSON.stringify({schema_version:1,records:[{metadata:{id:"nvidia.switchyard",kind:"rust_dynamic",generation:1},source:{manifest_ref:HERMES_SWITCHYARD_PLUGIN_MANIFEST},spec:{present:true,enabled:true},compatibility:{rust_dynamic:{relay:">=0.8.0, <1.0.0",native_api:"1"}},load:{rust_dynamic:{library:"libswitchyard_nemo_relay_plugin.so",symbol:"nemo_relay_register_plugin"}},status:{}}]},null,2)} +`;return{relayPluginsToml,relayPluginStateJson,switchyardDeploymentToml}}var ChannelManifestRegistry=class{manifests=new Map;constructor(manifests=[]){for(const manifest of manifests){this.register(manifest)}}register(manifest){if(this.manifests.has(manifest.id)){throw new Error(`Duplicate channel manifest id '${manifest.id}'`)}this.manifests.set(manifest.id,manifest);return this}get(channelId){return this.manifests.get(channelId)}list(){return Array.from(this.manifests.values())}listAvailable(ctx={}){const supportedChannelIds=Array.isArray(ctx.supportedChannelIds)?new Set(ctx.supportedChannelIds):null;return this.list().filter(manifest=>{if(ctx.agent&&!manifest.supportedAgents.includes(ctx.agent)){return false}if(supportedChannelIds&&!supportedChannelIds.has(manifest.id)){return false}return true})}};function createChannelManifestRegistry(manifests=[]){return new ChannelManifestRegistry(manifests)}var discordManifest={schemaVersion:1,id:"discord",displayName:"Discord",description:"Discord bot messaging",supportedAgents:["openclaw","hermes"],state:{hermes:["platforms/discord"]},auth:{mode:"token-paste"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"DISCORD_BOT_TOKEN",formatPattern:"^(?!$)\\S+$",formatHint:"Replace the documentation placeholder with your real Discord bot token.",prompt:{label:"Discord Bot Token",help:"Discord Developer Portal \u2192 Applications \u2192 Bot \u2192 Reset/Copy Token."}},{id:"serverId",kind:"config",required:false,envKey:"DISCORD_SERVER_ID",statePath:"discordGuilds.serverId",prompt:{label:"Discord Server ID (for guild workspace access)",help:"Enable Developer Mode in Discord, then right-click your server and copy the Server ID.",emptyValueMessage:"guild channels stay disabled"}},{id:"requireMention",kind:"config",required:false,envKey:"DISCORD_REQUIRE_MENTION",statePath:"discordGuilds.requireMention",promptWhenInput:"serverId",validValues:["0","1"],defaultValue:"1",prompt:{label:"Discord mention mode",help:"Choose whether the bot should reply only when @mentioned or to all messages in this server."}},{id:"userId",kind:"config",required:false,envKey:"DISCORD_USER_ID",statePath:"discordGuilds.userIds",promptWhenInput:"serverId",prompt:{label:"Discord User ID (optional guild allowlist)",help:"Optional: enable Developer Mode in Discord, then right-click your user/avatar and copy the User ID. Leave blank to allow any member of the configured server to message the bot.",emptyValueMessage:"any member in the configured server can message the bot"}}],credentials:[{id:"discordBotToken",sourceInput:"botToken",providerName:"{sandboxName}-discord-bridge",providerEnvKey:"DISCORD_BOT_TOKEN",placeholder:"openshell:resolve:env:DISCORD_BOT_TOKEN"}],policyPresets:[{name:"discord",requiredAtCreate:true,validationWarningLines:["For Discord preset validation, do not use curl as the success signal:","curl is not in the preset binary allowlist, so curl probes can fail even","when the policy is working. Validate the configured messaging bridge/gateway path.",'DNS-only checks such as dns.resolve("gateway.discord.gg")',"can also be inconclusive behind a proxy.","The agent-specific gateway probe prints an HTTP status when it reaches Discord.","Any HTTP response confirms reachability. A transport error or OpenShell policy","denial means validation failed."],validationWarningLinesByAgent:{openclaw:["OpenClaw validation uses its Node runtime:",`node -e "require('node:https').get('https://discord.com/api/v10/gateway',r=>console.log(r.statusCode)).on('error',e=>{console.error(e.message);process.exitCode=1})"`],hermes:["Hermes validation uses its virtual-environment Python runtime:",`nemohermes exec -- /opt/hermes/.venv/bin/python -c "import urllib.error, urllib.request; u='https://discord.com/api/v10/gateway'; try: print(urllib.request.urlopen(u, timeout=20).status) -except urllib.error.HTTPError as error: print(error.code)"`]}}],render:[{id:"discord-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.discord",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false},proxy:"{{discordProxyUrl}}",dmPolicy:"{{discord.allowedUsers.dmPolicy}}",allowFrom:"{{discord.allowedUsers.values}}"}}}}},{id:"discord-openclaw-guilds",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",when:"{{discord.hasGuilds}}",fragment:{path:"channels.discord",value:{groupPolicy:"allowlist",guilds:"{{discord.guilds}}"}}},{id:"discord-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.discord",value:{enabled:true}}},{id:"discord-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["NEMOCLAW_DISCORD_GUILD_IDS={{discord.guildIds.csv}}","DISCORD_ALLOWED_USERS={{discord.allowedUsers.csv}}","DISCORD_ALLOW_ALL_USERS={{discord.allowAllUsers}}"]},{id:"discord-hermes-config",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"discord",value:{require_mention:"{{discord.requireMention}}",free_response_channels:"",allowed_channels:"",auto_thread:true,reactions:true,channel_prompts:{}}}},{id:"discord-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.discord",value:{enabled:true}}}],runtime:{openclaw:{channelName:"discord",visibility:{configKeys:["discord"],logPatterns:["discord"]}}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/discord@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-j+fSHxbXA+DSxwbL8SvtsDNL6tvBX4+RmH+EerVW6dCbeIwSconXj6J/+AaN/mhzZI4g0jPPj30TUhdCRRbc2w==","2026.9.1":"sha512-qNmN2a8A9dET4igPp0RML171sEn8PDMyNCYNp/DqcJ4tn3XTHpacSOTkqBmv5yXTycJRC9rfFP8FT/SdW0Rldg=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/discord/-/discord-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/discord/-/discord-2026.9.1.tgz"},required:true}],hooks:[{id:"discord-hermes-status-health",phase:"status",handler:"discord.statusHealth",agents:["hermes"],outputs:[{id:"channelHealth",kind:"status"}]},{id:"discord-openclaw-bridge-health",phase:"health-check",handler:"discord.openclawBridgeHealth",agents:["openclaw"],onFailure:"abort"},{id:"discord-token-paste",phase:"enroll",handler:"discord.tokenPaste",outputs:[{id:"botToken",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"discord-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"serverId",kind:"config"},{id:"requireMention",kind:"config"},{id:"userId",kind:"config"}]}]};var googlechatManifest={schemaVersion:1,id:"googlechat",displayName:"Google Chat",description:"Google Chat (Chat API) bot messaging (experimental)",supportedAgents:["openclaw","hermes"],state:{hermes:["platforms/googlechat"]},auth:{mode:"token-paste"},inputs:[{id:"serviceAccount",kind:"secret",required:true,envKey:"GOOGLECHAT_SERVICE_ACCOUNT",allowLineBreaks:true,maskCap:40,formatHint:"Paste the entire service-account JSON key on one line (minified), or set GOOGLECHAT_SERVICE_ACCOUNT to the downloaded JSON, including line breaks.",maxTokenAttempts:3,prompt:{label:"Google Chat service account JSON",help:["\u2503 GOOGLE CHAT \u2014 service account key","\u2503","\u2503 Google Cloud Console \u2192 IAM & Admin \u2192 Service Accounts","\u2503 \u2192 your bot's SA \u2192 Keys \u2192 Add key \u2192 Create new key \u2192 JSON","\u2503","\u2503 A .json file downloads. Paste its contents below as ONE line (minified).","\u2503 Alternatively, GOOGLECHAT_SERVICE_ACCOUNT accepts the formatted JSON file contents.",""].join("\n")}},{id:"audienceType",kind:"config",required:false,envKey:"GOOGLECHAT_AUDIENCE_TYPE",statePath:"googlechatConfig.audienceType",validValues:["app-url","project-number"],defaultValue:"app-url"},{id:"audience",kind:"config",required:false,envKey:"GOOGLECHAT_AUDIENCE",statePath:"googlechatConfig.audience",prompt:{label:"Google Chat webhook audience",help:"Usually filled automatically from the public tunnel URL. For audienceType 'project-number', enter your GCP project number instead.",emptyValueMessage:"inbound webhook verification will be unconfigured"}},{id:"appPrincipal",kind:"config",required:false,envKey:"GOOGLECHAT_APP_PRINCIPAL",statePath:"googlechatConfig.appPrincipal",formatPattern:"^[0-9]{6,32}$",formatHint:"appPrincipal is the add-on's numeric OAuth client ID (uniqueId, ~21 digits), not an email.",prompt:{label:"Google Chat appPrincipal",help:[" Workspace account \u2192 leave blank, done."," Personal Gmail \u2192 needs the add-on's ~21-digit ID (not an email), stable across rebuilds.",""," If you already know it, paste it at the prompt and you're done."," If not, leave it blank \u2014 the first DM reveals it once the sandbox is live:",""," 1. Watch the gateway log:",' nemoclaw logs --follow | grep "unexpected add-on principal"'," 2. DM the bot once \u2014 it won't reply yet, that's expected. The log prints:"," unexpected add-on principal: "," 3. Save that and rebuild:"," GOOGLECHAT_APP_PRINCIPAL= nemoclaw channels add googlechat"," nemoclaw rebuild --yes"].join("\n"),emptyValueMessage:"Workspace accounts do not need it; personal accounts must set it later"}},{id:"allowFrom",kind:"config",required:false,envKey:"GOOGLECHAT_ALLOWED_USERS",statePath:"allowedIds.googlechat",prompt:{label:"Google Chat DM allowlist (comma-separated)",help:["Optional: restrict who can DM the bot."," OpenClaw: users/NNN (emails ignored)"," Hermes: email (users/NNN ignored)"," Blank: pairing mode (recommended) \u2014 OpenClaw's pairing reply shows your users/NNN"," Filling this switches DM policy to allowlist \u2014 a wrong-form entry is dropped silently, with no pairing code."].join("\n"),emptyValueMessage:"bot will require manual pairing"}},{id:"projectId",kind:"config",required:false,envKey:"GOOGLE_CHAT_PROJECT_ID",statePath:"googlechatConfig.projectId",prompt:{label:"Google Chat GCP project ID (Hermes Pub/Sub pull)",help:"The Google Cloud project that owns the Pub/Sub subscription Hermes pulls Chat events from. OpenClaw ignores this.",emptyValueMessage:"required for the Hermes Google Chat channel"}},{id:"subscriptionName",kind:"config",required:false,envKey:"GOOGLE_CHAT_SUBSCRIPTION_NAME",statePath:"googlechatConfig.subscriptionName",prompt:{label:"Google Chat Pub/Sub subscription (projects/

/subscriptions/)",help:["The pull subscription bound to the Chat events topic. Hermes pulls from it over the Pub/Sub REST API; the gateway-minted token is scoped to both chat.bot and pubsub."," Its topic must grant roles/pubsub.publisher to the app's push account:"," Interactive features service-@gcp-sa-gsuiteaddons.iam.gserviceaccount.com"," Classic bot chat-api-push@system.gserviceaccount.com"," Shown at Chat API \u2192 Configuration \u2192 Connection settings"," Missing it channel connects, no event arrives, Chat says the bot is not responding"].join("\n"),emptyValueMessage:"required for the Hermes Google Chat channel"}}],credentials:[],policyPresets:[{name:"googlechat",policyKeys:["googlechat"],agentPolicyKeys:{hermes:["googlechat_hermes"]}}],render:[{id:"googlechat-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.googlechat",value:{enabled:true,serviceAccount:{},audienceType:"{{googlechatConfig.audienceType}}",audience:"{{googlechatConfig.audience}}",appPrincipal:"{{googlechatConfig.appPrincipal}}",webhookPath:"/googlechat",healthMonitor:{enabled:false},dmPolicy:"{{allowedIds.googlechat.dmPolicy}}",allowFrom:"{{allowedIds.googlechat.values}}"}}},{id:"googlechat-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.googlechat",value:{enabled:true}}},{id:"googlechat-openclaw-gateway-reload-off",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"gateway.reload",value:{mode:"off"}}},{id:"googlechat-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["GOOGLE_CHAT_PROJECT_ID={{googlechatConfig.projectId}}","GOOGLE_CHAT_SUBSCRIPTION_NAME={{googlechatConfig.subscriptionName}}","GOOGLE_CHAT_ALLOWED_USERS={{allowedIds.googlechat.csv}}"]},{id:"googlechat-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.google_chat",value:{enabled:true}}}],runtime:{openclaw:{channelName:"googlechat",visibility:{configKeys:["googlechat"],logPatterns:["googlechat"]},nodePreloads:[{module:"googlechat-trusted-proxy-fetch",injectInto:["boot"],optional:false,installMessage:"[channels] Installing Google Chat trusted-proxy-fetch patch (route googleapis via trusted env proxy)",installedMessage:"[channels] Google Chat trusted-proxy-fetch patch installed (NODE_OPTIONS updated)"},{module:"googlechat-outbound-auth",injectInto:["boot"],optional:false,installMessage:"[channels] Installing Google Chat outbound-auth patch (gateway-minted bearer)",installedMessage:"[channels] Google Chat outbound-auth patch installed (NODE_OPTIONS updated)"}],secretScans:[{path:"/sandbox/.openclaw/openclaw.json",pattern:"-----BEGIN (?:RSA )?PRIVATE KEY-----",message:"[SECURITY] Google Chat service account private key leaked into {path} - refusing to serve",exitCode:78}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/googlechat@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-LUO8Lg07IhzJfEzxn+GSij8WMS/uX3hTmv0SydTy/0fKSN7iZksf74TaZg50kOoBi8FzeqeXo/zoGVuk0Mj1Ig==","2026.9.1":"sha512-Q5VTAJpfcrI7BSEw5Ugq3wf7JEg5QhTBwpi+BByGbfZsTTVjwZc7OIvNbKsVTh16I5/EWqHEnD+0WNeHqsteqw=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/googlechat/-/googlechat-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/googlechat/-/googlechat-2026.9.1.tgz"},required:true},{id:"hermesGooglePubsubPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"google-cloud-pubsub==2.39.0",required:true},{id:"hermesGoogleApiClientPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"google-api-python-client==2.194.0",required:true},{id:"hermesGoogleAuthPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"google-auth==2.55.1",required:true}],hooks:[{id:"googlechat-tunnel-audience-gate",phase:"enroll",handler:"googlechat.tunnelAudienceGate",agents:["openclaw"],inputs:["audienceType","audience"],outputs:[{id:"audience",kind:"config"}],onFailure:"skip-channel"},{id:"googlechat-service-account",phase:"enroll",handler:"googlechat.tokenPaste",outputs:[{id:"serviceAccount",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"googlechat-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"allowFrom",kind:"config"}]},{id:"googlechat-openclaw-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["openclaw"],outputs:[{id:"appPrincipal",kind:"config"}]},{id:"googlechat-hermes-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["hermes"],outputs:[{id:"projectId",kind:"config"},{id:"subscriptionName",kind:"config"}]}]};var slackManifest={schemaVersion:1,id:"slack",displayName:"Slack",description:"Slack bot messaging",supportedAgents:["openclaw","hermes"],state:{hermes:["platforms/slack"]},auth:{mode:"token-paste"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"SLACK_BOT_TOKEN",formatPattern:"^xoxb-[A-Za-z0-9_-]+$",formatHint:"Slack bot tokens start with 'xoxb-' (e.g. xoxb---).",prompt:{label:"Slack Bot Token",help:"Slack API \u2192 Your Apps \u2192 OAuth & Permissions \u2192 Bot User OAuth Token (xoxb-...)."}},{id:"appToken",kind:"secret",required:true,envKey:"SLACK_APP_TOKEN",formatPattern:"^xapp-[A-Za-z0-9_-]+$",formatHint:"Slack app tokens start with 'xapp-' (e.g. xapp----).",prompt:{label:"Slack App Token (Socket Mode)",help:"Slack API \u2192 Your Apps \u2192 Basic Information \u2192 App-Level Tokens (xapp-...)."}},{id:"allowedUsers",kind:"config",required:false,envKey:"SLACK_ALLOWED_USERS",statePath:"allowedIds.slack",prompt:{label:"Slack Member IDs (comma-separated allowlist)",help:"In Slack, open each allowed human user's profile -> More -> Copy member ID. Enter one or more comma-separated member IDs, not the app or bot user ID. Member IDs look like U01ABC2DEF3.",emptyValueMessage:"bot will require manual pairing"}},{id:"allowedChannels",kind:"config",required:false,envKey:"SLACK_ALLOWED_CHANNELS",statePath:"slackConfig.allowedChannels",prompt:{label:"Slack Channel IDs (comma-separated allowlist)",help:"Optional: enter comma-separated Slack channel IDs where the bot may answer @mentions. Channel IDs look like C012AB3CD.",emptyValueMessage:"channel @mentions stay unrestricted by channel ID"}}],credentials:[{id:"slackBotToken",sourceInput:"botToken",providerName:"{sandboxName}-slack-bridge",providerEnvKey:"SLACK_BOT_TOKEN",placeholder:"xoxb-OPENSHELL-RESOLVE-ENV-SLACK_BOT_TOKEN",primary:true},{id:"slackAppToken",sourceInput:"appToken",providerName:"{sandboxName}-slack-app",providerEnvKey:"SLACK_APP_TOKEN",placeholder:"xapp-OPENSHELL-RESOLVE-ENV-SLACK_APP_TOKEN"}],policyPresets:[{name:"slack",requiredAtCreate:true}],render:[{id:"slack-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.slack",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false},dmPolicy:"{{allowedIds.slack.dmPolicy}}",allowFrom:"{{allowedIds.slack.values}}",groupPolicy:"{{allowedIds.slack.groupPolicy}}",channels:"{{allowedIds.slack.channels}}"}}}}},{id:"slack-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.slack",value:{enabled:true}}},{id:"slack-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["SLACK_ALLOWED_USERS={{allowedIds.slack.csv}}","SLACK_ALLOWED_CHANNELS={{slackConfig.allowedChannels.csv}}"]},{id:"slack-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.slack",value:{enabled:true,extra:{rich_blocks:true}}}}],runtime:{openclaw:{channelName:"slack",visibility:{configKeys:["slack"],logPatterns:["slack"]},nodePreloads:[{module:"slack-channel-guard",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing Slack channel guard (unhandled-rejection safety net)",installedMessage:"[channels] Slack channel guard installed (NODE_OPTIONS updated)"}],secretScans:[{path:"/sandbox/.openclaw/openclaw.json",pattern:"(?:xoxb|xapp)-(?!OPENSHELL-RESOLVE-ENV-)",message:"[SECURITY] Slack token leaked into {path} - refusing to serve",exitCode:78}]},hermes:{}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/slack@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-6M1M6gL3iXahpalNsYAUuA+wvnV8lbMlNNH2ToegFvaSJcIll4S9kFa5mv3GFoquhMRHQjEjnkXPHP/pXwaWcA==","2026.9.1":"sha512-tU372jE40nnPcKQ6oxmDHf2/UhGtdz8ysi4JKsRZIO1QBAEkZd2YfsOw8aucmb2r0B0vjcFD3OmIV/Qzb57COg=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/slack/-/slack-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/slack/-/slack-2026.9.1.tgz"},required:true}],hooks:[{id:"slack-socket-mode-gateway-conflict",phase:"pre-enable",handler:"slack.socketModeGatewayConflict",onFailure:"abort"},{id:"slack-openclaw-bridge-health",phase:"health-check",handler:"slack.openclawBridgeHealth",agents:["openclaw"],onFailure:"abort"},{id:"slack-socket-mode-gateway-status",phase:"status",handler:"slack.socketModeGatewayStatus",outputs:[{id:"gatewayOverlaps",kind:"status"}]},{id:"slack-status-health",phase:"status",handler:"slack.statusHealth",providesReadiness:true,agents:["openclaw"],outputs:[{id:"channelHealth",kind:"status"}]},{id:"slack-token-paste",phase:"enroll",handler:"common.tokenPaste",outputs:[{id:"botToken",kind:"secret",required:true},{id:"appToken",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"slack-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"allowedUsers",kind:"config"},{id:"allowedChannels",kind:"config"}]},{id:"slack-credential-validation",phase:"reachability-check",handler:"slack.validateCredentials",inputs:["botToken","appToken"],onFailure:"skip-channel"}]};var TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT={channelId:"teams",renderId:"teams-openclaw-channel",hookId:"teams-openclaw-channel",handlerId:"common.staticOutputs",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",configPath:"channels.msteams",webhookPath:"/api/messages"};function authorizeTeamsOpenClawWebhookField(entry){if(!isPlainDataObject(entry))return[];const contract=TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT;if(ownDataPropertyValue(entry,"channelId")!==contract.channelId||ownDataPropertyValue(entry,"renderId")!==contract.renderId||ownDataPropertyValue(entry,"hookId")!==contract.hookId||ownDataPropertyValue(entry,"handler")!==contract.handlerId||ownDataPropertyValue(entry,"kind")!==contract.kind||ownDataPropertyValue(entry,"agent")!==contract.agent||ownDataPropertyValue(entry,"target")!==contract.target||ownDataPropertyValue(entry,"path")!==contract.configPath){return[]}const value=ownDataPropertyValue(entry,"value");if(!isPlainDataObject(value))return[];const webhook=ownDataPropertyValue(value,"webhook");if(!isPlainDataObject(webhook)||!hasExactlyOwnDataProperties(webhook,["path","port"])||!isTcpPort(ownDataPropertyValue(webhook,"port"))||ownDataPropertyValue(webhook,"path")!==contract.webhookPath){return[]}return[{path:["value","webhook"],value:webhook}]}function isPlainDataObject(value){if(value===null||typeof value!=="object"||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function ownDataPropertyValue(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}function hasExactlyOwnDataProperties(value,expected){const actual=Object.getOwnPropertyNames(value).sort();return actual.length===expected.length&&actual.every((key,index)=>key===expected[index])}function isTcpPort(value){return Number.isInteger(value)&&value>=1&&value<=65535}var teamsManifest={schemaVersion:1,id:"teams",displayName:"Microsoft Teams",description:"Microsoft Teams bot messaging (experimental)",enrollmentNotes:["Microsoft Teams requires a public HTTPS webhook endpoint at /api/messages; expose the configured Teams webhook port before installing the Teams app.","Use Azure AD object IDs in TEAMS_ALLOWED_USERS so only authorized users can interact with the bot."],supportedAgents:["openclaw","hermes"],auth:{mode:"token-paste"},inputs:[{id:"appId",kind:"config",required:true,envKey:"MSTEAMS_APP_ID",statePath:"teamsConfig.appId",prompt:{label:"Microsoft Teams Client ID",help:"Run `teams app create --endpoint https:///api/messages`, then copy CLIENT_ID."}},{id:"clientSecret",kind:"secret",required:true,envKey:"MSTEAMS_APP_PASSWORD",prompt:{label:"Microsoft Teams Client Secret",help:"Use the CLIENT_SECRET printed by `teams app create`. It is shown once; rotate it in Entra ID if it was lost."}},{id:"tenantId",kind:"config",required:true,envKey:"MSTEAMS_TENANT_ID",statePath:"teamsConfig.tenantId",prompt:{label:"Microsoft Teams Tenant ID",help:"Use the TENANT_ID printed by `teams app create` or shown by `teams status --verbose`."}},{id:"allowedUsers",kind:"config",required:false,envKey:"TEAMS_ALLOWED_USERS",statePath:"allowedIds.teams",prompt:{label:"Microsoft Teams AAD Object IDs (comma-separated allowlist)",help:"Recommended: run `teams status --verbose` and enter the Azure AD object IDs allowed to use the bot."}},{id:"webhookPort",kind:"config",required:false,envKey:"MSTEAMS_PORT",statePath:"teamsConfig.webhookPort",defaultValue:"3978",prompt:{label:"Microsoft Teams webhook port",help:"Local bot webhook port to expose publicly. Defaults to 3978 and serves /api/messages."}},{id:"requireMention",kind:"config",required:false,envKey:"TEAMS_REQUIRE_MENTION",statePath:"teamsConfig.requireMention",validValues:["0","1"],defaultValue:"1",prompt:{label:"Microsoft Teams mention mode",help:"Controls OpenClaw group and channel behavior only. Direct messages are unaffected."}}],credentials:[{id:"teamsClientSecret",sourceInput:"clientSecret",providerName:"{sandboxName}-teams-bridge",providerEnvKey:"MSTEAMS_APP_PASSWORD",placeholder:"openshell:resolve:env:MSTEAMS_APP_PASSWORD",primary:true}],policyPresets:[{name:"teams",policyKeys:["teams"],requiredAtCreate:true}],hostForward:{port:"{{teamsConfig.webhookPort}}",label:"Microsoft Teams webhook"},render:[{id:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.renderId,kind:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.kind,agent:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.agent,target:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.target,fragment:{path:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.configPath,value:{enabled:true,appId:"{{teamsConfig.appId}}",tenantId:"{{teamsConfig.tenantId}}",webhook:{port:"{{teamsConfig.webhookPort}}",path:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.webhookPath},healthMonitor:{enabled:false},streaming:{mode:"off"},dmPolicy:"{{allowedIds.teams.dmPolicy}}",allowFrom:"{{allowedIds.teams.values}}",groupPolicy:"open",requireMention:"{{teamsConfig.requireMention}}"}}},{id:"teams-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.msteams",value:{enabled:true}}},{id:"teams-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["TEAMS_CLIENT_ID={{teamsConfig.appId}}","TEAMS_TENANT_ID={{teamsConfig.tenantId}}","TEAMS_ALLOWED_USERS={{allowedIds.teams.csv}}","TEAMS_PORT={{teamsConfig.webhookPort}}"]},{id:"teams-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.teams",value:{enabled:true}}}],runtime:{openclaw:{channelName:"msteams",visibility:{configKeys:["msteams"],logPatterns:["msteams","teams"]},nodePreloads:[{module:"msteams-message-hints",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing Microsoft Teams message hint patch (native mentions)",installedMessage:"[channels] Microsoft Teams message hint patch installed (NODE_OPTIONS updated)"}]},hermes:{envAliases:[{envKey:"MSTEAMS_APP_PASSWORD",targetEnvKey:"TEAMS_CLIENT_SECRET",match:"^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_MSTEAMS_APP_PASSWORD$",value:"openshell:resolve:env:MSTEAMS_APP_PASSWORD"}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/msteams@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-py5KvGOTcd0qGGRf3EuqbH2jO+kZtvMquDMjwGkT6x9F4XZtaCBL/lmLitb4hDb5xaIpPP8ZLIbT8GIMXQr3Og==","2026.9.1":"sha512-seRGr9/X6Vk9xU5elLVpDwq8R+TO0QFvUmxPEitqkngqDnMoXW0LEEXkriG6jgue74w2YLcNnAv/Rjf0a9jong=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/msteams/-/msteams-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/msteams/-/msteams-2026.9.1.tgz"},required:true},{id:"hermesTeamsAppsPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"microsoft-teams-apps==2.0.13.4",required:true}],hooks:[{id:"teams-host-forward-port-conflict",phase:"pre-enable",handler:"teams.hostForwardPortConflict",inputs:["webhookPort"],onFailure:"abort"},{id:"teams-host-forward-port-status",phase:"status",handler:"teams.hostForwardPortStatus",outputs:[{id:"hostForwardPortOverlaps",kind:"status"}]},{id:"teams-token-paste",phase:"enroll",handler:"common.tokenPaste",outputs:[{id:"clientSecret",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"teams-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"appId",kind:"config",required:true},{id:"tenantId",kind:"config",required:true},{id:"allowedUsers",kind:"config"},{id:"webhookPort",kind:"config"},{id:"requireMention",kind:"config"}]}],state:{}};var telegramManifest={schemaVersion:1,id:"telegram",displayName:"Telegram",description:"Telegram bot messaging",diagnosticsProbe:"log-tail",enrollmentNotes:["For Telegram group chats, disable privacy mode in @BotFather (/setprivacy -> your bot -> Disable).","After changing privacy mode, remove and re-add the bot to each group before testing @mentions."],supportedAgents:["openclaw","hermes"],state:{openclaw:["telegram"],hermes:["platforms/telegram"]},auth:{mode:"token-paste"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"TELEGRAM_BOT_TOKEN",prompt:{label:"Telegram Bot Token",help:"Create a bot via @BotFather on Telegram, then copy the token."}},{id:"allowedIds",kind:"config",required:false,envKey:"TELEGRAM_ALLOWED_IDS",statePath:"allowedIds.telegram",prompt:{label:"Telegram User ID (for DM access)",help:"Send /start to @userinfobot on Telegram to get your numeric user ID.",emptyValueMessage:"bot will require manual pairing"}},{id:"requireMention",kind:"config",required:false,envKey:"TELEGRAM_REQUIRE_MENTION",statePath:"telegramConfig.requireMention",validValues:["0","1"],defaultValue:"1",prompt:{label:"Telegram group mention mode",help:"Controls Telegram group-chat behavior only \u2014 reply only when @mentioned vs. to all group messages. Direct messages are unaffected by this setting and remain subject to pairing and TELEGRAM_ALLOWED_IDS."}},{id:"groupPolicy",kind:"config",required:false,envKey:"TELEGRAM_GROUP_POLICY",statePath:"telegramConfig.groupPolicy",validValues:["open","allowlist","disabled"],defaultValue:"open",prompt:{label:"Telegram group policy",help:"Controls OpenClaw Telegram group access. Hermes does not expose an equivalent disable-groups policy."}}],credentials:[{id:"telegramBotToken",sourceInput:"botToken",providerName:"{sandboxName}-telegram-bridge",providerEnvKey:"TELEGRAM_BOT_TOKEN",placeholder:"openshell:resolve:env:TELEGRAM_BOT_TOKEN"}],policyPresets:[{name:"telegram",requiredAtCreate:true,policyKeys:["telegram_bot"],agentPolicyKeys:{hermes:["telegram"]}}],render:[{id:"telegram-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.telegram",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false},proxy:"{{proxyUrl}}",groupPolicy:"{{telegramConfig.groupPolicy}}",dmPolicy:"{{allowedIds.telegram.dmPolicy}}",allowFrom:"{{allowedIds.telegram.values}}"}}}}},{id:"telegram-openclaw-groups",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",when:"{{telegramConfig.openclawGroups}}",fragment:{path:"channels.telegram.groups",value:"{{telegramConfig.openclawGroups}}"}},{id:"telegram-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.telegram",value:{enabled:true}}},{id:"telegram-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["TELEGRAM_ALLOWED_USERS={{allowedIds.telegram.csv}}"]},{id:"telegram-hermes-config",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"telegram",value:{require_mention:"{{telegramConfig.requireMention}}"}}},{id:"telegram-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.telegram",value:{enabled:true}}}],runtime:{openclaw:{channelName:"telegram",visibility:{configKeys:["telegram"],logPatterns:["telegram"]},nodePreloads:[{module:"telegram-diagnostics",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing Telegram diagnostics (provider readiness + inference errors)",installedMessage:"[channels] Telegram diagnostics installed (NODE_OPTIONS updated)"}]}},hooks:[{id:"telegram-token-paste",phase:"enroll",handler:"common.tokenPaste",outputs:[{id:"botToken",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"telegram-allowlist-aliases",phase:"enroll",handler:"telegram.allowlistAliases",outputs:[{id:"allowedIds",kind:"config"}]},{id:"telegram-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"requireMention",kind:"config"},{id:"allowedIds",kind:"config"}]},{id:"telegram-openclaw-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["openclaw"],outputs:[{id:"groupPolicy",kind:"config"}]},{id:"telegram-get-me-reachability",phase:"reachability-check",handler:"telegram.getMeReachability",inputs:["botToken"],onFailure:"skip-channel"},{id:"telegram-openclaw-bridge-health",phase:"health-check",handler:"telegram.openclawBridgeHealth",agents:["openclaw"],onFailure:"abort"},{id:"telegram-gateway-conflict-status",phase:"status",handler:"telegram.gatewayConflictStatus",outputs:[{id:"bridgeHealth",kind:"status"}]},{id:"telegram-status-health",phase:"status",handler:"telegram.statusHealth",agents:["openclaw"],outputs:[{id:"channelHealth",kind:"status"}]}]};var WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT={channelId:"wechat",planHookId:"wechat-seed-openclaw-account",handlerId:"wechat.seedOpenClawAccount",outputId:"openclawWeixinAccountFile",kind:"build-file",required:true,mode:"0600"};var WECHAT_SEED_OPENCLAW_ACCOUNT_HOOK_ID=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.handlerId;var WECHAT_SEED_OPENCLAW_ACCOUNT_PLAN_HOOK_ID=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.planHookId;var WECHAT_OPENCLAW_ACCOUNT_FILE_OUTPUT_ID=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.outputId;var WECHAT_TOKEN_PLACEHOLDER="openshell:resolve:env:WECHAT_BOT_TOKEN";function authorizeWechatAccountFilePlaceholders(value){const content=isPlainDataObject2(value)?ownDataPropertyValue2(value,"content"):void 0;if(!isPlainDataObject2(value)||!hasExactlyOwnDataProperties2(value,["content","mode","path"])||!isWechatAccountFilePath(ownDataPropertyValue2(value,"path"))||ownDataPropertyValue2(value,"mode")!==WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.mode||!isPlainDataObject2(content)||!hasOnlyOwnDataProperties(content,["baseUrl","savedAt","token","userId"])||!hasOwnDataProperty(content,"savedAt")||!hasOwnDataProperty(content,"token")||ownDataPropertyValue2(content,"token")!==WECHAT_TOKEN_PLACEHOLDER||!isNonEmptyString(ownDataPropertyValue2(content,"savedAt"))||!isOptionalNonEmptyString(content,"baseUrl")||!isOptionalNonEmptyString(content,"userId")){return[]}return[{path:["content","token"],value:WECHAT_TOKEN_PLACEHOLDER}]}function isWechatAccountFilePath(value){if(typeof value!=="string")return false;const prefix="openclaw-weixin/accounts/";const suffix=".json";if(!value.startsWith(prefix)||!value.endsWith(suffix))return false;const accountId=value.slice(prefix.length,-suffix.length);return accountId===accountId.trim()&&isSafeWechatAccountId(accountId)}function isSafeWechatAccountId(accountId){return accountId.length>0&&accountId!=="."&&accountId!==".."&&!/[\\/\0-\x1F\x7F]/.test(accountId)&&!accountId.includes("..")}function isPlainDataObject2(value){if(value===null||typeof value!=="object"||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function ownDataPropertyValue2(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}function hasOwnDataProperty(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor!==void 0&&"value"in descriptor}function hasExactlyOwnDataProperties2(value,expected){const actual=Object.getOwnPropertyNames(value).sort();return actual.length===expected.length&&actual.every((key,index)=>key===expected[index])}function hasOnlyOwnDataProperties(value,allowed){return Object.getOwnPropertyNames(value).every(key=>allowed.includes(key))}function isNonEmptyString(value){return typeof value==="string"&&value.length>0}function isOptionalNonEmptyString(value,key){return!hasOwnDataProperty(value,key)||isNonEmptyString(ownDataPropertyValue2(value,key))}var wechatManifest={schemaVersion:1,id:"wechat",displayName:"WeChat",description:"WeChat (personal) bot messaging",enrollmentHelp:"Captured automatically via a host-side QR scan during onboard \u2014 pair the bot by scanning the QR with WeChat on your phone (Discover \u2192 Scan). DM-only.",supportedAgents:["openclaw","hermes"],auth:{mode:"host-qr"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"WECHAT_BOT_TOKEN",prompt:{label:"WeChat Bot Token",help:"Captured automatically via a host-side QR scan during onboard \u2014 pair the bot by scanning the QR with WeChat on your phone (Discover \u2192 Scan). DM-only."}},{id:"accountId",kind:"config",required:true,envKey:"WECHAT_ACCOUNT_ID",statePath:"wechatConfig.accountId"},{id:"baseUrl",kind:"config",required:false,envKey:"WECHAT_BASE_URL",statePath:"wechatConfig.baseUrl"},{id:"userId",kind:"config",required:false,envKey:"WECHAT_USER_ID",statePath:"wechatConfig.userId"},{id:"allowedIds",kind:"config",required:false,envKey:"WECHAT_ALLOWED_IDS",statePath:"allowedIds.wechat",prompt:{label:"WeChat User ID(s) (DM allowlist)",help:"Optional: restrict who can DM the bot. The WeChat user id of the operator who scanned is added automatically; supply additional ids as a comma-separated list.",emptyValueMessage:"bot will require manual pairing"}}],credentials:[{id:"wechatBotToken",sourceInput:"botToken",providerName:"{sandboxName}-wechat-bridge",providerEnvKey:"WECHAT_BOT_TOKEN",placeholder:"openshell:resolve:env:WECHAT_BOT_TOKEN"}],state:{openclaw:["wechat","openclaw-weixin"],hermes:["platforms/wechat"]},policyPresets:[{name:"wechat",policyKeys:["wechat_bridge"],requiredAtCreate:true}],render:[{id:"wechat-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.openclaw-weixin",value:{enabled:true}}},{id:"wechat-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.openclaw-weixin",value:{enabled:true}}},{id:"wechat-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["WEIXIN_ACCOUNT_ID={{wechatConfig.accountId}}","WEIXIN_BASE_URL={{wechatConfig.baseUrl}}","WEIXIN_ALLOWED_USERS={{allowedIds.wechat.csv}}"]},{id:"wechat-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.weixin",value:{enabled:true}}}],runtime:{openclaw:{channelName:"openclaw-weixin",visibility:{configKeys:["openclaw-weixin"],logPatterns:["wechat","openclaw-weixin"]},nodePreloads:[{module:"wechat-account-placeholder",injectInto:["boot"],optional:false},{module:"wechat-diagnostics",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing WeChat diagnostics (provider readiness + inference errors)",installedMessage:"[channels] WeChat diagnostics installed (NODE_OPTIONS updated)"}]},hermes:{envAliases:[{envKey:"WECHAT_BOT_TOKEN",targetEnvKey:"WEIXIN_TOKEN",match:"^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_WECHAT_BOT_TOKEN$",value:"openshell:resolve:env:WECHAT_BOT_TOKEN"}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@tencent-weixin/openclaw-weixin@2.4.9",pin:true,integrity:"sha512-SfaYehR1Cwq2VV5HxJBp9sVilMms420VfZlMbF4YjRbWomr5+GxfXp9HkeU6y5TbnOc4Ysq0qPw1yBvJwbenBA==",tarballUrl:"https://registry.npmjs.org/@tencent-weixin/openclaw-weixin/-/openclaw-weixin-2.4.9.tgz",runtimeLock:{cachePath:"/usr/local/share/nemoclaw/wechat-npm-cache",installCacheEnvKey:"NEMOCLAW_WECHAT_NPM_INSTALL_CACHE",lockFile:"/usr/local/lib/nemoclaw/wechat-runtime/package-lock.json",projectsRoot:"/sandbox/.openclaw/npm/projects",verifierPath:"/usr/local/lib/nemoclaw/verify-wechat-runtime-lock.mts",offline:true,legacyPeerDeps:true},required:true}],hooks:[{id:"wechat-host-qr",phase:"enroll",handler:"wechat.ilinkLogin",inputs:["allowedIds"],outputs:[{id:"botToken",kind:"secret",required:true},{id:"accountId",kind:"config",required:true},{id:"baseUrl",kind:"config"},{id:"userId",kind:"config"},{id:"allowedIds",kind:"config"}],onFailure:"skip-channel"},{id:"wechat-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"allowedIds",kind:"config"}]},{id:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.planHookId,phase:"post-agent-install",handler:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.handlerId,agents:["openclaw"],inputs:["wechatConfig.accountId","wechatConfig.baseUrl","wechatConfig.userId","credential.wechatBotToken.placeholder"],outputs:[{id:"openclawWeixinAccountsIndex",kind:"build-file",required:true},{id:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.outputId,kind:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.kind,required:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.required},{id:"openclawConfigPatch",kind:"build-file",required:true}],onFailure:"abort"},{id:"wechat-health-check",phase:"health-check",handler:"wechat.healthCheck",inputs:["wechatConfig.accountId"],onFailure:"abort"}]};var whatsappManifest={schemaVersion:1,id:"whatsapp",displayName:"WhatsApp",description:"WhatsApp Web messaging (QR pairing)",enrollmentHelp:"WhatsApp Web pairs via QR code scanned with your phone \u2014 no host-side token. After the sandbox is running, run `openshell term` and then use `openclaw channels login --channel whatsapp` for OpenClaw or `hermes whatsapp` for Hermes to display the QR.",enrollmentNotes:["After pairing, run `nemoclaw channels status --channel whatsapp`. OpenClaw reports inbound delivery evidence; Hermes reports gateway and dashboard session-path diagnostics."],supportedAgents:["openclaw","hermes"],auth:{mode:"in-sandbox-qr"},inputs:[{id:"mode",kind:"config",required:false,envKey:"WHATSAPP_MODE",statePath:"whatsappConfig.mode",validValues:["self-chat","bot"],defaultValue:"self-chat",prompt:{label:"WhatsApp reply mode",help:"self-chat replies only to messages the paired account sends to itself. bot replies to other senders and stops replying to that self-chat: an unknown sender receives a pairing code you approve with `hermes pairing approve whatsapp `, unless you set WHATSAPP_ALLOWED_IDS to a fixed sender list before this command.",emptyValueMessage:"the sandbox replies only in your own self-chat"}},{id:"allowedIds",kind:"config",required:false,envKey:"WHATSAPP_ALLOWED_IDS",statePath:"allowedIds.whatsapp"}],credentials:[],state:{openclaw:["whatsapp"],hermes:["platforms/whatsapp","profiles/dashboard-home/platforms/whatsapp/session","dashboard-home/platforms/whatsapp/session"]},policyPresets:["whatsapp"],render:[{id:"whatsapp-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.whatsapp",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false}}}}}},{id:"whatsapp-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.whatsapp",value:{enabled:true}}},{id:"whatsapp-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["WHATSAPP_ENABLED=true","WHATSAPP_MODE={{whatsappConfig.mode}}","WHATSAPP_DM_POLICY={{whatsappConfig.dmPolicy}}","WHATSAPP_ALLOWED_USERS={{allowedIds.whatsapp.csv}}"]},{id:"whatsapp-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.whatsapp",value:{enabled:true}}}],runtime:{openclaw:{channelName:"whatsapp",visibility:{configKeys:["whatsapp"],logPatterns:["whatsapp"]},nodePreloads:[{module:"whatsapp-qr-compact",injectInto:["connect"],optional:true,installMessage:"[channels] Installing WhatsApp compact-QR renderer (scan-friendly pairing)"}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/whatsapp@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-vOWQIk7FpLHrhMmO+FaLi+pnFB82hiWNJJFJONkBuofERh2SMEz7EMut/vECFFEjFnmOZSVlYfRlxhbNkd/R6g==","2026.9.1":"sha512-llIcoMa6FM4SgYn7GG1FQIeTTA5JDdcHW5D7PT+3aGYT3/E2eLFutKwDvD/w7G0hvDwSftzZgLi3iA8dzK7a3A=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/whatsapp/-/whatsapp-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/whatsapp/-/whatsapp-2026.9.1.tgz"},required:true}],hooks:[{id:"whatsapp-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["hermes"],outputs:[{id:"mode",kind:"config"}]},{id:"whatsapp-status-health",phase:"status",handler:"whatsapp.statusHealth",agents:["openclaw","hermes"],outputs:[{id:"channelHealth",kind:"status"}]}]};var BUILT_IN_CHANNEL_MANIFESTS=[telegramManifest,discordManifest,wechatManifest,slackManifest,whatsappManifest,teamsManifest,googlechatManifest];function createBuiltInChannelManifestRegistry(){return createChannelManifestRegistry(BUILT_IN_CHANNEL_MANIFESTS)}var EXACT_TEMPLATE_PATTERN=/^\{\{\s*([^}]+?)\s*\}\}$/;var TEMPLATE_REFERENCE_PATTERN=/\{\{\s*([^}]+?)\s*\}\}/g;function resolvedRenderTemplateReference(value){return{matched:true,value}}function resolveSandboxNameTemplate(value,sandboxName){return value.replaceAll("{sandboxName}",sandboxName)}function resolveRenderTemplatesInValue(value,context){if(typeof value==="string")return resolveRenderTemplatesInString(value,context);if(Array.isArray(value)){if(value.length===0)return value;const resolved=value.map(entry=>resolveRenderTemplatesInValue(entry,context)).filter(entry=>entry!==void 0);return resolved.length>0?resolved:void 0}if(value&&typeof value==="object"){const sourceEntries=Object.entries(value);if(sourceEntries.length===0)return value;const entries=sourceEntries.map(([key,entry])=>[key,resolveRenderTemplatesInValue(entry,context)]).filter(entry=>entry[1]!==void 0);return entries.length>0?Object.fromEntries(entries):void 0}return value}function isTruthyRenderTemplate(value,context){if(!value)return true;const resolved=resolveRenderTemplatesInString(value,context);if(resolved===void 0||resolved===null||resolved===false)return false;if(Array.isArray(resolved))return resolved.length>0;if(typeof resolved==="object")return Object.keys(resolved).length>0;if(typeof resolved==="string")return resolved.trim().length>0;return true}function resolveRenderTemplatesInString(value,context){const exact=value.match(EXACT_TEMPLATE_PATTERN);if(exact?.[1])return resolveTemplateReference(exact[1].trim(),context);let omitted=false;const resolved=value.replace(TEMPLATE_REFERENCE_PATTERN,(match,reference)=>{const replacement=resolveTemplateReference(reference.trim(),context);if(replacement===void 0||replacement===null){omitted=true;return""}if(Array.isArray(replacement))return replacement.map(String).join(",");if(typeof replacement==="object")return JSON.stringify(replacement);return String(replacement)});return omitted?void 0:resolved}function resolveTemplateReference(reference,context){const resolved=context.referenceResolver?.(reference,context);return resolved?.matched?resolved.value:"{{"+reference+"}}"}function allowedIds(context,channel){return parseList(stateValue(context,`allowedIds.${channel}`))}function stateValue(context,path4){const stateInput=context.inputs.find(input=>input.statePath===path4);if(stateInput?.value!==void 0)return stateInput.value;const inputId=path4.split(".").at(-1);return context.inputs.find(input=>input.inputId===inputId)?.value}function parseList(value){if(Array.isArray(value))return unique(value.map(String).map(cleanString).filter(Boolean));const text=cleanString(value);if(!text)return[];return unique(text.split(",").map(cleanString).filter(Boolean))}function parseBoolean(value){if(typeof value==="boolean")return value;const text=cleanString(value)?.toLowerCase();if(text==="1"||text==="true"||text==="yes"||text==="on")return true;if(text==="0"||text==="false"||text==="no"||text==="off")return false;return void 0}function nonEmptyString(value){return cleanString(value)||void 0}function cleanString(value){const text=String(value??"");if(/[\r\n]/.test(text)){throw new Error("Messaging template values must not contain line breaks.")}return text.trim()}function nonEmptyArray(values){return values.length>0?[...values]:void 0}function nonEmptyCsv(values){return values.length>0?values.join(","):void 0}function nonEmptyObject(value){return Object.keys(value).length>0?value:void 0}function unique(values){return[...new Set(values)]}var resolveDiscordTemplateReference=(reference,context)=>{if(reference==="discordProxyUrl")return resolvedRenderTemplateReference(void 0);switch(reference){case"discord.guilds":return resolvedRenderTemplateReference(nonEmptyObject(discordGuilds(context)));case"discord.hasGuilds":return resolvedRenderTemplateReference(Object.keys(discordGuilds(context)).length>0);case"discord.guildIds.csv":return resolvedRenderTemplateReference(nonEmptyCsv(Object.keys(discordGuilds(context))));case"discord.allowedUsers.values":return resolvedRenderTemplateReference(nonEmptyArray(discordAllowedUsers(context)));case"discord.allowedUsers.csv":return resolvedRenderTemplateReference(nonEmptyCsv(discordAllowedUsers(context)));case"discord.allowedUsers.dmPolicy":return resolvedRenderTemplateReference(discordAllowedUsers(context).length>0?"allowlist":void 0);case"discord.allowAllUsers":return resolvedRenderTemplateReference(Object.keys(discordGuilds(context)).length>0&&discordAllowedUsers(context).length===0?true:void 0);case"discord.requireMention":return resolvedRenderTemplateReference(discordRequireMention(context));default:return void 0}};function discordGuilds(context){const serverIds=parseList(stateValue(context,"discordGuilds.serverId"));if(serverIds.length===0)return{};const users=parseList(stateValue(context,"discordGuilds.userIds"));const requireMention=parseBoolean(stateValue(context,"discordGuilds.requireMention"))??true;return Object.fromEntries(serverIds.map(serverId=>[serverId,{requireMention,...users.length>0?{users}:{}}]))}function discordAllowedUsers(context){const users=new Set(allowedIds(context,"discord"));for(const guild of Object.values(discordGuilds(context))){for(const user of guild.users??[])users.add(String(user))}return[...users]}function discordRequireMention(context){for(const guild of Object.values(discordGuilds(context))){if(typeof guild.requireMention==="boolean")return guild.requireMention}return true}var DEFAULT_AUDIENCE_TYPE="app-url";var APP_PRINCIPAL_DISCOVERY_SENTINEL="000000000000000000000";var resolveGooglechatTemplateReference=(reference,context)=>{switch(reference){case"googlechatConfig.audienceType":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.audienceType"))??DEFAULT_AUDIENCE_TYPE);case"googlechatConfig.audience":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.audience")));case"googlechatConfig.appPrincipal":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.appPrincipal"))??APP_PRINCIPAL_DISCOVERY_SENTINEL);case"googlechatConfig.projectId":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.projectId")));case"googlechatConfig.subscriptionName":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.subscriptionName")));default:break}const allowReference=reference.match(/^allowedIds[.]googlechat[.](values|dmPolicy|csv)$/);if(!allowReference?.[1])return void 0;const ids=allowedIds(context,"googlechat");switch(allowReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);case"csv":return resolvedRenderTemplateReference(ids.length>0?ids.join(","):void 0);default:return void 0}};var resolveSlackTemplateReference=(reference,context)=>{if(reference==="slackConfig.allowedChannels.csv"){return resolvedRenderTemplateReference(nonEmptyCsv(slackAllowedChannels(context)))}const allowedIdsReference=reference.match(/^allowedIds[.]slack[.](values|csv|dmPolicy|groupPolicy|channels)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"slack");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);case"groupPolicy":return resolvedRenderTemplateReference(ids.length>0||slackAllowedChannels(context).length>0?"allowlist":void 0);case"channels":return resolvedRenderTemplateReference(slackChannelConfig(context,ids));default:return void 0}};function slackChannelConfig(context,users){const allowedChannels=slackAllowedChannels(context);const entry={enabled:true,requireMention:true,...users.length>0?{users:[...users]}:{}};if(allowedChannels.length>0){return Object.fromEntries(allowedChannels.map(channelId=>[channelId,{...entry}]))}return users.length>0?{"*":entry}:void 0}function slackAllowedChannels(context){return parseList(stateValue(context,"slackConfig.allowedChannels"))}var DEFAULT_TEAMS_WEBHOOK_PORT=3978;var resolveTeamsTemplateReference=(reference,context)=>{switch(reference){case"teamsConfig.appId":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"teamsConfig.appId")));case"teamsConfig.tenantId":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"teamsConfig.tenantId")));case"teamsConfig.webhookPort":return resolvedRenderTemplateReference(teamsWebhookPort(context));case"teamsConfig.requireMention":return resolvedRenderTemplateReference(parseBoolean(stateValue(context,"teamsConfig.requireMention")));default:break}const allowedIdsReference=reference.match(/^allowedIds[.]teams[.](values|csv|dmPolicy)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"teams");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);default:return void 0}};function teamsWebhookPort(context){const raw=nonEmptyString(stateValue(context,"teamsConfig.webhookPort"));if(!raw)return DEFAULT_TEAMS_WEBHOOK_PORT;const port=Number(raw);if(!Number.isInteger(port)||port<1||port>65535){throw new Error("Microsoft Teams webhook port must be an integer TCP port between 1 and 65535.")}return port}var DEFAULT_PROXY_HOST="10.200.0.1";var DEFAULT_PROXY_PORT="3128";var DEFAULT_TELEGRAM_GROUP_POLICY="open";var TELEGRAM_GROUP_POLICIES=new Set(["open","allowlist","disabled"]);var resolveTelegramTemplateReference=(reference,context)=>{if(reference==="proxyUrl")return resolvedRenderTemplateReference(proxyUrl(context.env));if(reference==="telegramConfig.groupPolicy"){return resolvedRenderTemplateReference(telegramGroupPolicy(context))}if(reference==="telegramConfig.openclawGroups"){return resolvedRenderTemplateReference(telegramOpenClawGroups(context))}if(reference==="telegramConfig.requireMention"){return resolvedRenderTemplateReference(parseBoolean(stateValue(context,"telegramConfig.requireMention")))}const allowedIdsReference=reference.match(/^allowedIds[.]telegram[.](values|csv|dmPolicy)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"telegram");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);default:return void 0}};function proxyUrl(env){const host=nonEmptyString(env?.NEMOCLAW_PROXY_HOST)??DEFAULT_PROXY_HOST;const port=nonEmptyString(env?.NEMOCLAW_PROXY_PORT)??DEFAULT_PROXY_PORT;return`http://${host}:${port}`}function telegramGroupPolicy(context){const value=nonEmptyString(stateValue(context,"telegramConfig.groupPolicy"));return value&&TELEGRAM_GROUP_POLICIES.has(value)?value:DEFAULT_TELEGRAM_GROUP_POLICY}function telegramOpenClawGroups(context){if(telegramGroupPolicy(context)!=="open")return void 0;const requireMention=parseBoolean(stateValue(context,"telegramConfig.requireMention"));return requireMention===true?{"*":{requireMention:true}}:void 0}var WECHAT_ILINK_HOSTS=new Set(["ilinkai.weixin.qq.com","ilinkai.wechat.com"]);var WECHAT_ILINK_IDC_HOST_PATTERN=/^idc-[0-9]+[.]weixin[.]qq[.]com$/u;function normalizeWechatIlinkBaseUrl(value){const raw=String(value??"");if(/[\r\n]/.test(raw)){throw new Error("WeChat baseUrl must not contain line breaks.")}const text=raw.trim();if(!text)return void 0;let url;try{url=new URL(text)}catch{throw new Error("WeChat baseUrl must be a valid URL.")}if(url.protocol!=="https:"){throw new Error("WeChat baseUrl must use HTTPS.")}if(url.username||url.password){throw new Error("WeChat baseUrl must not include credentials.")}const authority=text.match(/^[a-z][a-z0-9+.-]*:\/\/([^/?#]*)/iu)?.[1]??"";if(authority.includes(":")){throw new Error("WeChat baseUrl must not include an explicit port.")}if(!isWechatIlinkHost(url.hostname)){throw new Error("WeChat baseUrl must use an expected iLink host.")}if(url.pathname&&url.pathname!=="/"||url.search||url.hash){throw new Error("WeChat baseUrl must be an iLink origin URL.")}return url.origin}function isWechatIlinkHost(hostname){const normalized=hostname.toLowerCase();return WECHAT_ILINK_HOSTS.has(normalized)||isWechatIlinkIdcHost(normalized)}function isWechatIlinkIdcHost(hostname){return WECHAT_ILINK_IDC_HOST_PATTERN.test(hostname.toLowerCase())}var resolveWechatTemplateReference=(reference,context)=>{const wechatConfig=reference.match(/^wechatConfig[.](accountId|baseUrl|userId)$/);if(wechatConfig?.[1]){if(wechatConfig[1]==="baseUrl"){return resolvedRenderTemplateReference(normalizeWechatIlinkBaseUrl(stateValue(context,"wechatConfig.baseUrl")))}return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"wechatConfig."+wechatConfig[1])))}const allowedIdsReference=reference.match(/^allowedIds[.]wechat[.](values|csv|dmPolicy)$/);if(!allowedIdsReference?.[1])return void 0;const ids=wechatAllowedIds(context);switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);default:return void 0}};function wechatAllowedIds(context){const ids=allowedIds(context,"wechat");const userId=nonEmptyString(stateValue(context,"wechatConfig.userId"));return userId&&!ids.includes(userId)?[userId,...ids]:ids}var DEFAULT_WHATSAPP_MODE="self-chat";var BOT_WHATSAPP_MODE="bot";var WHATSAPP_MODES=new Set([DEFAULT_WHATSAPP_MODE,BOT_WHATSAPP_MODE]);var resolveWhatsappTemplateReference=(reference,context)=>{if(reference==="whatsappConfig.mode"){return resolvedRenderTemplateReference(whatsappMode(context))}if(reference==="whatsappConfig.dmPolicy"){return resolvedRenderTemplateReference(whatsappDmPolicy(context))}const allowedIdsReference=reference.match(/^allowedIds[.]whatsapp[.](values|csv)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"whatsapp");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));default:return void 0}};function whatsappMode(context){const value=nonEmptyString(stateValue(context,"whatsappConfig.mode"));return value&&WHATSAPP_MODES.has(value)?value:DEFAULT_WHATSAPP_MODE}function whatsappDmPolicy(context){if(whatsappMode(context)!==BOT_WHATSAPP_MODE)return void 0;return allowedIds(context,"whatsapp").length>0?"allowlist":"pairing"}var BUILT_IN_TEMPLATE_REFERENCE_RESOLVERS=[resolveTelegramTemplateReference,resolveDiscordTemplateReference,resolveWechatTemplateReference,resolveSlackTemplateReference,resolveWhatsappTemplateReference,resolveTeamsTemplateReference,resolveGooglechatTemplateReference];function createBuiltInRenderTemplateResolver(){return(reference,context)=>{for(const resolver of BUILT_IN_TEMPLATE_REFERENCE_RESOLVERS){const resolved=resolver(reference,context);if(resolved)return resolved}return void 0}}var import_node_crypto2=__toESM(require("node:crypto"));function hashCredential(value){const normalized=String(value??"").trim();if(!normalized)return null;return import_node_crypto2.default.createHash("sha256").update(normalized).digest("hex")}function planCredentialBindings(manifest,context,inputs,environment=process.env){return manifest.credentials.map(credential=>{const sourceInput=inputs.find(input=>input.inputId===credential.sourceInput);const credentialAvailable=sourceInput?.credentialAvailable===true||context.credentialAvailability?.[credential.id]===true||context.credentialAvailability?.[`${manifest.id}.${credential.id}`]===true;const envKey=sourceInput?.sourceEnv??credential.providerEnvKey;const credentialHash=credentialAvailable?hashCredential(environment[envKey])??void 0:void 0;return{channelId:manifest.id,credentialId:credential.id,sourceInput:credential.sourceInput,providerName:resolveSandboxNameTemplate(credential.providerName,context.sandboxName),providerEnvKey:credential.providerEnvKey,placeholder:credential.placeholder,credentialAvailable,...credentialHash!==void 0?{credentialHash}:{}}})}function planHostForward(manifest,inputs,active,referenceResolver,environment=process.env){if(!active||!manifest.hostForward)return void 0;const context={inputs,env:environment,referenceResolver};if(!isTruthyRenderTemplate(manifest.hostForward.when,context))return void 0;const portValue=resolveRenderTemplatesInValue(manifest.hostForward.port,context);const port=normalizeForwardPort(manifest.id,portValue);return{channelId:manifest.id,port,label:manifest.hostForward.label}}function normalizeForwardPort(channelId,value){const port=typeof value==="number"?value:Number(String(value??"").trim());if(!Number.isInteger(port)||port<1||port>65535){throw new Error(`Channel manifest '${channelId}' declares invalid host forward port '${String(value)}'.`)}return port}var OPENSHELL_ENV_PLACEHOLDER_PREFIX="openshell:resolve:env:";var OPENSHELL_ALIAS_PLACEHOLDER_RE=/^[A-Za-z0-9]+-OPENSHELL-RESOLVE-ENV-(.+)$/;function normalizeProviderPlaceholderForEnvKey(value,envKey){if(value.startsWith(OPENSHELL_ENV_PLACEHOLDER_PREFIX)){return placeholderSuffixMatchesEnvKey(value.slice(OPENSHELL_ENV_PLACEHOLDER_PREFIX.length),envKey)?`${OPENSHELL_ENV_PLACEHOLDER_PREFIX}${envKey}`:null}const aliasMatch=value.match(OPENSHELL_ALIAS_PLACEHOLDER_RE);if(!aliasMatch||!placeholderSuffixMatchesEnvKey(aliasMatch[1],envKey)){return null}return value.replace(/-OPENSHELL-RESOLVE-ENV-.+$/,`-OPENSHELL-RESOLVE-ENV-${envKey}`)}function placeholderSuffixMatchesEnvKey(suffix,envKey){if(suffix===envKey)return true;const generationMatch=suffix.match(/^(?:v[0-9]{1,20}|s[a-f0-9]{64})_(.+)$/);return generationMatch?.[1]===envKey}function hasFullPersistedCredentialBindingShape(binding){return typeof binding.channelId==="string"&&typeof binding.credentialId==="string"&&typeof binding.sourceInput==="string"&&typeof binding.providerName==="string"&&typeof binding.providerEnvKey==="string"&&typeof binding.placeholder==="string"&&typeof binding.credentialAvailable==="boolean"}function normalizeFullPersistedCredentialBindings(bindings){return bindings.map(binding=>({channelId:binding.channelId,credentialId:binding.credentialId,sourceInput:binding.sourceInput,providerName:binding.providerName,providerEnvKey:binding.providerEnvKey,placeholder:normalizeProviderPlaceholderForEnvKey(binding.placeholder,binding.providerEnvKey)??binding.placeholder,credentialAvailable:binding.credentialAvailable===true,...typeof binding.credentialHash==="string"?{credentialHash:binding.credentialHash}:{}}))}function normalizePersistedAgentCredentialPlaceholders(render,credentialBindings){const credentialEnvKeys=new Set(credentialBindings.map(binding=>binding.providerEnvKey).filter(Boolean));if(credentialEnvKeys.size===0)return[...render];return render.map(entry=>{if(entry.kind!=="env-lines")return entry;return{...entry,lines:entry.lines.map(line=>normalizeCredentialEnvLine(line,credentialEnvKeys))}})}function normalizeCredentialEnvLine(line,credentialEnvKeys){const index=line.indexOf("=");if(index<=0)return line;const envKey=line.slice(0,index).trim();if(!credentialEnvKeys.has(envKey))return line;const value=line.slice(index+1);const normalized=normalizeProviderPlaceholderForEnvKey(value,envKey);return normalized?`${envKey}=${normalized}`:line}function normalizePersistedSandboxMessagingPlanShape(plan,environment=process.env){const manifestRegistry=createBuiltInChannelManifestRegistry();const disabledChannels=plan.disabledChannels.filter(channelId=>typeof channelId==="string");const disabledSet=new Set(disabledChannels);const channels=plan.channels.map(channel=>normalizePersistedChannel(channel,disabledSet,manifestRegistry.get(channel.channelId),environment));const credentialBindings=normalizePersistedCredentialBindings(plan,channels,manifestRegistry,environment);const normalizedPlan={...plan,channels,disabledChannels,credentialBindings,networkPolicy:plan.networkPolicy&&Array.isArray(plan.networkPolicy.entries)?plan.networkPolicy:{presets:[],entries:[]},agentRender:normalizePersistedAgentCredentialPlaceholders(Array.isArray(plan.agentRender)?[...plan.agentRender]:[],credentialBindings),buildSteps:Array.isArray(plan.buildSteps)?[...plan.buildSteps]:[],...plan.runtimeSetup!==void 0?{runtimeSetup:normalizeRuntimeSetup(plan.runtimeSetup)}:{},stateUpdates:Array.isArray(plan.stateUpdates)?[...plan.stateUpdates]:[],healthChecks:Array.isArray(plan.healthChecks)?[...plan.healthChecks]:[]};return normalizedPlan}function normalizePersistedChannel(channel,disabledSet,manifest,environment){const disabled=channel.disabled??disabledSet.has(channel.channelId);const configured=channel.configured??true;const hasFullShape=hasFullChannelShape(channel);const inputs=hasFullShape?normalizeFullInputs(channel.channelId,channel.inputs??[]):normalizePersistedInputs(channel,manifest);const active=channel.active??(configured&&!disabled&&requiredInputsAvailable(manifest,inputs));const hostForward=manifest?planHostForward(manifest,inputs,active&&!disabled,createBuiltInRenderTemplateResolver(),environment):void 0;return{channelId:channel.channelId,displayName:channel.displayName??manifest?.displayName??channel.channelId,authMode:channel.authMode??manifest?.auth.mode??"none",active,selected:channel.selected??configured,configured,disabled,...channel.pendingRemoval===true?{pendingRemoval:true}:{},inputs,...hostForward?{hostForward}:{},hooks:Array.isArray(channel.hooks)?[...channel.hooks]:[]}}function normalizePersistedInputs(channel,manifest){const persistedById=new Map((channel.inputs??[]).filter(input=>typeof input.inputId==="string").map(input=>[input.inputId,input]));const fromManifest=(manifest?.inputs??[]).map(input=>inputReferenceFromManifest(channel.channelId,input,persistedById.get(input.id)));const manifestInputIds=new Set((manifest?.inputs??[]).map(input=>input.id));const unknownInputs=[...persistedById.values()].flatMap(input=>{if(!input.inputId||manifestInputIds.has(input.inputId))return[];return[normalizeUnknownInput(channel.channelId,input)]});return[...fromManifest,...unknownInputs]}function normalizeFullInputs(channelId,inputs){return inputs.filter(input=>typeof input.inputId==="string").map(input=>({channelId:typeof input.channelId==="string"?input.channelId:channelId,inputId:input.inputId,kind:input.kind==="secret"||input.kind==="config"?input.kind:"config",required:typeof input.required==="boolean"?input.required:false,...typeof input.sourceEnv==="string"?{sourceEnv:input.sourceEnv}:{},...typeof input.statePath==="string"?{statePath:input.statePath}:{},...input.credentialAvailable!==void 0?{credentialAvailable:input.credentialAvailable}:{},...input.value!==void 0?{value:input.value}:{}}))}function inputReferenceFromManifest(channelId,input,persisted){return{channelId,inputId:input.id,kind:input.kind,required:input.required,...input.envKey?{sourceEnv:input.envKey}:{},...input.kind==="config"&&input.statePath?{statePath:input.statePath}:{},...persisted?.credentialAvailable!==void 0?{credentialAvailable:persisted.credentialAvailable}:{},...persisted?.value!==void 0?{value:persisted.value}:{}}}function normalizeUnknownInput(channelId,input){const kind=input.kind==="secret"||input.kind==="config"?input.kind:"config";return{channelId,inputId:input.inputId,kind,required:input.required===true,...typeof input.sourceEnv==="string"?{sourceEnv:input.sourceEnv}:{},...typeof input.statePath==="string"?{statePath:input.statePath}:{},...input.credentialAvailable!==void 0?{credentialAvailable:input.credentialAvailable}:{},...input.value!==void 0?{value:input.value}:{}}}function requiredInputsAvailable(manifest,inputs){if(!manifest)return true;return manifest.inputs.every(manifestInput=>{if(!manifestInput.required)return true;const input=inputs.find(entry=>entry.inputId===manifestInput.id);if(!input)return false;if(input.kind==="secret")return input.credentialAvailable===true;if(input.value===void 0)return false;return typeof input.value==="string"?input.value.trim().length>0:true})}function normalizePersistedCredentialBindings(plan,channels,manifestRegistry,environment){const persisted=plan.credentialBindings??[];if(Array.isArray(plan.credentialBindings)&&plan.channels.every(hasFullChannelShape)&&persisted.every(hasFullPersistedCredentialBindingShape)){return normalizeFullPersistedCredentialBindings(persisted)}const manifests=channels.flatMap(channel=>{const manifest=manifestRegistry.get(channel.channelId);return manifest?[manifest]:[]});const planForBindings={...plan,channels,credentialBindings:[],networkPolicy:{presets:[],entries:[]},agentRender:[],buildSteps:[],runtimeSetup:{nodePreloads:[],envAliases:[],secretScans:[]},stateUpdates:[],healthChecks:[]};const generated=credentialBindingsFromManifests(planForBindings,manifests,new Map(channels.map(channel=>[channel.channelId,channel.inputs])),environment);return generated.map(binding=>overlayPersistedCredentialBinding(binding,persisted))}function credentialBindingsFromManifests(plan,manifests,inputRegistry,environment){const context=compilerContext(plan);return manifests.flatMap(manifest=>planCredentialBindings(manifest,context,inputRegistry.get(manifest.id)??[],environment).map(binding=>overlayPersistedCredentialBinding(binding,plan.credentialBindings)))}function overlayPersistedCredentialBinding(binding,persisted){const match=persisted.find(candidate=>credentialBindingMatches(binding,candidate));if(!match)return binding;return{...binding,credentialAvailable:typeof match.credentialAvailable==="boolean"?match.credentialAvailable:binding.credentialAvailable,...typeof match.credentialHash==="string"&&match.credentialHash.length>0?{credentialHash:match.credentialHash}:binding.credentialHash?{credentialHash:binding.credentialHash}:{}}}function credentialBindingMatches(binding,candidate){if(candidate.channelId&&candidate.channelId!==binding.channelId)return false;if(candidate.providerEnvKey&&candidate.providerEnvKey===binding.providerEnvKey)return true;if(candidate.credentialId&&candidate.credentialId===binding.credentialId)return true;if(candidate.sourceInput&&candidate.sourceInput===binding.sourceInput)return true;return false}function hasFullChannelShape(channel){return typeof channel.displayName==="string"&&typeof channel.authMode==="string"&&typeof channel.active==="boolean"&&typeof channel.selected==="boolean"&&typeof channel.configured==="boolean"&&typeof channel.disabled==="boolean"&&Array.isArray(channel.inputs)}function normalizeRuntimeSetup(setup){return{nodePreloads:Array.isArray(setup?.nodePreloads)?[...setup.nodePreloads]:[],envAliases:Array.isArray(setup?.envAliases)?[...setup.envAliases]:[],secretScans:Array.isArray(setup?.secretScans)?[...setup.secretScans]:[]}}function compilerContext(plan){return{sandboxName:plan.sandboxName,agent:plan.agent,workflow:plan.workflow,isInteractive:false,configuredChannels:plan.channels.map(channel=>channel.channelId),disabledChannels:plan.disabledChannels,credentialAvailability:credentialAvailabilityFromPlan(plan)}}function credentialAvailabilityFromPlan(plan){const availability={};for(const channel of plan.channels){for(const input of channel.inputs){if(input.kind!=="secret"||input.credentialAvailable!==true)continue;availability[`${channel.channelId}.${input.inputId}`]=true;if(input.sourceEnv)availability[input.sourceEnv]=true}}for(const credential of plan.credentialBindings){if(!credential.credentialAvailable)continue;availability[credential.credentialId]=true;availability[`${credential.channelId}.${credential.credentialId}`]=true;availability[`${credential.channelId}.${credential.sourceInput}`]=true;availability[credential.providerEnvKey]=true}return availability}function normalizeMessagingChannelId(channelId){return channelId.trim().toLowerCase()}function enabledPlanChannels(plan){const disabled=new Set((plan.disabledChannels??[]).map(normalizeMessagingChannelId).filter(Boolean));return plan.channels.filter(channel=>{const channelId=normalizeMessagingChannelId(channel.channelId);return channelId.length>0&&channel.active&&!channel.disabled&&!disabled.has(channelId)})}function selectActiveMessagingChannelIds(plan){const seen=new Set;const channels=[];for(const item of enabledPlanChannels(plan)){const channel=normalizeMessagingChannelId(item.channelId);if(!channel||seen.has(channel))continue;seen.add(channel);channels.push(channel)}return channels}function selectEnabledMessagingAgentRender(plan){const active=new Set(selectActiveMessagingChannelIds(plan));return plan.agentRender.filter(render=>render.agent===plan.agent&&active.has(normalizeMessagingChannelId(render.channelId)))}function selectEnabledPostAgentInstallBuildFiles(plan){const active=new Set(selectActiveMessagingChannelIds(plan));const channels=enabledPlanChannels(plan);return plan.buildSteps.filter(step=>{const channelId=normalizeMessagingChannelId(step.channelId);if(!active.has(channelId)||step.kind!=="build-file")return false;if(!step.hookId)return true;const matchingChannels=channels.filter(channel=>normalizeMessagingChannelId(channel.channelId)===channelId);if(matchingChannels.length!==1)return false;const matchedHook=matchingChannels[0]?.hooks?.find(hook=>hook.id===step.hookId);return matchedHook!==void 0&&matchedHook.phase==="post-agent-install"})}function parseSandboxMessagingPlan(value,options={}){if(!isObjectRecord(value)||value.schemaVersion!==1||typeof value.sandboxName!=="string"||typeof value.agent!=="string"||typeof value.workflow!=="string"||!Array.isArray(value.channels)||!Array.isArray(value.disabledChannels)||!isOptionalObjectArray(value,"credentialBindings")||Object.hasOwn(value,"networkPolicy")&&!isObjectRecord(value.networkPolicy)||!isOptionalObjectArray(value,"agentRender")||!isOptionalObjectArray(value,"buildSteps")||!isRuntimeSetup(value.runtimeSetup)||!isOptionalObjectArray(value,"stateUpdates")||!isOptionalObjectArray(value,"healthChecks")){return null}if(options.sandboxName&&value.sandboxName!==options.sandboxName)return null;if(options.agent&&value.agent!==options.agent)return null;const supported=Array.isArray(options.supportedChannelIds)?new Set(options.supportedChannelIds):null;const normalizedChannelIds=new Set;for(const channel of value.channels){if(!isObjectRecord(channel)||typeof channel.channelId!=="string")return null;const normalizedChannelId=normalizeMessagingChannelId(channel.channelId);if(!normalizedChannelId||normalizedChannelId!==channel.channelId||normalizedChannelIds.has(normalizedChannelId)){return null}if(Object.hasOwn(channel,"configured")&&typeof channel.configured!=="boolean"){return null}if(Object.hasOwn(channel,"active")&&typeof channel.active!=="boolean")return null;if(Object.hasOwn(channel,"disabled")&&typeof channel.disabled!=="boolean")return null;if(Object.hasOwn(channel,"pendingRemoval")&&typeof channel.pendingRemoval!=="boolean"){return null}if(Object.hasOwn(channel,"inputs")&&!Array.isArray(channel.inputs))return null;if(Object.hasOwn(channel,"hostForward")&&!isHostForward(channel.hostForward))return null;if(Object.hasOwn(channel,"hooks")&&!Array.isArray(channel.hooks))return null;if(Array.isArray(channel.inputs)&&channel.inputs.some(input=>!isObjectRecord(input)||typeof input.inputId!=="string"||Object.hasOwn(input,"channelId")&&input.channelId!==normalizedChannelId)){return null}if(Array.isArray(channel.hooks)&&channel.hooks.some(hook=>!isObjectRecord(hook)||Object.hasOwn(hook,"channelId")&&hook.channelId!==normalizedChannelId)){return null}if(Object.hasOwn(channel,"hostForward")&&isObjectRecord(channel.hostForward)&&channel.hostForward.channelId!==normalizedChannelId){return null}if(supported&&!supported.has(channel.channelId))return null;normalizedChannelIds.add(normalizedChannelId)}if(!value.disabledChannels.every(isCanonicalMessagingChannelId))return null;const disabledChannelIds=new Set(value.disabledChannels);if(disabledChannelIds.size!==value.disabledChannels.length||[...disabledChannelIds].some(channelId=>!normalizedChannelIds.has(channelId))||value.channels.some(channel=>isObjectRecord(channel)&&channel.disabled===true!==disabledChannelIds.has(String(channel.channelId)))){return null}if(!hasCanonicalChannelReferences(value.credentialBindings)||!hasMatchingAgentRenderEntries(value.agentRender,value.agent)||!hasCanonicalChannelReferences(value.agentRender)||!hasCanonicalChannelReferences(value.buildSteps)||!hasCanonicalChannelReferences(value.stateUpdates)||!hasCanonicalChannelReferences(value.healthChecks)||!hasCanonicalNetworkPolicyReferences(value.networkPolicy)||!hasCanonicalRuntimeSetupReferences(value.runtimeSetup)){return null}return cloneSandboxMessagingPlan(normalizePersistedSandboxMessagingPlanShape(value,options.environment))}function hasMatchingAgentRenderEntries(value,agent){return!Array.isArray(value)||value.every(render=>isObjectRecord(render)&&render.agent===agent)}function hasCanonicalNetworkPolicyReferences(value){if(!isObjectRecord(value)||!Object.hasOwn(value,"entries"))return true;return hasCanonicalChannelReferences(value.entries)}function cloneSandboxMessagingPlan(plan){return JSON.parse(JSON.stringify(plan))}function isOptionalObjectArray(value,key){if(!Object.hasOwn(value,key))return true;const entries=value[key];return Array.isArray(entries)&&entries.every(isObjectRecord)}function isHostForward(value){return isObjectRecord(value)&&typeof value.channelId==="string"&&typeof value.port==="number"&&Number.isInteger(value.port)&&value.port>=1&&value.port<=65535&&typeof value.label==="string"}function isRuntimeSetup(value){if(value===void 0)return true;return isObjectRecord(value)&&Array.isArray(value.nodePreloads)&&Array.isArray(value.envAliases)&&Array.isArray(value.secretScans)&&value.nodePreloads.every(isObjectRecord)&&value.envAliases.every(isObjectRecord)&&value.secretScans.every(isObjectRecord)}function isCanonicalMessagingChannelId(value){return typeof value==="string"&&value.length>0&&normalizeMessagingChannelId(value)===value}function hasCanonicalChannelReferences(value){return value===void 0||Array.isArray(value)&&value.every(entry=>isObjectRecord(entry)&&isCanonicalMessagingChannelId(entry.channelId))}function hasCanonicalRuntimeSetupReferences(value){if(value===void 0)return true;if(!isObjectRecord(value))return false;return["nodePreloads","envAliases","secretScans"].every(field=>hasCanonicalChannelReferences(value[field]))}var import_node_buffer2=require("node:buffer");var import_node_crypto3=require("node:crypto");var import_node_util=require("node:util");function isLoopbackHostname(hostname=""){const normalized=String(hostname||"").trim().toLowerCase().replace(/^\[|\]$/g,"");return normalized==="localhost"||normalized==="::1"||/^127(?:\.\d{1,3}){3}$/.test(normalized)}function isLoopbackDashboardUrl(value){return isLoopbackHostname(new URL(value).hostname)}function listMessagingCredentialEnvAssignments(options={}){return selectManifests(options).flatMap(manifest=>{const credentialsByTemplate=new Map(manifest.credentials.map(credential=>[`{{credential.${credential.id}.placeholder}}`,credential]));const renderedAssignments=manifest.render.flatMap(render=>{if(options.agent&&render.agent!==options.agent)return[];if(render.kind!=="env-lines")return[];return render.lines.flatMap(line=>{const separator=line.indexOf("=");if(separator<=0)return[];const credential=credentialsByTemplate.get(line.slice(separator+1));if(!credential)return[];return[{channelId:manifest.id,agent:render.agent,sourceEnvKey:credential.providerEnvKey,targetEnvKey:line.slice(0,separator),placeholder:credential.placeholder}]})});const runtimeAssignments=["openclaw","hermes"].flatMap(agent=>{if(options.agent&&agent!==options.agent)return[];if(!manifest.supportedAgents.includes(agent))return[];return(manifest.runtime?.[agent]?.envAliases??[]).flatMap(alias=>{if(!alias.targetEnvKey)return[];const credential=manifest.credentials.find(candidate=>candidate.providerEnvKey===alias.envKey);if(!credential)return[];return[{channelId:manifest.id,agent,sourceEnvKey:alias.envKey,targetEnvKey:alias.targetEnvKey,placeholder:credential.placeholder}]})});return[...renderedAssignments,...runtimeAssignments]})}function selectManifests(options){const manifests=options.manifests??BUILT_IN_CHANNEL_MANIFESTS;const agent=options.agent;const selected=agent?manifests.filter(manifest=>manifest.supportedAgents.includes(agent)):manifests;return[...selected]}function authorizeMessagingManagedStartupFields(entry,section){if(section==="agentRender")return authorizeTeamsOpenClawWebhookField(entry);if(!isPlainDataObject3(entry))return[];const contract=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT;if(ownDataPropertyValue3(entry,"channelId")!==contract.channelId||ownDataPropertyValue3(entry,"hookId")!==contract.planHookId||ownDataPropertyValue3(entry,"handler")!==contract.handlerId||ownDataPropertyValue3(entry,"outputId")!==contract.outputId||ownDataPropertyValue3(entry,"kind")!==contract.kind||ownDataPropertyValue3(entry,"required")!==contract.required){return[]}return authorizeWechatAccountFilePlaceholders(ownDataPropertyValue3(entry,"value")).map(authorization=>({...authorization,path:["value",...authorization.path]}))}function isPlainDataObject3(value){if(value===null||typeof value!=="object"||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function ownDataPropertyValue3(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}var DCODE_UPSTREAM_PROVIDER_RE=/^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/u;function isValidDcodeUpstreamProvider(value){return DCODE_UPSTREAM_PROVIDER_RE.test(value)}var LEGACY_MANAGED_STARTUP_PROFILE_SCHEMA_VERSION=1;var MANAGED_STARTUP_PROFILE_SCHEMA_VERSION=2;var MANAGED_STARTUP_PROFILE_MAX_BYTES=64*1024;var MANAGED_STARTUP_PROFILE_MAX_ENCODED_BYTES=Math.ceil(MANAGED_STARTUP_PROFILE_MAX_BYTES/3)*4;var MAX_IDENTIFIER_BYTES=256;var MAX_MODEL_BYTES2=1024;var MAX_URL_BYTES=2048;var MAX_LIST_ITEMS=128;var MAX_JSON_NODES=4096;var MAX_JSON_DEPTH=32;var MAX_TUNING_INTEGER=1e9;var MIN_HERMES_CONTEXT_WINDOW=64e3;var SHA256_RE=/^[a-f0-9]{64}$/;var CONTROL_CHARACTER_RE=/[\u0000-\u001f\u007f-\u009f]/u;var BASE64URL_RE=/^[A-Za-z0-9_-]+$/;var RAW_CA_PEM_RE=/-----BEGIN (?:TRUSTED )?CERTIFICATE-----/iu;var RAW_CA_PEM_BASE64_RE=/^LS0tLS1CRUdJTi(?:BDRVJUSUZJQ0FURS0tLS0t|BUlVTVEVEIENFUlRJRklDQVRFLS0tLS0)/u;var RAW_CA_DER_BASE64_RE=/^MII[A-Za-z0-9+/=\r\n]{253,}$/u;var RAW_CA_DATA_URI_RE=/data:application\/(?:pkix-cert|x-x509-ca-cert);base64,MII[A-Za-z0-9+/=]{253,}/iu;var URL_CANDIDATE_RE=/[A-Za-z][A-Za-z0-9+.-]*:\/\/[^\s"'<>]+/gu;var UTF8_DECODER=new import_node_util.TextDecoder("utf-8",{fatal:true});var CREDENTIAL_SHAPED_NAME_PATTERN=/(?:^|[_-])(?:api[_-]?key|access[_-]?key|secret[_-]?key|auth[_-]?token|refresh[_-]?token|access[_-]?token|client[_-]?secret|private[_-]?key|pass[_-]?code|personal[_-]?access[_-]?token|connection[_-]?string|webhook(?:[_-]?url)?|key|secret|token|password|passwd|passcode|auth|authorization|credential|credentials|bearer|bearer[_-]?token|cookie|cookies|pat|private|privatekey|pin|webhookurl|dsn|connectionstring)(?:$|[_-])/iu;var CREDENTIAL_COMPOUND_NAME_PATTERN=/^(?:access|refresh|client|bearer|auth|api|private|signing|session|bot|app|resolved)(?:token|key|secret|password)$/iu;var CREDENTIAL_CAMEL_SUFFIX_PATTERN=/(?:apiKey|accessKey|secretKey|authToken|refreshToken|accessToken|clientSecret|privateKey|passcode|password|passwd|passphrase|bearerToken|botToken|appToken|sessionToken|signingKey|secretPublicKey|personalAccessToken|connectionString|webhookUrl)$/iu;var CREDENTIAL_CAMEL_BOUNDARY_PATTERN=/[a-z0-9](?:Token|Key|Secret|Password|Passphrase|Pat)$/u;var CREDENTIAL_ENV_NAME_PATTERN=/^(?:[A-Z0-9]+_)*(?:TOKEN|KEY|SECRET|PASSWORD|PASSWD|PASS|PASSPHRASE|CREDENTIAL)S?$/u;var CREDENTIAL_HEADER_NAME_PATTERN=/^(?:authorization|proxy-authorization|cookie|set-cookie|.+-(?:key|token|secret|password|passphrase|credential|auth)s?)$/iu;var PUBLIC_KEY_NAME_PATTERN=/^public[-_]?keys?$/iu;var PASS_CREDENTIAL_NAME_PATTERN=/(?:^|[-_])pass(?:wd)?$/iu;var NON_SECRET_KEY_METADATA_NAMES=new Set(["envKey","installCacheEnvKey","providerEnvKey","stateKey","targetEnvKey"]);var MESSAGING_CREDENTIAL_PLACEHOLDER_RE=/^(?:openshell:resolve:env:|[A-Za-z0-9]+-OPENSHELL-RESOLVE-ENV-)(?:(?:v[0-9]{1,20}|s[a-f0-9]{64})_)?[A-Z][A-Z0-9_]*$/u;var MESSAGING_CREDENTIAL_ENV_ALIASES=new Set(listMessagingCredentialEnvAssignments().filter(({sourceEnvKey,targetEnvKey})=>sourceEnvKey!==targetEnvKey).map(({agent,sourceEnvKey,targetEnvKey})=>`${agent}\0${sourceEnvKey}\0${targetEnvKey}`));var MESSAGING_CREDENTIAL_RUNTIME_ENV_ALIASES=new Set(listMessagingCredentialEnvAssignments().filter(({sourceEnvKey,targetEnvKey})=>sourceEnvKey!==targetEnvKey).map(({agent,channelId,sourceEnvKey,targetEnvKey})=>`${agent}\0${channelId}\0${sourceEnvKey}\0${targetEnvKey}`));var JSON_ARRAY_INDEX_SEGMENT_RE=/^\[(?:0|[1-9][0-9]*)\]$/u;var SECRET_VALUE_PATTERNS2=[/nvapi-[A-Za-z0-9_-]{10,}/u,/nvcf-[A-Za-z0-9_-]{10,}/u,/ghp_[A-Za-z0-9_-]{10,}/u,/github_pat_[A-Za-z0-9_]{30,}/u,/sk-(?:proj-|ant-)?[A-Za-z0-9_-]{10,}/u,/(?:xox[bpas]|xapp)-[A-Za-z0-9-]{10,}/u,/A(?:K|S)IA[A-Z0-9]{16}/u,/hf_[A-Za-z0-9]{10,}/u,/glpat-[A-Za-z0-9_-]{10,}/u,/gsk_[A-Za-z0-9]{10,}/u,/pypi-[A-Za-z0-9_-]{10,}/u,/tvly-[A-Za-z0-9_-]{10,}/u,/lsv2_(?:pt|sk)_[A-Za-z0-9]{10,}(?:_[A-Za-z0-9]+)*/u,/\bbot\d{8,10}:[A-Za-z0-9_-]{35}\b/u,/\b\d{8,10}:[A-Za-z0-9_-]{35}\b/u,/\b[A-Za-z0-9]{24}\.[A-Za-z0-9_-]{6}\.[A-Za-z0-9_-]{27,}\b/u,/\beyJ[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{2,}\.[A-Za-z0-9_-]{10,}\b/u,/\bBearer\s+[A-Za-z0-9_.+/=-]{10,}/iu,/-----BEGIN (?:[A-Z0-9]+ )?PRIVATE KEY-----/u];var MANAGED_STARTUP_INFERENCE_APIS=["openai-completions","openai-responses","anthropic-messages"];var MANAGED_STARTUP_REASONING_EFFORTS=["default","low","medium","high"];var MANAGED_STARTUP_DCODE_AUTO_APPROVAL_MODES=["disabled","thread-opt-in"];var MANAGED_STARTUP_HERMES_TOOL_GATEWAYS=["nous-web","nous-image","nous-audio","nous-browser","nous-code"];var MANAGED_STARTUP_AGENTS=["openclaw","hermes","langchain-deepagents-code","pi"];var MANAGED_STARTUP_MESSAGING_AGENTS=["openclaw","hermes"];function freezeAgentCapabilities(capabilities){return Object.freeze({...capabilities,inferenceApis:Object.freeze([...capabilities.inferenceApis]),dashboardModes:Object.freeze([...capabilities.dashboardModes]),inputModalities:Object.freeze([...capabilities.inputModalities]),webSearchProviders:Object.freeze([...capabilities.webSearchProviders]),toolGateways:Object.freeze([...capabilities.toolGateways]),tuningFields:Object.freeze([...capabilities.tuningFields])})}var PROFILE_CAPABILITIES={openclaw:{inferenceApis:[...MANAGED_STARTUP_INFERENCE_APIS],dashboardModes:["loopback","remote"],inputModalities:["text","image"],webSearchProviders:["brave","tavily"],toolGateways:[],tuningFields:["contextWindow","maxTokens","reasoning","reasoningEffort"],supportsMessaging:true,supportsInferenceCompatibility:true,supportsUpstreamEndpoint:false,supportsHostProxyIntent:true,supportsPrimaryModelRef:true,supportsAgentTimeout:true,supportsHeartbeat:true,supportsExtraAgents:true,supportsDeviceAuth:false,observability:"openclaw-otel",supportsMinimalBootstrap:true},hermes:{inferenceApis:[...MANAGED_STARTUP_INFERENCE_APIS],dashboardModes:["disabled","loopback-forwarded"],inputModalities:[],webSearchProviders:["tavily"],toolGateways:[...MANAGED_STARTUP_HERMES_TOOL_GATEWAYS],tuningFields:["contextWindow"],supportsMessaging:true,supportsInferenceCompatibility:false,supportsUpstreamEndpoint:false,supportsHostProxyIntent:true,supportsPrimaryModelRef:false,supportsAgentTimeout:false,supportsHeartbeat:false,supportsExtraAgents:false,supportsDeviceAuth:false,observability:"none",supportsMinimalBootstrap:false},"langchain-deepagents-code":{inferenceApis:["openai-completions"],dashboardModes:["disabled"],inputModalities:[],webSearchProviders:[],toolGateways:[],tuningFields:["reasoningEffort"],supportsMessaging:false,supportsInferenceCompatibility:false,supportsUpstreamEndpoint:true,supportsHostProxyIntent:true,supportsPrimaryModelRef:false,supportsAgentTimeout:false,supportsHeartbeat:false,supportsExtraAgents:false,supportsDeviceAuth:false,observability:"dcode-marker",supportsMinimalBootstrap:false},pi:{inferenceApis:["openai-completions"],dashboardModes:["disabled"],inputModalities:[],webSearchProviders:[],toolGateways:[],tuningFields:["contextWindow","maxTokens","reasoning"],supportsMessaging:false,supportsInferenceCompatibility:false,supportsUpstreamEndpoint:false,supportsHostProxyIntent:true,supportsPrimaryModelRef:false,supportsAgentTimeout:false,supportsHeartbeat:false,supportsExtraAgents:false,supportsDeviceAuth:false,observability:"none",supportsMinimalBootstrap:false}};for(const agent of MANAGED_STARTUP_AGENTS){Object.defineProperty(PROFILE_CAPABILITIES,agent,{configurable:false,enumerable:true,value:freezeAgentCapabilities(PROFILE_CAPABILITIES[agent]),writable:false})}var MANAGED_STARTUP_PROFILE_CAPABILITIES=Object.freeze(PROFILE_CAPABILITIES);function affordance(input,profilePath,source="docker-arg",representation="value"){return{input,profilePath,source,representation}}var HOST_PROXY_AFFORDANCES=[affordance("HTTP_PROXY","proxy.hostHttpUrl","runtime-env"),affordance("http_proxy","proxy.hostHttpUrl","runtime-env","derived"),affordance("HTTPS_PROXY","proxy.hostHttpsUrl","runtime-env"),affordance("https_proxy","proxy.hostHttpsUrl","runtime-env","derived"),affordance("NO_PROXY","proxy.hostNoProxy","runtime-env"),affordance("no_proxy","proxy.hostNoProxy","runtime-env","derived")];var MANAGED_STARTUP_PROFILE_AFFORDANCE_INVENTORY={openclaw:[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_SERVING_PRESET","inference.servingPreset"),affordance("NEMOCLAW_PRIMARY_MODEL_REF","inference.primaryModelRef"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_INFERENCE_COMPAT_B64","inference.compatibility"),affordance("NEMOCLAW_INFERENCE_INPUTS","inference.inputModalities"),affordance("NEMOCLAW_CONTEXT_WINDOW","tuning.contextWindow"),affordance("NEMOCLAW_MAX_TOKENS","tuning.maxTokens"),affordance("NEMOCLAW_REASONING","tuning.reasoning"),affordance("NEMOCLAW_REASONING_EFFORT","tuning.reasoningEffort"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_AGENT_TIMEOUT","agentConfig.agentTimeoutSeconds"),affordance("NEMOCLAW_AGENT_HEARTBEAT_EVERY","agentConfig.heartbeatEvery"),affordance("NEMOCLAW_EXTRA_AGENTS_JSON_B64","agentConfig.extraAgents"),affordance("NEMOCLAW_WEB_SEARCH_ENABLED","agentConfig.webSearch.enabled"),affordance("NEMOCLAW_WEB_SEARCH_PROVIDER","agentConfig.webSearch.provider"),affordance("NEMOCLAW_OPENCLAW_OTEL","agentConfig.otel.enabled"),affordance("NEMOCLAW_OPENCLAW_OTEL_ENDPOINT","agentConfig.otel.endpointUrl"),affordance("NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME","agentConfig.otel.serviceName"),affordance("NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE","agentConfig.otel.sampleRate"),affordance("CHAT_UI_URL","dashboard.url"),affordance("NEMOCLAW_DASHBOARD_BIND","dashboard.bindAddress"),affordance("NEMOCLAW_WSL_DASHBOARD_EXPOSURE","dashboard.wslExposure"),affordance("NEMOCLAW_DASHBOARD_PORT","dashboard.port","runtime-env"),affordance("NEMOCLAW_PROXY_HOST","proxy.managedHost"),affordance("NEMOCLAW_PROXY_PORT","proxy.managedPort"),affordance("NEMOCLAW_MESSAGING_PLAN_B64","messaging.plan"),affordance("NEMOCLAW_MINIMAL_BOOTSTRAP","agentConfig.minimalBootstrap","runtime-env"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES],hermes:[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_CONTEXT_WINDOW","tuning.contextWindow"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_HERMES_TOOL_GATEWAY_BROKER","tools.enabledGateways","docker-arg","derived"),affordance("NEMOCLAW_HERMES_TOOL_GATEWAY_PRESETS_B64","tools.enabledGateways"),affordance("NEMOCLAW_HERMES_SWITCHYARD_ROUTING_B64","agentConfig.switchyardRouting","runtime-env"),affordance("NEMOCLAW_WEB_SEARCH_ENABLED","agentConfig.webSearch.enabled"),affordance("NEMOCLAW_WEB_SEARCH_PROVIDER","agentConfig.webSearch.provider"),affordance("NEMOCLAW_MESSAGING_PLAN_B64","messaging.plan"),affordance("CHAT_UI_URL","dashboard.browserUrl"),affordance("NEMOCLAW_DASHBOARD_PORT","dashboard.publicPort","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD","dashboard.mode","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD_PORT","dashboard.publicPort","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD_INTERNAL_PORT","dashboard.internalPort","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD_TUI","dashboard.tuiEnabled","runtime-env"),affordance("NEMOCLAW_PROXY_HOST","proxy.managedHost","runtime-env"),affordance("NEMOCLAW_PROXY_PORT","proxy.managedPort","runtime-env"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES],"langchain-deepagents-code":[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_UPSTREAM_ENDPOINT_URL","inference.upstreamEndpointUrl"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_REASONING_EFFORT","tuning.reasoningEffort"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_DCODE_AUTO_APPROVAL","agentConfig.autoApprovalMode"),affordance("NEMOCLAW_PROXY_HOST","proxy.managedHost"),affordance("NEMOCLAW_PROXY_PORT","proxy.managedPort"),affordance("NEMOCLAW_OBSERVABILITY","agentConfig.observabilityEnabled","runtime-env"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES],pi:[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_CONTEXT_WINDOW","tuning.contextWindow"),affordance("NEMOCLAW_MAX_TOKENS","tuning.maxTokens"),affordance("NEMOCLAW_REASONING","tuning.reasoning"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_PROXY_HOST","proxy.managedHost"),affordance("NEMOCLAW_PROXY_PORT","proxy.managedPort"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES]};function deferredRuntimeInput(input,owner,reason,admission="managed-launch-forwarded"){return Object.freeze({input,owner,admission,reason})}var MANAGED_STARTUP_PROFILE_DEFERRED_RUNTIME_INPUTS=Object.freeze({openclaw:Object.freeze([deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_DEADLINE_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_FAST_DEADLINE_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_FAST_REENTRY_INTERVAL_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_FAST_REENTRY_POLLS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_RUN_TIMEOUT_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_SLOW_INTERVAL_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_MCP_SHADOW_DIAGNOSTICS","application-environment","operator shadow-diagnostics tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_MCP_TOOLS_LIST_TIMEOUT_MS","application-environment","operator MCP discovery timeout tuning is applied by the application environment transaction"),deferredRuntimeInput("OPENCLAW_HOME","fixed-image-contract","the managed image and agent definition own this fixed runtime layout path"),deferredRuntimeInput("OPENCLAW_STATE_DIR","fixed-image-contract","the managed image and agent definition own this fixed runtime layout path"),deferredRuntimeInput("OPENCLAW_WORKSPACE_DIR","fixed-image-contract","the managed image and agent definition own this fixed runtime layout path"),deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")]),hermes:Object.freeze([deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")]),"langchain-deepagents-code":Object.freeze([deferredRuntimeInput("NEMOCLAW_SANDBOX_NAME","engine-identity","the lifecycle engine owns instance identity outside reusable startup intent"),deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")]),pi:Object.freeze([deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")])});function runtimeCleanupObligation(input,emittedFor,supportedFor,reason){return Object.freeze({input,emittedFor:Object.freeze([...emittedFor]),supportedFor:Object.freeze([...supportedFor]),owner:"application-environment",reason})}var MANAGED_STARTUP_RUNTIME_CLEANUP_OBLIGATIONS=Object.freeze([runtimeCleanupObligation("NEMOCLAW_DASHBOARD_BIND",["hermes"],["openclaw"],"generic managed-dashboard construction currently emits the OpenClaw-only bind control for Hermes"),runtimeCleanupObligation("NEMOCLAW_MINIMAL_BOOTSTRAP",["hermes","langchain-deepagents-code"],["openclaw"],"generic host-proxy construction currently emits the OpenClaw-only bootstrap control for other agents")]);var ManagedStartupProfileError=class extends Error{constructor(message){super(`Invalid managed startup profile: ${message}`);this.name="ManagedStartupProfileError"}};var PROFILE_KEYS=new Set(["schemaVersion","agent","agentConfig","inference","proxy","dashboard","tools","messaging","tuning","corporateCa"]);var INFERENCE_KEYS=new Set(["routeProvider","upstreamProvider","servingPreset","model","routedBaseUrl","upstreamEndpointUrl","api","primaryModelRef","compatibility","inputModalities"]);var PROXY_KEYS=new Set(["managedHost","managedPort","hostHttpUrl","hostHttpsUrl","hostNoProxy"]);var OPENCLAW_DASHBOARD_KEYS=new Set(["agent","mode","url","port","bindAddress","wslExposure"]);var HERMES_DASHBOARD_KEYS=new Set(["agent","mode","url","browserUrl","publicPort","internalPort","tuiEnabled"]);var DCODE_DASHBOARD_KEYS=new Set(["agent","mode"]);var TOOLS_KEYS=new Set(["disclosure","enabledGateways"]);var MESSAGING_KEYS=new Set(["plan"]);var TUNING_FIELD_ORDER=["contextWindow","maxTokens","reasoning","reasoningEffort"];var TUNING_KEYS=new Set(TUNING_FIELD_ORDER);var CORPORATE_CA_KEYS=new Set(["bundleSha256"]);var OPENCLAW_CONFIG_KEYS=new Set(["agent","webSearch","otel","agentTimeoutSeconds","heartbeatEvery","extraAgents","minimalBootstrap"]);var LEGACY_OPENCLAW_CONFIG_KEYS=new Set([...OPENCLAW_CONFIG_KEYS,"deviceAuth"]);var LEGACY_DEVICE_AUTH_KEYS=new Set(["disabled","optOutSource"]);var LEGACY_DEVICE_AUTH_OPT_OUT_SOURCES=new Set(["operator","managed-onboard"]);var HERMES_CONFIG_KEYS=new Set(["agent","webSearch","switchyardRouting"]);var DCODE_CONFIG_KEYS=new Set(["agent","autoApprovalMode","observabilityEnabled"]);var PI_CONFIG_KEYS=new Set(["agent"]);var PI_DASHBOARD_KEYS=new Set(["agent","mode"]);var WEB_SEARCH_KEYS=new Set(["enabled","provider"]);var OTEL_KEYS=new Set(["enabled","endpointUrl","serviceName","sampleRate"]);var EXTRA_AGENTS_KEYS=new Set(["agents","defaults","main"]);var MANAGED_STARTUP_AGENT_SET=new Set(MANAGED_STARTUP_AGENTS);var DCODE_AUTO_APPROVAL_MODE_SET=new Set(MANAGED_STARTUP_DCODE_AUTO_APPROVAL_MODES);var REASONING_EFFORT_SET=new Set(MANAGED_STARTUP_REASONING_EFFORTS);var HERMES_INTERNAL_API_PORT=18642;var HERMES_API_PORT_RANGE_START=8642;var HERMES_API_PORT_RANGE_END=8652;function isHermesApiPort(port){return port>=HERMES_API_PORT_RANGE_START&&port<=HERMES_API_PORT_RANGE_END}function isHermesReservedApiPort(port){return port===HERMES_INTERNAL_API_PORT||isHermesApiPort(port)}var HERMES_RESERVED_API_PORT_LABEL=`${HERMES_API_PORT_RANGE_START}-${HERMES_API_PORT_RANGE_END} or ${HERMES_INTERNAL_API_PORT}`;function isPlainObject(value){if(typeof value!=="object"||value===null||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function isCredentialShapedName(name){if(PUBLIC_KEY_NAME_PATTERN.test(name)||NON_SECRET_KEY_METADATA_NAMES.has(name))return false;return CREDENTIAL_SHAPED_NAME_PATTERN.test(name)||CREDENTIAL_COMPOUND_NAME_PATTERN.test(name)||CREDENTIAL_CAMEL_SUFFIX_PATTERN.test(name)||CREDENTIAL_CAMEL_BOUNDARY_PATTERN.test(name)||CREDENTIAL_ENV_NAME_PATTERN.test(name)||CREDENTIAL_HEADER_NAME_PATTERN.test(name)||PASS_CREDENTIAL_NAME_PATTERN.test(name)}function valueLooksLikeSecret(value){for(let index=0;index=5&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="agentRender"&&JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[3]??"")&&path4[4]==="value";const isAuthorizedBuildStepPlaceholder=allowedBuildStepPlaceholders.has(buildStepPlaceholderKey(path4,value));return isCredentialBindingPlaceholder||isAgentRenderValuePlaceholder||isAuthorizedBuildStepPlaceholder}function requiresMessagingSchemaFieldAuthorization(path4){const fieldName=path4[path4.length-1];return fieldName==="webhook"}function messagingAuthorizedFieldKey(path4){return JSON.stringify(path4)}function buildStepPlaceholderKey(path4,value){return JSON.stringify([path4,value])}function messagingCredentialPlaceholderEnvKey(value){if(!MESSAGING_CREDENTIAL_PLACEHOLDER_RE.test(value))return null;const marker=value.startsWith("openshell:resolve:env:")?"openshell:resolve:env:":"-OPENSHELL-RESOLVE-ENV-";const key=value.slice(value.indexOf(marker)+marker.length);return key.replace(/^(?:v[0-9]{1,20}|s[a-f0-9]{64})_/u,"")}function containsMessagingCredentialPlaceholder(value){return value.includes("openshell:resolve:env:")||value.includes("-OPENSHELL-RESOLVE-ENV-")}function isMessagingCredentialPlaceholderAssignment(selectedAgent,path4,value){if(path4.length!==6||path4[0]!=="messaging"||path4[1]!=="plan"||path4[2]!=="agentRender"||!JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[3]??"")||path4[4]!=="lines"||!JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[5]??"")){return false}const separator=value.indexOf("=");if(separator<=0||value.indexOf("=",separator+1)!==-1)return false;const envKey=value.slice(0,separator);const placeholder=value.slice(separator+1);const placeholderEnvKey=messagingCredentialPlaceholderEnvKey(placeholder);return CREDENTIAL_ENV_NAME_PATTERN.test(envKey)&&placeholderEnvKey!==null&&(envKey===placeholderEnvKey||typeof selectedAgent==="string"&&MESSAGING_CREDENTIAL_ENV_ALIASES.has(`${selectedAgent}\0${placeholderEnvKey}\0${envKey}`))}function isMessagingRuntimeEnvAliasPath(path4){return path4.length===5&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="runtimeSetup"&&path4[3]==="envAliases"&&JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[4]??"")}function ownDataPropertyValue4(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}function isCanonicalMessagingRuntimeEnvAlias(selectedAgent,path4,value){if(!isMessagingRuntimeEnvAliasPath(path4))return false;const channelId=ownDataPropertyValue4(value,"channelId");const envKey=ownDataPropertyValue4(value,"envKey");const targetEnvKey=ownDataPropertyValue4(value,"targetEnvKey");const match=ownDataPropertyValue4(value,"match");const placeholder=ownDataPropertyValue4(value,"value");const expectedMatch=targetEnvKey===void 0?`^openshell:resolve:env:((?:v[0-9]{1,20}|s[a-f0-9]{64})_)?${envKey}$`:`^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_${envKey}$`;return typeof envKey==="string"&&CREDENTIAL_ENV_NAME_PATTERN.test(envKey)&&match===expectedMatch&&typeof placeholder==="string"&&messagingCredentialPlaceholderEnvKey(placeholder)===envKey&&(targetEnvKey===void 0||typeof selectedAgent==="string"&&typeof channelId==="string"&&typeof targetEnvKey==="string"&&CREDENTIAL_ENV_NAME_PATTERN.test(targetEnvKey)&&MESSAGING_CREDENTIAL_RUNTIME_ENV_ALIASES.has(`${selectedAgent}\0${channelId}\0${envKey}\0${targetEnvKey}`))}function isAllowedMessagingRuntimeAliasStringPath(path4,allowedAliasIndexes){return path4.length===6&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="runtimeSetup"&&path4[3]==="envAliases"&&allowedAliasIndexes.has(path4[4]??"")&&(path4[5]==="match"||path4[5]==="value"||path4[5]==="targetEnvKey")}function isMessagingPackagePin(path4,value){return path4.length===6&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="buildSteps"&&JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[3]??"")&&path4[4]==="value"&&path4[5]==="pin"&&typeof value==="boolean"}function containsUrlWithCredentialMaterial(value){const candidates=value.match(URL_CANDIDATE_RE)??[];for(let index=0;index{if(isCredentialShapedName(key))credentialQuery=true});const fragment=url.hash.startsWith("#")?url.hash.slice(1):url.hash;const queryStart=fragment.indexOf("?");const fragmentParameters=new URLSearchParams(queryStart>=0?fragment.slice(queryStart+1):fragment);let credentialFragment=false;fragmentParameters.forEach((_fragmentValue,key)=>{if(isCredentialShapedName(key))credentialFragment=true});if(url.username||url.password||credentialQuery||credentialFragment)return true}catch{}}return false}function invalid(reason){throw new ManagedStartupProfileError(reason)}function payloadPath(path4){return path4.reduce((result,segment)=>segment.startsWith("[")?`${result}${segment}`:`${result}${result?".":""}${segment}`,"")}function mapArrayByIndex(values,mapper){const mapped=[];for(let index=0;index0&&values[insertion-1]>selected){Object.defineProperty(values,String(insertion),{configurable:true,enumerable:true,value:values[insertion-1],writable:true});insertion-=1}Object.defineProperty(values,String(insertion),{configurable:true,enumerable:true,value:selected,writable:true})}return values}function requireRecord(value,where){if(!isPlainObject(value))invalid(`${where} must be an object`);return value}function rejectUnknownKeys2(value,allowed,where){const keys=Object.keys(value);for(let index=0;indexmaxBytes||CONTROL_CHARACTER_RE.test(value)){invalid(`${where} must be a bounded, non-empty string without control characters`)}return value}function requireStringEnum(value,allowed,where){const normalized=requireBoundedString(value,where);if(!allowed.has(normalized))invalid(`${where} is not supported`);return normalized}function requireNullablePositiveInteger(value,where){if(value===null)return null;if(typeof value!=="number"||!Number.isSafeInteger(value)||value<1||value>MAX_TUNING_INTEGER){invalid(`${where} must be null or a bounded positive integer`)}return value}function requirePositiveInteger(value,where,maximum=MAX_TUNING_INTEGER){if(typeof value!=="number"||!Number.isSafeInteger(value)||value<1||value>maximum){invalid(`${where} must be a bounded positive integer`)}return value}function requirePort(value,where,minimum=1){if(typeof value!=="number"||!Number.isInteger(value)||value<1||value>65535){invalid(`${where} must be a valid TCP port`)}if(valueMAX_LIST_ITEMS){invalid(`${where} must be a bounded string list`)}const items=mapArrayByIndex(value,item=>requireBoundedString(item,`${where} item`));const unique2=new Set;for(let index=0;index{if(depth>MAX_JSON_DEPTH)invalid(`${where} exceeds the JSON depth limit`);if(current===null||typeof current==="string"||typeof current==="boolean"){return current}if(typeof current==="number"){if(!Number.isFinite(current))invalid(`${where} contains a non-finite number`);return current}if(Array.isArray(current)){return mapArrayByIndex(current,item=>clone(item,depth+1))}if(!isPlainObject(current))invalid(`${where} contains a non-JSON value`);const result=options.nullPrototypeObjects?Object.create(null):{};const keys=Object.getOwnPropertyNames(current);for(let index=0;indexMAX_IDENTIFIER_BYTES||CONTROL_CHARACTER_RE.test(key)){invalid(`${where} contains an invalid object key`)}const descriptor=Object.getOwnPropertyDescriptor(current,key);if(!descriptor||!("value"in descriptor)){invalid(`${where} contains a non-JSON value`)}Object.defineProperty(result,key,{configurable:true,enumerable:true,value:clone(descriptor.value,depth+1),writable:true})}return result};return clone(value,0)}function requireJsonObjectOrNull(value,where){if(value===null)return null;if(!isPlainObject(value))invalid(`${where} must be null or a plain JSON object`);return cloneJsonValue(value,where,{nullPrototypeObjects:true})}function requireJsonObject(value,where){const object=requireJsonObjectOrNull(value,where);if(object===null)invalid(`${where} must be a plain JSON object`);return object}function requireHttpUrl(value,where){const raw=requireBoundedString(value,where,MAX_URL_BYTES);let parsed;try{parsed=new URL(raw)}catch{invalid(`${where} must be a valid HTTP(S) URL`)}if(parsed.protocol!=="http:"&&parsed.protocol!=="https:"||parsed.username||parsed.password||parsed.search||parsed.hash){invalid(`${where} must be a credential-free HTTP(S) URL without query or fragment data`)}const pathname=parsed.pathname.replace(/\/+$/u,"");return pathname===""?parsed.origin:`${parsed.origin}${pathname}`}function requireProxyUrl(value,allowedSchemes,where){if(value===null)return null;const raw=requireBoundedString(value,where,MAX_URL_BYTES);let parsed;try{parsed=new URL(raw)}catch{invalid(`${where} must be a valid HTTP(S) proxy URL`)}if(!allowedSchemes.has(parsed.protocol)||parsed.username||parsed.password||parsed.pathname!=="/"||parsed.search||parsed.hash){invalid(`${where} must be a credential-free HTTP(S) proxy origin`)}return parsed.origin}function requireManagedProxyHost(value,where){const host=requireBoundedString(value,where);if(!/^[A-Za-z0-9._-]+$/u.test(host)){invalid(`${where} must be a hostname or IPv4 address without a scheme or separators`)}return host}function configuredDashboardPort(value){const explicit=new URL(value).port;return explicit===""?18789:Number(explicit)}function requireSampleRate(value,where){if(typeof value!=="number"||!Number.isFinite(value)||value<0||value>1){invalid(`${where} must be a number between 0 and 1`)}return value}function assertPayloadStructureAndCredentialShapes(root){const pending=[{value:root,depth:0,path:[]}];const allowedRuntimeAliasIndexes=new Set;const allowedMessagingCredentialFields=new Set;const allowedBuildStepPlaceholders=new Set;const selectedAgent=isPlainObject(root)?ownDataPropertyValue4(root,"agent"):void 0;let discoveredNodes=1;let observedBytes=0;const observeText=value=>{observedBytes+=import_node_buffer2.Buffer.byteLength(value,"utf8");if(observedBytes>MANAGED_STARTUP_PROFILE_MAX_BYTES){invalid(`payload exceeds ${String(MANAGED_STARTUP_PROFILE_MAX_BYTES)} bytes`)}};const reserveNode=depth=>{discoveredNodes+=1;if(discoveredNodes>MAX_JSON_NODES||depth>MAX_JSON_DEPTH){invalid("payload structure exceeds the complexity limit")}observedBytes+=1};while(pending.length>0){const current=pending.pop();if(!current)break;if(current.depth>MAX_JSON_DEPTH){invalid("payload structure exceeds the complexity limit")}if(typeof current.value==="string"){observeText(current.value);if(!isAllowedMessagingRuntimeAliasStringPath(current.path,allowedRuntimeAliasIndexes)&&!isMessagingCredentialPlaceholder(current.path,current.value,allowedBuildStepPlaceholders,allowedMessagingCredentialFields)&&!isMessagingCredentialPlaceholderAssignment(selectedAgent,current.path,current.value)&&(valueLooksLikeSecret(current.value)||containsMessagingCredentialPlaceholder(current.value))){invalid(`payload field ${payloadPath(current.path)} contains credential-shaped string data`)}if(RAW_CA_PEM_RE.test(current.value)||RAW_CA_PEM_BASE64_RE.test(current.value)||RAW_CA_DER_BASE64_RE.test(current.value)||RAW_CA_DATA_URI_RE.test(current.value)){invalid(`payload field ${payloadPath(current.path)} contains raw certificate data; provide only the CA SHA-256 digest`)}if(containsUrlWithCredentialMaterial(current.value)){invalid(`payload field ${payloadPath(current.path)} contains a URL with embedded credentials`)}continue}if(Array.isArray(current.value)){if(Object.getPrototypeOf(current.value)!==Array.prototype){invalid("payload arrays must use the standard JSON prototype")}if("toJSON"in current.value){invalid("payload must not define a custom JSON serializer")}if(Object.getOwnPropertySymbols(current.value).length>0||Object.getOwnPropertyNames(current.value).length!==current.value.length+1){invalid("payload arrays must contain only indexed JSON values")}for(let index=0;index0||discoveredNodes+keys.length>MAX_JSON_NODES){invalid("payload structure exceeds the complexity limit")}for(let index=0;indexMANAGED_STARTUP_PROFILE_MAX_BYTES){invalid(`payload exceeds ${String(MANAGED_STARTUP_PROFILE_MAX_BYTES)} bytes`)}}function validateWebSearch(value,agent){const webSearch=requireRecord(value,"agentConfig.webSearch");rejectUnknownKeys2(webSearch,WEB_SEARCH_KEYS,"agentConfig.webSearch");const provider2=requireStringEnum(webSearch.provider,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].webSearchProviders),"agentConfig.webSearch.provider");return{enabled:requireBoolean(webSearch.enabled,"agentConfig.webSearch.enabled"),provider:provider2}}function validateOpenClawOtel(value){const otel=requireRecord(value,"agentConfig.otel");rejectUnknownKeys2(otel,OTEL_KEYS,"agentConfig.otel");return{enabled:requireBoolean(otel.enabled,"agentConfig.otel.enabled"),endpointUrl:requireHttpUrl(otel.endpointUrl,"agentConfig.otel.endpointUrl"),serviceName:requireBoundedString(otel.serviceName,"agentConfig.otel.serviceName",MAX_IDENTIFIER_BYTES),sampleRate:requireSampleRate(otel.sampleRate,"agentConfig.otel.sampleRate")}}function validateExtraAgents(value){const extraAgents=requireRecord(value,"agentConfig.extraAgents");rejectUnknownKeys2(extraAgents,EXTRA_AGENTS_KEYS,"agentConfig.extraAgents");if(!Array.isArray(extraAgents.agents)||extraAgents.agents.length>MAX_LIST_ITEMS){invalid("agentConfig.extraAgents.agents must be a bounded JSON object list")}return{agents:mapArrayByIndex(extraAgents.agents,(agent,index)=>requireJsonObject(agent,`agentConfig.extraAgents.agents[${String(index)}]`)),defaults:requireJsonObject(extraAgents.defaults,"agentConfig.extraAgents.defaults"),main:requireJsonObject(extraAgents.main,"agentConfig.extraAgents.main")}}function validateAgentConfig(value,expectedAgent){const config=requireRecord(value,"agentConfig");const agent=requireStringEnum(config.agent,MANAGED_STARTUP_AGENT_SET,"agentConfig.agent");if(agent!==expectedAgent)invalid("agentConfig.agent must match agent");if(agent==="openclaw"){rejectUnknownKeys2(config,OPENCLAW_CONFIG_KEYS,"agentConfig");const heartbeatEvery=config.heartbeatEvery===null?null:requireBoundedString(config.heartbeatEvery,"agentConfig.heartbeatEvery",MAX_IDENTIFIER_BYTES);if(heartbeatEvery!==null&&!/^\d+(?:s|m|h)$/u.test(heartbeatEvery)){invalid("agentConfig.heartbeatEvery must be null or a duration ending in s, m, or h")}return{agent,webSearch:validateWebSearch(config.webSearch,agent),otel:validateOpenClawOtel(config.otel),agentTimeoutSeconds:requirePositiveInteger(config.agentTimeoutSeconds,"agentConfig.agentTimeoutSeconds"),heartbeatEvery,extraAgents:validateExtraAgents(config.extraAgents),minimalBootstrap:requireBoolean(config.minimalBootstrap,"agentConfig.minimalBootstrap")}}if(agent==="hermes"){rejectUnknownKeys2(config,HERMES_CONFIG_KEYS,"agentConfig");const hasSwitchyardRouting=Object.hasOwn(config,"switchyardRouting");const switchyardRouting=!hasSwitchyardRouting||config.switchyardRouting===null?null:validateHermesSwitchyardRoutingIntent(config.switchyardRouting);return{agent,webSearch:validateWebSearch(config.webSearch,agent),...hasSwitchyardRouting?{switchyardRouting}:{}}}if(agent==="pi"){rejectUnknownKeys2(config,PI_CONFIG_KEYS,"agentConfig");return{agent}}rejectUnknownKeys2(config,DCODE_CONFIG_KEYS,"agentConfig");return{agent,autoApprovalMode:requireStringEnum(config.autoApprovalMode,DCODE_AUTO_APPROVAL_MODE_SET,"agentConfig.autoApprovalMode"),observabilityEnabled:requireBoolean(config.observabilityEnabled,"agentConfig.observabilityEnabled")}}function validateDashboard(value,expectedAgent){const dashboard=requireRecord(value,"dashboard");const agent=requireStringEnum(dashboard.agent,MANAGED_STARTUP_AGENT_SET,"dashboard.agent");if(agent!==expectedAgent)invalid("dashboard.agent must match agent");if(agent==="openclaw"){rejectUnknownKeys2(dashboard,OPENCLAW_DASHBOARD_KEYS,"dashboard");const mode=requireStringEnum(dashboard.mode,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].dashboardModes),"dashboard.mode");const url=requireHttpUrl(dashboard.url,"dashboard.url");const bindAddress=requireStringEnum(dashboard.bindAddress,new Set(["127.0.0.1","0.0.0.0"]),"dashboard.bindAddress");const wslExposure=requireBoolean(dashboard.wslExposure,"dashboard.wslExposure");const hasRemoteExposure=!isLoopbackDashboardUrl(url)||bindAddress==="0.0.0.0"||wslExposure;if(mode==="remote"!==hasRemoteExposure){invalid("OpenClaw dashboard.mode must reflect its URL, bind address, and WSL exposure")}const port=requirePort(dashboard.port,"dashboard.port",1024);if(isHermesApiPort(port))invalid(`OpenClaw dashboard.port must not use a reserved Hermes API port (${HERMES_API_PORT_RANGE_START}-${HERMES_API_PORT_RANGE_END})`);if(configuredDashboardPort(url)!==port){invalid("OpenClaw dashboard.port must match dashboard.url")}return{agent,mode,url,port,bindAddress,wslExposure}}if(agent==="hermes"){rejectUnknownKeys2(dashboard,HERMES_DASHBOARD_KEYS,"dashboard");const mode=requireStringEnum(dashboard.mode,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].dashboardModes),"dashboard.mode");const url=requireHttpUrl(dashboard.url,"dashboard.url");if(!isLoopbackDashboardUrl(url)){invalid("Hermes dashboard.url must remain loopback; OpenShell owns the host forward")}const browserUrl=dashboard.browserUrl===void 0?void 0:requireHttpUrl(dashboard.browserUrl,"dashboard.browserUrl");if(browserUrl!==void 0&&!isLoopbackDashboardUrl(browserUrl)&&new URL(browserUrl).protocol!=="https:"){invalid("Hermes dashboard.browserUrl must use HTTPS unless it is loopback")}if(mode==="disabled"){if(dashboard.publicPort!==null||dashboard.internalPort!==null||dashboard.tuiEnabled!==false){invalid("disabled Hermes dashboard must not configure ports or TUI")}return{agent,mode,url,...browserUrl===void 0?{}:{browserUrl},publicPort:null,internalPort:null,tuiEnabled:false}}const publicPort=requirePort(dashboard.publicPort,"dashboard.publicPort",1024);const internalPort=requirePort(dashboard.internalPort,"dashboard.internalPort",1024);if(publicPort===internalPort){invalid("Hermes dashboard publicPort and internalPort must differ")}if(isHermesReservedApiPort(publicPort)||isHermesReservedApiPort(internalPort)){invalid(`Hermes dashboard ports must not use reserved API ports ${HERMES_RESERVED_API_PORT_LABEL}`)}if(configuredDashboardPort(url)!==publicPort){invalid("Hermes dashboard.publicPort must match dashboard.url")}if(browserUrl!==void 0&&isLoopbackDashboardUrl(browserUrl)&&configuredDashboardPort(browserUrl)!==publicPort){invalid("Hermes dashboard.publicPort must match dashboard.browserUrl")}return{agent,mode,url,...browserUrl===void 0?{}:{browserUrl},publicPort,internalPort,tuiEnabled:requireBoolean(dashboard.tuiEnabled,"dashboard.tuiEnabled")}}if(agent==="pi"){rejectUnknownKeys2(dashboard,PI_DASHBOARD_KEYS,"dashboard");if(dashboard.mode!=="disabled")invalid("pi dashboard.mode must be disabled");return{agent,mode:"disabled"}}rejectUnknownKeys2(dashboard,DCODE_DASHBOARD_KEYS,"dashboard");if(dashboard.mode!=="disabled"){invalid("langchain-deepagents-code dashboard.mode must be disabled")}return{agent,mode:"disabled"}}function validateInference(value,agent){if(value===null){if(agent!=="openclaw"&&agent!=="hermes")invalid(`${agent} requires inference configuration`);return null}const inference=requireRecord(value,"inference");rejectUnknownKeys2(inference,INFERENCE_KEYS,"inference");const routeProvider=requireBoundedString(inference.routeProvider,"inference.routeProvider");const upstreamProvider=requireBoundedString(inference.upstreamProvider,"inference.upstreamProvider");const servingPreset=inference.servingPreset===void 0?void 0:inference.servingPreset===null?null:requireBoundedString(inference.servingPreset,"inference.servingPreset");const model2=requireBoundedString(inference.model,"inference.model",MAX_MODEL_BYTES2);const api=requireStringEnum(inference.api,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].inferenceApis),"inference.api");const upstreamEndpointUrl=inference.upstreamEndpointUrl===null?null:requireHttpUrl(inference.upstreamEndpointUrl,"inference.upstreamEndpointUrl");const primaryModelRef=inference.primaryModelRef===null?null:requireBoundedString(inference.primaryModelRef,"inference.primaryModelRef",MAX_MODEL_BYTES2);const compatibility=requireJsonObjectOrNull(inference.compatibility,"inference.compatibility");const inputModalities=inference.inputModalities===null?null:requireEnumList(inference.inputModalities,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].inputModalities),"inference.inputModalities",{allowEmpty:false});if(upstreamEndpointUrl!==null&&!MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].supportsUpstreamEndpoint){invalid(`inference.upstreamEndpointUrl must be null for ${agent}`)}if(agent==="openclaw"){if(primaryModelRef===null||inputModalities===null){invalid("openclaw requires primaryModelRef and inputModalities")}if(primaryModelRef!==`${routeProvider}/${model2}`){invalid("openclaw primaryModelRef must match routeProvider and model")}}else{if(primaryModelRef!==null||compatibility!==null||inputModalities!==null){invalid(`${agent} does not support primaryModelRef, compatibility, or inputModalities`)}if(agent==="langchain-deepagents-code"&&!isValidDcodeUpstreamProvider(upstreamProvider)){invalid("inference.upstreamProvider must start with an ASCII letter or digit and contain 1-64 ASCII letters, digits, dots, underscores, or hyphens for DCode")}}return{routeProvider,upstreamProvider,...servingPreset===void 0?{}:{servingPreset},model:model2,routedBaseUrl:requireHttpUrl(inference.routedBaseUrl,"inference.routedBaseUrl"),upstreamEndpointUrl,api,primaryModelRef,compatibility,inputModalities}}function validateProxy(value,agent){const proxy=requireRecord(value,"proxy");rejectUnknownKeys2(proxy,PROXY_KEYS,"proxy");const hostHttpUrl=requireProxyUrl(proxy.hostHttpUrl,new Set(["http:"]),"proxy.hostHttpUrl");const hostHttpsUrl=requireProxyUrl(proxy.hostHttpsUrl,new Set(["http:","https:"]),"proxy.hostHttpsUrl");const hostNoProxy=requireStringList(proxy.hostNoProxy,"proxy.hostNoProxy");if(!MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].supportsHostProxyIntent&&(hostHttpUrl!==null||hostHttpsUrl!==null||hostNoProxy.length>0)){invalid(`${agent} rejects host proxy intent and accepts only its root-owned managed route`)}return{managedHost:requireManagedProxyHost(proxy.managedHost,"proxy.managedHost"),managedPort:requirePort(proxy.managedPort,"proxy.managedPort"),hostHttpUrl,hostHttpsUrl,hostNoProxy}}function validateTools(value,agent){const tools=requireRecord(value,"tools");rejectUnknownKeys2(tools,TOOLS_KEYS,"tools");const enabledGateways=requireEnumList(tools.enabledGateways,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].toolGateways),"tools.enabledGateways",{allowEmpty:true});return{disclosure:requireStringEnum(tools.disclosure,new Set(["progressive","direct"]),"tools.disclosure"),enabledGateways}}function validateTuning(value,agent){const tuning=requireRecord(value,"tuning");rejectUnknownKeys2(tuning,TUNING_KEYS,"tuning");const result={contextWindow:requireNullablePositiveInteger(tuning.contextWindow,"tuning.contextWindow"),maxTokens:requireNullablePositiveInteger(tuning.maxTokens,"tuning.maxTokens"),reasoning:requireNullableBoolean(tuning.reasoning,"tuning.reasoning"),reasoningEffort:tuning.reasoningEffort===null?null:requireStringEnum(tuning.reasoningEffort,REASONING_EFFORT_SET,"tuning.reasoningEffort")};const advertised=new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].tuningFields);const unsupported=TUNING_FIELD_ORDER.filter(field=>result[field]!==null&&!advertised.has(field));if(unsupported.length>0){invalid(`${agent} does not support startup tuning fields: ${unsupported.join(", ")}`)}if(agent==="openclaw"){const missing=TUNING_FIELD_ORDER.filter(field=>advertised.has(field)&&result[field]===null);if(missing.length>0){invalid(`openclaw requires ${missing.join(", ")} tuning`)}}if(agent==="hermes"&&result.contextWindow!==null&&result.contextWindowcanonicalizeJson(item));if(!isPlainObject(value))return value;const result={};const keys=sortStrings(Object.keys(value));for(let index=0;indexMANAGED_STARTUP_PROFILE_MAX_BYTES){invalid(`canonical payload exceeds ${String(MANAGED_STARTUP_PROFILE_MAX_BYTES)} bytes`)}return serialized}function decodeManagedStartupProfile(encoded){if(typeof encoded!=="string"||encoded.length===0||import_node_buffer2.Buffer.byteLength(encoded,"ascii")>MANAGED_STARTUP_PROFILE_MAX_ENCODED_BYTES||!BASE64URL_RE.test(encoded)||encoded.length%4===1){invalid("encoded payload is malformed or exceeds the size limit")}const bytes=import_node_buffer2.Buffer.from(encoded,"base64url");if(bytes.length===0||bytes.length>MANAGED_STARTUP_PROFILE_MAX_BYTES||bytes.toString("base64url")!==encoded){invalid("encoded payload is malformed or exceeds the size limit")}let raw;try{raw=UTF8_DECODER.decode(bytes)}catch{invalid("payload is not valid UTF-8")}let parsed;try{parsed=JSON.parse(raw)}catch{invalid("payload is not valid JSON")}const migration=migrateDecodedManagedStartupProfile(parsed);const profile=validateManagedStartupProfile(migration.value);const canonicalPayload=migration.migratedLegacyProfile?JSON.stringify(canonicalizeJson(parsed)):serializeManagedStartupProfile(profile);if(canonicalPayload!==raw){invalid("payload is not in canonical form")}return profile}function fingerprintManagedStartupProfile(profile){return(0,import_node_crypto3.createHash)("sha256").update(serializeManagedStartupProfile(profile),"utf8").digest("hex")}var ManagedStartupAgentEnvironmentError=class extends Error{constructor(message){super(`Cannot map managed startup profile: ${message}`);this.name="ManagedStartupAgentEnvironmentError"}};var EMPTY_APPLICATION_ENVIRONMENT=Object.freeze({});var OPENCLAW_APPLICATION_RUNTIME_INPUTS=Object.freeze([["NEMOCLAW_AUTO_PAIR_DEADLINE_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_FAST_DEADLINE_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_FAST_REENTRY_INTERVAL_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_FAST_REENTRY_POLLS","positive-safe-integer"],["NEMOCLAW_AUTO_PAIR_RUN_TIMEOUT_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_SLOW_INTERVAL_SECS","positive-finite-seconds"]]);function booleanFlag(value){return value?"1":"0"}function canonicalizeJson2(value){if(Array.isArray(value))return value.map(item=>canonicalizeJson2(item));if(value===null||typeof value!=="object")return value;const record2=value;return Object.fromEntries(Object.keys(record2).sort().map(key=>[key,canonicalizeJson2(record2[key])]))}function encodeCanonicalJson(value){return import_node_buffer3.Buffer.from(JSON.stringify(canonicalizeJson2(value)),"utf8").toString("base64")}function sortedEnvironment(environment){return Object.freeze(Object.fromEntries(Object.entries(environment).sort(([left],[right])=>leftright?1:0)))}function canonicalApplicationRuntimeValue(name,raw,kind){if(raw.includes("\0")||/[\r\n]/u.test(raw)){throw new ManagedStartupAgentEnvironmentError(`${name} must be single-line text`)}const value=Number(raw.trim());const valid=kind==="positive-safe-integer"?Number.isSafeInteger(value)&&value>0:Number.isFinite(value)&&value>0;if(!valid){throw new ManagedStartupAgentEnvironmentError(`${name} must be ${kind==="positive-safe-integer"?"a positive safe integer":"finite positive seconds"}`)}return String(value)}function applicationRuntimePlan(profile,environment){const exportEnvironment={};if(profile.agent==="openclaw"){for(const[name,kind]of OPENCLAW_APPLICATION_RUNTIME_INPUTS){const raw=environment[name];if(raw!==void 0){exportEnvironment[name]=canonicalApplicationRuntimeValue(name,raw,kind)}}}const unsetEnvironment=new Set(MANAGED_STARTUP_RUNTIME_CLEANUP_OBLIGATIONS.filter(({supportedFor})=>!supportedFor.includes(profile.agent)).map(({input})=>input));if(profile.agent!=="openclaw"){for(const[name]of OPENCLAW_APPLICATION_RUNTIME_INPUTS){unsetEnvironment.add(name)}}return Object.freeze({exportEnvironment:sortedEnvironment(exportEnvironment),unsetEnvironment:Object.freeze([...unsetEnvironment].sort())})}function commonConfigurationEnvironment(profile){if(profile.inference===null)return{...PROVIDERLESS_INFERENCE_ENV,NEMOCLAW_TOOL_DISCLOSURE:profile.tools.disclosure};return{NEMOCLAW_INFERENCE_API:profile.inference.api,NEMOCLAW_INFERENCE_BASE_URL:profile.inference.routedBaseUrl,NEMOCLAW_INFERENCE_PROVIDER_ID:profile.inference.routeProvider,NEMOCLAW_MODEL:profile.inference.model,NEMOCLAW_TOOL_DISCLOSURE:profile.tools.disclosure,NEMOCLAW_UPSTREAM_PROVIDER:profile.inference.upstreamProvider}}function appendHostProxyEnvironment(environment,profile,options={}){if(options.preserveAmbientWhenAbsent===true&&profile.proxy.hostHttpUrl===null&&profile.proxy.hostHttpsUrl===null&&profile.proxy.hostNoProxy.length===0){return}const httpProxy=profile.proxy.hostHttpUrl??"";const httpsProxy=profile.proxy.hostHttpsUrl??"";const noProxy=profile.proxy.hostNoProxy.join(",");environment.HTTP_PROXY=httpProxy;environment.HTTPS_PROXY=httpsProxy;environment.NO_PROXY=noProxy;environment.http_proxy=httpProxy;environment.https_proxy=httpsProxy;environment.no_proxy=noProxy}function messagingEnvironment(profile,expectedAgent){if(profile.messaging.plan===null)return{};const plan=parseSandboxMessagingPlan(profile.messaging.plan,{agent:expectedAgent});if(!plan){throw new ManagedStartupAgentEnvironmentError(`messaging.plan must contain a validated ${expectedAgent} messaging plan`)}const{workflow:_workflow,...imageBuildPlan}=plan;return{NEMOCLAW_MESSAGING_PLAN_B64:encodeCanonicalJson(imageBuildPlan)}}function corporateCaMaterial(profile){return Object.freeze({kind:"corporate-ca-handoff",legacyInput:"NEMOCLAW_CORPORATE_CA_B64",expectedSha256:profile.corporateCa.bundleSha256})}function rootOwnedFile(legacyInput,path4,value){return Object.freeze({kind:"root-owned-file",legacyInput,path:path4,contents:`${value} -`,owner:"root",group:"root",mode:292})}function dashboardAction(dashboard){return Object.freeze({kind:"configure-dashboard",dashboard:Object.freeze(structuredClone(dashboard))})}function applicationActions(profile,messagingAgent){const actions=[];if(messagingAgent!==null){actions.push(Object.freeze({kind:"apply-messaging-plan",agent:messagingAgent,mode:profile.messaging.plan===null?"clear":"apply",phase:"runtime-setup",runAs:"root"}))}actions.push(Object.freeze({kind:"generate-agent-config",agent:profile.agent,runAs:"sandbox"}));if(messagingAgent!==null){actions.push(Object.freeze({kind:"apply-messaging-plan",agent:messagingAgent,mode:profile.messaging.plan===null?"clear":"apply",phase:"post-agent-install",runAs:"sandbox"}))}actions.push(dashboardAction(profile.dashboard));return Object.freeze(actions)}function mapOpenClawProfile(profile,environment){if(profile.agent!=="openclaw"||profile.agentConfig.agent!=="openclaw"||profile.dashboard.agent!=="openclaw"||profile.inference!==null&&(profile.inference.primaryModelRef===null||profile.inference.inputModalities===null)||profile.tuning.contextWindow===null||profile.tuning.maxTokens===null||profile.tuning.reasoning===null||profile.tuning.reasoningEffort===null){throw new ManagedStartupAgentEnvironmentError("OpenClaw profile state is inconsistent")}const configurationEnvironment={...commonConfigurationEnvironment(profile),...messagingEnvironment(profile,"openclaw"),CHAT_UI_URL:profile.dashboard.url,NEMOCLAW_AGENT_HEARTBEAT_EVERY:profile.agentConfig.heartbeatEvery??"",NEMOCLAW_AGENT_TIMEOUT:String(profile.agentConfig.agentTimeoutSeconds),NEMOCLAW_CONTEXT_WINDOW:String(profile.tuning.contextWindow),NEMOCLAW_DASHBOARD_BIND:profile.dashboard.bindAddress==="0.0.0.0"?profile.dashboard.bindAddress:"",NEMOCLAW_EXTRA_AGENTS_JSON_B64:encodeCanonicalJson(profile.agentConfig.extraAgents),NEMOCLAW_INFERENCE_COMPAT_B64:encodeCanonicalJson(profile.inference?.compatibility??{}),NEMOCLAW_INFERENCE_INPUTS:profile.inference?.inputModalities?.join(",")??"text",NEMOCLAW_MAX_TOKENS:String(profile.tuning.maxTokens),NEMOCLAW_OPENCLAW_OTEL:booleanFlag(profile.agentConfig.otel.enabled),NEMOCLAW_OPENCLAW_OTEL_ENDPOINT:profile.agentConfig.otel.endpointUrl,NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE:String(profile.agentConfig.otel.sampleRate),NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME:profile.agentConfig.otel.serviceName,NEMOCLAW_PRIMARY_MODEL_REF:profile.inference?.primaryModelRef??"",NEMOCLAW_PROXY_HOST:profile.proxy.managedHost,NEMOCLAW_PROXY_PORT:String(profile.proxy.managedPort),NEMOCLAW_REASONING:String(profile.tuning.reasoning),NEMOCLAW_REASONING_EFFORT:profile.tuning.reasoningEffort,NEMOCLAW_SERVING_PRESET:profile.inference?.servingPreset??"",NEMOCLAW_WEB_SEARCH_ENABLED:booleanFlag(profile.agentConfig.webSearch.enabled),NEMOCLAW_WEB_SEARCH_PROVIDER:profile.agentConfig.webSearch.provider,NEMOCLAW_WSL_DASHBOARD_EXPOSURE:booleanFlag(profile.dashboard.wslExposure)};const runtimeEnvironment={...configurationEnvironment};delete runtimeEnvironment.NEMOCLAW_MESSAGING_PLAN_B64;runtimeEnvironment.NEMOCLAW_DASHBOARD_PORT=String(profile.dashboard.port);runtimeEnvironment.NEMOCLAW_MINIMAL_BOOTSTRAP=booleanFlag(profile.agentConfig.minimalBootstrap);appendHostProxyEnvironment(runtimeEnvironment,profile,{preserveAmbientWhenAbsent:true});return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials:Object.freeze([corporateCaMaterial(profile)]),actions:applicationActions(profile,"openclaw")})}function mapHermesProfile(profile,environment){if(profile.agent!=="hermes"||profile.agentConfig.agent!=="hermes"||profile.dashboard.agent!=="hermes"){throw new ManagedStartupAgentEnvironmentError("Hermes profile state is inconsistent")}let chatUiUrl=profile.dashboard.browserUrl??profile.dashboard.url;if(profile.dashboard.mode==="loopback-forwarded"){if(profile.dashboard.browserUrl===void 0){throw new ManagedStartupAgentEnvironmentError("Cannot start the Hermes dashboard because its managed startup profile has no recorded browser URL. Rerun onboarding before starting the sandbox.")}chatUiUrl=profile.dashboard.browserUrl}const configurationEnvironment={...commonConfigurationEnvironment(profile),...messagingEnvironment(profile,"hermes"),CHAT_UI_URL:chatUiUrl,NEMOCLAW_CONTEXT_WINDOW:profile.tuning.contextWindow===null?"":String(profile.tuning.contextWindow),NEMOCLAW_HERMES_TOOL_GATEWAY_BROKER:booleanFlag(profile.tools.enabledGateways.length>0),NEMOCLAW_HERMES_TOOL_GATEWAY_PRESETS_B64:encodeCanonicalJson(profile.tools.enabledGateways),NEMOCLAW_WEB_SEARCH_ENABLED:booleanFlag(profile.agentConfig.webSearch.enabled),NEMOCLAW_WEB_SEARCH_PROVIDER:profile.agentConfig.webSearch.provider,[HERMES_SWITCHYARD_ROUTING_ENV]:profile.agentConfig.switchyardRouting?encodeHermesSwitchyardRoutingIntent(profile.agentConfig.switchyardRouting):""};const runtimeEnvironment={...configurationEnvironment,HERMES_HOME:"/sandbox/.hermes",HERMES_LAZY_INSTALL_TARGET:"/sandbox/.hermes/lazy-packages"};delete runtimeEnvironment.NEMOCLAW_MESSAGING_PLAN_B64;delete runtimeEnvironment[HERMES_SWITCHYARD_ROUTING_ENV];runtimeEnvironment.NEMOCLAW_DASHBOARD_PORT=profile.dashboard.publicPort===null?"":String(profile.dashboard.publicPort);runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD=profile.dashboard.mode==="loopback-forwarded"?"1":"0";runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_INTERNAL_PORT=profile.dashboard.internalPort===null?"":String(profile.dashboard.internalPort);runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_PORT=profile.dashboard.publicPort===null?"":String(profile.dashboard.publicPort);runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_TUI=booleanFlag(profile.dashboard.tuiEnabled);runtimeEnvironment.NEMOCLAW_PROXY_HOST=profile.proxy.managedHost;runtimeEnvironment.NEMOCLAW_PROXY_PORT=String(profile.proxy.managedPort);appendHostProxyEnvironment(runtimeEnvironment,profile,{preserveAmbientWhenAbsent:true});return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials:Object.freeze([corporateCaMaterial(profile)]),actions:applicationActions(profile,"hermes")})}function mapDcodeProfile(profile,environment){if(profile.agent!=="langchain-deepagents-code"||profile.agentConfig.agent!=="langchain-deepagents-code"||profile.dashboard.agent!=="langchain-deepagents-code"||profile.messaging.plan!==null||profile.inference===null){throw new ManagedStartupAgentEnvironmentError("LangChain Deep Agents Code profile state is inconsistent")}const reasoningEffort=profile.tuning.reasoningEffort===null||profile.tuning.reasoningEffort==="default"?"":profile.tuning.reasoningEffort;const configurationEnvironment={...commonConfigurationEnvironment(profile),NEMOCLAW_REASONING_EFFORT:reasoningEffort,NEMOCLAW_UPSTREAM_ENDPOINT_URL:profile.inference.upstreamEndpointUrl??""};appendHostProxyEnvironment(configurationEnvironment,profile);const runtimeEnvironment={...configurationEnvironment,NEMOCLAW_OBSERVABILITY:booleanFlag(profile.agentConfig.observabilityEnabled)};delete runtimeEnvironment.NEMOCLAW_INFERENCE_BASE_URL;delete runtimeEnvironment.NEMOCLAW_REASONING_EFFORT;delete runtimeEnvironment.NEMOCLAW_UPSTREAM_PROVIDER;for(const name of["HTTP_PROXY","HTTPS_PROXY","NO_PROXY","http_proxy","https_proxy","no_proxy"]){delete runtimeEnvironment[name]}const materials=Object.freeze([corporateCaMaterial(profile),rootOwnedFile("NEMOCLAW_DCODE_AUTO_APPROVAL","/usr/local/share/nemoclaw/dcode-auto-approval",profile.agentConfig.autoApprovalMode),rootOwnedFile("NEMOCLAW_INFERENCE_BASE_URL","/usr/local/share/nemoclaw/dcode-inference-base-url",profile.inference.routedBaseUrl),rootOwnedFile("NEMOCLAW_UPSTREAM_PROVIDER","/usr/local/share/nemoclaw/dcode-upstream-provider",profile.inference.upstreamProvider),rootOwnedFile("NEMOCLAW_INFERENCE_PROVIDER_ID","/usr/local/share/nemoclaw/dcode-provider-route",profile.inference.routeProvider),rootOwnedFile("NEMOCLAW_PROXY_HOST","/usr/local/share/nemoclaw/dcode-proxy-host",profile.proxy.managedHost),rootOwnedFile("NEMOCLAW_PROXY_PORT","/usr/local/share/nemoclaw/dcode-proxy-port",String(profile.proxy.managedPort)),rootOwnedFile("NEMOCLAW_REASONING_EFFORT","/usr/local/share/nemoclaw/dcode-reasoning-effort",reasoningEffort)]);return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials,actions:applicationActions(profile,null)})}function mapPiProfile(profile,environment){if(profile.agent!=="pi"||profile.agentConfig.agent!=="pi"||profile.dashboard.agent!=="pi"||profile.messaging.plan!==null){throw new ManagedStartupAgentEnvironmentError("Pi profile state is inconsistent")}const configurationEnvironment={...commonConfigurationEnvironment(profile),NEMOCLAW_CONTEXT_WINDOW:profile.tuning.contextWindow===null?"":String(profile.tuning.contextWindow),NEMOCLAW_MAX_TOKENS:profile.tuning.maxTokens===null?"":String(profile.tuning.maxTokens),NEMOCLAW_REASONING:profile.tuning.reasoning===null?"":String(profile.tuning.reasoning)};appendHostProxyEnvironment(configurationEnvironment,profile);const runtimeEnvironment={...configurationEnvironment};delete runtimeEnvironment.NEMOCLAW_INFERENCE_BASE_URL;delete runtimeEnvironment.NEMOCLAW_CONTEXT_WINDOW;delete runtimeEnvironment.NEMOCLAW_MAX_TOKENS;delete runtimeEnvironment.NEMOCLAW_REASONING;for(const name of["HTTP_PROXY","HTTPS_PROXY","NO_PROXY","http_proxy","https_proxy","no_proxy"]){delete runtimeEnvironment[name]}const materials=Object.freeze([corporateCaMaterial(profile),rootOwnedFile("NEMOCLAW_PROXY_HOST","/usr/local/share/nemoclaw/pi-proxy-host",profile.proxy.managedHost),rootOwnedFile("NEMOCLAW_PROXY_PORT","/usr/local/share/nemoclaw/pi-proxy-port",String(profile.proxy.managedPort))]);return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials,actions:applicationActions(profile,null)})}function mapManagedStartupProfileToAgentEnvironment(profile,environment=EMPTY_APPLICATION_ENVIRONMENT){const validated=validateManagedStartupProfile(profile);switch(validated.agent){case"openclaw":return mapOpenClawProfile(validated,environment);case"hermes":return mapHermesProfile(validated,environment);case"langchain-deepagents-code":return mapDcodeProfile(validated,environment);case"pi":return mapPiProfile(validated,environment)}}var import_node_buffer4=require("node:buffer");var import_node_crypto4=require("node:crypto");var import_node_fs=__toESM(require("node:fs"));var import_node_path=__toESM(require("node:path"));var import_node_util2=require("node:util");var MANAGED_STARTUP_APPLICATION_STATE_DIR="/var/lib/nemoclaw/startup-profile";var MANAGED_STARTUP_CA_MAX_BYTES=128*1024;var MANAGED_STARTUP_CA_MAX_CERTIFICATES=24;var STATE_SCHEMA_VERSION=1;var STATE_DIRECTORY_MODE=448;var STATE_FILE_MODE=384;var MAX_CONTROL_FILE_BYTES=512;var MAX_STATE_ENTRIES=32;var SHA256_RE2=/^[a-f0-9]{64}$/u;var GENERATION_RE=/^generation-([a-f0-9]{64})$/u;var PREPARE_TEMP_RE=/^\.prepare-[0-9]+-[a-f0-9]{24}$/u;var CONTROL_TEMP_RE=/^\.(?:committed|pending)\.json-[a-f0-9]{24}\.tmp$/u;var PEM_CERTIFICATE_RE=/-----BEGIN CERTIFICATE-----\r?\n[A-Za-z0-9+/=\r\n]+?-----END CERTIFICATE-----/gu;var UTF8_DECODER2=new import_node_util2.TextDecoder("utf-8",{fatal:true});var DEFAULT_RUNTIME={rootUid:0,rootGid:0};var ManagedStartupApplicationError=class extends Error{constructor(message){super(`Managed startup application failed: ${message}`);this.name="ManagedStartupApplicationError"}};function fail2(message){throw new ManagedStartupApplicationError(message)}function runtimeFor(override){return override??DEFAULT_RUNTIME}function requireContainerRoot(){if(process.geteuid?.()!==0){fail2("the image-side applicator must run with effective uid 0")}}function modeOf(stat){return stat.mode&511}function requireOwner(stat,target2,runtime){if(stat.uid!==runtime.rootUid||stat.gid!==runtime.rootGid){fail2(`${target2} must be owned by root:root`)}}function requireSecureDirectory(target2,runtime,exactMode){let stat;try{stat=import_node_fs.default.lstatSync(target2)}catch{fail2(`state directory component is missing or unreadable: ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail2(`state directory component must be a real directory: ${target2}`)}const runtimeOwned=stat.uid===runtime.rootUid&&stat.gid===runtime.rootGid;const systemRootOwned=stat.uid===0&&stat.gid===0;if(exactMode){requireOwner(stat,target2,runtime)}else if(!runtimeOwned&&!systemRootOwned){fail2(`state directory ancestor is not owned by a trusted identity: ${target2}`)}const mode=modeOf(stat);const writableByUntrustedIdentity=(mode&18)!==0;const trustedStickyRoot=(stat.mode&512)!==0&&(runtimeOwned||systemRootOwned);if(exactMode&&mode!==STATE_DIRECTORY_MODE||!exactMode&&writableByUntrustedIdentity&&!trustedStickyRoot){fail2(exactMode?`${target2} must have mode 0700`:`${target2} is a replaceable group- or world-writable ancestor`)}}function requireSecureAncestors(target2,runtime){const root=import_node_path.default.parse(target2).root;let current=root;requireSecureDirectory(current,runtime,false);for(const segment of import_node_path.default.relative(root,target2).split(import_node_path.default.sep).filter(Boolean)){current=import_node_path.default.join(current,segment);let stat;try{stat=import_node_fs.default.lstatSync(current)}catch{fail2(`state directory component is missing or unreadable: ${current}`)}if(stat.isSymbolicLink()){const runtimeOwned=stat.uid===runtime.rootUid&&stat.gid===runtime.rootGid;const systemRootOwned=stat.uid===0&&stat.gid===0;if(!runtimeOwned&&!systemRootOwned){fail2(`state directory ancestor is a replaceable symlink: ${current}`)}let resolved;try{resolved=import_node_fs.default.realpathSync(current)}catch{fail2(`state directory symlink is missing or unreadable: ${current}`)}requireSecureAncestors(resolved,runtime);continue}requireSecureDirectory(current,runtime,false)}}function ensureStateDirectory(rawStateDirectory,runtime){const stateDirectory=rawStateDirectory??MANAGED_STARTUP_APPLICATION_STATE_DIR;if(!import_node_path.default.isAbsolute(stateDirectory)||stateDirectory.includes("\0")){fail2("stateDirectory must be an absolute path")}const normalized=import_node_path.default.resolve(stateDirectory);const parent=import_node_path.default.dirname(normalized);requireSecureAncestors(parent,runtime);try{import_node_fs.default.mkdirSync(normalized,{mode:STATE_DIRECTORY_MODE});import_node_fs.default.chownSync(normalized,runtime.rootUid,runtime.rootGid);import_node_fs.default.chmodSync(normalized,STATE_DIRECTORY_MODE)}catch(error){if(error.code!=="EEXIST"){fail2(`could not create the managed startup state directory: ${normalized}`)}}requireSecureDirectory(normalized,runtime,true);return normalized}function requireSecureRegularFileStat(stat,target2,runtime){if(!stat.isFile()||stat.isSymbolicLink()){fail2(`${target2} must be a regular file`)}if(stat.nlink!==1){fail2(`${target2} must not be hardlinked`)}requireOwner(stat,target2,runtime);if(modeOf(stat)!==STATE_FILE_MODE){fail2(`${target2} must have mode 0600`)}}function readSecureFile(target2,maxBytes,runtime){let descriptor;try{descriptor=import_node_fs.default.openSync(target2,import_node_fs.default.constants.O_RDONLY|import_node_fs.default.constants.O_NOFOLLOW)}catch{fail2(`state file is missing, unreadable, or a symlink: ${target2}`)}try{const stat=import_node_fs.default.fstatSync(descriptor);requireSecureRegularFileStat(stat,target2,runtime);if(stat.size<1||stat.size>maxBytes){fail2(`${target2} is empty or exceeds its size limit`)}const content=import_node_fs.default.readFileSync(descriptor);if(content.length!==stat.size){fail2(`${target2} changed while it was being read`)}return content}finally{import_node_fs.default.closeSync(descriptor)}}function writeSecureNewFile(target2,content,runtime){let descriptor;try{descriptor=import_node_fs.default.openSync(target2,import_node_fs.default.constants.O_CREAT|import_node_fs.default.constants.O_EXCL|import_node_fs.default.constants.O_WRONLY|import_node_fs.default.constants.O_NOFOLLOW,STATE_FILE_MODE)}catch{fail2(`refused to replace an existing state file: ${target2}`)}try{import_node_fs.default.fchownSync(descriptor,runtime.rootUid,runtime.rootGid);import_node_fs.default.fchmodSync(descriptor,STATE_FILE_MODE);import_node_fs.default.writeFileSync(descriptor,content);import_node_fs.default.fsyncSync(descriptor)}finally{import_node_fs.default.closeSync(descriptor)}}function syncDirectory(target2){const descriptor=import_node_fs.default.openSync(target2,import_node_fs.default.constants.O_RDONLY);try{import_node_fs.default.fsyncSync(descriptor)}finally{import_node_fs.default.closeSync(descriptor)}}function randomToken(){return(0,import_node_crypto4.randomBytes)(12).toString("hex")}function stateControl(fingerprint){return{schemaVersion:STATE_SCHEMA_VERSION,fingerprint,generation:`generation-${fingerprint}`}}function serializeStateControl(control){return JSON.stringify({fingerprint:control.fingerprint,generation:control.generation,schemaVersion:control.schemaVersion})}function parseStateControl(target2,runtime){const bytes=readSecureFile(target2,MAX_CONTROL_FILE_BYTES,runtime);let raw;try{raw=UTF8_DECODER2.decode(bytes)}catch{fail2(`${target2} is not valid UTF-8`)}let parsed;try{parsed=JSON.parse(raw)}catch{fail2(`${target2} is not valid JSON`)}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail2(`${target2} does not contain a valid state control`)}const record2=parsed;if(Object.keys(record2).sort().join(",")!=="fingerprint,generation,schemaVersion"||record2.schemaVersion!==STATE_SCHEMA_VERSION||typeof record2.fingerprint!=="string"||!SHA256_RE2.test(record2.fingerprint)||record2.generation!==`generation-${record2.fingerprint}`){fail2(`${target2} does not contain a valid state control`)}const control=stateControl(record2.fingerprint);if(serializeStateControl(control)!==raw){fail2(`${target2} is not in canonical form`)}return control}function publishStateControlIfAbsent(stateDirectory,basename,control,runtime){const target2=import_node_path.default.join(stateDirectory,basename);const temporary=import_node_path.default.join(stateDirectory,`.${basename}-${randomToken()}.tmp`);writeSecureNewFile(temporary,serializeStateControl(control),runtime);try{import_node_fs.default.linkSync(temporary,target2)}catch(error){try{unlinkSecureControlOrTemp(temporary,runtime)}catch{}if(error.code==="EEXIST"){return{control:parseStateControl(target2,runtime),created:false}}fail2(`could not atomically publish ${basename}`)}try{import_node_fs.default.unlinkSync(temporary)}catch(error){if(error.code!=="ENOENT"){fail2(`could not finalize atomic publication of ${basename}`)}}syncDirectory(stateDirectory);return{control,created:true}}function validateCorporateCaBytes(bytes){if(bytes.length<1||bytes.length>MANAGED_STARTUP_CA_MAX_BYTES){fail2(`corporate CA bundle must contain 1-${String(MANAGED_STARTUP_CA_MAX_BYTES)} bytes`)}let pem;try{pem=UTF8_DECODER2.decode(bytes)}catch{fail2("corporate CA bundle must be valid UTF-8 PEM")}const matches=[...pem.matchAll(PEM_CERTIFICATE_RE)];if(matches.length<1||matches.length>MANAGED_STARTUP_CA_MAX_CERTIFICATES||matches[0]?.index!==0){fail2(`corporate CA bundle must contain 1-${String(MANAGED_STARTUP_CA_MAX_CERTIFICATES)} PEM CA certificates`)}let cursor=0;for(const match of matches){const index=match.index;if(index===void 0||!/^(?:\r?\n)+$/u.test(pem.slice(cursor,index))&&index!==0){fail2("corporate CA bundle contains non-PEM material between certificates")}const block=match[0];let certificate;try{certificate=new import_node_crypto4.X509Certificate(block)}catch{fail2("corporate CA bundle contains an invalid X.509 certificate")}if(!certificate.ca){fail2("corporate CA bundle contains a certificate without basicConstraints CA:TRUE")}cursor=index+block.length}if(!/^(?:\r?\n)?$/u.test(pem.slice(cursor))){fail2("corporate CA bundle contains trailing non-PEM material")}}function validateManagedStartupCorporateCaTransport(encoded,profile){const expectedDigest=profile.corporateCa.bundleSha256;if(expectedDigest===null){if(encoded!==void 0){fail2("corporate CA transport must be absent when the profile has no CA digest")}return null}if(typeof encoded!=="string"||encoded.length===0||encoded.length>Math.ceil(MANAGED_STARTUP_CA_MAX_BYTES/3)*4||!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u.test(encoded)){fail2("corporate CA transport must be canonical standard base64")}const bytes=import_node_buffer4.Buffer.from(encoded,"base64");if(bytes.toString("base64")!==encoded){fail2("corporate CA transport must be canonical standard base64")}validateCorporateCaBytes(bytes);const actualDigest=(0,import_node_crypto4.createHash)("sha256").update(bytes).digest("hex");if(actualDigest!==expectedDigest){fail2("corporate CA bundle does not match the profile SHA-256 digest")}return bytes}function readCanonicalProfile(profilePath,runtime){const bytes=readSecureFile(profilePath,MANAGED_STARTUP_PROFILE_MAX_BYTES,runtime);let raw;try{raw=UTF8_DECODER2.decode(bytes)}catch{fail2(`${profilePath} is not valid UTF-8`)}let parsed;try{parsed=JSON.parse(raw)}catch{fail2(`${profilePath} is not valid JSON`)}let profile;try{profile=validateManagedStartupProfile(parsed)}catch(error){fail2(`${profilePath} is invalid: ${error.message}`)}if(serializeManagedStartupProfile(profile)!==raw){fail2(`${profilePath} is not a canonical managed startup profile`)}return{profile,fingerprint:fingerprintManagedStartupProfile(profile)}}function validateGeneration(stateDirectory,control,runtime,expectedAgent){if(!GENERATION_RE.test(control.generation)){fail2("state control names an invalid generation")}const directory=import_node_path.default.join(stateDirectory,control.generation);requireSecureDirectory(directory,runtime,true);const entries=import_node_fs.default.readdirSync(directory).sort();if(entries.some(entry=>entry!=="profile.json"&&entry!=="corporate-ca.pem")||!entries.includes("profile.json")){fail2(`${directory} contains missing or unsupported state files`)}const profilePath=import_node_path.default.join(directory,"profile.json");const{profile,fingerprint}=readCanonicalProfile(profilePath,runtime);if(fingerprint!==control.fingerprint){fail2(`${directory} does not match its recorded profile fingerprint`)}if(expectedAgent!==void 0&&profile.agent!==expectedAgent){fail2(`managed startup profile targets ${profile.agent}, expected ${expectedAgent}`)}const caPath=import_node_path.default.join(directory,"corporate-ca.pem");let corporateCaPath=null;if(profile.corporateCa.bundleSha256===null){if(entries.includes("corporate-ca.pem")){fail2(`${directory} contains a CA bundle that is absent from the profile`)}}else{if(!entries.includes("corporate-ca.pem")){fail2(`${directory} is missing the CA bundle recorded by the profile`)}const caBytes=readSecureFile(caPath,MANAGED_STARTUP_CA_MAX_BYTES,runtime);validateCorporateCaBytes(caBytes);if((0,import_node_crypto4.createHash)("sha256").update(caBytes).digest("hex")!==profile.corporateCa.bundleSha256){fail2(`${directory} contains a CA bundle with the wrong SHA-256 digest`)}corporateCaPath=caPath}return{directory,profilePath,corporateCaPath,profile,fingerprint}}function validateDisposableDirectory(target2,runtime){requireSecureDirectory(target2,runtime,true);const entries=import_node_fs.default.readdirSync(target2);if(entries.length>2||entries.some(entry=>entry!=="profile.json"&&entry!=="corporate-ca.pem")){fail2(`${target2} is not a recognized disposable generation`)}for(const entry of entries){const file=import_node_path.default.join(target2,entry);const stat=import_node_fs.default.lstatSync(file);requireSecureRegularFileStat(stat,file,runtime)}}function discardDirectory(target2,runtime){validateDisposableDirectory(target2,runtime);import_node_fs.default.rmSync(target2,{recursive:true})}function discardDirectoryIfPresent(target2,runtime){try{import_node_fs.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return false;fail2(`could not inspect disposable generation ${target2}`)}discardDirectory(target2,runtime);return true}function unlinkSecureControlOrTemp(target2,runtime){const stat=import_node_fs.default.lstatSync(target2);requireSecureRegularFileStat(stat,target2,runtime);if(stat.size>MAX_CONTROL_FILE_BYTES){fail2(`${target2} exceeds the state-control size limit`)}import_node_fs.default.unlinkSync(target2)}function listStateEntries(stateDirectory){const entries=import_node_fs.default.readdirSync(stateDirectory).sort();if(entries.length>MAX_STATE_ENTRIES){fail2(`state directory exceeds ${String(MAX_STATE_ENTRIES)} entries`)}return entries}function unlinkRecoverableControlTemp(stateDirectory,entry,runtime){const temporary=import_node_path.default.join(stateDirectory,entry);const stat=import_node_fs.default.lstatSync(temporary);if(stat.nlink===1){unlinkSecureControlOrTemp(temporary,runtime);return}const basename=entry.startsWith(".committed.json-")?"committed.json":entry.startsWith(".pending.json-")?"pending.json":null;const target2=basename===null?null:import_node_path.default.join(stateDirectory,basename);let targetStat=null;try{targetStat=target2===null?null:import_node_fs.default.lstatSync(target2)}catch{fail2(`refused to remove an unpaired atomic-control temporary file: ${temporary}`)}if(stat.nlink!==2||targetStat===null||stat.dev!==targetStat.dev||stat.ino!==targetStat.ino||!stat.isFile()||stat.isSymbolicLink()||modeOf(stat)!==STATE_FILE_MODE||stat.size<1||stat.size>MAX_CONTROL_FILE_BYTES){fail2(`refused to remove an unpaired atomic-control temporary file: ${temporary}`)}requireOwner(stat,temporary,runtime);requireOwner(targetStat,target2,runtime);import_node_fs.default.unlinkSync(temporary)}function cleanAtomicTemps(stateDirectory,entries,runtime){let changed=false;for(const entry of entries){const target2=import_node_path.default.join(stateDirectory,entry);if(PREPARE_TEMP_RE.test(entry)){discardDirectory(target2,runtime);changed=true}else if(CONTROL_TEMP_RE.test(entry)){unlinkRecoverableControlTemp(stateDirectory,entry,runtime);changed=true}}if(changed)syncDirectory(stateDirectory)}function requireKnownStateEntries(stateDirectory,entries){for(const entry of entries){if(entry==="committed.json"||entry==="pending.json"||GENERATION_RE.test(entry)||PREPARE_TEMP_RE.test(entry)||CONTROL_TEMP_RE.test(entry)){continue}fail2(`${stateDirectory} contains unsupported state component ${entry}`)}}function discardGenerationsExcept(stateDirectory,keepGeneration,runtime){for(const entry of listStateEntries(stateDirectory)){if(GENERATION_RE.test(entry)&&entry!==keepGeneration){discardDirectoryIfPresent(import_node_path.default.join(stateDirectory,entry),runtime)}}}function optionalStateControl(stateDirectory,basename,runtime){const target2=import_node_path.default.join(stateDirectory,basename);try{import_node_fs.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return null;fail2(`could not inspect ${target2}`)}return parseStateControl(target2,runtime)}function removePendingControl(stateDirectory,runtime){try{unlinkSecureControlOrTemp(import_node_path.default.join(stateDirectory,"pending.json"),runtime)}catch(error){if(error.code==="ENOENT")return;throw error}syncDirectory(stateDirectory)}function stateControlsMatch(left,right){return left.fingerprint===right.fingerprint&&left.generation===right.generation}function recoverCommittedState(stateDirectory,committedControl,pendingControl,requested,expectedAgent,runtime){const committed=validateGeneration(stateDirectory,committedControl,runtime,expectedAgent);if(pendingControl)removePendingControl(stateDirectory,runtime);discardGenerationsExcept(stateDirectory,committedControl.generation,runtime);syncDirectory(stateDirectory);if(!stateControlsMatch(committedControl,requested)){fail2("a different startup profile is already committed; recreate the sandbox to change it")}return committed}function recoverState(stateDirectory,requested,expectedAgent,runtime){const initialEntries=listStateEntries(stateDirectory);requireKnownStateEntries(stateDirectory,initialEntries);cleanAtomicTemps(stateDirectory,initialEntries,runtime);const initiallyCommittedControl=optionalStateControl(stateDirectory,"committed.json",runtime);const pendingControl=optionalStateControl(stateDirectory,"pending.json",runtime);const committedAfterPendingRead=optionalStateControl(stateDirectory,"committed.json",runtime);const committedControl=committedAfterPendingRead??initiallyCommittedControl;if(committedControl){return{committed:recoverCommittedState(stateDirectory,committedControl,pendingControl,requested,expectedAgent,runtime),pending:null}}if(pendingControl){if(stateControlsMatch(pendingControl,requested)){const pending=validateGeneration(stateDirectory,pendingControl,runtime,expectedAgent);const committedAfterPendingValidation=optionalStateControl(stateDirectory,"committed.json",runtime);if(committedAfterPendingValidation){return{committed:recoverCommittedState(stateDirectory,committedAfterPendingValidation,pendingControl,requested,expectedAgent,runtime),pending:null}}discardGenerationsExcept(stateDirectory,pendingControl.generation,runtime);return{committed:null,pending}}fail2("a different startup profile is already pending; wait for it to commit or recreate")}return{committed:null,pending:null}}function createGeneration(stateDirectory,control,profileJson,corporateCa,runtime){const temporaryName=`.prepare-${String(process.pid)}-${randomToken()}`;const temporary=import_node_path.default.join(stateDirectory,temporaryName);const generation=import_node_path.default.join(stateDirectory,control.generation);let renameAttempted=false;try{import_node_fs.default.mkdirSync(temporary,{mode:STATE_DIRECTORY_MODE});import_node_fs.default.chownSync(temporary,runtime.rootUid,runtime.rootGid);import_node_fs.default.chmodSync(temporary,STATE_DIRECTORY_MODE);writeSecureNewFile(import_node_path.default.join(temporary,"profile.json"),profileJson,runtime);if(corporateCa){writeSecureNewFile(import_node_path.default.join(temporary,"corporate-ca.pem"),corporateCa,runtime)}syncDirectory(temporary);renameAttempted=true;import_node_fs.default.renameSync(temporary,generation);syncDirectory(stateDirectory)}catch(error){try{import_node_fs.default.lstatSync(temporary);discardDirectory(temporary,runtime)}catch{}if(error instanceof ManagedStartupApplicationError)throw error;if(renameAttempted&&(error.code==="EEXIST"||error.code==="ENOTEMPTY")){return validateGeneration(stateDirectory,control,runtime)}fail2(`could not atomically prepare generation ${control.generation}`)}return validateGeneration(stateDirectory,control,runtime)}function toPrepared(status,stateDirectory,generation,expectedAgent){return{status,stateDirectory,generationDirectory:generation.directory,profilePath:generation.profilePath,corporateCaPath:generation.corporateCaPath,fingerprint:generation.fingerprint,expectedAgent,profile:generation.profile}}function prepareManagedStartupApplication(input,testRuntime){const runtime=runtimeFor(testRuntime);requireContainerRoot();let profile;try{profile=decodeManagedStartupProfile(input.encodedProfile)}catch(error){fail2(error.message)}if(profile.agent!==input.expectedAgent){fail2(`managed startup profile targets ${profile.agent}, expected ${input.expectedAgent}`)}const corporateCa=validateManagedStartupCorporateCaTransport(input.corporateCaB64,profile);const profileJson=serializeManagedStartupProfile(profile);const control=stateControl(fingerprintManagedStartupProfile(profile));const stateDirectory=ensureStateDirectory(input.stateDirectory,runtime);const recovered=recoverState(stateDirectory,control,input.expectedAgent,runtime);if(recovered.committed){return toPrepared("already-committed",stateDirectory,recovered.committed,input.expectedAgent)}if(recovered.pending){return toPrepared("prepared",stateDirectory,recovered.pending,input.expectedAgent)}const generation=createGeneration(stateDirectory,control,profileJson,corporateCa,runtime);const publication=publishStateControlIfAbsent(stateDirectory,"pending.json",control,runtime);if(publication.control.fingerprint!==control.fingerprint||publication.control.generation!==control.generation){discardDirectoryIfPresent(generation.directory,runtime);syncDirectory(stateDirectory);fail2("a different startup profile won the pending-state transaction")}const committedAfterPublication=optionalStateControl(stateDirectory,"committed.json",runtime);if(committedAfterPublication){if(committedAfterPublication.fingerprint!==control.fingerprint||committedAfterPublication.generation!==control.generation){if(publication.created){removePendingControl(stateDirectory,runtime);discardDirectoryIfPresent(generation.directory,runtime);syncDirectory(stateDirectory)}fail2("a different startup profile committed during pending-state publication")}const committedGeneration=validateGeneration(stateDirectory,committedAfterPublication,runtime,input.expectedAgent);removePendingControl(stateDirectory,runtime);discardGenerationsExcept(stateDirectory,committedAfterPublication.generation,runtime);return toPrepared("already-committed",stateDirectory,committedGeneration,input.expectedAgent)}const activeGeneration=publication.created?generation:validateGeneration(stateDirectory,publication.control,runtime,input.expectedAgent);return toPrepared("prepared",stateDirectory,activeGeneration,input.expectedAgent)}function validatePreparedHandle(handle){if(!import_node_path.default.isAbsolute(handle.stateDirectory)||!SHA256_RE2.test(handle.fingerprint)||handle.generationDirectory!==import_node_path.default.join(handle.stateDirectory,`generation-${handle.fingerprint}`)||handle.profilePath!==import_node_path.default.join(handle.generationDirectory,"profile.json")||handle.corporateCaPath!==null&&handle.corporateCaPath!==import_node_path.default.join(handle.generationDirectory,"corporate-ca.pem")){fail2("prepared startup handle is malformed")}return stateControl(handle.fingerprint)}function commitManagedStartupApplication(prepared,testRuntime){const runtime=runtimeFor(testRuntime);requireContainerRoot();const requested=validatePreparedHandle(prepared);const stateDirectory=ensureStateDirectory(prepared.stateDirectory,runtime);const committedControl=optionalStateControl(stateDirectory,"committed.json",runtime);if(committedControl){if(committedControl.fingerprint!==requested.fingerprint||committedControl.generation!==requested.generation){fail2("a different startup profile is already committed")}const generation2=validateGeneration(stateDirectory,committedControl,runtime,prepared.expectedAgent);return{...toPrepared("already-committed",stateDirectory,generation2,prepared.expectedAgent),status:"committed"}}const pendingControl=optionalStateControl(stateDirectory,"pending.json",runtime);if(!pendingControl||pendingControl.fingerprint!==requested.fingerprint||pendingControl.generation!==requested.generation){fail2("the prepared startup generation is not the active pending generation")}const generation=validateGeneration(stateDirectory,pendingControl,runtime,prepared.expectedAgent);const publication=publishStateControlIfAbsent(stateDirectory,"committed.json",pendingControl,runtime);if(publication.control.fingerprint!==requested.fingerprint||publication.control.generation!==requested.generation){fail2("a different startup profile won the committed-state transaction")}removePendingControl(stateDirectory,runtime);discardGenerationsExcept(stateDirectory,publication.control.generation,runtime);syncDirectory(stateDirectory);return{...toPrepared("already-committed",stateDirectory,generation,prepared.expectedAgent),status:"committed"}}var SHIPPED_AGENT_SET=new Set(MANAGED_STARTUP_AGENTS);var DEFAULT_DEPENDENCIES={prepareApplication:input=>prepareManagedStartupApplication(input),commitApplication:prepared=>commitManagedStartupApplication(prepared)};var ManagedStartupCoordinatorError=class extends Error{constructor(message){super(`Managed startup coordination failed: ${message}`);this.name="ManagedStartupCoordinatorError"}};function fail3(message){throw new ManagedStartupCoordinatorError(message)}function createAdapterRegistry(adapters2){const byAgent=new Map;for(const adapter of adapters2){if(typeof adapter!=="object"||adapter===null||!SHIPPED_AGENT_SET.has(adapter.agent)||typeof adapter.apply!=="function"){fail3("every adapter must identify one shipped agent and provide an apply function")}if(byAgent.has(adapter.agent)){fail3(`duplicate adapter registered for ${adapter.agent}`)}byAgent.set(adapter.agent,adapter)}const missing=MANAGED_STARTUP_AGENTS.filter(agent=>!byAgent.has(agent));if(missing.length>0){fail3(`missing adapter for ${missing.join(", ")}`)}if(byAgent.size!==MANAGED_STARTUP_AGENTS.length){fail3("adapter registry must contain exactly the shipped agents")}return Object.freeze(Object.fromEntries(MANAGED_STARTUP_AGENTS.map(agent=>{const adapter=byAgent.get(agent);if(!adapter)fail3(`missing adapter for ${agent}`);return[agent,adapter]})))}function requirePreparedIdentity(prepared,requestedAgent){if(prepared.expectedAgent!==requestedAgent||prepared.profile.agent!==requestedAgent){fail3(`prepared profile targets ${prepared.profile.agent}, expected ${requestedAgent}`)}}function adapterContext(prepared){return Object.freeze({agent:prepared.profile.agent,profile:prepared.profile,fingerprint:prepared.fingerprint,generationDirectory:prepared.generationDirectory,profilePath:prepared.profilePath,corporateCaPath:prepared.corporateCaPath})}async function coordinateManagedStartupApplication(input,adapters2,dependencies=DEFAULT_DEPENDENCIES){const registry=createAdapterRegistry(adapters2);const prepared=await dependencies.prepareApplication(input);requirePreparedIdentity(prepared,input.expectedAgent);if(prepared.status==="already-committed"){return{adapterApplied:false,application:await dependencies.commitApplication(prepared)}}const adapter=registry[prepared.profile.agent];if(adapter.agent!==prepared.profile.agent){fail3(`adapter registry cross-dispatch detected for ${prepared.profile.agent}`)}await adapter.apply(adapterContext(prepared));return{adapterApplied:true,application:await dependencies.commitApplication(prepared)}}var import_node_crypto5=require("node:crypto");var MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION=1;var MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES=320*1024;var MAX_CORPORATE_CA_ENCODED_BYTES=4*Math.ceil(128*1024/3);var SHA256_RE3=/^[a-f0-9]{64}$/u;var STANDARD_BASE64_RE2=/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u;var MCP_SHADOW_DIAGNOSTICS_ENV="NEMOCLAW_MCP_SHADOW_DIAGNOSTICS";var MANAGED_STARTUP_APPLICATION_RUNTIME_ENV_KEYS=Object.freeze(MANAGED_STARTUP_PROFILE_DEFERRED_RUNTIME_INPUTS.openclaw.filter(({admission,owner})=>admission==="managed-launch-forwarded"&&owner==="application-environment").map(({input})=>input));function selectManagedStartupApplicationRuntimeEnvironment(environment){const selected={};for(const name of MANAGED_STARTUP_APPLICATION_RUNTIME_ENV_KEYS){const value=environment[name];if(name===MCP_SHADOW_DIAGNOSTICS_ENV){if(value?.trim()==="1")selected[name]="1";continue}if(value!==void 0)selected[name]=value}return Object.freeze(selected)}function fail4(message){throw new Error(`Managed startup root application request is invalid: ${message}`)}function isManagedStartupRootApplyAgent(value){return typeof value==="string"&&MANAGED_STARTUP_AGENTS.includes(value)}function exactAgent(value){if(isManagedStartupRootApplyAgent(value))return value;return fail4("agent is unsupported")}function createManagedStartupRootApplyRequest(input){const agent=exactAgent(input.agent);if(input.encodedProfile.length===0||input.encodedProfile.length>MANAGED_STARTUP_PROFILE_MAX_ENCODED_BYTES){fail4("encoded profile exceeds its bounded transport")}const profile=decodeManagedStartupProfile(input.encodedProfile);if(profile.agent!==agent){fail4(`profile targets ${profile.agent}, expected ${agent}`)}const corporateCaB64=input.corporateCaB64??null;if(corporateCaB64!==null&&(corporateCaB64.length===0||corporateCaB64.length>MAX_CORPORATE_CA_ENCODED_BYTES||!STANDARD_BASE64_RE2.test(corporateCaB64)||Buffer.from(corporateCaB64,"base64").toString("base64")!==corporateCaB64)){fail4("corporate CA is not canonical bounded base64")}if(profile.corporateCa.bundleSha256!==null!==(corporateCaB64!==null)){fail4("corporate CA transport does not match the profile")}if(corporateCaB64!==null&&(0,import_node_crypto5.createHash)("sha256").update(Buffer.from(corporateCaB64,"base64")).digest("hex")!==profile.corporateCa.bundleSha256){fail4("corporate CA does not match the profile digest")}return Object.freeze({schemaVersion:MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION,agent,encodedProfile:input.encodedProfile,profileFingerprint:fingerprintManagedStartupProfile(profile),corporateCaB64})}function serializeManagedStartupRootApplyRequest(request){const normalized=createManagedStartupRootApplyRequest({agent:request.agent,encodedProfile:request.encodedProfile,...request.corporateCaB64===null?{}:{corporateCaB64:request.corporateCaB64}});if(request.schemaVersion!==MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION||request.profileFingerprint!==normalized.profileFingerprint||!SHA256_RE3.test(request.profileFingerprint)){fail4("schema version or profile fingerprint is invalid")}const serialized=`${JSON.stringify({agent:normalized.agent,corporateCaB64:normalized.corporateCaB64,encodedProfile:normalized.encodedProfile,profileFingerprint:normalized.profileFingerprint,schemaVersion:normalized.schemaVersion})} -`;if(Buffer.byteLength(serialized,"utf8")>MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES){fail4("serialized request exceeds its bounded transport")}return serialized}function parseManagedStartupRootApplyRequest(text){if(text.length===0||Buffer.byteLength(text,"utf8")>MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES){fail4("serialized request is empty or too large")}let parsed;try{parsed=JSON.parse(text)}catch{fail4("serialized request is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail4("serialized request must be an object")}const record2=parsed;const expectedKeys=["agent","corporateCaB64","encodedProfile","profileFingerprint","schemaVersion"];if(Object.keys(record2).sort().join(",")!==expectedKeys.sort().join(",")||record2.schemaVersion!==MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION||typeof record2.encodedProfile!=="string"||typeof record2.profileFingerprint!=="string"||record2.corporateCaB64!==null&&typeof record2.corporateCaB64!=="string"){fail4("serialized request has an invalid schema")}const request=createManagedStartupRootApplyRequest({agent:exactAgent(record2.agent),encodedProfile:record2.encodedProfile,...record2.corporateCaB64===null?{}:{corporateCaB64:record2.corporateCaB64}});if(record2.profileFingerprint!==request.profileFingerprint||!SHA256_RE3.test(record2.profileFingerprint)){fail4("profile fingerprint does not match the encoded profile")}if(serializeManagedStartupRootApplyRequest(request)!==text){fail4("serialized request is not canonical")}return request}var import_node_crypto6=require("node:crypto");var import_node_fs2=__toESM(require("node:fs"));var import_node_path2=__toESM(require("node:path"));var MANAGED_HERMES_STATE_ROOT="/sandbox/.hermes";var MANAGED_OPENCLAW_STATE_ROOT="/sandbox/.openclaw";var HERMES_STATE_VOLUME_NAME_PREFIX="nemoclaw-hermes-state-v1";var OPENCLAW_STATE_VOLUME_NAME_PREFIX="nemoclaw-openclaw-state-v1";var MANAGED_AGENT_STATE_ROOTS=Object.freeze({openclaw:Object.freeze([Object.freeze({mountTarget:MANAGED_OPENCLAW_STATE_ROOT,resourceIdentity:sandboxName=>`${OPENCLAW_STATE_VOLUME_NAME_PREFIX}-${sandboxName}`,ownershipLabels:(sandboxName,mountTarget)=>Object.freeze({"io.nvidia.nemoclaw.openclaw-state.managed":"true","io.nvidia.nemoclaw.openclaw-state.schema":"1","io.nvidia.nemoclaw.openclaw-state.sandbox":sandboxName,"io.nvidia.nemoclaw.openclaw-state.target":mountTarget}),uidAuthority:"agent",gidAuthority:"agent",mode:1528,readWrite:true})]),hermes:Object.freeze([Object.freeze({mountTarget:MANAGED_HERMES_STATE_ROOT,resourceIdentity:sandboxName=>`${HERMES_STATE_VOLUME_NAME_PREFIX}-${sandboxName}`,ownershipLabels:(sandboxName,mountTarget)=>Object.freeze({"io.nvidia.nemoclaw.hermes-state.managed":"true","io.nvidia.nemoclaw.hermes-state.schema":"1","io.nvidia.nemoclaw.hermes-state.sandbox":sandboxName,"io.nvidia.nemoclaw.hermes-state.target":mountTarget}),uidAuthority:"agent",gidAuthority:"agent",mode:2040,readWrite:true})]),"langchain-deepagents-code":Object.freeze([]),pi:Object.freeze([])});function managedStartupStateRootMountTargets(agent){return Object.freeze(MANAGED_AGENT_STATE_ROOTS[agent].map(({mountTarget})=>mountTarget))}var MANAGED_AGENT_WORKSPACE_ROOTS=Object.freeze({openclaw:Object.freeze({uidAuthority:"agent",gidAuthority:"agent",mode:493}),hermes:Object.freeze({uidAuthority:"agent",gidAuthority:"agent",mode:493}),"langchain-deepagents-code":Object.freeze({uidAuthority:"root",gidAuthority:"agent",mode:1021}),pi:Object.freeze({uidAuthority:"agent",gidAuthority:"agent",mode:493})});function exactAgentIdentity(value,label){if(!Number.isSafeInteger(value)||value<0||value>2147483647){throw new Error(`Managed startup state-root ${label} authority is invalid.`)}return value}function managedStartupWorkspaceRoot(input){const uid=exactAgentIdentity(input.agentIdentity.uid,"workspace UID");const gid=exactAgentIdentity(input.agentIdentity.gid,"workspace GID");const declaration=MANAGED_AGENT_WORKSPACE_ROOTS[input.agent];return Object.freeze({uid:declaration.uidAuthority==="root"?0:uid,gid,mode:declaration.mode})}var TRANSACTION_SCHEMA_VERSION=1;var MAX_TRANSACTION_FILES=128;var MAX_TRANSACTION_FILE_BYTES=8*1024*1024;var MAX_TRANSACTION_TOTAL_BYTES=32*1024*1024;var MAX_MANIFEST_BYTES=256*1024;var MAX_COMMIT_RECEIPT_BYTES=4096;var MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES=4*Math.ceil(MAX_TRANSACTION_TOTAL_BYTES/3)+MAX_MANIFEST_BYTES*2;var TRANSACTION_PARENT_DIRECTORY_MODE=493;var TRANSACTION_DIRECTORY_MODE=448;var TRANSACTION_FILE_MODE=256;var ATOMIC_TEMPORARY_FILE_MODE=384;var MANAGED_STARTUP_SHARED_TRANSACTION_DIRECTORY="/var/lib/nemoclaw/managed-startup-shared-state-transaction-v1";var MANAGED_STARTUP_SHARED_ROLLBACK_RECEIPT_DIRECTORY="/run/nemoclaw/managed-startup-shared-rollback-receipt-v1";var MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_DIRECTORY="/var/lib/nemoclaw/managed-startup-shared-state-commit-v1";var MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE="receipt.json";function fail5(message){throw new Error(`Managed startup shared-state transaction failed: ${message}`)}function resolveOptions(options={}){const sandboxRoot=import_node_path2.default.resolve(options.sandboxRoot??"/sandbox");const transactionDirectory=import_node_path2.default.resolve(options.transactionDirectory??MANAGED_STARTUP_SHARED_TRANSACTION_DIRECTORY);const commitReceiptDirectory=import_node_path2.default.resolve(options.commitReceiptDirectory??(options.transactionDirectory?import_node_path2.default.join(import_node_path2.default.dirname(transactionDirectory),import_node_path2.default.basename(MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_DIRECTORY)):MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_DIRECTORY));if(transactionDirectory===sandboxRoot||transactionDirectory.startsWith(`${sandboxRoot}${import_node_path2.default.sep}`)||commitReceiptDirectory===sandboxRoot||commitReceiptDirectory.startsWith(`${sandboxRoot}${import_node_path2.default.sep}`)||import_node_path2.default.dirname(commitReceiptDirectory)!==import_node_path2.default.dirname(transactionDirectory)||commitReceiptDirectory===transactionDirectory){fail5("transaction and commit receipts require distinct paths outside sandbox-shared state")}const bootstrapIdentity=options.bootstrapIdentity??null;if(bootstrapIdentity!==null&&!/^[a-f0-9]{64}$/u.test(bootstrapIdentity)){fail5("bootstrap identity must encode 32 lowercase-hex bytes")}return{sandboxRoot,transactionParentDirectory:import_node_path2.default.dirname(transactionDirectory),transactionDirectory,backupDirectory:import_node_path2.default.join(transactionDirectory,"backups"),manifestFile:import_node_path2.default.join(transactionDirectory,"manifest.json"),commitReceiptDirectory,commitReceiptFile:import_node_path2.default.join(commitReceiptDirectory,MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE),trustedUid:options.trustedUid??0,trustedGid:options.trustedGid??0,readOnlyReceipt:options.readOnlyReceipt??false,bootstrapIdentity}}function modeOf2(stat){if(typeof stat.mode==="bigint"){return Number(stat.mode&0o7777n)}return stat.mode&4095}function requireTransactionIdentity(options){const expectedUid=options.readOnlyReceipt?0:options.trustedUid;const expectedGid=options.readOnlyReceipt?0:options.trustedGid;if(process.geteuid?.()!==expectedUid||process.getegid?.()!==expectedGid){fail5("transaction control requires the trusted effective identity")}}function pathExistsNoFollow(target2){try{import_node_fs2.default.lstatSync(target2);return true}catch(error){if(error.code==="ENOENT")return false;fail5(`could not inspect ${target2}`)}}function requireDirectory(target2,options,expectedMode=null){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch{fail5(`required directory is missing: ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`required directory is unsafe: ${target2}`)}if(expectedMode!==null&&(stat.uid!==options.trustedUid||stat.gid!==options.trustedGid||modeOf2(stat)!==expectedMode)){fail5(`${target2} must be ${options.trustedUid}:${options.trustedGid} mode ${expectedMode.toString(8)}`)}return stat}function requireTransactionBoundaries(options){requireDirectory(options.sandboxRoot,options);requireDirectory(options.transactionParentDirectory,options,TRANSACTION_PARENT_DIRECTORY_MODE)}function sameStableMetadata(left,right){return left.dev===right.dev&&left.ino===right.ino&&left.mode===right.mode&&left.nlink===right.nlink&&left.uid===right.uid&&left.gid===right.gid&&left.size===right.size&&left.mtimeNs===right.mtimeNs&&left.ctimeNs===right.ctimeNs}function readStableFile(target2,maxBytes){const noFollow=import_node_fs2.default.constants.O_NOFOLLOW;if(typeof noFollow!=="number")fail5("O_NOFOLLOW is unavailable");let descriptor;try{descriptor=import_node_fs2.default.openSync(target2,import_node_fs2.default.constants.O_RDONLY|noFollow)}catch{fail5(`could not safely open ${target2}`)}try{const before=import_node_fs2.default.fstatSync(descriptor,{bigint:true});if(!before.isFile()||before.nlink!==1n||before.size<0n||before.size>BigInt(maxBytes)){fail5(`refusing unsafe or oversized transaction file ${target2}`)}const bytes=Buffer.alloc(Number(before.size));let offset=0;while(offset!segment||segment==="."||segment==="..")){fail5(`unsafe transaction path ${JSON.stringify(value)}`)}return segments.join("/")}function absoluteTarget(relativePath,options){const safe=safeRelativePath(relativePath);const target2=import_node_path2.default.resolve(options.sandboxRoot,safe);if(!target2.startsWith(`${options.sandboxRoot}${import_node_path2.default.sep}`)){fail5(`transaction target escapes the sandbox root: ${relativePath}`)}return target2}function relativeTarget(target2,options){return safeRelativePath(import_node_path2.default.relative(options.sandboxRoot,target2))}function isDeclaredAgentStateRoot(expectedAgent,outputRoot,options){const relative=import_node_path2.default.relative(options.sandboxRoot,outputRoot).split(import_node_path2.default.sep).join("/");const canonicalTarget=import_node_path2.default.posix.join("/sandbox",relative);return managedStartupStateRootMountTargets(expectedAgent).includes(canonicalTarget)}function validateExistingAncestors(target2,expectedAgent,options){const relative=relativeTarget(target2,options);const sandboxStat=requireDirectory(options.sandboxRoot,options);const outputRoot=agentRoot(expectedAgent,options.sandboxRoot);if(target2!==outputRoot&&!target2.startsWith(`${outputRoot}${import_node_path2.default.sep}`)){fail5(`transaction target escapes the ${expectedAgent} state root: ${target2}`)}let current=options.sandboxRoot;let expectedDevice=sandboxStat.dev;const segments=relative.split("/").slice(0,-1);for(const segment of segments){current=import_node_path2.default.join(current,segment);let stat;try{stat=import_node_fs2.default.lstatSync(current)}catch(error){if(error.code==="ENOENT")return;fail5(`could not inspect transaction path ancestor ${current}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`transaction path ancestor is unsafe: ${current}`)}if(current===outputRoot&&isDeclaredAgentStateRoot(expectedAgent,outputRoot,options)){expectedDevice=stat.dev}else if(stat.dev!==expectedDevice){fail5(`transaction path crosses a nested filesystem mount: ${current}`)}}}function managedOutputDevice(expectedAgent,options){const sandboxStat=requireDirectory(options.sandboxRoot,options);const outputRoot=agentRoot(expectedAgent,options.sandboxRoot);let stat;try{stat=import_node_fs2.default.lstatSync(outputRoot)}catch(error){if(error.code==="ENOENT")return sandboxStat.dev;fail5(`could not inspect managed output root ${outputRoot}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`managed output root is unsafe: ${outputRoot}`)}if(!isDeclaredAgentStateRoot(expectedAgent,outputRoot,options)&&stat.dev!==sandboxStat.dev){fail5(`managed output root crosses a nested filesystem mount: ${outputRoot}`)}return stat.dev}function agentRoot(agent,sandboxRoot){switch(agent){case"openclaw":return import_node_path2.default.join(sandboxRoot,".openclaw");case"hermes":return import_node_path2.default.join(sandboxRoot,".hermes");case"langchain-deepagents-code":return import_node_path2.default.join(sandboxRoot,".deepagents");case"pi":return import_node_path2.default.join(sandboxRoot,".pi")}}function resolveUnderAgentRoot(root,relativePath){const safe=safeRelativePath(relativePath);const target2=import_node_path2.default.resolve(root,safe);if(!target2.startsWith(`${root}${import_node_path2.default.sep}`)){fail5(`managed output escapes the agent root: ${relativePath}`)}return target2}function renderTarget(root,agent,target2){if(agent==="openclaw"&&target2==="openclaw.json"){return import_node_path2.default.join(root,"openclaw.json")}const prefix=agent==="openclaw"?"~/.openclaw/":agent==="hermes"?"~/.hermes/":null;if(!prefix||!target2.startsWith(prefix)){fail5(`unsupported managed messaging render target ${JSON.stringify(target2)}`)}return resolveUnderAgentRoot(root,target2.slice(prefix.length))}function managedOutputTargets(profile,options){const root=agentRoot(profile.agent,options.sandboxRoot);const files=new Set;const directories=new Set([root]);switch(profile.agent){case"openclaw":files.add(import_node_path2.default.join(root,"openclaw.json"));break;case"hermes":files.add(import_node_path2.default.join(root,"config.yaml"));files.add(import_node_path2.default.join(root,".env"));files.add(import_node_path2.default.join(root,".config-hash"));break;case"langchain-deepagents-code":files.add(import_node_path2.default.join(root,"config.toml"));directories.add(import_node_path2.default.join(root,".state"));break;case"pi":directories.add(import_node_path2.default.join(root,"agent"));files.add(import_node_path2.default.join(root,"agent","models.json"));files.add(import_node_path2.default.join(root,"agent","settings.json"));break}if(profile.messaging.plan!==null){const plan=parseSandboxMessagingPlan(profile.messaging.plan,{agent:profile.agent});if(!plan)fail5("managed messaging plan is invalid");for(const render of selectEnabledMessagingAgentRender(plan)){if(typeof render.target!=="string")continue;files.add(renderTarget(root,profile.agent,render.target))}for(const step of selectEnabledPostAgentInstallBuildFiles(plan)){if(typeof step.value!=="object"||step.value===null){continue}const outputPath=step.value.path;if(typeof outputPath==="string"){files.add(resolveUnderAgentRoot(root,outputPath))}}}for(const file of files){let parent=import_node_path2.default.dirname(file);while(parent!==options.sandboxRoot&&parent.startsWith(`${root}${import_node_path2.default.sep}`)){directories.add(parent);if(parent===root)break;parent=import_node_path2.default.dirname(parent)}}return{files:[...files].sort(),directories:[...directories].sort((left,right)=>left.split(import_node_path2.default.sep).length-right.split(import_node_path2.default.sep).length)}}function snapshotFile(target2,index,expectedAgent,options){validateExistingAncestors(target2,expectedAgent,options);let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT"){return{receipt:{path:relativeTarget(target2,options),state:"absent"},bytes:null}}fail5(`could not inspect managed output ${target2}`)}if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1){fail5(`managed output is not a safe regular file: ${target2}`)}if(stat.dev!==managedOutputDevice(expectedAgent,options)){fail5(`managed output crosses a nested filesystem mount: ${target2}`)}const stable=readStableFile(target2,MAX_TRANSACTION_FILE_BYTES);const size=Number(stable.stat.size);const backup=`${String(index).padStart(3,"0")}.bin`;return{receipt:{path:relativeTarget(target2,options),state:"file",backup,sha256:(0,import_node_crypto6.createHash)("sha256").update(stable.bytes).digest("hex"),size,uid:Number(stable.stat.uid),gid:Number(stable.stat.gid),mode:Number(stable.stat.mode&0o7777n)},bytes:stable.bytes}}function snapshotDirectory(target2,expectedAgent,options){validateExistingAncestors(import_node_path2.default.join(target2,".receipt"),expectedAgent,options);let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT"){return{path:relativeTarget(target2,options),state:"absent"}}fail5(`could not inspect managed output directory ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`managed output directory is unsafe: ${target2}`)}if(stat.dev!==managedOutputDevice(expectedAgent,options)){fail5(`managed output directory crosses a nested filesystem mount: ${target2}`)}return{path:relativeTarget(target2,options),state:"directory",uid:stat.uid,gid:stat.gid,mode:modeOf2(stat)}}function atomicWriteTrustedFile(target2,contents,mode,uid,gid){const parent=import_node_path2.default.dirname(target2);const temporary=import_node_path2.default.join(parent,`.${import_node_path2.default.basename(target2)}.${(0,import_node_crypto6.randomBytes)(12).toString("hex")}`);let descriptor;try{descriptor=import_node_fs2.default.openSync(temporary,import_node_fs2.default.constants.O_CREAT|import_node_fs2.default.constants.O_EXCL|import_node_fs2.default.constants.O_WRONLY|import_node_fs2.default.constants.O_NOFOLLOW,384);import_node_fs2.default.writeFileSync(descriptor,contents);import_node_fs2.default.fchownSync(descriptor,uid,gid);import_node_fs2.default.fchmodSync(descriptor,mode);import_node_fs2.default.fsyncSync(descriptor);import_node_fs2.default.closeSync(descriptor);descriptor=void 0;import_node_fs2.default.renameSync(temporary,target2)}catch(error){if(descriptor!==void 0)import_node_fs2.default.closeSync(descriptor);try{import_node_fs2.default.unlinkSync(temporary)}catch{}fail5(`could not atomically write ${target2}: ${error.message}`)}}function fsyncDirectory(directory){const descriptor=import_node_fs2.default.openSync(directory,import_node_fs2.default.constants.O_RDONLY);try{import_node_fs2.default.fsyncSync(descriptor)}finally{import_node_fs2.default.closeSync(descriptor)}}function canonicalManifest(manifest){return`${JSON.stringify(manifest,null,2)} +except urllib.error.HTTPError as error: print(error.code)"`]}}],render:[{id:"discord-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.discord",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false},proxy:"{{discordProxyUrl}}",dmPolicy:"{{discord.allowedUsers.dmPolicy}}",allowFrom:"{{discord.allowedUsers.values}}"}}}}},{id:"discord-openclaw-guilds",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",when:"{{discord.hasGuilds}}",fragment:{path:"channels.discord",value:{groupPolicy:"allowlist",guilds:"{{discord.guilds}}"}}},{id:"discord-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.discord",value:{enabled:true}}},{id:"discord-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["NEMOCLAW_DISCORD_GUILD_IDS={{discord.guildIds.csv}}","DISCORD_ALLOWED_USERS={{discord.allowedUsers.csv}}","DISCORD_ALLOW_ALL_USERS={{discord.allowAllUsers}}"]},{id:"discord-hermes-config",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"discord",value:{require_mention:"{{discord.requireMention}}",free_response_channels:"",allowed_channels:"",auto_thread:true,reactions:true,channel_prompts:{}}}},{id:"discord-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.discord",value:{enabled:true}}}],runtime:{openclaw:{channelName:"discord",visibility:{configKeys:["discord"],logPatterns:["discord"]}}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/discord@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-j+fSHxbXA+DSxwbL8SvtsDNL6tvBX4+RmH+EerVW6dCbeIwSconXj6J/+AaN/mhzZI4g0jPPj30TUhdCRRbc2w==","2026.9.1":"sha512-qNmN2a8A9dET4igPp0RML171sEn8PDMyNCYNp/DqcJ4tn3XTHpacSOTkqBmv5yXTycJRC9rfFP8FT/SdW0Rldg=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/discord/-/discord-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/discord/-/discord-2026.9.1.tgz"},required:true}],hooks:[{id:"discord-hermes-status-health",phase:"status",handler:"discord.statusHealth",agents:["hermes"],outputs:[{id:"channelHealth",kind:"status"}]},{id:"discord-openclaw-bridge-health",phase:"health-check",handler:"discord.openclawBridgeHealth",agents:["openclaw"],onFailure:"abort"},{id:"discord-token-paste",phase:"enroll",handler:"discord.tokenPaste",outputs:[{id:"botToken",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"discord-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"serverId",kind:"config"},{id:"requireMention",kind:"config"},{id:"userId",kind:"config"}]}]};var googlechatManifest={schemaVersion:1,id:"googlechat",displayName:"Google Chat",description:"Google Chat (Chat API) bot messaging (experimental)",supportedAgents:["openclaw","hermes"],state:{hermes:["platforms/googlechat"]},auth:{mode:"token-paste"},inputs:[{id:"serviceAccount",kind:"secret",required:true,envKey:"GOOGLECHAT_SERVICE_ACCOUNT",allowLineBreaks:true,maskCap:40,formatHint:"Paste the entire service-account JSON key on one line (minified), or set GOOGLECHAT_SERVICE_ACCOUNT to the downloaded JSON, including line breaks.",maxTokenAttempts:3,prompt:{label:"Google Chat service account JSON",help:["\u2503 GOOGLE CHAT \u2014 service account key","\u2503","\u2503 Google Cloud Console \u2192 IAM & Admin \u2192 Service Accounts","\u2503 \u2192 your bot's SA \u2192 Keys \u2192 Add key \u2192 Create new key \u2192 JSON","\u2503","\u2503 A .json file downloads. Paste its contents below as ONE line (minified).","\u2503 Alternatively, GOOGLECHAT_SERVICE_ACCOUNT accepts the formatted JSON file contents.",""].join("\n")}},{id:"audienceType",kind:"config",required:false,envKey:"GOOGLECHAT_AUDIENCE_TYPE",statePath:"googlechatConfig.audienceType",validValues:["app-url","project-number"],defaultValue:"app-url"},{id:"audience",kind:"config",required:false,envKey:"GOOGLECHAT_AUDIENCE",statePath:"googlechatConfig.audience",prompt:{label:"Google Chat webhook audience",help:"Usually filled automatically from the public tunnel URL. For audienceType 'project-number', enter your GCP project number instead.",emptyValueMessage:"inbound webhook verification will be unconfigured"}},{id:"appPrincipal",kind:"config",required:false,envKey:"GOOGLECHAT_APP_PRINCIPAL",statePath:"googlechatConfig.appPrincipal",formatPattern:"^[0-9]{6,32}$",formatHint:"appPrincipal is the add-on's numeric OAuth client ID (uniqueId, ~21 digits), not an email.",prompt:{label:"Google Chat appPrincipal",help:[" Workspace account \u2192 leave blank, done."," Personal Gmail \u2192 needs the add-on's ~21-digit ID (not an email), stable across rebuilds.",""," If you already know it, paste it at the prompt and you're done."," If not, leave it blank \u2014 the first DM reveals it once the sandbox is live:",""," 1. Watch the gateway log:",' nemoclaw logs --follow | grep "unexpected add-on principal"'," 2. DM the bot once \u2014 it won't reply yet, that's expected. The log prints:"," unexpected add-on principal: "," 3. Save that and rebuild:"," GOOGLECHAT_APP_PRINCIPAL= nemoclaw channels add googlechat"," nemoclaw rebuild --yes"].join("\n"),emptyValueMessage:"Workspace accounts do not need it; personal accounts must set it later"}},{id:"allowFrom",kind:"config",required:false,envKey:"GOOGLECHAT_ALLOWED_USERS",statePath:"allowedIds.googlechat",prompt:{label:"Google Chat DM allowlist (comma-separated)",help:["Optional: restrict who can DM the bot."," OpenClaw: users/NNN (emails ignored)"," Hermes: email (users/NNN ignored)"," Blank: pairing mode (recommended) \u2014 OpenClaw's pairing reply shows your users/NNN"," Filling this switches DM policy to allowlist \u2014 a wrong-form entry is dropped silently, with no pairing code."].join("\n"),emptyValueMessage:"bot will require manual pairing"}},{id:"projectId",kind:"config",required:false,envKey:"GOOGLE_CHAT_PROJECT_ID",statePath:"googlechatConfig.projectId",prompt:{label:"Google Chat GCP project ID (Hermes Pub/Sub pull)",help:"The Google Cloud project that owns the Pub/Sub subscription Hermes pulls Chat events from. OpenClaw ignores this.",emptyValueMessage:"required for the Hermes Google Chat channel"}},{id:"subscriptionName",kind:"config",required:false,envKey:"GOOGLE_CHAT_SUBSCRIPTION_NAME",statePath:"googlechatConfig.subscriptionName",prompt:{label:"Google Chat Pub/Sub subscription (projects/

/subscriptions/)",help:["The pull subscription bound to the Chat events topic. Hermes pulls from it over the Pub/Sub REST API; the gateway-minted token is scoped to both chat.bot and pubsub."," Its topic must grant roles/pubsub.publisher to the app's push account:"," Interactive features service-@gcp-sa-gsuiteaddons.iam.gserviceaccount.com"," Classic bot chat-api-push@system.gserviceaccount.com"," Shown at Chat API \u2192 Configuration \u2192 Connection settings"," Missing it channel connects, no event arrives, Chat says the bot is not responding"].join("\n"),emptyValueMessage:"required for the Hermes Google Chat channel"}}],credentials:[],policyPresets:[{name:"googlechat",policyKeys:["googlechat"],agentPolicyKeys:{hermes:["googlechat_hermes"]}}],render:[{id:"googlechat-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.googlechat",value:{enabled:true,serviceAccount:{},audienceType:"{{googlechatConfig.audienceType}}",audience:"{{googlechatConfig.audience}}",appPrincipal:"{{googlechatConfig.appPrincipal}}",webhookPath:"/googlechat",healthMonitor:{enabled:false},dmPolicy:"{{allowedIds.googlechat.dmPolicy}}",allowFrom:"{{allowedIds.googlechat.values}}"}}},{id:"googlechat-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.googlechat",value:{enabled:true}}},{id:"googlechat-openclaw-gateway-reload-off",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"gateway.reload",value:{mode:"off"}}},{id:"googlechat-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["GOOGLE_CHAT_PROJECT_ID={{googlechatConfig.projectId}}","GOOGLE_CHAT_SUBSCRIPTION_NAME={{googlechatConfig.subscriptionName}}","GOOGLE_CHAT_ALLOWED_USERS={{allowedIds.googlechat.csv}}"]},{id:"googlechat-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.google_chat",value:{enabled:true}}}],runtime:{openclaw:{channelName:"googlechat",visibility:{configKeys:["googlechat"],logPatterns:["googlechat"]},nodePreloads:[{module:"googlechat-outbound-auth",injectInto:["boot"],optional:false,installMessage:"[channels] Installing Google Chat outbound-auth patch (gateway-minted bearer)",installedMessage:"[channels] Google Chat outbound-auth patch installed (NODE_OPTIONS updated)"}],secretScans:[{path:"/sandbox/.openclaw/openclaw.json",pattern:"-----BEGIN (?:RSA )?PRIVATE KEY-----",message:"[SECURITY] Google Chat service account private key leaked into {path} - refusing to serve",exitCode:78}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/googlechat@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-LUO8Lg07IhzJfEzxn+GSij8WMS/uX3hTmv0SydTy/0fKSN7iZksf74TaZg50kOoBi8FzeqeXo/zoGVuk0Mj1Ig==","2026.9.1":"sha512-Q5VTAJpfcrI7BSEw5Ugq3wf7JEg5QhTBwpi+BByGbfZsTTVjwZc7OIvNbKsVTh16I5/EWqHEnD+0WNeHqsteqw=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/googlechat/-/googlechat-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/googlechat/-/googlechat-2026.9.1.tgz"},required:true},{id:"hermesGooglePubsubPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"google-cloud-pubsub==2.39.0",required:true},{id:"hermesGoogleApiClientPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"google-api-python-client==2.194.0",required:true},{id:"hermesGoogleAuthPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"google-auth==2.55.1",required:true}],hooks:[{id:"googlechat-tunnel-audience-gate",phase:"enroll",handler:"googlechat.tunnelAudienceGate",agents:["openclaw"],inputs:["audienceType","audience"],outputs:[{id:"audience",kind:"config"}],onFailure:"skip-channel"},{id:"googlechat-service-account",phase:"enroll",handler:"googlechat.tokenPaste",outputs:[{id:"serviceAccount",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"googlechat-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"allowFrom",kind:"config"}]},{id:"googlechat-openclaw-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["openclaw"],outputs:[{id:"appPrincipal",kind:"config"}]},{id:"googlechat-hermes-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["hermes"],outputs:[{id:"projectId",kind:"config"},{id:"subscriptionName",kind:"config"}]}]};var slackManifest={schemaVersion:1,id:"slack",displayName:"Slack",description:"Slack bot messaging",supportedAgents:["openclaw","hermes"],state:{hermes:["platforms/slack"]},auth:{mode:"token-paste"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"SLACK_BOT_TOKEN",formatPattern:"^xoxb-[A-Za-z0-9_-]+$",formatHint:"Slack bot tokens start with 'xoxb-' (e.g. xoxb---).",prompt:{label:"Slack Bot Token",help:"Slack API \u2192 Your Apps \u2192 OAuth & Permissions \u2192 Bot User OAuth Token (xoxb-...)."}},{id:"appToken",kind:"secret",required:true,envKey:"SLACK_APP_TOKEN",formatPattern:"^xapp-[A-Za-z0-9_-]+$",formatHint:"Slack app tokens start with 'xapp-' (e.g. xapp----).",prompt:{label:"Slack App Token (Socket Mode)",help:"Slack API \u2192 Your Apps \u2192 Basic Information \u2192 App-Level Tokens (xapp-...)."}},{id:"allowedUsers",kind:"config",required:false,envKey:"SLACK_ALLOWED_USERS",statePath:"allowedIds.slack",prompt:{label:"Slack Member IDs (comma-separated allowlist)",help:"In Slack, open each allowed human user's profile -> More -> Copy member ID. Enter one or more comma-separated member IDs, not the app or bot user ID. Member IDs look like U01ABC2DEF3.",emptyValueMessage:"bot will require manual pairing"}},{id:"allowedChannels",kind:"config",required:false,envKey:"SLACK_ALLOWED_CHANNELS",statePath:"slackConfig.allowedChannels",prompt:{label:"Slack Channel IDs (comma-separated allowlist)",help:"Optional: enter comma-separated Slack channel IDs where the bot may answer @mentions. Channel IDs look like C012AB3CD.",emptyValueMessage:"channel @mentions stay unrestricted by channel ID"}}],credentials:[{id:"slackBotToken",sourceInput:"botToken",providerName:"{sandboxName}-slack-bridge",providerEnvKey:"SLACK_BOT_TOKEN",placeholder:"xoxb-OPENSHELL-RESOLVE-ENV-SLACK_BOT_TOKEN",primary:true},{id:"slackAppToken",sourceInput:"appToken",providerName:"{sandboxName}-slack-app",providerEnvKey:"SLACK_APP_TOKEN",placeholder:"xapp-OPENSHELL-RESOLVE-ENV-SLACK_APP_TOKEN"}],policyPresets:[{name:"slack",requiredAtCreate:true}],render:[{id:"slack-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.slack",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false},dmPolicy:"{{allowedIds.slack.dmPolicy}}",allowFrom:"{{allowedIds.slack.values}}",groupPolicy:"{{allowedIds.slack.groupPolicy}}",channels:"{{allowedIds.slack.channels}}"}}}}},{id:"slack-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.slack",value:{enabled:true}}},{id:"slack-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["SLACK_ALLOWED_USERS={{allowedIds.slack.csv}}","SLACK_ALLOWED_CHANNELS={{slackConfig.allowedChannels.csv}}"]},{id:"slack-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.slack",value:{enabled:true,extra:{rich_blocks:true}}}}],runtime:{openclaw:{channelName:"slack",visibility:{configKeys:["slack"],logPatterns:["slack"]},nodePreloads:[{module:"slack-channel-guard",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing Slack channel guard (unhandled-rejection safety net)",installedMessage:"[channels] Slack channel guard installed (NODE_OPTIONS updated)"}],secretScans:[{path:"/sandbox/.openclaw/openclaw.json",pattern:"(?:xoxb|xapp)-(?!OPENSHELL-RESOLVE-ENV-)",message:"[SECURITY] Slack token leaked into {path} - refusing to serve",exitCode:78}]},hermes:{}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/slack@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-6M1M6gL3iXahpalNsYAUuA+wvnV8lbMlNNH2ToegFvaSJcIll4S9kFa5mv3GFoquhMRHQjEjnkXPHP/pXwaWcA==","2026.9.1":"sha512-tU372jE40nnPcKQ6oxmDHf2/UhGtdz8ysi4JKsRZIO1QBAEkZd2YfsOw8aucmb2r0B0vjcFD3OmIV/Qzb57COg=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/slack/-/slack-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/slack/-/slack-2026.9.1.tgz"},required:true}],hooks:[{id:"slack-socket-mode-gateway-conflict",phase:"pre-enable",handler:"slack.socketModeGatewayConflict",onFailure:"abort"},{id:"slack-openclaw-bridge-health",phase:"health-check",handler:"slack.openclawBridgeHealth",agents:["openclaw"],onFailure:"abort"},{id:"slack-socket-mode-gateway-status",phase:"status",handler:"slack.socketModeGatewayStatus",outputs:[{id:"gatewayOverlaps",kind:"status"}]},{id:"slack-status-health",phase:"status",handler:"slack.statusHealth",providesReadiness:true,agents:["openclaw"],outputs:[{id:"channelHealth",kind:"status"}]},{id:"slack-token-paste",phase:"enroll",handler:"common.tokenPaste",outputs:[{id:"botToken",kind:"secret",required:true},{id:"appToken",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"slack-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"allowedUsers",kind:"config"},{id:"allowedChannels",kind:"config"}]},{id:"slack-credential-validation",phase:"reachability-check",handler:"slack.validateCredentials",inputs:["botToken","appToken"],onFailure:"skip-channel"}]};var TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT={channelId:"teams",renderId:"teams-openclaw-channel",hookId:"teams-openclaw-channel",handlerId:"common.staticOutputs",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",configPath:"channels.msteams",webhookPath:"/api/messages"};function authorizeTeamsOpenClawWebhookField(entry){if(!isPlainDataObject(entry))return[];const contract=TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT;if(ownDataPropertyValue(entry,"channelId")!==contract.channelId||ownDataPropertyValue(entry,"renderId")!==contract.renderId||ownDataPropertyValue(entry,"hookId")!==contract.hookId||ownDataPropertyValue(entry,"handler")!==contract.handlerId||ownDataPropertyValue(entry,"kind")!==contract.kind||ownDataPropertyValue(entry,"agent")!==contract.agent||ownDataPropertyValue(entry,"target")!==contract.target||ownDataPropertyValue(entry,"path")!==contract.configPath){return[]}const value=ownDataPropertyValue(entry,"value");if(!isPlainDataObject(value))return[];const webhook=ownDataPropertyValue(value,"webhook");if(!isPlainDataObject(webhook)||!hasExactlyOwnDataProperties(webhook,["path","port"])||!isTcpPort(ownDataPropertyValue(webhook,"port"))||ownDataPropertyValue(webhook,"path")!==contract.webhookPath){return[]}return[{path:["value","webhook"],value:webhook}]}function isPlainDataObject(value){if(value===null||typeof value!=="object"||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function ownDataPropertyValue(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}function hasExactlyOwnDataProperties(value,expected){const actual=Object.getOwnPropertyNames(value).sort();return actual.length===expected.length&&actual.every((key,index)=>key===expected[index])}function isTcpPort(value){return Number.isInteger(value)&&value>=1&&value<=65535}var teamsManifest={schemaVersion:1,id:"teams",displayName:"Microsoft Teams",description:"Microsoft Teams bot messaging (experimental)",enrollmentNotes:["Microsoft Teams requires a public HTTPS webhook endpoint at /api/messages; expose the configured Teams webhook port before installing the Teams app.","Use Azure AD object IDs in TEAMS_ALLOWED_USERS so only authorized users can interact with the bot."],supportedAgents:["openclaw","hermes"],auth:{mode:"token-paste"},inputs:[{id:"appId",kind:"config",required:true,envKey:"MSTEAMS_APP_ID",statePath:"teamsConfig.appId",prompt:{label:"Microsoft Teams Client ID",help:"Run `teams app create --endpoint https:///api/messages`, then copy CLIENT_ID."}},{id:"clientSecret",kind:"secret",required:true,envKey:"MSTEAMS_APP_PASSWORD",prompt:{label:"Microsoft Teams Client Secret",help:"Use the CLIENT_SECRET printed by `teams app create`. It is shown once; rotate it in Entra ID if it was lost."}},{id:"tenantId",kind:"config",required:true,envKey:"MSTEAMS_TENANT_ID",statePath:"teamsConfig.tenantId",prompt:{label:"Microsoft Teams Tenant ID",help:"Use the TENANT_ID printed by `teams app create` or shown by `teams status --verbose`."}},{id:"allowedUsers",kind:"config",required:false,envKey:"TEAMS_ALLOWED_USERS",statePath:"allowedIds.teams",prompt:{label:"Microsoft Teams AAD Object IDs (comma-separated allowlist)",help:"Recommended: run `teams status --verbose` and enter the Azure AD object IDs allowed to use the bot."}},{id:"webhookPort",kind:"config",required:false,envKey:"MSTEAMS_PORT",statePath:"teamsConfig.webhookPort",defaultValue:"3978",prompt:{label:"Microsoft Teams webhook port",help:"Local bot webhook port to expose publicly. Defaults to 3978 and serves /api/messages."}},{id:"requireMention",kind:"config",required:false,envKey:"TEAMS_REQUIRE_MENTION",statePath:"teamsConfig.requireMention",validValues:["0","1"],defaultValue:"1",prompt:{label:"Microsoft Teams mention mode",help:"Controls OpenClaw group and channel behavior only. Direct messages are unaffected."}}],credentials:[{id:"teamsClientSecret",sourceInput:"clientSecret",providerName:"{sandboxName}-teams-bridge",providerEnvKey:"MSTEAMS_APP_PASSWORD",placeholder:"openshell:resolve:env:MSTEAMS_APP_PASSWORD",primary:true}],policyPresets:[{name:"teams",policyKeys:["teams"],requiredAtCreate:true}],hostForward:{port:"{{teamsConfig.webhookPort}}",label:"Microsoft Teams webhook"},render:[{id:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.renderId,kind:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.kind,agent:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.agent,target:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.target,fragment:{path:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.configPath,value:{enabled:true,appId:"{{teamsConfig.appId}}",tenantId:"{{teamsConfig.tenantId}}",webhook:{port:"{{teamsConfig.webhookPort}}",path:TEAMS_OPENCLAW_WEBHOOK_RENDER_CONTRACT.webhookPath},healthMonitor:{enabled:false},streaming:{mode:"off"},dmPolicy:"{{allowedIds.teams.dmPolicy}}",allowFrom:"{{allowedIds.teams.values}}",groupPolicy:"open",requireMention:"{{teamsConfig.requireMention}}"}}},{id:"teams-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.msteams",value:{enabled:true}}},{id:"teams-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["TEAMS_CLIENT_ID={{teamsConfig.appId}}","TEAMS_TENANT_ID={{teamsConfig.tenantId}}","TEAMS_ALLOWED_USERS={{allowedIds.teams.csv}}","TEAMS_PORT={{teamsConfig.webhookPort}}"]},{id:"teams-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.teams",value:{enabled:true}}}],runtime:{openclaw:{channelName:"msteams",visibility:{configKeys:["msteams"],logPatterns:["msteams","teams"]},nodePreloads:[{module:"msteams-message-hints",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing Microsoft Teams message hint patch (native mentions)",installedMessage:"[channels] Microsoft Teams message hint patch installed (NODE_OPTIONS updated)"}]},hermes:{envAliases:[{envKey:"MSTEAMS_APP_PASSWORD",targetEnvKey:"TEAMS_CLIENT_SECRET",match:"^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_MSTEAMS_APP_PASSWORD$",value:"openshell:resolve:env:MSTEAMS_APP_PASSWORD"}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/msteams@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-py5KvGOTcd0qGGRf3EuqbH2jO+kZtvMquDMjwGkT6x9F4XZtaCBL/lmLitb4hDb5xaIpPP8ZLIbT8GIMXQr3Og==","2026.9.1":"sha512-seRGr9/X6Vk9xU5elLVpDwq8R+TO0QFvUmxPEitqkngqDnMoXW0LEEXkriG6jgue74w2YLcNnAv/Rjf0a9jong=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/msteams/-/msteams-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/msteams/-/msteams-2026.9.1.tgz"},required:true},{id:"hermesTeamsAppsPackage",agent:"hermes",manager:"hermes-uv-pip",spec:"microsoft-teams-apps==2.0.13.4",required:true}],hooks:[{id:"teams-host-forward-port-conflict",phase:"pre-enable",handler:"teams.hostForwardPortConflict",inputs:["webhookPort"],onFailure:"abort"},{id:"teams-host-forward-port-status",phase:"status",handler:"teams.hostForwardPortStatus",outputs:[{id:"hostForwardPortOverlaps",kind:"status"}]},{id:"teams-token-paste",phase:"enroll",handler:"common.tokenPaste",outputs:[{id:"clientSecret",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"teams-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"appId",kind:"config",required:true},{id:"tenantId",kind:"config",required:true},{id:"allowedUsers",kind:"config"},{id:"webhookPort",kind:"config"},{id:"requireMention",kind:"config"}]}],state:{}};var telegramManifest={schemaVersion:1,id:"telegram",displayName:"Telegram",description:"Telegram bot messaging",diagnosticsProbe:"log-tail",enrollmentNotes:["For Telegram group chats, disable privacy mode in @BotFather (/setprivacy -> your bot -> Disable).","After changing privacy mode, remove and re-add the bot to each group before testing @mentions."],supportedAgents:["openclaw","hermes"],state:{openclaw:["telegram"],hermes:["platforms/telegram"]},auth:{mode:"token-paste"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"TELEGRAM_BOT_TOKEN",prompt:{label:"Telegram Bot Token",help:"Create a bot via @BotFather on Telegram, then copy the token."}},{id:"allowedIds",kind:"config",required:false,envKey:"TELEGRAM_ALLOWED_IDS",statePath:"allowedIds.telegram",prompt:{label:"Telegram User ID (for DM access)",help:"Send /start to @userinfobot on Telegram to get your numeric user ID.",emptyValueMessage:"bot will require manual pairing"}},{id:"requireMention",kind:"config",required:false,envKey:"TELEGRAM_REQUIRE_MENTION",statePath:"telegramConfig.requireMention",validValues:["0","1"],defaultValue:"1",prompt:{label:"Telegram group mention mode",help:"Controls Telegram group-chat behavior only \u2014 reply only when @mentioned vs. to all group messages. Direct messages are unaffected by this setting and remain subject to pairing and TELEGRAM_ALLOWED_IDS."}},{id:"groupPolicy",kind:"config",required:false,envKey:"TELEGRAM_GROUP_POLICY",statePath:"telegramConfig.groupPolicy",validValues:["open","allowlist","disabled"],defaultValue:"open",prompt:{label:"Telegram group policy",help:"Controls OpenClaw Telegram group access. Hermes does not expose an equivalent disable-groups policy."}}],credentials:[{id:"telegramBotToken",sourceInput:"botToken",providerName:"{sandboxName}-telegram-bridge",providerEnvKey:"TELEGRAM_BOT_TOKEN",placeholder:"openshell:resolve:env:TELEGRAM_BOT_TOKEN"}],policyPresets:[{name:"telegram",requiredAtCreate:true,policyKeys:["telegram_bot"],agentPolicyKeys:{hermes:["telegram"]}}],render:[{id:"telegram-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.telegram",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false},groupPolicy:"{{telegramConfig.groupPolicy}}",dmPolicy:"{{allowedIds.telegram.dmPolicy}}",allowFrom:"{{allowedIds.telegram.values}}"}}}}},{id:"telegram-openclaw-groups",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",when:"{{telegramConfig.openclawGroups}}",fragment:{path:"channels.telegram.groups",value:"{{telegramConfig.openclawGroups}}"}},{id:"telegram-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.telegram",value:{enabled:true}}},{id:"telegram-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["TELEGRAM_ALLOWED_USERS={{allowedIds.telegram.csv}}"]},{id:"telegram-hermes-config",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"telegram",value:{require_mention:"{{telegramConfig.requireMention}}"}}},{id:"telegram-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.telegram",value:{enabled:true}}}],runtime:{openclaw:{channelName:"telegram",visibility:{configKeys:["telegram"],logPatterns:["telegram"]},nodePreloads:[{module:"telegram-diagnostics",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing Telegram diagnostics (provider readiness + inference errors)",installedMessage:"[channels] Telegram diagnostics installed (NODE_OPTIONS updated)"}]}},hooks:[{id:"telegram-token-paste",phase:"enroll",handler:"common.tokenPaste",outputs:[{id:"botToken",kind:"secret",required:true}],onFailure:"skip-channel"},{id:"telegram-allowlist-aliases",phase:"enroll",handler:"telegram.allowlistAliases",outputs:[{id:"allowedIds",kind:"config"}]},{id:"telegram-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"requireMention",kind:"config"},{id:"allowedIds",kind:"config"}]},{id:"telegram-openclaw-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["openclaw"],outputs:[{id:"groupPolicy",kind:"config"}]},{id:"telegram-get-me-reachability",phase:"reachability-check",handler:"telegram.getMeReachability",inputs:["botToken"],onFailure:"skip-channel"},{id:"telegram-openclaw-bridge-health",phase:"health-check",handler:"telegram.openclawBridgeHealth",agents:["openclaw"],onFailure:"abort"},{id:"telegram-gateway-conflict-status",phase:"status",handler:"telegram.gatewayConflictStatus",outputs:[{id:"bridgeHealth",kind:"status"}]},{id:"telegram-status-health",phase:"status",handler:"telegram.statusHealth",agents:["openclaw"],outputs:[{id:"channelHealth",kind:"status"}]}]};var WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT={channelId:"wechat",planHookId:"wechat-seed-openclaw-account",handlerId:"wechat.seedOpenClawAccount",outputId:"openclawWeixinAccountFile",kind:"build-file",required:true,mode:"0600"};var WECHAT_SEED_OPENCLAW_ACCOUNT_HOOK_ID=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.handlerId;var WECHAT_SEED_OPENCLAW_ACCOUNT_PLAN_HOOK_ID=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.planHookId;var WECHAT_OPENCLAW_ACCOUNT_FILE_OUTPUT_ID=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.outputId;var WECHAT_TOKEN_PLACEHOLDER="openshell:resolve:env:WECHAT_BOT_TOKEN";function authorizeWechatAccountFilePlaceholders(value){const content=isPlainDataObject2(value)?ownDataPropertyValue2(value,"content"):void 0;if(!isPlainDataObject2(value)||!hasExactlyOwnDataProperties2(value,["content","mode","path"])||!isWechatAccountFilePath(ownDataPropertyValue2(value,"path"))||ownDataPropertyValue2(value,"mode")!==WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.mode||!isPlainDataObject2(content)||!hasOnlyOwnDataProperties(content,["baseUrl","savedAt","token","userId"])||!hasOwnDataProperty(content,"savedAt")||!hasOwnDataProperty(content,"token")||ownDataPropertyValue2(content,"token")!==WECHAT_TOKEN_PLACEHOLDER||!isNonEmptyString(ownDataPropertyValue2(content,"savedAt"))||!isOptionalNonEmptyString(content,"baseUrl")||!isOptionalNonEmptyString(content,"userId")){return[]}return[{path:["content","token"],value:WECHAT_TOKEN_PLACEHOLDER}]}function isWechatAccountFilePath(value){if(typeof value!=="string")return false;const prefix="openclaw-weixin/accounts/";const suffix=".json";if(!value.startsWith(prefix)||!value.endsWith(suffix))return false;const accountId=value.slice(prefix.length,-suffix.length);return accountId===accountId.trim()&&isSafeWechatAccountId(accountId)}function isSafeWechatAccountId(accountId){return accountId.length>0&&accountId!=="."&&accountId!==".."&&!/[\\/\0-\x1F\x7F]/.test(accountId)&&!accountId.includes("..")}function isPlainDataObject2(value){if(value===null||typeof value!=="object"||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function ownDataPropertyValue2(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}function hasOwnDataProperty(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor!==void 0&&"value"in descriptor}function hasExactlyOwnDataProperties2(value,expected){const actual=Object.getOwnPropertyNames(value).sort();return actual.length===expected.length&&actual.every((key,index)=>key===expected[index])}function hasOnlyOwnDataProperties(value,allowed){return Object.getOwnPropertyNames(value).every(key=>allowed.includes(key))}function isNonEmptyString(value){return typeof value==="string"&&value.length>0}function isOptionalNonEmptyString(value,key){return!hasOwnDataProperty(value,key)||isNonEmptyString(ownDataPropertyValue2(value,key))}var wechatManifest={schemaVersion:1,id:"wechat",displayName:"WeChat",description:"WeChat (personal) bot messaging",enrollmentHelp:"Captured automatically via a host-side QR scan during onboard \u2014 pair the bot by scanning the QR with WeChat on your phone (Discover \u2192 Scan). DM-only.",supportedAgents:["openclaw","hermes"],auth:{mode:"host-qr"},inputs:[{id:"botToken",kind:"secret",required:true,envKey:"WECHAT_BOT_TOKEN",prompt:{label:"WeChat Bot Token",help:"Captured automatically via a host-side QR scan during onboard \u2014 pair the bot by scanning the QR with WeChat on your phone (Discover \u2192 Scan). DM-only."}},{id:"accountId",kind:"config",required:true,envKey:"WECHAT_ACCOUNT_ID",statePath:"wechatConfig.accountId"},{id:"baseUrl",kind:"config",required:false,envKey:"WECHAT_BASE_URL",statePath:"wechatConfig.baseUrl"},{id:"userId",kind:"config",required:false,envKey:"WECHAT_USER_ID",statePath:"wechatConfig.userId"},{id:"allowedIds",kind:"config",required:false,envKey:"WECHAT_ALLOWED_IDS",statePath:"allowedIds.wechat",prompt:{label:"WeChat User ID(s) (DM allowlist)",help:"Optional: restrict who can DM the bot. The WeChat user id of the operator who scanned is added automatically; supply additional ids as a comma-separated list.",emptyValueMessage:"bot will require manual pairing"}}],credentials:[{id:"wechatBotToken",sourceInput:"botToken",providerName:"{sandboxName}-wechat-bridge",providerEnvKey:"WECHAT_BOT_TOKEN",placeholder:"openshell:resolve:env:WECHAT_BOT_TOKEN"}],state:{openclaw:["wechat","openclaw-weixin"],hermes:["platforms/wechat"]},policyPresets:[{name:"wechat",policyKeys:["wechat_bridge"],requiredAtCreate:true}],render:[{id:"wechat-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.openclaw-weixin",value:{enabled:true}}},{id:"wechat-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.openclaw-weixin",value:{enabled:true}}},{id:"wechat-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["WEIXIN_ACCOUNT_ID={{wechatConfig.accountId}}","WEIXIN_BASE_URL={{wechatConfig.baseUrl}}","WEIXIN_ALLOWED_USERS={{allowedIds.wechat.csv}}"]},{id:"wechat-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.weixin",value:{enabled:true}}}],runtime:{openclaw:{channelName:"openclaw-weixin",visibility:{configKeys:["openclaw-weixin"],logPatterns:["wechat","openclaw-weixin"]},nodePreloads:[{module:"wechat-account-placeholder",injectInto:["boot"],optional:false},{module:"wechat-diagnostics",injectInto:["boot","connect"],optional:false,installMessage:"[channels] Installing WeChat diagnostics (provider readiness + inference errors)",installedMessage:"[channels] WeChat diagnostics installed (NODE_OPTIONS updated)"}]},hermes:{envAliases:[{envKey:"WECHAT_BOT_TOKEN",targetEnvKey:"WEIXIN_TOKEN",match:"^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_WECHAT_BOT_TOKEN$",value:"openshell:resolve:env:WECHAT_BOT_TOKEN"}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@tencent-weixin/openclaw-weixin@2.4.9",pin:true,integrity:"sha512-SfaYehR1Cwq2VV5HxJBp9sVilMms420VfZlMbF4YjRbWomr5+GxfXp9HkeU6y5TbnOc4Ysq0qPw1yBvJwbenBA==",tarballUrl:"https://registry.npmjs.org/@tencent-weixin/openclaw-weixin/-/openclaw-weixin-2.4.9.tgz",runtimeLock:{cachePath:"/usr/local/share/nemoclaw/wechat-npm-cache",installCacheEnvKey:"NEMOCLAW_WECHAT_NPM_INSTALL_CACHE",lockFile:"/usr/local/lib/nemoclaw/wechat-runtime/package-lock.json",projectsRoot:"/sandbox/.openclaw/npm/projects",verifierPath:"/usr/local/lib/nemoclaw/verify-wechat-runtime-lock.mts",offline:true,legacyPeerDeps:true},required:true}],hooks:[{id:"wechat-host-qr",phase:"enroll",handler:"wechat.ilinkLogin",inputs:["allowedIds"],outputs:[{id:"botToken",kind:"secret",required:true},{id:"accountId",kind:"config",required:true},{id:"baseUrl",kind:"config"},{id:"userId",kind:"config"},{id:"allowedIds",kind:"config"}],onFailure:"skip-channel"},{id:"wechat-config-prompt",phase:"enroll",handler:"common.configPrompt",outputs:[{id:"allowedIds",kind:"config"}]},{id:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.planHookId,phase:"post-agent-install",handler:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.handlerId,agents:["openclaw"],inputs:["wechatConfig.accountId","wechatConfig.baseUrl","wechatConfig.userId","credential.wechatBotToken.placeholder"],outputs:[{id:"openclawWeixinAccountsIndex",kind:"build-file",required:true},{id:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.outputId,kind:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.kind,required:WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT.required},{id:"openclawConfigPatch",kind:"build-file",required:true}],onFailure:"abort"},{id:"wechat-health-check",phase:"health-check",handler:"wechat.healthCheck",inputs:["wechatConfig.accountId"],onFailure:"abort"}]};var whatsappManifest={schemaVersion:1,id:"whatsapp",displayName:"WhatsApp",description:"WhatsApp Web messaging (QR pairing)",enrollmentHelp:"WhatsApp Web pairs via QR code scanned with your phone \u2014 no host-side token. After the sandbox is running, run `openshell term` and then use `openclaw channels login --channel whatsapp` for OpenClaw or `hermes whatsapp` for Hermes to display the QR.",enrollmentNotes:["After pairing, run `nemoclaw channels status --channel whatsapp`. OpenClaw reports inbound delivery evidence; Hermes reports gateway and dashboard session-path diagnostics."],supportedAgents:["openclaw","hermes"],auth:{mode:"in-sandbox-qr"},inputs:[{id:"mode",kind:"config",required:false,envKey:"WHATSAPP_MODE",statePath:"whatsappConfig.mode",validValues:["self-chat","bot"],defaultValue:"self-chat",prompt:{label:"WhatsApp reply mode",help:"self-chat replies only to messages the paired account sends to itself. bot replies to other senders and stops replying to that self-chat: an unknown sender receives a pairing code you approve with `hermes pairing approve whatsapp `, unless you set WHATSAPP_ALLOWED_IDS to a fixed sender list before this command.",emptyValueMessage:"the sandbox replies only in your own self-chat"}},{id:"allowedIds",kind:"config",required:false,envKey:"WHATSAPP_ALLOWED_IDS",statePath:"allowedIds.whatsapp"}],credentials:[],state:{openclaw:["whatsapp"],hermes:["platforms/whatsapp","profiles/dashboard-home/platforms/whatsapp/session","dashboard-home/platforms/whatsapp/session"]},policyPresets:["whatsapp"],render:[{id:"whatsapp-openclaw-channel",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"channels.whatsapp",value:{enabled:true,accounts:{default:{enabled:true,healthMonitor:{enabled:false}}}}}},{id:"whatsapp-openclaw-plugin",kind:"json-fragment",agent:"openclaw",target:"openclaw.json",fragment:{path:"plugins.entries.whatsapp",value:{enabled:true}}},{id:"whatsapp-hermes-env",kind:"env-lines",agent:"hermes",target:"~/.hermes/.env",lines:["WHATSAPP_ENABLED=true","WHATSAPP_MODE={{whatsappConfig.mode}}","WHATSAPP_DM_POLICY={{whatsappConfig.dmPolicy}}","WHATSAPP_ALLOWED_USERS={{allowedIds.whatsapp.csv}}"]},{id:"whatsapp-hermes-platform",kind:"json-fragment",agent:"hermes",target:"~/.hermes/config.yaml",fragment:{path:"platforms.whatsapp",value:{enabled:true}}}],runtime:{openclaw:{channelName:"whatsapp",visibility:{configKeys:["whatsapp"],logPatterns:["whatsapp"]},nodePreloads:[{module:"whatsapp-qr-compact",injectInto:["connect"],optional:true,installMessage:"[channels] Installing WhatsApp compact-QR renderer (scan-friendly pairing)"}]}},agentPackages:[{id:"openclawPluginPackage",agent:"openclaw",manager:"openclaw-plugin",spec:"npm:@openclaw/whatsapp@{{openclaw.version}}",pin:true,integrityByVersion:{"2026.9.2":"sha512-vOWQIk7FpLHrhMmO+FaLi+pnFB82hiWNJJFJONkBuofERh2SMEz7EMut/vECFFEjFnmOZSVlYfRlxhbNkd/R6g==","2026.9.1":"sha512-llIcoMa6FM4SgYn7GG1FQIeTTA5JDdcHW5D7PT+3aGYT3/E2eLFutKwDvD/w7G0hvDwSftzZgLi3iA8dzK7a3A=="},tarballUrlByVersion:{"2026.9.2":"https://registry.npmjs.org/@openclaw/whatsapp/-/whatsapp-2026.9.2.tgz","2026.9.1":"https://registry.npmjs.org/@openclaw/whatsapp/-/whatsapp-2026.9.1.tgz"},required:true}],hooks:[{id:"whatsapp-config-prompt",phase:"enroll",handler:"common.configPrompt",agents:["hermes"],outputs:[{id:"mode",kind:"config"}]},{id:"whatsapp-status-health",phase:"status",handler:"whatsapp.statusHealth",agents:["openclaw","hermes"],outputs:[{id:"channelHealth",kind:"status"}]}]};var BUILT_IN_CHANNEL_MANIFESTS=[telegramManifest,discordManifest,wechatManifest,slackManifest,whatsappManifest,teamsManifest,googlechatManifest];function createBuiltInChannelManifestRegistry(){return createChannelManifestRegistry(BUILT_IN_CHANNEL_MANIFESTS)}function listMessagingCredentialEnvAssignments(options={}){return selectManifests(options).flatMap(manifest=>{const credentialsByTemplate=new Map(manifest.credentials.map(credential=>[`{{credential.${credential.id}.placeholder}}`,credential]));const renderedAssignments=manifest.render.flatMap(render=>{if(options.agent&&render.agent!==options.agent)return[];if(render.kind!=="env-lines")return[];return render.lines.flatMap(line=>{const separator=line.indexOf("=");if(separator<=0)return[];const credential=credentialsByTemplate.get(line.slice(separator+1));if(!credential)return[];return[{channelId:manifest.id,agent:render.agent,sourceEnvKey:credential.providerEnvKey,targetEnvKey:line.slice(0,separator),placeholder:credential.placeholder}]})});const runtimeAssignments=["openclaw","hermes"].flatMap(agent=>{if(options.agent&&agent!==options.agent)return[];if(!manifest.supportedAgents.includes(agent))return[];return(manifest.runtime?.[agent]?.envAliases??[]).flatMap(alias=>{if(!alias.targetEnvKey)return[];const credential=manifest.credentials.find(candidate=>candidate.providerEnvKey===alias.envKey);if(!credential)return[];return[{channelId:manifest.id,agent,sourceEnvKey:alias.envKey,targetEnvKey:alias.targetEnvKey,placeholder:credential.placeholder}]})});return[...renderedAssignments,...runtimeAssignments]})}function selectManifests(options){const manifests=options.manifests??BUILT_IN_CHANNEL_MANIFESTS;const agent=options.agent;const selected=agent?manifests.filter(manifest=>manifest.supportedAgents.includes(agent)):manifests;return[...selected]}function authorizeMessagingManagedStartupFields(entry,section){if(section==="agentRender")return authorizeTeamsOpenClawWebhookField(entry);if(!isPlainDataObject3(entry))return[];const contract=WECHAT_OPENCLAW_ACCOUNT_FILE_CONTRACT;if(ownDataPropertyValue3(entry,"channelId")!==contract.channelId||ownDataPropertyValue3(entry,"hookId")!==contract.planHookId||ownDataPropertyValue3(entry,"handler")!==contract.handlerId||ownDataPropertyValue3(entry,"outputId")!==contract.outputId||ownDataPropertyValue3(entry,"kind")!==contract.kind||ownDataPropertyValue3(entry,"required")!==contract.required){return[]}return authorizeWechatAccountFilePlaceholders(ownDataPropertyValue3(entry,"value")).map(authorization=>({...authorization,path:["value",...authorization.path]}))}function isPlainDataObject3(value){if(value===null||typeof value!=="object"||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function ownDataPropertyValue3(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}var DCODE_UPSTREAM_PROVIDER_RE=/^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/u;function isValidDcodeUpstreamProvider(value){return DCODE_UPSTREAM_PROVIDER_RE.test(value)}var LEGACY_MANAGED_STARTUP_PROFILE_SCHEMA_VERSION=1;var MANAGED_STARTUP_PROFILE_SCHEMA_VERSION=2;var MANAGED_STARTUP_PROFILE_MAX_BYTES=64*1024;var MANAGED_STARTUP_PROFILE_MAX_ENCODED_BYTES=Math.ceil(MANAGED_STARTUP_PROFILE_MAX_BYTES/3)*4;var MAX_IDENTIFIER_BYTES=256;var MAX_MODEL_BYTES2=1024;var MAX_URL_BYTES=2048;var MAX_LIST_ITEMS=128;var MAX_JSON_NODES=4096;var MAX_JSON_DEPTH=32;var MAX_TUNING_INTEGER=1e9;var MIN_HERMES_CONTEXT_WINDOW=64e3;var SHA256_RE=/^[a-f0-9]{64}$/;var CONTROL_CHARACTER_RE=/[\u0000-\u001f\u007f-\u009f]/u;var BASE64URL_RE=/^[A-Za-z0-9_-]+$/;var RAW_CA_PEM_RE=/-----BEGIN (?:TRUSTED )?CERTIFICATE-----/iu;var RAW_CA_PEM_BASE64_RE=/^LS0tLS1CRUdJTi(?:BDRVJUSUZJQ0FURS0tLS0t|BUlVTVEVEIENFUlRJRklDQVRFLS0tLS0)/u;var RAW_CA_DER_BASE64_RE=/^MII[A-Za-z0-9+/=\r\n]{253,}$/u;var RAW_CA_DATA_URI_RE=/data:application\/(?:pkix-cert|x-x509-ca-cert);base64,MII[A-Za-z0-9+/=]{253,}/iu;var URL_CANDIDATE_RE=/[A-Za-z][A-Za-z0-9+.-]*:\/\/[^\s"'<>]+/gu;var UTF8_DECODER=new import_node_util.TextDecoder("utf-8",{fatal:true});var CREDENTIAL_SHAPED_NAME_PATTERN=/(?:^|[_-])(?:api[_-]?key|access[_-]?key|secret[_-]?key|auth[_-]?token|refresh[_-]?token|access[_-]?token|client[_-]?secret|private[_-]?key|pass[_-]?code|personal[_-]?access[_-]?token|connection[_-]?string|webhook(?:[_-]?url)?|key|secret|token|password|passwd|passcode|auth|authorization|credential|credentials|bearer|bearer[_-]?token|cookie|cookies|pat|private|privatekey|pin|webhookurl|dsn|connectionstring)(?:$|[_-])/iu;var CREDENTIAL_COMPOUND_NAME_PATTERN=/^(?:access|refresh|client|bearer|auth|api|private|signing|session|bot|app|resolved)(?:token|key|secret|password)$/iu;var CREDENTIAL_CAMEL_SUFFIX_PATTERN=/(?:apiKey|accessKey|secretKey|authToken|refreshToken|accessToken|clientSecret|privateKey|passcode|password|passwd|passphrase|bearerToken|botToken|appToken|sessionToken|signingKey|secretPublicKey|personalAccessToken|connectionString|webhookUrl)$/iu;var CREDENTIAL_CAMEL_BOUNDARY_PATTERN=/[a-z0-9](?:Token|Key|Secret|Password|Passphrase|Pat)$/u;var CREDENTIAL_ENV_NAME_PATTERN=/^(?:[A-Z0-9]+_)*(?:TOKEN|KEY|SECRET|PASSWORD|PASSWD|PASS|PASSPHRASE|CREDENTIAL)S?$/u;var CREDENTIAL_HEADER_NAME_PATTERN=/^(?:authorization|proxy-authorization|cookie|set-cookie|.+-(?:key|token|secret|password|passphrase|credential|auth)s?)$/iu;var PUBLIC_KEY_NAME_PATTERN=/^public[-_]?keys?$/iu;var PASS_CREDENTIAL_NAME_PATTERN=/(?:^|[-_])pass(?:wd)?$/iu;var NON_SECRET_KEY_METADATA_NAMES=new Set(["envKey","installCacheEnvKey","providerEnvKey","stateKey","targetEnvKey"]);var MESSAGING_CREDENTIAL_PLACEHOLDER_RE=/^(?:openshell:resolve:env:|[A-Za-z0-9]+-OPENSHELL-RESOLVE-ENV-)(?:(?:v[0-9]{1,20}|s[a-f0-9]{64})_)?[A-Z][A-Z0-9_]*$/u;var MESSAGING_CREDENTIAL_ENV_ALIASES=new Set(listMessagingCredentialEnvAssignments().filter(({sourceEnvKey,targetEnvKey})=>sourceEnvKey!==targetEnvKey).map(({agent,sourceEnvKey,targetEnvKey})=>`${agent}\0${sourceEnvKey}\0${targetEnvKey}`));var MESSAGING_CREDENTIAL_RUNTIME_ENV_ALIASES=new Set(listMessagingCredentialEnvAssignments().filter(({sourceEnvKey,targetEnvKey})=>sourceEnvKey!==targetEnvKey).map(({agent,channelId,sourceEnvKey,targetEnvKey})=>`${agent}\0${channelId}\0${sourceEnvKey}\0${targetEnvKey}`));var JSON_ARRAY_INDEX_SEGMENT_RE=/^\[(?:0|[1-9][0-9]*)\]$/u;var SECRET_VALUE_PATTERNS2=[/nvapi-[A-Za-z0-9_-]{10,}/u,/nvcf-[A-Za-z0-9_-]{10,}/u,/ghp_[A-Za-z0-9_-]{10,}/u,/github_pat_[A-Za-z0-9_]{30,}/u,/sk-(?:proj-|ant-)?[A-Za-z0-9_-]{10,}/u,/(?:xox[bpas]|xapp)-[A-Za-z0-9-]{10,}/u,/A(?:K|S)IA[A-Z0-9]{16}/u,/hf_[A-Za-z0-9]{10,}/u,/glpat-[A-Za-z0-9_-]{10,}/u,/gsk_[A-Za-z0-9]{10,}/u,/pypi-[A-Za-z0-9_-]{10,}/u,/tvly-[A-Za-z0-9_-]{10,}/u,/lsv2_(?:pt|sk)_[A-Za-z0-9]{10,}(?:_[A-Za-z0-9]+)*/u,/\bbot\d{8,10}:[A-Za-z0-9_-]{35}\b/u,/\b\d{8,10}:[A-Za-z0-9_-]{35}\b/u,/\b[A-Za-z0-9]{24}\.[A-Za-z0-9_-]{6}\.[A-Za-z0-9_-]{27,}\b/u,/\beyJ[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{2,}\.[A-Za-z0-9_-]{10,}\b/u,/\bBearer\s+[A-Za-z0-9_.+/=-]{10,}/iu,/-----BEGIN (?:[A-Z0-9]+ )?PRIVATE KEY-----/u];var MANAGED_STARTUP_INFERENCE_APIS=["openai-completions","openai-responses","anthropic-messages"];var MANAGED_STARTUP_REASONING_EFFORTS=["default","low","medium","high"];var MANAGED_STARTUP_DCODE_AUTO_APPROVAL_MODES=["disabled","thread-opt-in"];var MANAGED_STARTUP_HERMES_TOOL_GATEWAYS=["nous-web","nous-image","nous-audio","nous-browser","nous-code"];var MANAGED_STARTUP_AGENTS=["openclaw","hermes","langchain-deepagents-code","pi"];var MANAGED_STARTUP_MESSAGING_AGENTS=["openclaw","hermes"];function freezeAgentCapabilities(capabilities){return Object.freeze({...capabilities,inferenceApis:Object.freeze([...capabilities.inferenceApis]),dashboardModes:Object.freeze([...capabilities.dashboardModes]),inputModalities:Object.freeze([...capabilities.inputModalities]),webSearchProviders:Object.freeze([...capabilities.webSearchProviders]),toolGateways:Object.freeze([...capabilities.toolGateways]),tuningFields:Object.freeze([...capabilities.tuningFields])})}var PROFILE_CAPABILITIES={openclaw:{inferenceApis:[...MANAGED_STARTUP_INFERENCE_APIS],dashboardModes:["loopback","remote"],inputModalities:["text","image"],webSearchProviders:["brave","tavily"],toolGateways:[],tuningFields:["contextWindow","maxTokens","reasoning","reasoningEffort"],supportsMessaging:true,supportsInferenceCompatibility:true,supportsUpstreamEndpoint:false,supportsHostProxyIntent:true,supportsPrimaryModelRef:true,supportsAgentTimeout:true,supportsHeartbeat:true,supportsExtraAgents:true,supportsDeviceAuth:false,observability:"openclaw-otel",supportsMinimalBootstrap:true},hermes:{inferenceApis:[...MANAGED_STARTUP_INFERENCE_APIS],dashboardModes:["disabled","loopback-forwarded"],inputModalities:[],webSearchProviders:["tavily"],toolGateways:[...MANAGED_STARTUP_HERMES_TOOL_GATEWAYS],tuningFields:["contextWindow"],supportsMessaging:true,supportsInferenceCompatibility:false,supportsUpstreamEndpoint:false,supportsHostProxyIntent:true,supportsPrimaryModelRef:false,supportsAgentTimeout:false,supportsHeartbeat:false,supportsExtraAgents:false,supportsDeviceAuth:false,observability:"none",supportsMinimalBootstrap:false},"langchain-deepagents-code":{inferenceApis:["openai-completions"],dashboardModes:["disabled"],inputModalities:[],webSearchProviders:[],toolGateways:[],tuningFields:["reasoningEffort"],supportsMessaging:false,supportsInferenceCompatibility:false,supportsUpstreamEndpoint:true,supportsHostProxyIntent:true,supportsPrimaryModelRef:false,supportsAgentTimeout:false,supportsHeartbeat:false,supportsExtraAgents:false,supportsDeviceAuth:false,observability:"dcode-marker",supportsMinimalBootstrap:false},pi:{inferenceApis:["openai-completions"],dashboardModes:["disabled"],inputModalities:[],webSearchProviders:[],toolGateways:[],tuningFields:["contextWindow","maxTokens","reasoning"],supportsMessaging:false,supportsInferenceCompatibility:false,supportsUpstreamEndpoint:false,supportsHostProxyIntent:true,supportsPrimaryModelRef:false,supportsAgentTimeout:false,supportsHeartbeat:false,supportsExtraAgents:false,supportsDeviceAuth:false,observability:"none",supportsMinimalBootstrap:false}};for(const agent of MANAGED_STARTUP_AGENTS){Object.defineProperty(PROFILE_CAPABILITIES,agent,{configurable:false,enumerable:true,value:freezeAgentCapabilities(PROFILE_CAPABILITIES[agent]),writable:false})}var MANAGED_STARTUP_PROFILE_CAPABILITIES=Object.freeze(PROFILE_CAPABILITIES);function affordance(input,profilePath,source="docker-arg",representation="value"){return{input,profilePath,source,representation}}var HOST_PROXY_AFFORDANCES=[affordance("HTTP_PROXY","proxy.hostHttpUrl","runtime-env"),affordance("http_proxy","proxy.hostHttpUrl","runtime-env","derived"),affordance("HTTPS_PROXY","proxy.hostHttpsUrl","runtime-env"),affordance("https_proxy","proxy.hostHttpsUrl","runtime-env","derived"),affordance("NO_PROXY","proxy.hostNoProxy","runtime-env"),affordance("no_proxy","proxy.hostNoProxy","runtime-env","derived")];var MANAGED_STARTUP_PROFILE_AFFORDANCE_INVENTORY={openclaw:[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_SERVING_PRESET","inference.servingPreset"),affordance("NEMOCLAW_PRIMARY_MODEL_REF","inference.primaryModelRef"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_INFERENCE_COMPAT_B64","inference.compatibility"),affordance("NEMOCLAW_INFERENCE_INPUTS","inference.inputModalities"),affordance("NEMOCLAW_CONTEXT_WINDOW","tuning.contextWindow"),affordance("NEMOCLAW_MAX_TOKENS","tuning.maxTokens"),affordance("NEMOCLAW_REASONING","tuning.reasoning"),affordance("NEMOCLAW_REASONING_EFFORT","tuning.reasoningEffort"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_AGENT_TIMEOUT","agentConfig.agentTimeoutSeconds"),affordance("NEMOCLAW_AGENT_HEARTBEAT_EVERY","agentConfig.heartbeatEvery"),affordance("NEMOCLAW_EXTRA_AGENTS_JSON_B64","agentConfig.extraAgents"),affordance("NEMOCLAW_WEB_SEARCH_ENABLED","agentConfig.webSearch.enabled"),affordance("NEMOCLAW_WEB_SEARCH_PROVIDER","agentConfig.webSearch.provider"),affordance("NEMOCLAW_OPENCLAW_OTEL","agentConfig.otel.enabled"),affordance("NEMOCLAW_OPENCLAW_OTEL_ENDPOINT","agentConfig.otel.endpointUrl"),affordance("NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME","agentConfig.otel.serviceName"),affordance("NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE","agentConfig.otel.sampleRate"),affordance("CHAT_UI_URL","dashboard.url"),affordance("NEMOCLAW_DASHBOARD_BIND","dashboard.bindAddress"),affordance("NEMOCLAW_WSL_DASHBOARD_EXPOSURE","dashboard.wslExposure"),affordance("NEMOCLAW_DASHBOARD_PORT","dashboard.port","runtime-env"),affordance("NEMOCLAW_MESSAGING_PLAN_B64","messaging.plan"),affordance("NEMOCLAW_MINIMAL_BOOTSTRAP","agentConfig.minimalBootstrap","runtime-env"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES],hermes:[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_CONTEXT_WINDOW","tuning.contextWindow"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_HERMES_TOOL_GATEWAY_BROKER","tools.enabledGateways","docker-arg","derived"),affordance("NEMOCLAW_HERMES_TOOL_GATEWAY_PRESETS_B64","tools.enabledGateways"),affordance("NEMOCLAW_HERMES_SWITCHYARD_ROUTING_B64","agentConfig.switchyardRouting","runtime-env"),affordance("NEMOCLAW_WEB_SEARCH_ENABLED","agentConfig.webSearch.enabled"),affordance("NEMOCLAW_WEB_SEARCH_PROVIDER","agentConfig.webSearch.provider"),affordance("NEMOCLAW_MESSAGING_PLAN_B64","messaging.plan"),affordance("CHAT_UI_URL","dashboard.browserUrl"),affordance("NEMOCLAW_DASHBOARD_PORT","dashboard.publicPort","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD","dashboard.mode","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD_PORT","dashboard.publicPort","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD_INTERNAL_PORT","dashboard.internalPort","runtime-env"),affordance("NEMOCLAW_HERMES_DASHBOARD_TUI","dashboard.tuiEnabled","runtime-env"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES],"langchain-deepagents-code":[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_UPSTREAM_ENDPOINT_URL","inference.upstreamEndpointUrl"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_REASONING_EFFORT","tuning.reasoningEffort"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_DCODE_AUTO_APPROVAL","agentConfig.autoApprovalMode"),affordance("NEMOCLAW_OBSERVABILITY","agentConfig.observabilityEnabled","runtime-env"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES],pi:[affordance("NEMOCLAW_MODEL","inference.model"),affordance("NEMOCLAW_INFERENCE_PROVIDER_ID","inference.routeProvider"),affordance("NEMOCLAW_UPSTREAM_PROVIDER","inference.upstreamProvider"),affordance("NEMOCLAW_INFERENCE_BASE_URL","inference.routedBaseUrl"),affordance("NEMOCLAW_INFERENCE_API","inference.api"),affordance("NEMOCLAW_CONTEXT_WINDOW","tuning.contextWindow"),affordance("NEMOCLAW_MAX_TOKENS","tuning.maxTokens"),affordance("NEMOCLAW_REASONING","tuning.reasoning"),affordance("NEMOCLAW_TOOL_DISCLOSURE","tools.disclosure"),affordance("NEMOCLAW_CORPORATE_CA_B64","corporateCa.bundleSha256","host-material","digest-handoff"),...HOST_PROXY_AFFORDANCES]};function deferredRuntimeInput(input,owner,reason,admission="managed-launch-forwarded"){return Object.freeze({input,owner,admission,reason})}var MANAGED_STARTUP_PROFILE_DEFERRED_RUNTIME_INPUTS=Object.freeze({openclaw:Object.freeze([deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_DEADLINE_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_FAST_DEADLINE_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_FAST_REENTRY_INTERVAL_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_FAST_REENTRY_POLLS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_RUN_TIMEOUT_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_AUTO_PAIR_SLOW_INTERVAL_SECS","application-environment","operator scheduler tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_MCP_SHADOW_DIAGNOSTICS","application-environment","operator shadow-diagnostics tuning is applied by the application environment transaction"),deferredRuntimeInput("NEMOCLAW_MCP_TOOLS_LIST_TIMEOUT_MS","application-environment","operator MCP discovery timeout tuning is applied by the application environment transaction"),deferredRuntimeInput("OPENCLAW_HOME","fixed-image-contract","the managed image and agent definition own this fixed runtime layout path"),deferredRuntimeInput("OPENCLAW_STATE_DIR","fixed-image-contract","the managed image and agent definition own this fixed runtime layout path"),deferredRuntimeInput("OPENCLAW_WORKSPACE_DIR","fixed-image-contract","the managed image and agent definition own this fixed runtime layout path"),deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")]),hermes:Object.freeze([deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")]),"langchain-deepagents-code":Object.freeze([deferredRuntimeInput("NEMOCLAW_SANDBOX_NAME","engine-identity","the lifecycle engine owns instance identity outside reusable startup intent"),deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")]),pi:Object.freeze([deferredRuntimeInput("NEMOCLAW_EXTRA_PLACEHOLDER_KEYS","credential-plumbing","credential provider construction owns key metadata outside the secret-free profile")])});function runtimeCleanupObligation(input,emittedFor,supportedFor,reason){return Object.freeze({input,emittedFor:Object.freeze([...emittedFor]),supportedFor:Object.freeze([...supportedFor]),owner:"application-environment",reason})}var MANAGED_STARTUP_RUNTIME_CLEANUP_OBLIGATIONS=Object.freeze([runtimeCleanupObligation("NEMOCLAW_DASHBOARD_BIND",["hermes"],["openclaw"],"generic managed-dashboard construction currently emits the OpenClaw-only bind control for Hermes"),runtimeCleanupObligation("NEMOCLAW_MINIMAL_BOOTSTRAP",["hermes","langchain-deepagents-code"],["openclaw"],"generic host-proxy construction currently emits the OpenClaw-only bootstrap control for other agents")]);var ManagedStartupProfileError=class extends Error{constructor(message){super(`Invalid managed startup profile: ${message}`);this.name="ManagedStartupProfileError"}};var PROFILE_KEYS=new Set(["schemaVersion","agent","agentConfig","inference","proxy","dashboard","tools","messaging","tuning","corporateCa"]);var INFERENCE_KEYS=new Set(["routeProvider","upstreamProvider","servingPreset","model","routedBaseUrl","upstreamEndpointUrl","api","primaryModelRef","compatibility","inputModalities"]);var PROXY_KEYS=new Set(["managedHost","managedPort","hostHttpUrl","hostHttpsUrl","hostNoProxy"]);var OPENCLAW_DASHBOARD_KEYS=new Set(["agent","mode","url","port","bindAddress","wslExposure"]);var HERMES_DASHBOARD_KEYS=new Set(["agent","mode","url","browserUrl","publicPort","internalPort","tuiEnabled"]);var DCODE_DASHBOARD_KEYS=new Set(["agent","mode"]);var TOOLS_KEYS=new Set(["disclosure","enabledGateways"]);var MESSAGING_KEYS=new Set(["plan"]);var TUNING_FIELD_ORDER=["contextWindow","maxTokens","reasoning","reasoningEffort"];var TUNING_KEYS=new Set(TUNING_FIELD_ORDER);var CORPORATE_CA_KEYS=new Set(["bundleSha256"]);var OPENCLAW_CONFIG_KEYS=new Set(["agent","webSearch","otel","agentTimeoutSeconds","heartbeatEvery","extraAgents","minimalBootstrap"]);var LEGACY_OPENCLAW_CONFIG_KEYS=new Set([...OPENCLAW_CONFIG_KEYS,"deviceAuth"]);var LEGACY_DEVICE_AUTH_KEYS=new Set(["disabled","optOutSource"]);var LEGACY_DEVICE_AUTH_OPT_OUT_SOURCES=new Set(["operator","managed-onboard"]);var HERMES_CONFIG_KEYS=new Set(["agent","webSearch","switchyardRouting"]);var DCODE_CONFIG_KEYS=new Set(["agent","autoApprovalMode","observabilityEnabled"]);var PI_CONFIG_KEYS=new Set(["agent"]);var PI_DASHBOARD_KEYS=new Set(["agent","mode"]);var WEB_SEARCH_KEYS=new Set(["enabled","provider"]);var OTEL_KEYS=new Set(["enabled","endpointUrl","serviceName","sampleRate"]);var EXTRA_AGENTS_KEYS=new Set(["agents","defaults","main"]);var MANAGED_STARTUP_AGENT_SET=new Set(MANAGED_STARTUP_AGENTS);var DCODE_AUTO_APPROVAL_MODE_SET=new Set(MANAGED_STARTUP_DCODE_AUTO_APPROVAL_MODES);var REASONING_EFFORT_SET=new Set(MANAGED_STARTUP_REASONING_EFFORTS);var HERMES_INTERNAL_API_PORT=18642;var HERMES_API_PORT_RANGE_START=8642;var HERMES_API_PORT_RANGE_END=8652;function isHermesApiPort(port){return port>=HERMES_API_PORT_RANGE_START&&port<=HERMES_API_PORT_RANGE_END}function isHermesReservedApiPort(port){return port===HERMES_INTERNAL_API_PORT||isHermesApiPort(port)}var HERMES_RESERVED_API_PORT_LABEL=`${HERMES_API_PORT_RANGE_START}-${HERMES_API_PORT_RANGE_END} or ${HERMES_INTERNAL_API_PORT}`;function isPlainObject(value){if(typeof value!=="object"||value===null||Array.isArray(value))return false;const prototype=Object.getPrototypeOf(value);return prototype===Object.prototype||prototype===null}function isCredentialShapedName(name){if(PUBLIC_KEY_NAME_PATTERN.test(name)||NON_SECRET_KEY_METADATA_NAMES.has(name))return false;return CREDENTIAL_SHAPED_NAME_PATTERN.test(name)||CREDENTIAL_COMPOUND_NAME_PATTERN.test(name)||CREDENTIAL_CAMEL_SUFFIX_PATTERN.test(name)||CREDENTIAL_CAMEL_BOUNDARY_PATTERN.test(name)||CREDENTIAL_ENV_NAME_PATTERN.test(name)||CREDENTIAL_HEADER_NAME_PATTERN.test(name)||PASS_CREDENTIAL_NAME_PATTERN.test(name)}function valueLooksLikeSecret(value){for(let index=0;index=5&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="agentRender"&&JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[3]??"")&&path4[4]==="value";const isAuthorizedBuildStepPlaceholder=allowedBuildStepPlaceholders.has(buildStepPlaceholderKey(path4,value));return isCredentialBindingPlaceholder||isAgentRenderValuePlaceholder||isAuthorizedBuildStepPlaceholder}function requiresMessagingSchemaFieldAuthorization(path4){const fieldName=path4[path4.length-1];return fieldName==="webhook"}function messagingAuthorizedFieldKey(path4){return JSON.stringify(path4)}function buildStepPlaceholderKey(path4,value){return JSON.stringify([path4,value])}function messagingCredentialPlaceholderEnvKey(value){if(!MESSAGING_CREDENTIAL_PLACEHOLDER_RE.test(value))return null;const marker=value.startsWith("openshell:resolve:env:")?"openshell:resolve:env:":"-OPENSHELL-RESOLVE-ENV-";const key=value.slice(value.indexOf(marker)+marker.length);return key.replace(/^(?:v[0-9]{1,20}|s[a-f0-9]{64})_/u,"")}function containsMessagingCredentialPlaceholder(value){return value.includes("openshell:resolve:env:")||value.includes("-OPENSHELL-RESOLVE-ENV-")}function isMessagingCredentialPlaceholderAssignment(selectedAgent,path4,value){if(path4.length!==6||path4[0]!=="messaging"||path4[1]!=="plan"||path4[2]!=="agentRender"||!JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[3]??"")||path4[4]!=="lines"||!JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[5]??"")){return false}const separator=value.indexOf("=");if(separator<=0||value.indexOf("=",separator+1)!==-1)return false;const envKey=value.slice(0,separator);const placeholder=value.slice(separator+1);const placeholderEnvKey=messagingCredentialPlaceholderEnvKey(placeholder);return CREDENTIAL_ENV_NAME_PATTERN.test(envKey)&&placeholderEnvKey!==null&&(envKey===placeholderEnvKey||typeof selectedAgent==="string"&&MESSAGING_CREDENTIAL_ENV_ALIASES.has(`${selectedAgent}\0${placeholderEnvKey}\0${envKey}`))}function isMessagingRuntimeEnvAliasPath(path4){return path4.length===5&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="runtimeSetup"&&path4[3]==="envAliases"&&JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[4]??"")}function ownDataPropertyValue4(value,key){const descriptor=Object.getOwnPropertyDescriptor(value,key);return descriptor&&"value"in descriptor?descriptor.value:void 0}function isCanonicalMessagingRuntimeEnvAlias(selectedAgent,path4,value){if(!isMessagingRuntimeEnvAliasPath(path4))return false;const channelId=ownDataPropertyValue4(value,"channelId");const envKey=ownDataPropertyValue4(value,"envKey");const targetEnvKey=ownDataPropertyValue4(value,"targetEnvKey");const match=ownDataPropertyValue4(value,"match");const placeholder=ownDataPropertyValue4(value,"value");const expectedMatch=targetEnvKey===void 0?`^openshell:resolve:env:((?:v[0-9]{1,20}|s[a-f0-9]{64})_)?${envKey}$`:`^openshell:resolve:env:(?:v[0-9]{1,20}|s[a-f0-9]{64})_${envKey}$`;return typeof envKey==="string"&&CREDENTIAL_ENV_NAME_PATTERN.test(envKey)&&match===expectedMatch&&typeof placeholder==="string"&&messagingCredentialPlaceholderEnvKey(placeholder)===envKey&&(targetEnvKey===void 0||typeof selectedAgent==="string"&&typeof channelId==="string"&&typeof targetEnvKey==="string"&&CREDENTIAL_ENV_NAME_PATTERN.test(targetEnvKey)&&MESSAGING_CREDENTIAL_RUNTIME_ENV_ALIASES.has(`${selectedAgent}\0${channelId}\0${envKey}\0${targetEnvKey}`))}function isAllowedMessagingRuntimeAliasStringPath(path4,allowedAliasIndexes){return path4.length===6&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="runtimeSetup"&&path4[3]==="envAliases"&&allowedAliasIndexes.has(path4[4]??"")&&(path4[5]==="match"||path4[5]==="value"||path4[5]==="targetEnvKey")}function isMessagingPackagePin(path4,value){return path4.length===6&&path4[0]==="messaging"&&path4[1]==="plan"&&path4[2]==="buildSteps"&&JSON_ARRAY_INDEX_SEGMENT_RE.test(path4[3]??"")&&path4[4]==="value"&&path4[5]==="pin"&&typeof value==="boolean"}function containsUrlWithCredentialMaterial(value){const candidates=value.match(URL_CANDIDATE_RE)??[];for(let index=0;index{if(isCredentialShapedName(key))credentialQuery=true});const fragment=url.hash.startsWith("#")?url.hash.slice(1):url.hash;const queryStart=fragment.indexOf("?");const fragmentParameters=new URLSearchParams(queryStart>=0?fragment.slice(queryStart+1):fragment);let credentialFragment=false;fragmentParameters.forEach((_fragmentValue,key)=>{if(isCredentialShapedName(key))credentialFragment=true});if(url.username||url.password||credentialQuery||credentialFragment)return true}catch{}}return false}function invalid(reason){throw new ManagedStartupProfileError(reason)}function payloadPath(path4){return path4.reduce((result,segment)=>segment.startsWith("[")?`${result}${segment}`:`${result}${result?".":""}${segment}`,"")}function mapArrayByIndex(values,mapper){const mapped=[];for(let index=0;index0&&values[insertion-1]>selected){Object.defineProperty(values,String(insertion),{configurable:true,enumerable:true,value:values[insertion-1],writable:true});insertion-=1}Object.defineProperty(values,String(insertion),{configurable:true,enumerable:true,value:selected,writable:true})}return values}function requireRecord(value,where){if(!isPlainObject(value))invalid(`${where} must be an object`);return value}function rejectUnknownKeys2(value,allowed,where){const keys=Object.keys(value);for(let index=0;indexmaxBytes||CONTROL_CHARACTER_RE.test(value)){invalid(`${where} must be a bounded, non-empty string without control characters`)}return value}function requireStringEnum(value,allowed,where){const normalized=requireBoundedString(value,where);if(!allowed.has(normalized))invalid(`${where} is not supported`);return normalized}function requireNullablePositiveInteger(value,where){if(value===null)return null;if(typeof value!=="number"||!Number.isSafeInteger(value)||value<1||value>MAX_TUNING_INTEGER){invalid(`${where} must be null or a bounded positive integer`)}return value}function requirePositiveInteger(value,where,maximum=MAX_TUNING_INTEGER){if(typeof value!=="number"||!Number.isSafeInteger(value)||value<1||value>maximum){invalid(`${where} must be a bounded positive integer`)}return value}function requirePort(value,where,minimum=1){if(typeof value!=="number"||!Number.isInteger(value)||value<1||value>65535){invalid(`${where} must be a valid TCP port`)}if(valueMAX_LIST_ITEMS){invalid(`${where} must be a bounded string list`)}const items=mapArrayByIndex(value,item=>requireBoundedString(item,`${where} item`));const unique2=new Set;for(let index=0;index{if(depth>MAX_JSON_DEPTH)invalid(`${where} exceeds the JSON depth limit`);if(current===null||typeof current==="string"||typeof current==="boolean"){return current}if(typeof current==="number"){if(!Number.isFinite(current))invalid(`${where} contains a non-finite number`);return current}if(Array.isArray(current)){return mapArrayByIndex(current,item=>clone(item,depth+1))}if(!isPlainObject(current))invalid(`${where} contains a non-JSON value`);const result=options.nullPrototypeObjects?Object.create(null):{};const keys=Object.getOwnPropertyNames(current);for(let index=0;indexMAX_IDENTIFIER_BYTES||CONTROL_CHARACTER_RE.test(key)){invalid(`${where} contains an invalid object key`)}const descriptor=Object.getOwnPropertyDescriptor(current,key);if(!descriptor||!("value"in descriptor)){invalid(`${where} contains a non-JSON value`)}Object.defineProperty(result,key,{configurable:true,enumerable:true,value:clone(descriptor.value,depth+1),writable:true})}return result};return clone(value,0)}function requireJsonObjectOrNull(value,where){if(value===null)return null;if(!isPlainObject(value))invalid(`${where} must be null or a plain JSON object`);return cloneJsonValue(value,where,{nullPrototypeObjects:true})}function requireJsonObject(value,where){const object=requireJsonObjectOrNull(value,where);if(object===null)invalid(`${where} must be a plain JSON object`);return object}function requireHttpUrl(value,where){const raw=requireBoundedString(value,where,MAX_URL_BYTES);let parsed;try{parsed=new URL(raw)}catch{invalid(`${where} must be a valid HTTP(S) URL`)}if(parsed.protocol!=="http:"&&parsed.protocol!=="https:"||parsed.username||parsed.password||parsed.search||parsed.hash){invalid(`${where} must be a credential-free HTTP(S) URL without query or fragment data`)}const pathname=parsed.pathname.replace(/\/+$/u,"");return pathname===""?parsed.origin:`${parsed.origin}${pathname}`}function requireProxyUrl(value,allowedSchemes,where){if(value===null)return null;const raw=requireBoundedString(value,where,MAX_URL_BYTES);let parsed;try{parsed=new URL(raw)}catch{invalid(`${where} must be a valid HTTP(S) proxy URL`)}if(!allowedSchemes.has(parsed.protocol)||parsed.username||parsed.password||parsed.pathname!=="/"||parsed.search||parsed.hash){invalid(`${where} must be a credential-free HTTP(S) proxy origin`)}return parsed.origin}function requireManagedProxyHost(value,where){const host=requireBoundedString(value,where);if(!/^[A-Za-z0-9._-]+$/u.test(host)){invalid(`${where} must be a hostname or IPv4 address without a scheme or separators`)}return host}function configuredDashboardPort(value){const explicit=new URL(value).port;return explicit===""?18789:Number(explicit)}function requireSampleRate(value,where){if(typeof value!=="number"||!Number.isFinite(value)||value<0||value>1){invalid(`${where} must be a number between 0 and 1`)}return value}function assertPayloadStructureAndCredentialShapes(root){const pending=[{value:root,depth:0,path:[]}];const allowedRuntimeAliasIndexes=new Set;const allowedMessagingCredentialFields=new Set;const allowedBuildStepPlaceholders=new Set;const selectedAgent=isPlainObject(root)?ownDataPropertyValue4(root,"agent"):void 0;let discoveredNodes=1;let observedBytes=0;const observeText=value=>{observedBytes+=import_node_buffer2.Buffer.byteLength(value,"utf8");if(observedBytes>MANAGED_STARTUP_PROFILE_MAX_BYTES){invalid(`payload exceeds ${String(MANAGED_STARTUP_PROFILE_MAX_BYTES)} bytes`)}};const reserveNode=depth=>{discoveredNodes+=1;if(discoveredNodes>MAX_JSON_NODES||depth>MAX_JSON_DEPTH){invalid("payload structure exceeds the complexity limit")}observedBytes+=1};while(pending.length>0){const current=pending.pop();if(!current)break;if(current.depth>MAX_JSON_DEPTH){invalid("payload structure exceeds the complexity limit")}if(typeof current.value==="string"){observeText(current.value);if(!isAllowedMessagingRuntimeAliasStringPath(current.path,allowedRuntimeAliasIndexes)&&!isMessagingCredentialPlaceholder(current.path,current.value,allowedBuildStepPlaceholders,allowedMessagingCredentialFields)&&!isMessagingCredentialPlaceholderAssignment(selectedAgent,current.path,current.value)&&(valueLooksLikeSecret(current.value)||containsMessagingCredentialPlaceholder(current.value))){invalid(`payload field ${payloadPath(current.path)} contains credential-shaped string data`)}if(RAW_CA_PEM_RE.test(current.value)||RAW_CA_PEM_BASE64_RE.test(current.value)||RAW_CA_DER_BASE64_RE.test(current.value)||RAW_CA_DATA_URI_RE.test(current.value)){invalid(`payload field ${payloadPath(current.path)} contains raw certificate data; provide only the CA SHA-256 digest`)}if(containsUrlWithCredentialMaterial(current.value)){invalid(`payload field ${payloadPath(current.path)} contains a URL with embedded credentials`)}continue}if(Array.isArray(current.value)){if(Object.getPrototypeOf(current.value)!==Array.prototype){invalid("payload arrays must use the standard JSON prototype")}if("toJSON"in current.value){invalid("payload must not define a custom JSON serializer")}if(Object.getOwnPropertySymbols(current.value).length>0||Object.getOwnPropertyNames(current.value).length!==current.value.length+1){invalid("payload arrays must contain only indexed JSON values")}for(let index=0;index0||discoveredNodes+keys.length>MAX_JSON_NODES){invalid("payload structure exceeds the complexity limit")}for(let index=0;indexMANAGED_STARTUP_PROFILE_MAX_BYTES){invalid(`payload exceeds ${String(MANAGED_STARTUP_PROFILE_MAX_BYTES)} bytes`)}}function validateWebSearch(value,agent){const webSearch=requireRecord(value,"agentConfig.webSearch");rejectUnknownKeys2(webSearch,WEB_SEARCH_KEYS,"agentConfig.webSearch");const provider2=requireStringEnum(webSearch.provider,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].webSearchProviders),"agentConfig.webSearch.provider");return{enabled:requireBoolean(webSearch.enabled,"agentConfig.webSearch.enabled"),provider:provider2}}function validateOpenClawOtel(value){const otel=requireRecord(value,"agentConfig.otel");rejectUnknownKeys2(otel,OTEL_KEYS,"agentConfig.otel");return{enabled:requireBoolean(otel.enabled,"agentConfig.otel.enabled"),endpointUrl:requireHttpUrl(otel.endpointUrl,"agentConfig.otel.endpointUrl"),serviceName:requireBoundedString(otel.serviceName,"agentConfig.otel.serviceName",MAX_IDENTIFIER_BYTES),sampleRate:requireSampleRate(otel.sampleRate,"agentConfig.otel.sampleRate")}}function validateExtraAgents(value){const extraAgents=requireRecord(value,"agentConfig.extraAgents");rejectUnknownKeys2(extraAgents,EXTRA_AGENTS_KEYS,"agentConfig.extraAgents");if(!Array.isArray(extraAgents.agents)||extraAgents.agents.length>MAX_LIST_ITEMS){invalid("agentConfig.extraAgents.agents must be a bounded JSON object list")}return{agents:mapArrayByIndex(extraAgents.agents,(agent,index)=>requireJsonObject(agent,`agentConfig.extraAgents.agents[${String(index)}]`)),defaults:requireJsonObject(extraAgents.defaults,"agentConfig.extraAgents.defaults"),main:requireJsonObject(extraAgents.main,"agentConfig.extraAgents.main")}}function validateAgentConfig(value,expectedAgent){const config=requireRecord(value,"agentConfig");const agent=requireStringEnum(config.agent,MANAGED_STARTUP_AGENT_SET,"agentConfig.agent");if(agent!==expectedAgent)invalid("agentConfig.agent must match agent");if(agent==="openclaw"){rejectUnknownKeys2(config,OPENCLAW_CONFIG_KEYS,"agentConfig");const heartbeatEvery=config.heartbeatEvery===null?null:requireBoundedString(config.heartbeatEvery,"agentConfig.heartbeatEvery",MAX_IDENTIFIER_BYTES);if(heartbeatEvery!==null&&!/^\d+(?:s|m|h)$/u.test(heartbeatEvery)){invalid("agentConfig.heartbeatEvery must be null or a duration ending in s, m, or h")}return{agent,webSearch:validateWebSearch(config.webSearch,agent),otel:validateOpenClawOtel(config.otel),agentTimeoutSeconds:requirePositiveInteger(config.agentTimeoutSeconds,"agentConfig.agentTimeoutSeconds"),heartbeatEvery,extraAgents:validateExtraAgents(config.extraAgents),minimalBootstrap:requireBoolean(config.minimalBootstrap,"agentConfig.minimalBootstrap")}}if(agent==="hermes"){rejectUnknownKeys2(config,HERMES_CONFIG_KEYS,"agentConfig");const hasSwitchyardRouting=Object.hasOwn(config,"switchyardRouting");const switchyardRouting=!hasSwitchyardRouting||config.switchyardRouting===null?null:validateHermesSwitchyardRoutingIntent(config.switchyardRouting);return{agent,webSearch:validateWebSearch(config.webSearch,agent),...hasSwitchyardRouting?{switchyardRouting}:{}}}if(agent==="pi"){rejectUnknownKeys2(config,PI_CONFIG_KEYS,"agentConfig");return{agent}}rejectUnknownKeys2(config,DCODE_CONFIG_KEYS,"agentConfig");return{agent,autoApprovalMode:requireStringEnum(config.autoApprovalMode,DCODE_AUTO_APPROVAL_MODE_SET,"agentConfig.autoApprovalMode"),observabilityEnabled:requireBoolean(config.observabilityEnabled,"agentConfig.observabilityEnabled")}}function validateDashboard(value,expectedAgent){const dashboard=requireRecord(value,"dashboard");const agent=requireStringEnum(dashboard.agent,MANAGED_STARTUP_AGENT_SET,"dashboard.agent");if(agent!==expectedAgent)invalid("dashboard.agent must match agent");if(agent==="openclaw"){rejectUnknownKeys2(dashboard,OPENCLAW_DASHBOARD_KEYS,"dashboard");const mode=requireStringEnum(dashboard.mode,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].dashboardModes),"dashboard.mode");const url=requireHttpUrl(dashboard.url,"dashboard.url");const bindAddress=requireStringEnum(dashboard.bindAddress,new Set(["127.0.0.1","0.0.0.0"]),"dashboard.bindAddress");const wslExposure=requireBoolean(dashboard.wslExposure,"dashboard.wslExposure");const hasRemoteExposure=!isLoopbackDashboardUrl(url)||bindAddress==="0.0.0.0"||wslExposure;if(mode==="remote"!==hasRemoteExposure){invalid("OpenClaw dashboard.mode must reflect its URL, bind address, and WSL exposure")}const port=requirePort(dashboard.port,"dashboard.port",1024);if(isHermesApiPort(port))invalid(`OpenClaw dashboard.port must not use a reserved Hermes API port (${HERMES_API_PORT_RANGE_START}-${HERMES_API_PORT_RANGE_END})`);if(configuredDashboardPort(url)!==port){invalid("OpenClaw dashboard.port must match dashboard.url")}return{agent,mode,url,port,bindAddress,wslExposure}}if(agent==="hermes"){rejectUnknownKeys2(dashboard,HERMES_DASHBOARD_KEYS,"dashboard");const mode=requireStringEnum(dashboard.mode,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].dashboardModes),"dashboard.mode");const url=requireHttpUrl(dashboard.url,"dashboard.url");if(!isLoopbackDashboardUrl(url)){invalid("Hermes dashboard.url must remain loopback; OpenShell owns the host forward")}const browserUrl=dashboard.browserUrl===void 0?void 0:requireHttpUrl(dashboard.browserUrl,"dashboard.browserUrl");if(browserUrl!==void 0&&!isLoopbackDashboardUrl(browserUrl)&&new URL(browserUrl).protocol!=="https:"){invalid("Hermes dashboard.browserUrl must use HTTPS unless it is loopback")}if(mode==="disabled"){if(dashboard.publicPort!==null||dashboard.internalPort!==null||dashboard.tuiEnabled!==false){invalid("disabled Hermes dashboard must not configure ports or TUI")}return{agent,mode,url,...browserUrl===void 0?{}:{browserUrl},publicPort:null,internalPort:null,tuiEnabled:false}}const publicPort=requirePort(dashboard.publicPort,"dashboard.publicPort",1024);const internalPort=requirePort(dashboard.internalPort,"dashboard.internalPort",1024);if(publicPort===internalPort){invalid("Hermes dashboard publicPort and internalPort must differ")}if(isHermesReservedApiPort(publicPort)||isHermesReservedApiPort(internalPort)){invalid(`Hermes dashboard ports must not use reserved API ports ${HERMES_RESERVED_API_PORT_LABEL}`)}if(configuredDashboardPort(url)!==publicPort){invalid("Hermes dashboard.publicPort must match dashboard.url")}if(browserUrl!==void 0&&isLoopbackDashboardUrl(browserUrl)&&configuredDashboardPort(browserUrl)!==publicPort){invalid("Hermes dashboard.publicPort must match dashboard.browserUrl")}return{agent,mode,url,...browserUrl===void 0?{}:{browserUrl},publicPort,internalPort,tuiEnabled:requireBoolean(dashboard.tuiEnabled,"dashboard.tuiEnabled")}}if(agent==="pi"){rejectUnknownKeys2(dashboard,PI_DASHBOARD_KEYS,"dashboard");if(dashboard.mode!=="disabled")invalid("pi dashboard.mode must be disabled");return{agent,mode:"disabled"}}rejectUnknownKeys2(dashboard,DCODE_DASHBOARD_KEYS,"dashboard");if(dashboard.mode!=="disabled"){invalid("langchain-deepagents-code dashboard.mode must be disabled")}return{agent,mode:"disabled"}}function validateInference(value,agent){if(value===null){if(agent!=="openclaw"&&agent!=="hermes")invalid(`${agent} requires inference configuration`);return null}const inference=requireRecord(value,"inference");rejectUnknownKeys2(inference,INFERENCE_KEYS,"inference");const routeProvider=requireBoundedString(inference.routeProvider,"inference.routeProvider");const upstreamProvider=requireBoundedString(inference.upstreamProvider,"inference.upstreamProvider");const servingPreset=inference.servingPreset===void 0?void 0:inference.servingPreset===null?null:requireBoundedString(inference.servingPreset,"inference.servingPreset");const model2=requireBoundedString(inference.model,"inference.model",MAX_MODEL_BYTES2);const api=requireStringEnum(inference.api,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].inferenceApis),"inference.api");const upstreamEndpointUrl=inference.upstreamEndpointUrl===null?null:requireHttpUrl(inference.upstreamEndpointUrl,"inference.upstreamEndpointUrl");const primaryModelRef=inference.primaryModelRef===null?null:requireBoundedString(inference.primaryModelRef,"inference.primaryModelRef",MAX_MODEL_BYTES2);const compatibility=requireJsonObjectOrNull(inference.compatibility,"inference.compatibility");const inputModalities=inference.inputModalities===null?null:requireEnumList(inference.inputModalities,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].inputModalities),"inference.inputModalities",{allowEmpty:false});if(upstreamEndpointUrl!==null&&!MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].supportsUpstreamEndpoint){invalid(`inference.upstreamEndpointUrl must be null for ${agent}`)}if(agent==="openclaw"){if(primaryModelRef===null||inputModalities===null){invalid("openclaw requires primaryModelRef and inputModalities")}if(primaryModelRef!==`${routeProvider}/${model2}`){invalid("openclaw primaryModelRef must match routeProvider and model")}}else{if(primaryModelRef!==null||compatibility!==null||inputModalities!==null){invalid(`${agent} does not support primaryModelRef, compatibility, or inputModalities`)}if(agent==="langchain-deepagents-code"&&!isValidDcodeUpstreamProvider(upstreamProvider)){invalid("inference.upstreamProvider must start with an ASCII letter or digit and contain 1-64 ASCII letters, digits, dots, underscores, or hyphens for DCode")}}return{routeProvider,upstreamProvider,...servingPreset===void 0?{}:{servingPreset},model:model2,routedBaseUrl:requireHttpUrl(inference.routedBaseUrl,"inference.routedBaseUrl"),upstreamEndpointUrl,api,primaryModelRef,compatibility,inputModalities}}function validateProxy(value,agent){const proxy=requireRecord(value,"proxy");rejectUnknownKeys2(proxy,PROXY_KEYS,"proxy");const hostHttpUrl=requireProxyUrl(proxy.hostHttpUrl,new Set(["http:"]),"proxy.hostHttpUrl");const hostHttpsUrl=requireProxyUrl(proxy.hostHttpsUrl,new Set(["http:","https:"]),"proxy.hostHttpsUrl");const hostNoProxy=requireStringList(proxy.hostNoProxy,"proxy.hostNoProxy");if(!MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].supportsHostProxyIntent&&(hostHttpUrl!==null||hostHttpsUrl!==null||hostNoProxy.length>0)){invalid(`${agent} rejects host proxy intent and accepts only its root-owned managed route`)}return{managedHost:requireManagedProxyHost(proxy.managedHost,"proxy.managedHost"),managedPort:requirePort(proxy.managedPort,"proxy.managedPort"),hostHttpUrl,hostHttpsUrl,hostNoProxy}}function validateTools(value,agent){const tools=requireRecord(value,"tools");rejectUnknownKeys2(tools,TOOLS_KEYS,"tools");const enabledGateways=requireEnumList(tools.enabledGateways,new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].toolGateways),"tools.enabledGateways",{allowEmpty:true});return{disclosure:requireStringEnum(tools.disclosure,new Set(["progressive","direct"]),"tools.disclosure"),enabledGateways}}function validateTuning(value,agent){const tuning=requireRecord(value,"tuning");rejectUnknownKeys2(tuning,TUNING_KEYS,"tuning");const result={contextWindow:requireNullablePositiveInteger(tuning.contextWindow,"tuning.contextWindow"),maxTokens:requireNullablePositiveInteger(tuning.maxTokens,"tuning.maxTokens"),reasoning:requireNullableBoolean(tuning.reasoning,"tuning.reasoning"),reasoningEffort:tuning.reasoningEffort===null?null:requireStringEnum(tuning.reasoningEffort,REASONING_EFFORT_SET,"tuning.reasoningEffort")};const advertised=new Set(MANAGED_STARTUP_PROFILE_CAPABILITIES[agent].tuningFields);const unsupported=TUNING_FIELD_ORDER.filter(field=>result[field]!==null&&!advertised.has(field));if(unsupported.length>0){invalid(`${agent} does not support startup tuning fields: ${unsupported.join(", ")}`)}if(agent==="openclaw"){const missing=TUNING_FIELD_ORDER.filter(field=>advertised.has(field)&&result[field]===null);if(missing.length>0){invalid(`openclaw requires ${missing.join(", ")} tuning`)}}if(agent==="hermes"&&result.contextWindow!==null&&result.contextWindowcanonicalizeJson(item));if(!isPlainObject(value))return value;const result={};const keys=sortStrings(Object.keys(value));for(let index=0;indexMANAGED_STARTUP_PROFILE_MAX_BYTES){invalid(`canonical payload exceeds ${String(MANAGED_STARTUP_PROFILE_MAX_BYTES)} bytes`)}return serialized}function decodeManagedStartupProfile(encoded){if(typeof encoded!=="string"||encoded.length===0||import_node_buffer2.Buffer.byteLength(encoded,"ascii")>MANAGED_STARTUP_PROFILE_MAX_ENCODED_BYTES||!BASE64URL_RE.test(encoded)||encoded.length%4===1){invalid("encoded payload is malformed or exceeds the size limit")}const bytes=import_node_buffer2.Buffer.from(encoded,"base64url");if(bytes.length===0||bytes.length>MANAGED_STARTUP_PROFILE_MAX_BYTES||bytes.toString("base64url")!==encoded){invalid("encoded payload is malformed or exceeds the size limit")}let raw;try{raw=UTF8_DECODER.decode(bytes)}catch{invalid("payload is not valid UTF-8")}let parsed;try{parsed=JSON.parse(raw)}catch{invalid("payload is not valid JSON")}const migration=migrateDecodedManagedStartupProfile(parsed);const profile=validateManagedStartupProfile(migration.value);const canonicalPayload=migration.migratedLegacyProfile?JSON.stringify(canonicalizeJson(parsed)):serializeManagedStartupProfile(profile);if(canonicalPayload!==raw){invalid("payload is not in canonical form")}return profile}function fingerprintManagedStartupProfile(profile){return(0,import_node_crypto.createHash)("sha256").update(serializeManagedStartupProfile(profile),"utf8").digest("hex")}var MANAGED_STARTUP_APPLICATION_STATE_DIR="/var/lib/nemoclaw/startup-profile";var MANAGED_STARTUP_CA_MAX_BYTES=128*1024;var MANAGED_STARTUP_CA_MAX_CERTIFICATES=24;var STATE_SCHEMA_VERSION=1;var STATE_DIRECTORY_MODE=448;var STATE_FILE_MODE=384;var MAX_CONTROL_FILE_BYTES=512;var MAX_STATE_ENTRIES=32;var SHA256_RE2=/^[a-f0-9]{64}$/u;var GENERATION_RE=/^generation-([a-f0-9]{64})$/u;var PREPARE_TEMP_RE=/^\.prepare-[0-9]+-[a-f0-9]{24}$/u;var CONTROL_TEMP_RE=/^\.(?:committed|pending)\.json-[a-f0-9]{24}\.tmp$/u;var PEM_CERTIFICATE_RE=/-----BEGIN CERTIFICATE-----\r?\n[A-Za-z0-9+/=\r\n]+?-----END CERTIFICATE-----/gu;var UTF8_DECODER2=new import_node_util2.TextDecoder("utf-8",{fatal:true});var DEFAULT_RUNTIME={rootUid:0,rootGid:0};var ManagedStartupApplicationError=class extends Error{constructor(message){super(`Managed startup application failed: ${message}`);this.name="ManagedStartupApplicationError"}};function fail2(message){throw new ManagedStartupApplicationError(message)}function runtimeFor(override){return override??DEFAULT_RUNTIME}function requireContainerRoot(){if(process.geteuid?.()!==0){fail2("the image-side applicator must run with effective uid 0")}}function modeOf(stat){return stat.mode&511}function requireOwner(stat,target2,runtime){if(stat.uid!==runtime.rootUid||stat.gid!==runtime.rootGid){fail2(`${target2} must be owned by root:root`)}}function requireSecureDirectory(target2,runtime,exactMode){let stat;try{stat=import_node_fs.default.lstatSync(target2)}catch{fail2(`state directory component is missing or unreadable: ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail2(`state directory component must be a real directory: ${target2}`)}const runtimeOwned=stat.uid===runtime.rootUid&&stat.gid===runtime.rootGid;const systemRootOwned=stat.uid===0&&stat.gid===0;if(exactMode){requireOwner(stat,target2,runtime)}else if(!runtimeOwned&&!systemRootOwned){fail2(`state directory ancestor is not owned by a trusted identity: ${target2}`)}const mode=modeOf(stat);const writableByUntrustedIdentity=(mode&18)!==0;const trustedStickyRoot=(stat.mode&512)!==0&&(runtimeOwned||systemRootOwned);if(exactMode&&mode!==STATE_DIRECTORY_MODE||!exactMode&&writableByUntrustedIdentity&&!trustedStickyRoot){fail2(exactMode?`${target2} must have mode 0700`:`${target2} is a replaceable group- or world-writable ancestor`)}}function requireSecureAncestors(target2,runtime){const root=import_node_path.default.parse(target2).root;let current=root;requireSecureDirectory(current,runtime,false);for(const segment of import_node_path.default.relative(root,target2).split(import_node_path.default.sep).filter(Boolean)){current=import_node_path.default.join(current,segment);let stat;try{stat=import_node_fs.default.lstatSync(current)}catch{fail2(`state directory component is missing or unreadable: ${current}`)}if(stat.isSymbolicLink()){const runtimeOwned=stat.uid===runtime.rootUid&&stat.gid===runtime.rootGid;const systemRootOwned=stat.uid===0&&stat.gid===0;if(!runtimeOwned&&!systemRootOwned){fail2(`state directory ancestor is a replaceable symlink: ${current}`)}let resolved;try{resolved=import_node_fs.default.realpathSync(current)}catch{fail2(`state directory symlink is missing or unreadable: ${current}`)}requireSecureAncestors(resolved,runtime);continue}requireSecureDirectory(current,runtime,false)}}function ensureStateDirectory(rawStateDirectory,runtime){const stateDirectory=rawStateDirectory??MANAGED_STARTUP_APPLICATION_STATE_DIR;if(!import_node_path.default.isAbsolute(stateDirectory)||stateDirectory.includes("\0")){fail2("stateDirectory must be an absolute path")}const normalized=import_node_path.default.resolve(stateDirectory);const parent=import_node_path.default.dirname(normalized);requireSecureAncestors(parent,runtime);try{import_node_fs.default.mkdirSync(normalized,{mode:STATE_DIRECTORY_MODE});import_node_fs.default.chownSync(normalized,runtime.rootUid,runtime.rootGid);import_node_fs.default.chmodSync(normalized,STATE_DIRECTORY_MODE)}catch(error){if(error.code!=="EEXIST"){fail2(`could not create the managed startup state directory: ${normalized}`)}}requireSecureDirectory(normalized,runtime,true);return normalized}function requireSecureRegularFileStat(stat,target2,runtime){if(!stat.isFile()||stat.isSymbolicLink()){fail2(`${target2} must be a regular file`)}if(stat.nlink!==1){fail2(`${target2} must not be hardlinked`)}requireOwner(stat,target2,runtime);if(modeOf(stat)!==STATE_FILE_MODE){fail2(`${target2} must have mode 0600`)}}function readSecureFile(target2,maxBytes,runtime){let descriptor;try{descriptor=import_node_fs.default.openSync(target2,import_node_fs.default.constants.O_RDONLY|import_node_fs.default.constants.O_NOFOLLOW)}catch{fail2(`state file is missing, unreadable, or a symlink: ${target2}`)}try{const stat=import_node_fs.default.fstatSync(descriptor);requireSecureRegularFileStat(stat,target2,runtime);if(stat.size<1||stat.size>maxBytes){fail2(`${target2} is empty or exceeds its size limit`)}const content=import_node_fs.default.readFileSync(descriptor);if(content.length!==stat.size){fail2(`${target2} changed while it was being read`)}return content}finally{import_node_fs.default.closeSync(descriptor)}}function writeSecureNewFile(target2,content,runtime){let descriptor;try{descriptor=import_node_fs.default.openSync(target2,import_node_fs.default.constants.O_CREAT|import_node_fs.default.constants.O_EXCL|import_node_fs.default.constants.O_WRONLY|import_node_fs.default.constants.O_NOFOLLOW,STATE_FILE_MODE)}catch{fail2(`refused to replace an existing state file: ${target2}`)}try{import_node_fs.default.fchownSync(descriptor,runtime.rootUid,runtime.rootGid);import_node_fs.default.fchmodSync(descriptor,STATE_FILE_MODE);import_node_fs.default.writeFileSync(descriptor,content);import_node_fs.default.fsyncSync(descriptor)}finally{import_node_fs.default.closeSync(descriptor)}}function syncDirectory(target2){const descriptor=import_node_fs.default.openSync(target2,import_node_fs.default.constants.O_RDONLY);try{import_node_fs.default.fsyncSync(descriptor)}finally{import_node_fs.default.closeSync(descriptor)}}function randomToken(){return(0,import_node_crypto2.randomBytes)(12).toString("hex")}function stateControl(fingerprint){return{schemaVersion:STATE_SCHEMA_VERSION,fingerprint,generation:`generation-${fingerprint}`}}function serializeStateControl(control){return JSON.stringify({fingerprint:control.fingerprint,generation:control.generation,schemaVersion:control.schemaVersion})}function parseStateControl(target2,runtime){const bytes=readSecureFile(target2,MAX_CONTROL_FILE_BYTES,runtime);let raw;try{raw=UTF8_DECODER2.decode(bytes)}catch{fail2(`${target2} is not valid UTF-8`)}let parsed;try{parsed=JSON.parse(raw)}catch{fail2(`${target2} is not valid JSON`)}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail2(`${target2} does not contain a valid state control`)}const record2=parsed;if(Object.keys(record2).sort().join(",")!=="fingerprint,generation,schemaVersion"||record2.schemaVersion!==STATE_SCHEMA_VERSION||typeof record2.fingerprint!=="string"||!SHA256_RE2.test(record2.fingerprint)||record2.generation!==`generation-${record2.fingerprint}`){fail2(`${target2} does not contain a valid state control`)}const control=stateControl(record2.fingerprint);if(serializeStateControl(control)!==raw){fail2(`${target2} is not in canonical form`)}return control}function publishStateControlIfAbsent(stateDirectory,basename,control,runtime){const target2=import_node_path.default.join(stateDirectory,basename);const temporary=import_node_path.default.join(stateDirectory,`.${basename}-${randomToken()}.tmp`);writeSecureNewFile(temporary,serializeStateControl(control),runtime);try{import_node_fs.default.linkSync(temporary,target2)}catch(error){try{unlinkSecureControlOrTemp(temporary,runtime)}catch{}if(error.code==="EEXIST"){return{control:parseStateControl(target2,runtime),created:false}}fail2(`could not atomically publish ${basename}`)}try{import_node_fs.default.unlinkSync(temporary)}catch(error){if(error.code!=="ENOENT"){fail2(`could not finalize atomic publication of ${basename}`)}}syncDirectory(stateDirectory);return{control,created:true}}function validateCorporateCaBytes(bytes){if(bytes.length<1||bytes.length>MANAGED_STARTUP_CA_MAX_BYTES){fail2(`corporate CA bundle must contain 1-${String(MANAGED_STARTUP_CA_MAX_BYTES)} bytes`)}let pem;try{pem=UTF8_DECODER2.decode(bytes)}catch{fail2("corporate CA bundle must be valid UTF-8 PEM")}const matches=[...pem.matchAll(PEM_CERTIFICATE_RE)];if(matches.length<1||matches.length>MANAGED_STARTUP_CA_MAX_CERTIFICATES||matches[0]?.index!==0){fail2(`corporate CA bundle must contain 1-${String(MANAGED_STARTUP_CA_MAX_CERTIFICATES)} PEM CA certificates`)}let cursor=0;for(const match of matches){const index=match.index;if(index===void 0||!/^(?:\r?\n)+$/u.test(pem.slice(cursor,index))&&index!==0){fail2("corporate CA bundle contains non-PEM material between certificates")}const block=match[0];let certificate;try{certificate=new import_node_crypto2.X509Certificate(block)}catch{fail2("corporate CA bundle contains an invalid X.509 certificate")}if(!certificate.ca){fail2("corporate CA bundle contains a certificate without basicConstraints CA:TRUE")}cursor=index+block.length}if(!/^(?:\r?\n)?$/u.test(pem.slice(cursor))){fail2("corporate CA bundle contains trailing non-PEM material")}}function validateManagedStartupCorporateCaTransport(encoded,profile){const expectedDigest=profile.corporateCa.bundleSha256;if(expectedDigest===null){if(encoded!==void 0){fail2("corporate CA transport must be absent when the profile has no CA digest")}return null}if(typeof encoded!=="string"||encoded.length===0||encoded.length>Math.ceil(MANAGED_STARTUP_CA_MAX_BYTES/3)*4||!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u.test(encoded)){fail2("corporate CA transport must be canonical standard base64")}const bytes=import_node_buffer3.Buffer.from(encoded,"base64");if(bytes.toString("base64")!==encoded){fail2("corporate CA transport must be canonical standard base64")}validateCorporateCaBytes(bytes);const actualDigest=(0,import_node_crypto2.createHash)("sha256").update(bytes).digest("hex");if(actualDigest!==expectedDigest){fail2("corporate CA bundle does not match the profile SHA-256 digest")}return bytes}function readCanonicalProfile(profilePath,runtime){const bytes=readSecureFile(profilePath,MANAGED_STARTUP_PROFILE_MAX_BYTES,runtime);let raw;try{raw=UTF8_DECODER2.decode(bytes)}catch{fail2(`${profilePath} is not valid UTF-8`)}let parsed;try{parsed=JSON.parse(raw)}catch{fail2(`${profilePath} is not valid JSON`)}let profile;try{profile=validateManagedStartupProfile(parsed)}catch(error){fail2(`${profilePath} is invalid: ${error.message}`)}if(serializeManagedStartupProfile(profile)!==raw){fail2(`${profilePath} is not a canonical managed startup profile`)}return{profile,fingerprint:fingerprintManagedStartupProfile(profile)}}function validateGeneration(stateDirectory,control,runtime,expectedAgent){if(!GENERATION_RE.test(control.generation)){fail2("state control names an invalid generation")}const directory=import_node_path.default.join(stateDirectory,control.generation);requireSecureDirectory(directory,runtime,true);const entries=import_node_fs.default.readdirSync(directory).sort();if(entries.some(entry=>entry!=="profile.json"&&entry!=="corporate-ca.pem")||!entries.includes("profile.json")){fail2(`${directory} contains missing or unsupported state files`)}const profilePath=import_node_path.default.join(directory,"profile.json");const{profile,fingerprint}=readCanonicalProfile(profilePath,runtime);if(fingerprint!==control.fingerprint){fail2(`${directory} does not match its recorded profile fingerprint`)}if(expectedAgent!==void 0&&profile.agent!==expectedAgent){fail2(`managed startup profile targets ${profile.agent}, expected ${expectedAgent}`)}const caPath=import_node_path.default.join(directory,"corporate-ca.pem");let corporateCaPath=null;if(profile.corporateCa.bundleSha256===null){if(entries.includes("corporate-ca.pem")){fail2(`${directory} contains a CA bundle that is absent from the profile`)}}else{if(!entries.includes("corporate-ca.pem")){fail2(`${directory} is missing the CA bundle recorded by the profile`)}const caBytes=readSecureFile(caPath,MANAGED_STARTUP_CA_MAX_BYTES,runtime);validateCorporateCaBytes(caBytes);if((0,import_node_crypto2.createHash)("sha256").update(caBytes).digest("hex")!==profile.corporateCa.bundleSha256){fail2(`${directory} contains a CA bundle with the wrong SHA-256 digest`)}corporateCaPath=caPath}return{directory,profilePath,corporateCaPath,profile,fingerprint}}function validateDisposableDirectory(target2,runtime){requireSecureDirectory(target2,runtime,true);const entries=import_node_fs.default.readdirSync(target2);if(entries.length>2||entries.some(entry=>entry!=="profile.json"&&entry!=="corporate-ca.pem")){fail2(`${target2} is not a recognized disposable generation`)}for(const entry of entries){const file=import_node_path.default.join(target2,entry);const stat=import_node_fs.default.lstatSync(file);requireSecureRegularFileStat(stat,file,runtime)}}function discardDirectory(target2,runtime){validateDisposableDirectory(target2,runtime);import_node_fs.default.rmSync(target2,{recursive:true})}function discardDirectoryIfPresent(target2,runtime){try{import_node_fs.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return false;fail2(`could not inspect disposable generation ${target2}`)}discardDirectory(target2,runtime);return true}function unlinkSecureControlOrTemp(target2,runtime){const stat=import_node_fs.default.lstatSync(target2);requireSecureRegularFileStat(stat,target2,runtime);if(stat.size>MAX_CONTROL_FILE_BYTES){fail2(`${target2} exceeds the state-control size limit`)}import_node_fs.default.unlinkSync(target2)}function listStateEntries(stateDirectory){const entries=import_node_fs.default.readdirSync(stateDirectory).sort();if(entries.length>MAX_STATE_ENTRIES){fail2(`state directory exceeds ${String(MAX_STATE_ENTRIES)} entries`)}return entries}function unlinkRecoverableControlTemp(stateDirectory,entry,runtime){const temporary=import_node_path.default.join(stateDirectory,entry);const stat=import_node_fs.default.lstatSync(temporary);if(stat.nlink===1){unlinkSecureControlOrTemp(temporary,runtime);return}const basename=entry.startsWith(".committed.json-")?"committed.json":entry.startsWith(".pending.json-")?"pending.json":null;const target2=basename===null?null:import_node_path.default.join(stateDirectory,basename);let targetStat=null;try{targetStat=target2===null?null:import_node_fs.default.lstatSync(target2)}catch{fail2(`refused to remove an unpaired atomic-control temporary file: ${temporary}`)}if(stat.nlink!==2||targetStat===null||stat.dev!==targetStat.dev||stat.ino!==targetStat.ino||!stat.isFile()||stat.isSymbolicLink()||modeOf(stat)!==STATE_FILE_MODE||stat.size<1||stat.size>MAX_CONTROL_FILE_BYTES){fail2(`refused to remove an unpaired atomic-control temporary file: ${temporary}`)}requireOwner(stat,temporary,runtime);requireOwner(targetStat,target2,runtime);import_node_fs.default.unlinkSync(temporary)}function cleanAtomicTemps(stateDirectory,entries,runtime){let changed=false;for(const entry of entries){const target2=import_node_path.default.join(stateDirectory,entry);if(PREPARE_TEMP_RE.test(entry)){discardDirectory(target2,runtime);changed=true}else if(CONTROL_TEMP_RE.test(entry)){unlinkRecoverableControlTemp(stateDirectory,entry,runtime);changed=true}}if(changed)syncDirectory(stateDirectory)}function requireKnownStateEntries(stateDirectory,entries){for(const entry of entries){if(entry==="committed.json"||entry==="pending.json"||GENERATION_RE.test(entry)||PREPARE_TEMP_RE.test(entry)||CONTROL_TEMP_RE.test(entry)){continue}fail2(`${stateDirectory} contains unsupported state component ${entry}`)}}function discardGenerationsExcept(stateDirectory,keepGeneration,runtime){for(const entry of listStateEntries(stateDirectory)){if(GENERATION_RE.test(entry)&&entry!==keepGeneration){discardDirectoryIfPresent(import_node_path.default.join(stateDirectory,entry),runtime)}}}function optionalStateControl(stateDirectory,basename,runtime){const target2=import_node_path.default.join(stateDirectory,basename);try{import_node_fs.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return null;fail2(`could not inspect ${target2}`)}return parseStateControl(target2,runtime)}function removePendingControl(stateDirectory,runtime){try{unlinkSecureControlOrTemp(import_node_path.default.join(stateDirectory,"pending.json"),runtime)}catch(error){if(error.code==="ENOENT")return;throw error}syncDirectory(stateDirectory)}function stateControlsMatch(left,right){return left.fingerprint===right.fingerprint&&left.generation===right.generation}function recoverCommittedState(stateDirectory,committedControl,pendingControl,requested,expectedAgent,runtime){const committed=validateGeneration(stateDirectory,committedControl,runtime,expectedAgent);if(pendingControl)removePendingControl(stateDirectory,runtime);discardGenerationsExcept(stateDirectory,committedControl.generation,runtime);syncDirectory(stateDirectory);if(!stateControlsMatch(committedControl,requested)){fail2("a different startup profile is already committed; recreate the sandbox to change it")}return committed}function recoverState(stateDirectory,requested,expectedAgent,runtime){const initialEntries=listStateEntries(stateDirectory);requireKnownStateEntries(stateDirectory,initialEntries);cleanAtomicTemps(stateDirectory,initialEntries,runtime);const initiallyCommittedControl=optionalStateControl(stateDirectory,"committed.json",runtime);const pendingControl=optionalStateControl(stateDirectory,"pending.json",runtime);const committedAfterPendingRead=optionalStateControl(stateDirectory,"committed.json",runtime);const committedControl=committedAfterPendingRead??initiallyCommittedControl;if(committedControl){return{committed:recoverCommittedState(stateDirectory,committedControl,pendingControl,requested,expectedAgent,runtime),pending:null}}if(pendingControl){if(stateControlsMatch(pendingControl,requested)){const pending=validateGeneration(stateDirectory,pendingControl,runtime,expectedAgent);const committedAfterPendingValidation=optionalStateControl(stateDirectory,"committed.json",runtime);if(committedAfterPendingValidation){return{committed:recoverCommittedState(stateDirectory,committedAfterPendingValidation,pendingControl,requested,expectedAgent,runtime),pending:null}}discardGenerationsExcept(stateDirectory,pendingControl.generation,runtime);return{committed:null,pending}}fail2("a different startup profile is already pending; wait for it to commit or recreate")}return{committed:null,pending:null}}function createGeneration(stateDirectory,control,profileJson,corporateCa,runtime){const temporaryName=`.prepare-${String(process.pid)}-${randomToken()}`;const temporary=import_node_path.default.join(stateDirectory,temporaryName);const generation=import_node_path.default.join(stateDirectory,control.generation);let renameAttempted=false;try{import_node_fs.default.mkdirSync(temporary,{mode:STATE_DIRECTORY_MODE});import_node_fs.default.chownSync(temporary,runtime.rootUid,runtime.rootGid);import_node_fs.default.chmodSync(temporary,STATE_DIRECTORY_MODE);writeSecureNewFile(import_node_path.default.join(temporary,"profile.json"),profileJson,runtime);if(corporateCa){writeSecureNewFile(import_node_path.default.join(temporary,"corporate-ca.pem"),corporateCa,runtime)}syncDirectory(temporary);renameAttempted=true;import_node_fs.default.renameSync(temporary,generation);syncDirectory(stateDirectory)}catch(error){try{import_node_fs.default.lstatSync(temporary);discardDirectory(temporary,runtime)}catch{}if(error instanceof ManagedStartupApplicationError)throw error;if(renameAttempted&&(error.code==="EEXIST"||error.code==="ENOTEMPTY")){return validateGeneration(stateDirectory,control,runtime)}fail2(`could not atomically prepare generation ${control.generation}`)}return validateGeneration(stateDirectory,control,runtime)}function toPrepared(status,stateDirectory,generation,expectedAgent){return{status,stateDirectory,generationDirectory:generation.directory,profilePath:generation.profilePath,corporateCaPath:generation.corporateCaPath,fingerprint:generation.fingerprint,expectedAgent,profile:generation.profile}}function prepareManagedStartupApplication(input,testRuntime){const runtime=runtimeFor(testRuntime);requireContainerRoot();let profile;try{profile=decodeManagedStartupProfile(input.encodedProfile)}catch(error){fail2(error.message)}if(profile.agent!==input.expectedAgent){fail2(`managed startup profile targets ${profile.agent}, expected ${input.expectedAgent}`)}const corporateCa=validateManagedStartupCorporateCaTransport(input.corporateCaB64,profile);const profileJson=serializeManagedStartupProfile(profile);const control=stateControl(fingerprintManagedStartupProfile(profile));const stateDirectory=ensureStateDirectory(input.stateDirectory,runtime);const recovered=recoverState(stateDirectory,control,input.expectedAgent,runtime);if(recovered.committed){return toPrepared("already-committed",stateDirectory,recovered.committed,input.expectedAgent)}if(recovered.pending){return toPrepared("prepared",stateDirectory,recovered.pending,input.expectedAgent)}const generation=createGeneration(stateDirectory,control,profileJson,corporateCa,runtime);const publication=publishStateControlIfAbsent(stateDirectory,"pending.json",control,runtime);if(publication.control.fingerprint!==control.fingerprint||publication.control.generation!==control.generation){discardDirectoryIfPresent(generation.directory,runtime);syncDirectory(stateDirectory);fail2("a different startup profile won the pending-state transaction")}const committedAfterPublication=optionalStateControl(stateDirectory,"committed.json",runtime);if(committedAfterPublication){if(committedAfterPublication.fingerprint!==control.fingerprint||committedAfterPublication.generation!==control.generation){if(publication.created){removePendingControl(stateDirectory,runtime);discardDirectoryIfPresent(generation.directory,runtime);syncDirectory(stateDirectory)}fail2("a different startup profile committed during pending-state publication")}const committedGeneration=validateGeneration(stateDirectory,committedAfterPublication,runtime,input.expectedAgent);removePendingControl(stateDirectory,runtime);discardGenerationsExcept(stateDirectory,committedAfterPublication.generation,runtime);return toPrepared("already-committed",stateDirectory,committedGeneration,input.expectedAgent)}const activeGeneration=publication.created?generation:validateGeneration(stateDirectory,publication.control,runtime,input.expectedAgent);return toPrepared("prepared",stateDirectory,activeGeneration,input.expectedAgent)}function validatePreparedHandle(handle){if(!import_node_path.default.isAbsolute(handle.stateDirectory)||!SHA256_RE2.test(handle.fingerprint)||handle.generationDirectory!==import_node_path.default.join(handle.stateDirectory,`generation-${handle.fingerprint}`)||handle.profilePath!==import_node_path.default.join(handle.generationDirectory,"profile.json")||handle.corporateCaPath!==null&&handle.corporateCaPath!==import_node_path.default.join(handle.generationDirectory,"corporate-ca.pem")){fail2("prepared startup handle is malformed")}return stateControl(handle.fingerprint)}function commitManagedStartupApplication(prepared,testRuntime){const runtime=runtimeFor(testRuntime);requireContainerRoot();const requested=validatePreparedHandle(prepared);const stateDirectory=ensureStateDirectory(prepared.stateDirectory,runtime);const committedControl=optionalStateControl(stateDirectory,"committed.json",runtime);if(committedControl){if(committedControl.fingerprint!==requested.fingerprint||committedControl.generation!==requested.generation){fail2("a different startup profile is already committed")}const generation2=validateGeneration(stateDirectory,committedControl,runtime,prepared.expectedAgent);return{...toPrepared("already-committed",stateDirectory,generation2,prepared.expectedAgent),status:"committed"}}const pendingControl=optionalStateControl(stateDirectory,"pending.json",runtime);if(!pendingControl||pendingControl.fingerprint!==requested.fingerprint||pendingControl.generation!==requested.generation){fail2("the prepared startup generation is not the active pending generation")}const generation=validateGeneration(stateDirectory,pendingControl,runtime,prepared.expectedAgent);const publication=publishStateControlIfAbsent(stateDirectory,"committed.json",pendingControl,runtime);if(publication.control.fingerprint!==requested.fingerprint||publication.control.generation!==requested.generation){fail2("a different startup profile won the committed-state transaction")}removePendingControl(stateDirectory,runtime);discardGenerationsExcept(stateDirectory,publication.control.generation,runtime);syncDirectory(stateDirectory);return{...toPrepared("already-committed",stateDirectory,generation,prepared.expectedAgent),status:"committed"}}var import_node_buffer4=require("node:buffer");var PROVIDERLESS_INFERENCE_ENV=Object.freeze({NEMOCLAW_MODEL:"",NEMOCLAW_INFERENCE_PROVIDER_ID:"",NEMOCLAW_UPSTREAM_PROVIDER:"",NEMOCLAW_INFERENCE_BASE_URL:"",NEMOCLAW_INFERENCE_API:""});function isObjectRecord(value){return typeof value==="object"&&value!==null&&!Array.isArray(value)}var EXACT_TEMPLATE_PATTERN=/^\{\{\s*([^}]+?)\s*\}\}$/;var TEMPLATE_REFERENCE_PATTERN=/\{\{\s*([^}]+?)\s*\}\}/g;function resolvedRenderTemplateReference(value){return{matched:true,value}}function resolveSandboxNameTemplate(value,sandboxName){return value.replaceAll("{sandboxName}",sandboxName)}function resolveRenderTemplatesInValue(value,context){if(typeof value==="string")return resolveRenderTemplatesInString(value,context);if(Array.isArray(value)){if(value.length===0)return value;const resolved=value.map(entry=>resolveRenderTemplatesInValue(entry,context)).filter(entry=>entry!==void 0);return resolved.length>0?resolved:void 0}if(value&&typeof value==="object"){const sourceEntries=Object.entries(value);if(sourceEntries.length===0)return value;const entries=sourceEntries.map(([key,entry])=>[key,resolveRenderTemplatesInValue(entry,context)]).filter(entry=>entry[1]!==void 0);return entries.length>0?Object.fromEntries(entries):void 0}return value}function isTruthyRenderTemplate(value,context){if(!value)return true;const resolved=resolveRenderTemplatesInString(value,context);if(resolved===void 0||resolved===null||resolved===false)return false;if(Array.isArray(resolved))return resolved.length>0;if(typeof resolved==="object")return Object.keys(resolved).length>0;if(typeof resolved==="string")return resolved.trim().length>0;return true}function resolveRenderTemplatesInString(value,context){const exact=value.match(EXACT_TEMPLATE_PATTERN);if(exact?.[1])return resolveTemplateReference(exact[1].trim(),context);let omitted=false;const resolved=value.replace(TEMPLATE_REFERENCE_PATTERN,(match,reference)=>{const replacement=resolveTemplateReference(reference.trim(),context);if(replacement===void 0||replacement===null){omitted=true;return""}if(Array.isArray(replacement))return replacement.map(String).join(",");if(typeof replacement==="object")return JSON.stringify(replacement);return String(replacement)});return omitted?void 0:resolved}function resolveTemplateReference(reference,context){const resolved=context.referenceResolver?.(reference,context);return resolved?.matched?resolved.value:"{{"+reference+"}}"}function allowedIds(context,channel){return parseList(stateValue(context,`allowedIds.${channel}`))}function stateValue(context,path4){const stateInput=context.inputs.find(input=>input.statePath===path4);if(stateInput?.value!==void 0)return stateInput.value;const inputId=path4.split(".").at(-1);return context.inputs.find(input=>input.inputId===inputId)?.value}function parseList(value){if(Array.isArray(value))return unique(value.map(String).map(cleanString).filter(Boolean));const text=cleanString(value);if(!text)return[];return unique(text.split(",").map(cleanString).filter(Boolean))}function parseBoolean(value){if(typeof value==="boolean")return value;const text=cleanString(value)?.toLowerCase();if(text==="1"||text==="true"||text==="yes"||text==="on")return true;if(text==="0"||text==="false"||text==="no"||text==="off")return false;return void 0}function nonEmptyString(value){return cleanString(value)||void 0}function cleanString(value){const text=String(value??"");if(/[\r\n]/.test(text)){throw new Error("Messaging template values must not contain line breaks.")}return text.trim()}function nonEmptyArray(values){return values.length>0?[...values]:void 0}function nonEmptyCsv(values){return values.length>0?values.join(","):void 0}function nonEmptyObject(value){return Object.keys(value).length>0?value:void 0}function unique(values){return[...new Set(values)]}var resolveDiscordTemplateReference=(reference,context)=>{if(reference==="discordProxyUrl")return resolvedRenderTemplateReference(void 0);switch(reference){case"discord.guilds":return resolvedRenderTemplateReference(nonEmptyObject(discordGuilds(context)));case"discord.hasGuilds":return resolvedRenderTemplateReference(Object.keys(discordGuilds(context)).length>0);case"discord.guildIds.csv":return resolvedRenderTemplateReference(nonEmptyCsv(Object.keys(discordGuilds(context))));case"discord.allowedUsers.values":return resolvedRenderTemplateReference(nonEmptyArray(discordAllowedUsers(context)));case"discord.allowedUsers.csv":return resolvedRenderTemplateReference(nonEmptyCsv(discordAllowedUsers(context)));case"discord.allowedUsers.dmPolicy":return resolvedRenderTemplateReference(discordAllowedUsers(context).length>0?"allowlist":void 0);case"discord.allowAllUsers":return resolvedRenderTemplateReference(Object.keys(discordGuilds(context)).length>0&&discordAllowedUsers(context).length===0?true:void 0);case"discord.requireMention":return resolvedRenderTemplateReference(discordRequireMention(context));default:return void 0}};function discordGuilds(context){const serverIds=parseList(stateValue(context,"discordGuilds.serverId"));if(serverIds.length===0)return{};const users=parseList(stateValue(context,"discordGuilds.userIds"));const requireMention=parseBoolean(stateValue(context,"discordGuilds.requireMention"))??true;return Object.fromEntries(serverIds.map(serverId=>[serverId,{requireMention,...users.length>0?{users}:{}}]))}function discordAllowedUsers(context){const users=new Set(allowedIds(context,"discord"));for(const guild of Object.values(discordGuilds(context))){for(const user of guild.users??[])users.add(String(user))}return[...users]}function discordRequireMention(context){for(const guild of Object.values(discordGuilds(context))){if(typeof guild.requireMention==="boolean")return guild.requireMention}return true}var DEFAULT_AUDIENCE_TYPE="app-url";var APP_PRINCIPAL_DISCOVERY_SENTINEL="000000000000000000000";var resolveGooglechatTemplateReference=(reference,context)=>{switch(reference){case"googlechatConfig.audienceType":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.audienceType"))??DEFAULT_AUDIENCE_TYPE);case"googlechatConfig.audience":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.audience")));case"googlechatConfig.appPrincipal":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.appPrincipal"))??APP_PRINCIPAL_DISCOVERY_SENTINEL);case"googlechatConfig.projectId":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.projectId")));case"googlechatConfig.subscriptionName":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"googlechatConfig.subscriptionName")));default:break}const allowReference=reference.match(/^allowedIds[.]googlechat[.](values|dmPolicy|csv)$/);if(!allowReference?.[1])return void 0;const ids=allowedIds(context,"googlechat");switch(allowReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);case"csv":return resolvedRenderTemplateReference(ids.length>0?ids.join(","):void 0);default:return void 0}};var resolveSlackTemplateReference=(reference,context)=>{if(reference==="slackConfig.allowedChannels.csv"){return resolvedRenderTemplateReference(nonEmptyCsv(slackAllowedChannels(context)))}const allowedIdsReference=reference.match(/^allowedIds[.]slack[.](values|csv|dmPolicy|groupPolicy|channels)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"slack");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);case"groupPolicy":return resolvedRenderTemplateReference(ids.length>0||slackAllowedChannels(context).length>0?"allowlist":void 0);case"channels":return resolvedRenderTemplateReference(slackChannelConfig(context,ids));default:return void 0}};function slackChannelConfig(context,users){const allowedChannels=slackAllowedChannels(context);const entry={enabled:true,requireMention:true,...users.length>0?{users:[...users]}:{}};if(allowedChannels.length>0){return Object.fromEntries(allowedChannels.map(channelId=>[channelId,{...entry}]))}return users.length>0?{"*":entry}:void 0}function slackAllowedChannels(context){return parseList(stateValue(context,"slackConfig.allowedChannels"))}var DEFAULT_TEAMS_WEBHOOK_PORT=3978;var resolveTeamsTemplateReference=(reference,context)=>{switch(reference){case"teamsConfig.appId":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"teamsConfig.appId")));case"teamsConfig.tenantId":return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"teamsConfig.tenantId")));case"teamsConfig.webhookPort":return resolvedRenderTemplateReference(teamsWebhookPort(context));case"teamsConfig.requireMention":return resolvedRenderTemplateReference(parseBoolean(stateValue(context,"teamsConfig.requireMention")));default:break}const allowedIdsReference=reference.match(/^allowedIds[.]teams[.](values|csv|dmPolicy)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"teams");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);default:return void 0}};function teamsWebhookPort(context){const raw=nonEmptyString(stateValue(context,"teamsConfig.webhookPort"));if(!raw)return DEFAULT_TEAMS_WEBHOOK_PORT;const port=Number(raw);if(!Number.isInteger(port)||port<1||port>65535){throw new Error("Microsoft Teams webhook port must be an integer TCP port between 1 and 65535.")}return port}var DEFAULT_TELEGRAM_GROUP_POLICY="open";var TELEGRAM_GROUP_POLICIES=new Set(["open","allowlist","disabled"]);var resolveTelegramTemplateReference=(reference,context)=>{if(reference==="telegramConfig.groupPolicy"){return resolvedRenderTemplateReference(telegramGroupPolicy(context))}if(reference==="telegramConfig.openclawGroups"){return resolvedRenderTemplateReference(telegramOpenClawGroups(context))}if(reference==="telegramConfig.requireMention"){return resolvedRenderTemplateReference(parseBoolean(stateValue(context,"telegramConfig.requireMention")))}const allowedIdsReference=reference.match(/^allowedIds[.]telegram[.](values|csv|dmPolicy)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"telegram");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);default:return void 0}};function telegramGroupPolicy(context){const value=nonEmptyString(stateValue(context,"telegramConfig.groupPolicy"));return value&&TELEGRAM_GROUP_POLICIES.has(value)?value:DEFAULT_TELEGRAM_GROUP_POLICY}function telegramOpenClawGroups(context){if(telegramGroupPolicy(context)!=="open")return void 0;const requireMention=parseBoolean(stateValue(context,"telegramConfig.requireMention"));return requireMention===true?{"*":{requireMention:true}}:void 0}var WECHAT_ILINK_HOSTS=new Set(["ilinkai.weixin.qq.com","ilinkai.wechat.com"]);var WECHAT_ILINK_IDC_HOST_PATTERN=/^idc-[0-9]+[.]weixin[.]qq[.]com$/u;function normalizeWechatIlinkBaseUrl(value){const raw=String(value??"");if(/[\r\n]/.test(raw)){throw new Error("WeChat baseUrl must not contain line breaks.")}const text=raw.trim();if(!text)return void 0;let url;try{url=new URL(text)}catch{throw new Error("WeChat baseUrl must be a valid URL.")}if(url.protocol!=="https:"){throw new Error("WeChat baseUrl must use HTTPS.")}if(url.username||url.password){throw new Error("WeChat baseUrl must not include credentials.")}const authority=text.match(/^[a-z][a-z0-9+.-]*:\/\/([^/?#]*)/iu)?.[1]??"";if(authority.includes(":")){throw new Error("WeChat baseUrl must not include an explicit port.")}if(!isWechatIlinkHost(url.hostname)){throw new Error("WeChat baseUrl must use an expected iLink host.")}if(url.pathname&&url.pathname!=="/"||url.search||url.hash){throw new Error("WeChat baseUrl must be an iLink origin URL.")}return url.origin}function isWechatIlinkHost(hostname){const normalized=hostname.toLowerCase();return WECHAT_ILINK_HOSTS.has(normalized)||isWechatIlinkIdcHost(normalized)}function isWechatIlinkIdcHost(hostname){return WECHAT_ILINK_IDC_HOST_PATTERN.test(hostname.toLowerCase())}var resolveWechatTemplateReference=(reference,context)=>{const wechatConfig=reference.match(/^wechatConfig[.](accountId|baseUrl|userId)$/);if(wechatConfig?.[1]){if(wechatConfig[1]==="baseUrl"){return resolvedRenderTemplateReference(normalizeWechatIlinkBaseUrl(stateValue(context,"wechatConfig.baseUrl")))}return resolvedRenderTemplateReference(nonEmptyString(stateValue(context,"wechatConfig."+wechatConfig[1])))}const allowedIdsReference=reference.match(/^allowedIds[.]wechat[.](values|csv|dmPolicy)$/);if(!allowedIdsReference?.[1])return void 0;const ids=wechatAllowedIds(context);switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));case"dmPolicy":return resolvedRenderTemplateReference(ids.length>0?"allowlist":void 0);default:return void 0}};function wechatAllowedIds(context){const ids=allowedIds(context,"wechat");const userId=nonEmptyString(stateValue(context,"wechatConfig.userId"));return userId&&!ids.includes(userId)?[userId,...ids]:ids}var DEFAULT_WHATSAPP_MODE="self-chat";var BOT_WHATSAPP_MODE="bot";var WHATSAPP_MODES=new Set([DEFAULT_WHATSAPP_MODE,BOT_WHATSAPP_MODE]);var resolveWhatsappTemplateReference=(reference,context)=>{if(reference==="whatsappConfig.mode"){return resolvedRenderTemplateReference(whatsappMode(context))}if(reference==="whatsappConfig.dmPolicy"){return resolvedRenderTemplateReference(whatsappDmPolicy(context))}const allowedIdsReference=reference.match(/^allowedIds[.]whatsapp[.](values|csv)$/);if(!allowedIdsReference?.[1])return void 0;const ids=allowedIds(context,"whatsapp");switch(allowedIdsReference[1]){case"values":return resolvedRenderTemplateReference(nonEmptyArray(ids));case"csv":return resolvedRenderTemplateReference(nonEmptyCsv(ids));default:return void 0}};function whatsappMode(context){const value=nonEmptyString(stateValue(context,"whatsappConfig.mode"));return value&&WHATSAPP_MODES.has(value)?value:DEFAULT_WHATSAPP_MODE}function whatsappDmPolicy(context){if(whatsappMode(context)!==BOT_WHATSAPP_MODE)return void 0;return allowedIds(context,"whatsapp").length>0?"allowlist":"pairing"}var BUILT_IN_TEMPLATE_REFERENCE_RESOLVERS=[resolveTelegramTemplateReference,resolveDiscordTemplateReference,resolveWechatTemplateReference,resolveSlackTemplateReference,resolveWhatsappTemplateReference,resolveTeamsTemplateReference,resolveGooglechatTemplateReference];function createBuiltInRenderTemplateResolver(){return(reference,context)=>{for(const resolver of BUILT_IN_TEMPLATE_REFERENCE_RESOLVERS){const resolved=resolver(reference,context);if(resolved)return resolved}return void 0}}var import_node_crypto3=__toESM(require("node:crypto"));function hashCredential(value){const normalized=String(value??"").trim();if(!normalized)return null;return import_node_crypto3.default.createHash("sha256").update(normalized).digest("hex")}function planCredentialBindings(manifest,context,inputs,environment=process.env){return manifest.credentials.map(credential=>{const sourceInput=inputs.find(input=>input.inputId===credential.sourceInput);const credentialAvailable=sourceInput?.credentialAvailable===true||context.credentialAvailability?.[credential.id]===true||context.credentialAvailability?.[`${manifest.id}.${credential.id}`]===true;const envKey=sourceInput?.sourceEnv??credential.providerEnvKey;const credentialHash=credentialAvailable?hashCredential(environment[envKey])??void 0:void 0;return{channelId:manifest.id,credentialId:credential.id,sourceInput:credential.sourceInput,providerName:resolveSandboxNameTemplate(credential.providerName,context.sandboxName),providerEnvKey:credential.providerEnvKey,placeholder:credential.placeholder,credentialAvailable,...credentialHash!==void 0?{credentialHash}:{}}})}function planHostForward(manifest,inputs,active,referenceResolver,environment=process.env){if(!active||!manifest.hostForward)return void 0;const context={inputs,env:environment,referenceResolver};if(!isTruthyRenderTemplate(manifest.hostForward.when,context))return void 0;const portValue=resolveRenderTemplatesInValue(manifest.hostForward.port,context);const port=normalizeForwardPort(manifest.id,portValue);return{channelId:manifest.id,port,label:manifest.hostForward.label}}function normalizeForwardPort(channelId,value){const port=typeof value==="number"?value:Number(String(value??"").trim());if(!Number.isInteger(port)||port<1||port>65535){throw new Error(`Channel manifest '${channelId}' declares invalid host forward port '${String(value)}'.`)}return port}var OPENSHELL_ENV_PLACEHOLDER_PREFIX="openshell:resolve:env:";var OPENSHELL_ALIAS_PLACEHOLDER_RE=/^[A-Za-z0-9]+-OPENSHELL-RESOLVE-ENV-(.+)$/;function normalizeProviderPlaceholderForEnvKey(value,envKey){if(value.startsWith(OPENSHELL_ENV_PLACEHOLDER_PREFIX)){return placeholderSuffixMatchesEnvKey(value.slice(OPENSHELL_ENV_PLACEHOLDER_PREFIX.length),envKey)?`${OPENSHELL_ENV_PLACEHOLDER_PREFIX}${envKey}`:null}const aliasMatch=value.match(OPENSHELL_ALIAS_PLACEHOLDER_RE);if(!aliasMatch||!placeholderSuffixMatchesEnvKey(aliasMatch[1],envKey)){return null}return value.replace(/-OPENSHELL-RESOLVE-ENV-.+$/,`-OPENSHELL-RESOLVE-ENV-${envKey}`)}function placeholderSuffixMatchesEnvKey(suffix,envKey){if(suffix===envKey)return true;const generationMatch=suffix.match(/^(?:v[0-9]{1,20}|s[a-f0-9]{64})_(.+)$/);return generationMatch?.[1]===envKey}function hasFullPersistedCredentialBindingShape(binding){return typeof binding.channelId==="string"&&typeof binding.credentialId==="string"&&typeof binding.sourceInput==="string"&&typeof binding.providerName==="string"&&typeof binding.providerEnvKey==="string"&&typeof binding.placeholder==="string"&&typeof binding.credentialAvailable==="boolean"}function normalizeFullPersistedCredentialBindings(bindings){return bindings.map(binding=>({channelId:binding.channelId,credentialId:binding.credentialId,sourceInput:binding.sourceInput,providerName:binding.providerName,providerEnvKey:binding.providerEnvKey,placeholder:normalizeProviderPlaceholderForEnvKey(binding.placeholder,binding.providerEnvKey)??binding.placeholder,credentialAvailable:binding.credentialAvailable===true,...typeof binding.credentialHash==="string"?{credentialHash:binding.credentialHash}:{}}))}function normalizePersistedAgentCredentialPlaceholders(render,credentialBindings){const credentialEnvKeys=new Set(credentialBindings.map(binding=>binding.providerEnvKey).filter(Boolean));if(credentialEnvKeys.size===0)return[...render];return render.map(entry=>{if(entry.kind!=="env-lines")return entry;return{...entry,lines:entry.lines.map(line=>normalizeCredentialEnvLine(line,credentialEnvKeys))}})}function normalizeCredentialEnvLine(line,credentialEnvKeys){const index=line.indexOf("=");if(index<=0)return line;const envKey=line.slice(0,index).trim();if(!credentialEnvKeys.has(envKey))return line;const value=line.slice(index+1);const normalized=normalizeProviderPlaceholderForEnvKey(value,envKey);return normalized?`${envKey}=${normalized}`:line}function normalizePersistedSandboxMessagingPlanShape(plan,environment=process.env){const manifestRegistry=createBuiltInChannelManifestRegistry();const disabledChannels=plan.disabledChannels.filter(channelId=>typeof channelId==="string");const disabledSet=new Set(disabledChannels);const channels=plan.channels.map(channel=>normalizePersistedChannel(channel,disabledSet,manifestRegistry.get(channel.channelId),environment));const credentialBindings=normalizePersistedCredentialBindings(plan,channels,manifestRegistry,environment);const normalizedPlan={...plan,channels,disabledChannels,credentialBindings,networkPolicy:plan.networkPolicy&&Array.isArray(plan.networkPolicy.entries)?plan.networkPolicy:{presets:[],entries:[]},agentRender:normalizePersistedAgentCredentialPlaceholders(Array.isArray(plan.agentRender)?[...plan.agentRender]:[],credentialBindings),buildSteps:Array.isArray(plan.buildSteps)?[...plan.buildSteps]:[],...plan.runtimeSetup!==void 0?{runtimeSetup:normalizeRuntimeSetup(plan.runtimeSetup)}:{},stateUpdates:Array.isArray(plan.stateUpdates)?[...plan.stateUpdates]:[],healthChecks:Array.isArray(plan.healthChecks)?[...plan.healthChecks]:[]};return normalizedPlan}function normalizePersistedChannel(channel,disabledSet,manifest,environment){const disabled=channel.disabled??disabledSet.has(channel.channelId);const configured=channel.configured??true;const hasFullShape=hasFullChannelShape(channel);const inputs=hasFullShape?normalizeFullInputs(channel.channelId,channel.inputs??[]):normalizePersistedInputs(channel,manifest);const active=channel.active??(configured&&!disabled&&requiredInputsAvailable(manifest,inputs));const hostForward=manifest?planHostForward(manifest,inputs,active&&!disabled,createBuiltInRenderTemplateResolver(),environment):void 0;return{channelId:channel.channelId,displayName:channel.displayName??manifest?.displayName??channel.channelId,authMode:channel.authMode??manifest?.auth.mode??"none",active,selected:channel.selected??configured,configured,disabled,...channel.pendingRemoval===true?{pendingRemoval:true}:{},inputs,...hostForward?{hostForward}:{},hooks:Array.isArray(channel.hooks)?[...channel.hooks]:[]}}function normalizePersistedInputs(channel,manifest){const persistedById=new Map((channel.inputs??[]).filter(input=>typeof input.inputId==="string").map(input=>[input.inputId,input]));const fromManifest=(manifest?.inputs??[]).map(input=>inputReferenceFromManifest(channel.channelId,input,persistedById.get(input.id)));const manifestInputIds=new Set((manifest?.inputs??[]).map(input=>input.id));const unknownInputs=[...persistedById.values()].flatMap(input=>{if(!input.inputId||manifestInputIds.has(input.inputId))return[];return[normalizeUnknownInput(channel.channelId,input)]});return[...fromManifest,...unknownInputs]}function normalizeFullInputs(channelId,inputs){return inputs.filter(input=>typeof input.inputId==="string").map(input=>({channelId:typeof input.channelId==="string"?input.channelId:channelId,inputId:input.inputId,kind:input.kind==="secret"||input.kind==="config"?input.kind:"config",required:typeof input.required==="boolean"?input.required:false,...typeof input.sourceEnv==="string"?{sourceEnv:input.sourceEnv}:{},...typeof input.statePath==="string"?{statePath:input.statePath}:{},...input.credentialAvailable!==void 0?{credentialAvailable:input.credentialAvailable}:{},...input.value!==void 0?{value:input.value}:{}}))}function inputReferenceFromManifest(channelId,input,persisted){return{channelId,inputId:input.id,kind:input.kind,required:input.required,...input.envKey?{sourceEnv:input.envKey}:{},...input.kind==="config"&&input.statePath?{statePath:input.statePath}:{},...persisted?.credentialAvailable!==void 0?{credentialAvailable:persisted.credentialAvailable}:{},...persisted?.value!==void 0?{value:persisted.value}:{}}}function normalizeUnknownInput(channelId,input){const kind=input.kind==="secret"||input.kind==="config"?input.kind:"config";return{channelId,inputId:input.inputId,kind,required:input.required===true,...typeof input.sourceEnv==="string"?{sourceEnv:input.sourceEnv}:{},...typeof input.statePath==="string"?{statePath:input.statePath}:{},...input.credentialAvailable!==void 0?{credentialAvailable:input.credentialAvailable}:{},...input.value!==void 0?{value:input.value}:{}}}function requiredInputsAvailable(manifest,inputs){if(!manifest)return true;return manifest.inputs.every(manifestInput=>{if(!manifestInput.required)return true;const input=inputs.find(entry=>entry.inputId===manifestInput.id);if(!input)return false;if(input.kind==="secret")return input.credentialAvailable===true;if(input.value===void 0)return false;return typeof input.value==="string"?input.value.trim().length>0:true})}function normalizePersistedCredentialBindings(plan,channels,manifestRegistry,environment){const persisted=plan.credentialBindings??[];if(Array.isArray(plan.credentialBindings)&&plan.channels.every(hasFullChannelShape)&&persisted.every(hasFullPersistedCredentialBindingShape)){return normalizeFullPersistedCredentialBindings(persisted)}const manifests=channels.flatMap(channel=>{const manifest=manifestRegistry.get(channel.channelId);return manifest?[manifest]:[]});const planForBindings={...plan,channels,credentialBindings:[],networkPolicy:{presets:[],entries:[]},agentRender:[],buildSteps:[],runtimeSetup:{nodePreloads:[],envAliases:[],secretScans:[]},stateUpdates:[],healthChecks:[]};const generated=credentialBindingsFromManifests(planForBindings,manifests,new Map(channels.map(channel=>[channel.channelId,channel.inputs])),environment);return generated.map(binding=>overlayPersistedCredentialBinding(binding,persisted))}function credentialBindingsFromManifests(plan,manifests,inputRegistry,environment){const context=compilerContext(plan);return manifests.flatMap(manifest=>planCredentialBindings(manifest,context,inputRegistry.get(manifest.id)??[],environment).map(binding=>overlayPersistedCredentialBinding(binding,plan.credentialBindings)))}function overlayPersistedCredentialBinding(binding,persisted){const match=persisted.find(candidate=>credentialBindingMatches(binding,candidate));if(!match)return binding;return{...binding,credentialAvailable:typeof match.credentialAvailable==="boolean"?match.credentialAvailable:binding.credentialAvailable,...typeof match.credentialHash==="string"&&match.credentialHash.length>0?{credentialHash:match.credentialHash}:binding.credentialHash?{credentialHash:binding.credentialHash}:{}}}function credentialBindingMatches(binding,candidate){if(candidate.channelId&&candidate.channelId!==binding.channelId)return false;if(candidate.providerEnvKey&&candidate.providerEnvKey===binding.providerEnvKey)return true;if(candidate.credentialId&&candidate.credentialId===binding.credentialId)return true;if(candidate.sourceInput&&candidate.sourceInput===binding.sourceInput)return true;return false}function hasFullChannelShape(channel){return typeof channel.displayName==="string"&&typeof channel.authMode==="string"&&typeof channel.active==="boolean"&&typeof channel.selected==="boolean"&&typeof channel.configured==="boolean"&&typeof channel.disabled==="boolean"&&Array.isArray(channel.inputs)}function normalizeRuntimeSetup(setup){return{nodePreloads:Array.isArray(setup?.nodePreloads)?[...setup.nodePreloads]:[],envAliases:Array.isArray(setup?.envAliases)?[...setup.envAliases]:[],secretScans:Array.isArray(setup?.secretScans)?[...setup.secretScans]:[]}}function compilerContext(plan){return{sandboxName:plan.sandboxName,agent:plan.agent,workflow:plan.workflow,isInteractive:false,configuredChannels:plan.channels.map(channel=>channel.channelId),disabledChannels:plan.disabledChannels,credentialAvailability:credentialAvailabilityFromPlan(plan)}}function credentialAvailabilityFromPlan(plan){const availability={};for(const channel of plan.channels){for(const input of channel.inputs){if(input.kind!=="secret"||input.credentialAvailable!==true)continue;availability[`${channel.channelId}.${input.inputId}`]=true;if(input.sourceEnv)availability[input.sourceEnv]=true}}for(const credential of plan.credentialBindings){if(!credential.credentialAvailable)continue;availability[credential.credentialId]=true;availability[`${credential.channelId}.${credential.credentialId}`]=true;availability[`${credential.channelId}.${credential.sourceInput}`]=true;availability[credential.providerEnvKey]=true}return availability}function normalizeMessagingChannelId(channelId){return channelId.trim().toLowerCase()}function enabledPlanChannels(plan){const disabled=new Set((plan.disabledChannels??[]).map(normalizeMessagingChannelId).filter(Boolean));return plan.channels.filter(channel=>{const channelId=normalizeMessagingChannelId(channel.channelId);return channelId.length>0&&channel.active&&!channel.disabled&&!disabled.has(channelId)})}function selectActiveMessagingChannelIds(plan){const seen=new Set;const channels=[];for(const item of enabledPlanChannels(plan)){const channel=normalizeMessagingChannelId(item.channelId);if(!channel||seen.has(channel))continue;seen.add(channel);channels.push(channel)}return channels}function selectEnabledMessagingAgentRender(plan){const active=new Set(selectActiveMessagingChannelIds(plan));return plan.agentRender.filter(render=>render.agent===plan.agent&&active.has(normalizeMessagingChannelId(render.channelId)))}function selectEnabledPostAgentInstallBuildFiles(plan){const active=new Set(selectActiveMessagingChannelIds(plan));const channels=enabledPlanChannels(plan);return plan.buildSteps.filter(step=>{const channelId=normalizeMessagingChannelId(step.channelId);if(!active.has(channelId)||step.kind!=="build-file")return false;if(!step.hookId)return true;const matchingChannels=channels.filter(channel=>normalizeMessagingChannelId(channel.channelId)===channelId);if(matchingChannels.length!==1)return false;const matchedHook=matchingChannels[0]?.hooks?.find(hook=>hook.id===step.hookId);return matchedHook!==void 0&&matchedHook.phase==="post-agent-install"})}function parseSandboxMessagingPlan(value,options={}){if(!isObjectRecord(value)||value.schemaVersion!==1||typeof value.sandboxName!=="string"||typeof value.agent!=="string"||typeof value.workflow!=="string"||!Array.isArray(value.channels)||!Array.isArray(value.disabledChannels)||!isOptionalObjectArray(value,"credentialBindings")||Object.hasOwn(value,"networkPolicy")&&!isObjectRecord(value.networkPolicy)||!isOptionalObjectArray(value,"agentRender")||!isOptionalObjectArray(value,"buildSteps")||!isRuntimeSetup(value.runtimeSetup)||!isOptionalObjectArray(value,"stateUpdates")||!isOptionalObjectArray(value,"healthChecks")){return null}if(options.sandboxName&&value.sandboxName!==options.sandboxName)return null;if(options.agent&&value.agent!==options.agent)return null;const supported=Array.isArray(options.supportedChannelIds)?new Set(options.supportedChannelIds):null;const normalizedChannelIds=new Set;for(const channel of value.channels){if(!isObjectRecord(channel)||typeof channel.channelId!=="string")return null;const normalizedChannelId=normalizeMessagingChannelId(channel.channelId);if(!normalizedChannelId||normalizedChannelId!==channel.channelId||normalizedChannelIds.has(normalizedChannelId)){return null}if(Object.hasOwn(channel,"configured")&&typeof channel.configured!=="boolean"){return null}if(Object.hasOwn(channel,"active")&&typeof channel.active!=="boolean")return null;if(Object.hasOwn(channel,"disabled")&&typeof channel.disabled!=="boolean")return null;if(Object.hasOwn(channel,"pendingRemoval")&&typeof channel.pendingRemoval!=="boolean"){return null}if(Object.hasOwn(channel,"inputs")&&!Array.isArray(channel.inputs))return null;if(Object.hasOwn(channel,"hostForward")&&!isHostForward(channel.hostForward))return null;if(Object.hasOwn(channel,"hooks")&&!Array.isArray(channel.hooks))return null;if(Array.isArray(channel.inputs)&&channel.inputs.some(input=>!isObjectRecord(input)||typeof input.inputId!=="string"||Object.hasOwn(input,"channelId")&&input.channelId!==normalizedChannelId)){return null}if(Array.isArray(channel.hooks)&&channel.hooks.some(hook=>!isObjectRecord(hook)||Object.hasOwn(hook,"channelId")&&hook.channelId!==normalizedChannelId)){return null}if(Object.hasOwn(channel,"hostForward")&&isObjectRecord(channel.hostForward)&&channel.hostForward.channelId!==normalizedChannelId){return null}if(supported&&!supported.has(channel.channelId))return null;normalizedChannelIds.add(normalizedChannelId)}if(!value.disabledChannels.every(isCanonicalMessagingChannelId))return null;const disabledChannelIds=new Set(value.disabledChannels);if(disabledChannelIds.size!==value.disabledChannels.length||[...disabledChannelIds].some(channelId=>!normalizedChannelIds.has(channelId))||value.channels.some(channel=>isObjectRecord(channel)&&channel.disabled===true!==disabledChannelIds.has(String(channel.channelId)))){return null}if(!hasCanonicalChannelReferences(value.credentialBindings)||!hasMatchingAgentRenderEntries(value.agentRender,value.agent)||!hasCanonicalChannelReferences(value.agentRender)||!hasCanonicalChannelReferences(value.buildSteps)||!hasCanonicalChannelReferences(value.stateUpdates)||!hasCanonicalChannelReferences(value.healthChecks)||!hasCanonicalNetworkPolicyReferences(value.networkPolicy)||!hasCanonicalRuntimeSetupReferences(value.runtimeSetup)){return null}return cloneSandboxMessagingPlan(normalizePersistedSandboxMessagingPlanShape(value,options.environment))}function hasMatchingAgentRenderEntries(value,agent){return!Array.isArray(value)||value.every(render=>isObjectRecord(render)&&render.agent===agent)}function hasCanonicalNetworkPolicyReferences(value){if(!isObjectRecord(value)||!Object.hasOwn(value,"entries"))return true;return hasCanonicalChannelReferences(value.entries)}function cloneSandboxMessagingPlan(plan){return JSON.parse(JSON.stringify(plan))}function isOptionalObjectArray(value,key){if(!Object.hasOwn(value,key))return true;const entries=value[key];return Array.isArray(entries)&&entries.every(isObjectRecord)}function isHostForward(value){return isObjectRecord(value)&&typeof value.channelId==="string"&&typeof value.port==="number"&&Number.isInteger(value.port)&&value.port>=1&&value.port<=65535&&typeof value.label==="string"}function isRuntimeSetup(value){if(value===void 0)return true;return isObjectRecord(value)&&Array.isArray(value.nodePreloads)&&Array.isArray(value.envAliases)&&Array.isArray(value.secretScans)&&value.nodePreloads.every(isObjectRecord)&&value.envAliases.every(isObjectRecord)&&value.secretScans.every(isObjectRecord)}function isCanonicalMessagingChannelId(value){return typeof value==="string"&&value.length>0&&normalizeMessagingChannelId(value)===value}function hasCanonicalChannelReferences(value){return value===void 0||Array.isArray(value)&&value.every(entry=>isObjectRecord(entry)&&isCanonicalMessagingChannelId(entry.channelId))}function hasCanonicalRuntimeSetupReferences(value){if(value===void 0)return true;if(!isObjectRecord(value))return false;return["nodePreloads","envAliases","secretScans"].every(field=>hasCanonicalChannelReferences(value[field]))}var ManagedStartupAgentEnvironmentError=class extends Error{constructor(message){super(`Cannot map managed startup profile: ${message}`);this.name="ManagedStartupAgentEnvironmentError"}};var EMPTY_APPLICATION_ENVIRONMENT=Object.freeze({});var OPENCLAW_APPLICATION_RUNTIME_INPUTS=Object.freeze([["NEMOCLAW_AUTO_PAIR_DEADLINE_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_FAST_DEADLINE_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_FAST_REENTRY_INTERVAL_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_FAST_REENTRY_POLLS","positive-safe-integer"],["NEMOCLAW_AUTO_PAIR_RUN_TIMEOUT_SECS","positive-finite-seconds"],["NEMOCLAW_AUTO_PAIR_SLOW_INTERVAL_SECS","positive-finite-seconds"]]);function booleanFlag(value){return value?"1":"0"}function canonicalizeJson2(value){if(Array.isArray(value))return value.map(item=>canonicalizeJson2(item));if(value===null||typeof value!=="object")return value;const record2=value;return Object.fromEntries(Object.keys(record2).sort().map(key=>[key,canonicalizeJson2(record2[key])]))}function encodeCanonicalJson(value){return import_node_buffer4.Buffer.from(JSON.stringify(canonicalizeJson2(value)),"utf8").toString("base64")}function sortedEnvironment(environment){return Object.freeze(Object.fromEntries(Object.entries(environment).sort(([left],[right])=>leftright?1:0)))}function canonicalApplicationRuntimeValue(name,raw,kind){if(raw.includes("\0")||/[\r\n]/u.test(raw)){throw new ManagedStartupAgentEnvironmentError(`${name} must be single-line text`)}const value=Number(raw.trim());const valid=kind==="positive-safe-integer"?Number.isSafeInteger(value)&&value>0:Number.isFinite(value)&&value>0;if(!valid){throw new ManagedStartupAgentEnvironmentError(`${name} must be ${kind==="positive-safe-integer"?"a positive safe integer":"finite positive seconds"}`)}return String(value)}function applicationRuntimePlan(profile,environment){const exportEnvironment={};if(profile.agent==="openclaw"){for(const[name,kind]of OPENCLAW_APPLICATION_RUNTIME_INPUTS){const raw=environment[name];if(raw!==void 0){exportEnvironment[name]=canonicalApplicationRuntimeValue(name,raw,kind)}}}const unsetEnvironment=new Set(MANAGED_STARTUP_RUNTIME_CLEANUP_OBLIGATIONS.filter(({supportedFor})=>!supportedFor.includes(profile.agent)).map(({input})=>input));if(profile.agent!=="openclaw"){for(const[name]of OPENCLAW_APPLICATION_RUNTIME_INPUTS){unsetEnvironment.add(name)}}return Object.freeze({exportEnvironment:sortedEnvironment(exportEnvironment),unsetEnvironment:Object.freeze([...unsetEnvironment].sort())})}function commonConfigurationEnvironment(profile){if(profile.inference===null)return{...PROVIDERLESS_INFERENCE_ENV,NEMOCLAW_TOOL_DISCLOSURE:profile.tools.disclosure};return{NEMOCLAW_INFERENCE_API:profile.inference.api,NEMOCLAW_INFERENCE_BASE_URL:profile.inference.routedBaseUrl,NEMOCLAW_INFERENCE_PROVIDER_ID:profile.inference.routeProvider,NEMOCLAW_MODEL:profile.inference.model,NEMOCLAW_TOOL_DISCLOSURE:profile.tools.disclosure,NEMOCLAW_UPSTREAM_PROVIDER:profile.inference.upstreamProvider}}function appendHostProxyEnvironment(environment,profile,options={}){if(options.preserveAmbientWhenAbsent===true&&profile.proxy.hostHttpUrl===null&&profile.proxy.hostHttpsUrl===null&&profile.proxy.hostNoProxy.length===0){return}const httpProxy=profile.proxy.hostHttpUrl??"";const httpsProxy=profile.proxy.hostHttpsUrl??"";const noProxy=profile.proxy.hostNoProxy.join(",");environment.HTTP_PROXY=httpProxy;environment.HTTPS_PROXY=httpsProxy;environment.NO_PROXY=noProxy;environment.http_proxy=httpProxy;environment.https_proxy=httpsProxy;environment.no_proxy=noProxy}function messagingEnvironment(profile,expectedAgent){if(profile.messaging.plan===null)return{};const plan=parseSandboxMessagingPlan(profile.messaging.plan,{agent:expectedAgent});if(!plan){throw new ManagedStartupAgentEnvironmentError(`messaging.plan must contain a validated ${expectedAgent} messaging plan`)}const{workflow:_workflow,...imageBuildPlan}=plan;return{NEMOCLAW_MESSAGING_PLAN_B64:encodeCanonicalJson(imageBuildPlan)}}function corporateCaMaterial(profile){return Object.freeze({kind:"corporate-ca-handoff",legacyInput:"NEMOCLAW_CORPORATE_CA_B64",expectedSha256:profile.corporateCa.bundleSha256})}function rootOwnedFile(legacyInput,path4,value){return Object.freeze({kind:"root-owned-file",legacyInput,path:path4,contents:`${value} +`,owner:"root",group:"root",mode:292})}function dashboardAction(dashboard){return Object.freeze({kind:"configure-dashboard",dashboard:Object.freeze(structuredClone(dashboard))})}function applicationActions(profile,messagingAgent){const actions=[];if(messagingAgent!==null){actions.push(Object.freeze({kind:"apply-messaging-plan",agent:messagingAgent,mode:profile.messaging.plan===null?"clear":"apply",phase:"runtime-setup",runAs:"root"}))}actions.push(Object.freeze({kind:"generate-agent-config",agent:profile.agent,runAs:"sandbox"}));if(messagingAgent!==null){actions.push(Object.freeze({kind:"apply-messaging-plan",agent:messagingAgent,mode:profile.messaging.plan===null?"clear":"apply",phase:"post-agent-install",runAs:"sandbox"}))}actions.push(dashboardAction(profile.dashboard));return Object.freeze(actions)}function mapOpenClawProfile(profile,environment){if(profile.agent!=="openclaw"||profile.agentConfig.agent!=="openclaw"||profile.dashboard.agent!=="openclaw"||profile.inference!==null&&(profile.inference.primaryModelRef===null||profile.inference.inputModalities===null)||profile.tuning.contextWindow===null||profile.tuning.maxTokens===null||profile.tuning.reasoning===null||profile.tuning.reasoningEffort===null){throw new ManagedStartupAgentEnvironmentError("OpenClaw profile state is inconsistent")}const configurationEnvironment={...commonConfigurationEnvironment(profile),...messagingEnvironment(profile,"openclaw"),CHAT_UI_URL:profile.dashboard.url,NEMOCLAW_AGENT_HEARTBEAT_EVERY:profile.agentConfig.heartbeatEvery??"",NEMOCLAW_AGENT_TIMEOUT:String(profile.agentConfig.agentTimeoutSeconds),NEMOCLAW_CONTEXT_WINDOW:String(profile.tuning.contextWindow),NEMOCLAW_DASHBOARD_BIND:profile.dashboard.bindAddress==="0.0.0.0"?profile.dashboard.bindAddress:"",NEMOCLAW_EXTRA_AGENTS_JSON_B64:encodeCanonicalJson(profile.agentConfig.extraAgents),NEMOCLAW_INFERENCE_COMPAT_B64:encodeCanonicalJson(profile.inference?.compatibility??{}),NEMOCLAW_INFERENCE_INPUTS:profile.inference?.inputModalities?.join(",")??"text",NEMOCLAW_MAX_TOKENS:String(profile.tuning.maxTokens),NEMOCLAW_OPENCLAW_OTEL:booleanFlag(profile.agentConfig.otel.enabled),NEMOCLAW_OPENCLAW_OTEL_ENDPOINT:profile.agentConfig.otel.endpointUrl,NEMOCLAW_OPENCLAW_OTEL_SAMPLE_RATE:String(profile.agentConfig.otel.sampleRate),NEMOCLAW_OPENCLAW_OTEL_SERVICE_NAME:profile.agentConfig.otel.serviceName,NEMOCLAW_PRIMARY_MODEL_REF:profile.inference?.primaryModelRef??"",NEMOCLAW_REASONING:String(profile.tuning.reasoning),NEMOCLAW_REASONING_EFFORT:profile.tuning.reasoningEffort,NEMOCLAW_SERVING_PRESET:profile.inference?.servingPreset??"",NEMOCLAW_WEB_SEARCH_ENABLED:booleanFlag(profile.agentConfig.webSearch.enabled),NEMOCLAW_WEB_SEARCH_PROVIDER:profile.agentConfig.webSearch.provider,NEMOCLAW_WSL_DASHBOARD_EXPOSURE:booleanFlag(profile.dashboard.wslExposure)};const runtimeEnvironment={...configurationEnvironment};delete runtimeEnvironment.NEMOCLAW_MESSAGING_PLAN_B64;runtimeEnvironment.NEMOCLAW_DASHBOARD_PORT=String(profile.dashboard.port);runtimeEnvironment.NEMOCLAW_MINIMAL_BOOTSTRAP=booleanFlag(profile.agentConfig.minimalBootstrap);appendHostProxyEnvironment(runtimeEnvironment,profile,{preserveAmbientWhenAbsent:true});return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials:Object.freeze([corporateCaMaterial(profile)]),actions:applicationActions(profile,"openclaw")})}function mapHermesProfile(profile,environment){if(profile.agent!=="hermes"||profile.agentConfig.agent!=="hermes"||profile.dashboard.agent!=="hermes"){throw new ManagedStartupAgentEnvironmentError("Hermes profile state is inconsistent")}let chatUiUrl=profile.dashboard.browserUrl??profile.dashboard.url;if(profile.dashboard.mode==="loopback-forwarded"){if(profile.dashboard.browserUrl===void 0){throw new ManagedStartupAgentEnvironmentError("Cannot start the Hermes dashboard because its managed startup profile has no recorded browser URL. Rerun onboarding before starting the sandbox.")}chatUiUrl=profile.dashboard.browserUrl}const configurationEnvironment={...commonConfigurationEnvironment(profile),...messagingEnvironment(profile,"hermes"),CHAT_UI_URL:chatUiUrl,NEMOCLAW_CONTEXT_WINDOW:profile.tuning.contextWindow===null?"":String(profile.tuning.contextWindow),NEMOCLAW_HERMES_TOOL_GATEWAY_BROKER:booleanFlag(profile.tools.enabledGateways.length>0),NEMOCLAW_HERMES_TOOL_GATEWAY_PRESETS_B64:encodeCanonicalJson(profile.tools.enabledGateways),NEMOCLAW_WEB_SEARCH_ENABLED:booleanFlag(profile.agentConfig.webSearch.enabled),NEMOCLAW_WEB_SEARCH_PROVIDER:profile.agentConfig.webSearch.provider,[HERMES_SWITCHYARD_ROUTING_ENV]:profile.agentConfig.switchyardRouting?encodeHermesSwitchyardRoutingIntent(profile.agentConfig.switchyardRouting):""};const runtimeEnvironment={...configurationEnvironment,HERMES_HOME:"/sandbox/.hermes",HERMES_LAZY_INSTALL_TARGET:"/sandbox/.hermes/lazy-packages"};delete runtimeEnvironment.NEMOCLAW_MESSAGING_PLAN_B64;delete runtimeEnvironment[HERMES_SWITCHYARD_ROUTING_ENV];runtimeEnvironment.NEMOCLAW_DASHBOARD_PORT=profile.dashboard.publicPort===null?"":String(profile.dashboard.publicPort);runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD=profile.dashboard.mode==="loopback-forwarded"?"1":"0";runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_INTERNAL_PORT=profile.dashboard.internalPort===null?"":String(profile.dashboard.internalPort);runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_PORT=profile.dashboard.publicPort===null?"":String(profile.dashboard.publicPort);runtimeEnvironment.NEMOCLAW_HERMES_DASHBOARD_TUI=booleanFlag(profile.dashboard.tuiEnabled);appendHostProxyEnvironment(runtimeEnvironment,profile,{preserveAmbientWhenAbsent:true});return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials:Object.freeze([corporateCaMaterial(profile)]),actions:applicationActions(profile,"hermes")})}function mapDcodeProfile(profile,environment){if(profile.agent!=="langchain-deepagents-code"||profile.agentConfig.agent!=="langchain-deepagents-code"||profile.dashboard.agent!=="langchain-deepagents-code"||profile.messaging.plan!==null||profile.inference===null){throw new ManagedStartupAgentEnvironmentError("LangChain Deep Agents Code profile state is inconsistent")}const reasoningEffort=profile.tuning.reasoningEffort===null||profile.tuning.reasoningEffort==="default"?"":profile.tuning.reasoningEffort;const configurationEnvironment={...commonConfigurationEnvironment(profile),NEMOCLAW_REASONING_EFFORT:reasoningEffort,NEMOCLAW_UPSTREAM_ENDPOINT_URL:profile.inference.upstreamEndpointUrl??""};appendHostProxyEnvironment(configurationEnvironment,profile);const runtimeEnvironment={...configurationEnvironment,NEMOCLAW_OBSERVABILITY:booleanFlag(profile.agentConfig.observabilityEnabled)};delete runtimeEnvironment.NEMOCLAW_INFERENCE_BASE_URL;delete runtimeEnvironment.NEMOCLAW_REASONING_EFFORT;delete runtimeEnvironment.NEMOCLAW_UPSTREAM_PROVIDER;for(const name of["HTTP_PROXY","HTTPS_PROXY","NO_PROXY","http_proxy","https_proxy","no_proxy"]){delete runtimeEnvironment[name]}const materials=Object.freeze([corporateCaMaterial(profile),rootOwnedFile("NEMOCLAW_DCODE_AUTO_APPROVAL","/usr/local/share/nemoclaw/dcode-auto-approval",profile.agentConfig.autoApprovalMode),rootOwnedFile("NEMOCLAW_INFERENCE_BASE_URL","/usr/local/share/nemoclaw/dcode-inference-base-url",profile.inference.routedBaseUrl),rootOwnedFile("NEMOCLAW_UPSTREAM_PROVIDER","/usr/local/share/nemoclaw/dcode-upstream-provider",profile.inference.upstreamProvider),rootOwnedFile("NEMOCLAW_INFERENCE_PROVIDER_ID","/usr/local/share/nemoclaw/dcode-provider-route",profile.inference.routeProvider),rootOwnedFile("NEMOCLAW_REASONING_EFFORT","/usr/local/share/nemoclaw/dcode-reasoning-effort",reasoningEffort)]);return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials,actions:applicationActions(profile,null)})}function mapPiProfile(profile,environment){if(profile.agent!=="pi"||profile.agentConfig.agent!=="pi"||profile.dashboard.agent!=="pi"||profile.messaging.plan!==null){throw new ManagedStartupAgentEnvironmentError("Pi profile state is inconsistent")}const configurationEnvironment={...commonConfigurationEnvironment(profile),NEMOCLAW_CONTEXT_WINDOW:profile.tuning.contextWindow===null?"":String(profile.tuning.contextWindow),NEMOCLAW_MAX_TOKENS:profile.tuning.maxTokens===null?"":String(profile.tuning.maxTokens),NEMOCLAW_REASONING:profile.tuning.reasoning===null?"":String(profile.tuning.reasoning)};appendHostProxyEnvironment(configurationEnvironment,profile);const runtimeEnvironment={...configurationEnvironment};delete runtimeEnvironment.NEMOCLAW_INFERENCE_BASE_URL;delete runtimeEnvironment.NEMOCLAW_CONTEXT_WINDOW;delete runtimeEnvironment.NEMOCLAW_MAX_TOKENS;delete runtimeEnvironment.NEMOCLAW_REASONING;for(const name of["HTTP_PROXY","HTTPS_PROXY","NO_PROXY","http_proxy","https_proxy","no_proxy"]){delete runtimeEnvironment[name]}const materials=Object.freeze([corporateCaMaterial(profile)]);return Object.freeze({schemaVersion:profile.schemaVersion,agent:profile.agent,configurationEnvironment:sortedEnvironment(configurationEnvironment),runtimeEnvironment:sortedEnvironment(runtimeEnvironment),applicationRuntime:applicationRuntimePlan(profile,environment),materials,actions:applicationActions(profile,null)})}function mapManagedStartupProfileToAgentEnvironment(profile,environment=EMPTY_APPLICATION_ENVIRONMENT){const validated=validateManagedStartupProfile(profile);switch(validated.agent){case"openclaw":return mapOpenClawProfile(validated,environment);case"hermes":return mapHermesProfile(validated,environment);case"langchain-deepagents-code":return mapDcodeProfile(validated,environment);case"pi":return mapPiProfile(validated,environment)}}var SHIPPED_AGENT_SET=new Set(MANAGED_STARTUP_AGENTS);var DEFAULT_DEPENDENCIES={prepareApplication:input=>prepareManagedStartupApplication(input),commitApplication:prepared=>commitManagedStartupApplication(prepared)};var ManagedStartupCoordinatorError=class extends Error{constructor(message){super(`Managed startup coordination failed: ${message}`);this.name="ManagedStartupCoordinatorError"}};function fail3(message){throw new ManagedStartupCoordinatorError(message)}function createAdapterRegistry(adapters2){const byAgent=new Map;for(const adapter of adapters2){if(typeof adapter!=="object"||adapter===null||!SHIPPED_AGENT_SET.has(adapter.agent)||typeof adapter.apply!=="function"){fail3("every adapter must identify one shipped agent and provide an apply function")}if(byAgent.has(adapter.agent)){fail3(`duplicate adapter registered for ${adapter.agent}`)}byAgent.set(adapter.agent,adapter)}const missing=MANAGED_STARTUP_AGENTS.filter(agent=>!byAgent.has(agent));if(missing.length>0){fail3(`missing adapter for ${missing.join(", ")}`)}if(byAgent.size!==MANAGED_STARTUP_AGENTS.length){fail3("adapter registry must contain exactly the shipped agents")}return Object.freeze(Object.fromEntries(MANAGED_STARTUP_AGENTS.map(agent=>{const adapter=byAgent.get(agent);if(!adapter)fail3(`missing adapter for ${agent}`);return[agent,adapter]})))}function requirePreparedIdentity(prepared,requestedAgent){if(prepared.expectedAgent!==requestedAgent||prepared.profile.agent!==requestedAgent){fail3(`prepared profile targets ${prepared.profile.agent}, expected ${requestedAgent}`)}}function adapterContext(prepared){return Object.freeze({agent:prepared.profile.agent,profile:prepared.profile,fingerprint:prepared.fingerprint,generationDirectory:prepared.generationDirectory,profilePath:prepared.profilePath,corporateCaPath:prepared.corporateCaPath})}async function coordinateManagedStartupApplication(input,adapters2,dependencies=DEFAULT_DEPENDENCIES){const registry=createAdapterRegistry(adapters2);const prepared=await dependencies.prepareApplication(input);requirePreparedIdentity(prepared,input.expectedAgent);if(prepared.status==="already-committed"){return{adapterApplied:false,application:await dependencies.commitApplication(prepared)}}const adapter=registry[prepared.profile.agent];if(adapter.agent!==prepared.profile.agent){fail3(`adapter registry cross-dispatch detected for ${prepared.profile.agent}`)}await adapter.apply(adapterContext(prepared));return{adapterApplied:true,application:await dependencies.commitApplication(prepared)}}var import_node_crypto4=require("node:crypto");var MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION=1;var MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES=320*1024;var MAX_CORPORATE_CA_ENCODED_BYTES=4*Math.ceil(128*1024/3);var SHA256_RE3=/^[a-f0-9]{64}$/u;var STANDARD_BASE64_RE2=/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u;var MCP_SHADOW_DIAGNOSTICS_ENV="NEMOCLAW_MCP_SHADOW_DIAGNOSTICS";var MANAGED_STARTUP_APPLICATION_RUNTIME_ENV_KEYS=Object.freeze(MANAGED_STARTUP_PROFILE_DEFERRED_RUNTIME_INPUTS.openclaw.filter(({admission,owner})=>admission==="managed-launch-forwarded"&&owner==="application-environment").map(({input})=>input));function selectManagedStartupApplicationRuntimeEnvironment(environment){const selected={};for(const name of MANAGED_STARTUP_APPLICATION_RUNTIME_ENV_KEYS){const value=environment[name];if(name===MCP_SHADOW_DIAGNOSTICS_ENV){if(value?.trim()==="1")selected[name]="1";continue}if(value!==void 0)selected[name]=value}return Object.freeze(selected)}function fail4(message){throw new Error(`Managed startup root application request is invalid: ${message}`)}function isManagedStartupRootApplyAgent(value){return typeof value==="string"&&MANAGED_STARTUP_AGENTS.includes(value)}function exactAgent(value){if(isManagedStartupRootApplyAgent(value))return value;return fail4("agent is unsupported")}function createManagedStartupRootApplyRequest(input){const agent=exactAgent(input.agent);if(input.encodedProfile.length===0||input.encodedProfile.length>MANAGED_STARTUP_PROFILE_MAX_ENCODED_BYTES){fail4("encoded profile exceeds its bounded transport")}const profile=decodeManagedStartupProfile(input.encodedProfile);if(profile.agent!==agent){fail4(`profile targets ${profile.agent}, expected ${agent}`)}const corporateCaB64=input.corporateCaB64??null;if(corporateCaB64!==null&&(corporateCaB64.length===0||corporateCaB64.length>MAX_CORPORATE_CA_ENCODED_BYTES||!STANDARD_BASE64_RE2.test(corporateCaB64)||Buffer.from(corporateCaB64,"base64").toString("base64")!==corporateCaB64)){fail4("corporate CA is not canonical bounded base64")}if(profile.corporateCa.bundleSha256!==null!==(corporateCaB64!==null)){fail4("corporate CA transport does not match the profile")}if(corporateCaB64!==null&&(0,import_node_crypto4.createHash)("sha256").update(Buffer.from(corporateCaB64,"base64")).digest("hex")!==profile.corporateCa.bundleSha256){fail4("corporate CA does not match the profile digest")}return Object.freeze({schemaVersion:MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION,agent,encodedProfile:input.encodedProfile,profileFingerprint:fingerprintManagedStartupProfile(profile),corporateCaB64})}function serializeManagedStartupRootApplyRequest(request){const normalized=createManagedStartupRootApplyRequest({agent:request.agent,encodedProfile:request.encodedProfile,...request.corporateCaB64===null?{}:{corporateCaB64:request.corporateCaB64}});if(request.schemaVersion!==MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION||request.profileFingerprint!==normalized.profileFingerprint||!SHA256_RE3.test(request.profileFingerprint)){fail4("schema version or profile fingerprint is invalid")}const serialized=`${JSON.stringify({agent:normalized.agent,corporateCaB64:normalized.corporateCaB64,encodedProfile:normalized.encodedProfile,profileFingerprint:normalized.profileFingerprint,schemaVersion:normalized.schemaVersion})} +`;if(Buffer.byteLength(serialized,"utf8")>MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES){fail4("serialized request exceeds its bounded transport")}return serialized}function parseManagedStartupRootApplyRequest(text){if(text.length===0||Buffer.byteLength(text,"utf8")>MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES){fail4("serialized request is empty or too large")}let parsed;try{parsed=JSON.parse(text)}catch{fail4("serialized request is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail4("serialized request must be an object")}const record2=parsed;const expectedKeys=["agent","corporateCaB64","encodedProfile","profileFingerprint","schemaVersion"];if(Object.keys(record2).sort().join(",")!==expectedKeys.sort().join(",")||record2.schemaVersion!==MANAGED_STARTUP_ROOT_APPLY_SCHEMA_VERSION||typeof record2.encodedProfile!=="string"||typeof record2.profileFingerprint!=="string"||record2.corporateCaB64!==null&&typeof record2.corporateCaB64!=="string"){fail4("serialized request has an invalid schema")}const request=createManagedStartupRootApplyRequest({agent:exactAgent(record2.agent),encodedProfile:record2.encodedProfile,...record2.corporateCaB64===null?{}:{corporateCaB64:record2.corporateCaB64}});if(record2.profileFingerprint!==request.profileFingerprint||!SHA256_RE3.test(record2.profileFingerprint)){fail4("profile fingerprint does not match the encoded profile")}if(serializeManagedStartupRootApplyRequest(request)!==text){fail4("serialized request is not canonical")}return request}var import_node_crypto5=require("node:crypto");var import_node_fs2=__toESM(require("node:fs"));var import_node_path2=__toESM(require("node:path"));var MANAGED_HERMES_STATE_ROOT="/sandbox/.hermes";var MANAGED_OPENCLAW_STATE_ROOT="/sandbox/.openclaw";var HERMES_STATE_VOLUME_NAME_PREFIX="nemoclaw-hermes-state-v1";var OPENCLAW_STATE_VOLUME_NAME_PREFIX="nemoclaw-openclaw-state-v1";var MANAGED_AGENT_STATE_ROOTS=Object.freeze({openclaw:Object.freeze([Object.freeze({mountTarget:MANAGED_OPENCLAW_STATE_ROOT,resourceIdentity:sandboxName=>`${OPENCLAW_STATE_VOLUME_NAME_PREFIX}-${sandboxName}`,ownershipLabels:(sandboxName,mountTarget)=>Object.freeze({"io.nvidia.nemoclaw.openclaw-state.managed":"true","io.nvidia.nemoclaw.openclaw-state.schema":"1","io.nvidia.nemoclaw.openclaw-state.sandbox":sandboxName,"io.nvidia.nemoclaw.openclaw-state.target":mountTarget}),uidAuthority:"agent",gidAuthority:"agent",mode:1528,readWrite:true})]),hermes:Object.freeze([Object.freeze({mountTarget:MANAGED_HERMES_STATE_ROOT,resourceIdentity:sandboxName=>`${HERMES_STATE_VOLUME_NAME_PREFIX}-${sandboxName}`,ownershipLabels:(sandboxName,mountTarget)=>Object.freeze({"io.nvidia.nemoclaw.hermes-state.managed":"true","io.nvidia.nemoclaw.hermes-state.schema":"1","io.nvidia.nemoclaw.hermes-state.sandbox":sandboxName,"io.nvidia.nemoclaw.hermes-state.target":mountTarget}),uidAuthority:"agent",gidAuthority:"agent",mode:2040,readWrite:true})]),"langchain-deepagents-code":Object.freeze([]),pi:Object.freeze([])});function managedStartupStateRootMountTargets(agent){return Object.freeze(MANAGED_AGENT_STATE_ROOTS[agent].map(({mountTarget})=>mountTarget))}var MANAGED_AGENT_WORKSPACE_ROOTS=Object.freeze({openclaw:Object.freeze({uidAuthority:"agent",gidAuthority:"agent",mode:493}),hermes:Object.freeze({uidAuthority:"agent",gidAuthority:"agent",mode:493}),"langchain-deepagents-code":Object.freeze({uidAuthority:"root",gidAuthority:"agent",mode:1021}),pi:Object.freeze({uidAuthority:"agent",gidAuthority:"agent",mode:493})});function exactAgentIdentity(value,label){if(!Number.isSafeInteger(value)||value<0||value>2147483647){throw new Error(`Managed startup state-root ${label} authority is invalid.`)}return value}function managedStartupWorkspaceRoot(input){const uid=exactAgentIdentity(input.agentIdentity.uid,"workspace UID");const gid=exactAgentIdentity(input.agentIdentity.gid,"workspace GID");const declaration=MANAGED_AGENT_WORKSPACE_ROOTS[input.agent];return Object.freeze({uid:declaration.uidAuthority==="root"?0:uid,gid,mode:declaration.mode})}var TRANSACTION_SCHEMA_VERSION=1;var MAX_TRANSACTION_FILES=128;var MAX_TRANSACTION_FILE_BYTES=8*1024*1024;var MAX_TRANSACTION_TOTAL_BYTES=32*1024*1024;var MAX_MANIFEST_BYTES=256*1024;var MAX_COMMIT_RECEIPT_BYTES=4096;var MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES=4*Math.ceil(MAX_TRANSACTION_TOTAL_BYTES/3)+MAX_MANIFEST_BYTES*2;var TRANSACTION_PARENT_DIRECTORY_MODE=493;var TRANSACTION_DIRECTORY_MODE=448;var TRANSACTION_FILE_MODE=256;var ATOMIC_TEMPORARY_FILE_MODE=384;var MANAGED_STARTUP_SHARED_TRANSACTION_DIRECTORY="/var/lib/nemoclaw/managed-startup-shared-state-transaction-v1";var MANAGED_STARTUP_SHARED_ROLLBACK_RECEIPT_DIRECTORY="/run/nemoclaw/managed-startup-shared-rollback-receipt-v1";var MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_DIRECTORY="/var/lib/nemoclaw/managed-startup-shared-state-commit-v1";var MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE="receipt.json";function fail5(message){throw new Error(`Managed startup shared-state transaction failed: ${message}`)}function resolveOptions(options={}){const sandboxRoot=import_node_path2.default.resolve(options.sandboxRoot??"/sandbox");const transactionDirectory=import_node_path2.default.resolve(options.transactionDirectory??MANAGED_STARTUP_SHARED_TRANSACTION_DIRECTORY);const commitReceiptDirectory=import_node_path2.default.resolve(options.commitReceiptDirectory??(options.transactionDirectory?import_node_path2.default.join(import_node_path2.default.dirname(transactionDirectory),import_node_path2.default.basename(MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_DIRECTORY)):MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_DIRECTORY));if(transactionDirectory===sandboxRoot||transactionDirectory.startsWith(`${sandboxRoot}${import_node_path2.default.sep}`)||commitReceiptDirectory===sandboxRoot||commitReceiptDirectory.startsWith(`${sandboxRoot}${import_node_path2.default.sep}`)||import_node_path2.default.dirname(commitReceiptDirectory)!==import_node_path2.default.dirname(transactionDirectory)||commitReceiptDirectory===transactionDirectory){fail5("transaction and commit receipts require distinct paths outside sandbox-shared state")}const bootstrapIdentity=options.bootstrapIdentity??null;if(bootstrapIdentity!==null&&!/^[a-f0-9]{64}$/u.test(bootstrapIdentity)){fail5("bootstrap identity must encode 32 lowercase-hex bytes")}return{sandboxRoot,transactionParentDirectory:import_node_path2.default.dirname(transactionDirectory),transactionDirectory,backupDirectory:import_node_path2.default.join(transactionDirectory,"backups"),manifestFile:import_node_path2.default.join(transactionDirectory,"manifest.json"),commitReceiptDirectory,commitReceiptFile:import_node_path2.default.join(commitReceiptDirectory,MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE),trustedUid:options.trustedUid??0,trustedGid:options.trustedGid??0,readOnlyReceipt:options.readOnlyReceipt??false,bootstrapIdentity}}function modeOf2(stat){if(typeof stat.mode==="bigint"){return Number(stat.mode&0o7777n)}return stat.mode&4095}function requireTransactionIdentity(options){const expectedUid=options.readOnlyReceipt?0:options.trustedUid;const expectedGid=options.readOnlyReceipt?0:options.trustedGid;if(process.geteuid?.()!==expectedUid||process.getegid?.()!==expectedGid){fail5("transaction control requires the trusted effective identity")}}function pathExistsNoFollow(target2){try{import_node_fs2.default.lstatSync(target2);return true}catch(error){if(error.code==="ENOENT")return false;fail5(`could not inspect ${target2}`)}}function requireDirectory(target2,options,expectedMode=null){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch{fail5(`required directory is missing: ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`required directory is unsafe: ${target2}`)}if(expectedMode!==null&&(stat.uid!==options.trustedUid||stat.gid!==options.trustedGid||modeOf2(stat)!==expectedMode)){fail5(`${target2} must be ${options.trustedUid}:${options.trustedGid} mode ${expectedMode.toString(8)}`)}return stat}function requireTransactionBoundaries(options){requireDirectory(options.sandboxRoot,options);requireDirectory(options.transactionParentDirectory,options,TRANSACTION_PARENT_DIRECTORY_MODE)}function sameStableMetadata(left,right){return left.dev===right.dev&&left.ino===right.ino&&left.mode===right.mode&&left.nlink===right.nlink&&left.uid===right.uid&&left.gid===right.gid&&left.size===right.size&&left.mtimeNs===right.mtimeNs&&left.ctimeNs===right.ctimeNs}function readStableFile(target2,maxBytes){const noFollow=import_node_fs2.default.constants.O_NOFOLLOW;if(typeof noFollow!=="number")fail5("O_NOFOLLOW is unavailable");let descriptor;try{descriptor=import_node_fs2.default.openSync(target2,import_node_fs2.default.constants.O_RDONLY|noFollow)}catch{fail5(`could not safely open ${target2}`)}try{const before=import_node_fs2.default.fstatSync(descriptor,{bigint:true});if(!before.isFile()||before.nlink!==1n||before.size<0n||before.size>BigInt(maxBytes)){fail5(`refusing unsafe or oversized transaction file ${target2}`)}const bytes=Buffer.alloc(Number(before.size));let offset=0;while(offset!segment||segment==="."||segment==="..")){fail5(`unsafe transaction path ${JSON.stringify(value)}`)}return segments.join("/")}function absoluteTarget(relativePath,options){const safe=safeRelativePath(relativePath);const target2=import_node_path2.default.resolve(options.sandboxRoot,safe);if(!target2.startsWith(`${options.sandboxRoot}${import_node_path2.default.sep}`)){fail5(`transaction target escapes the sandbox root: ${relativePath}`)}return target2}function relativeTarget(target2,options){return safeRelativePath(import_node_path2.default.relative(options.sandboxRoot,target2))}function isDeclaredAgentStateRoot(expectedAgent,outputRoot,options){const relative=import_node_path2.default.relative(options.sandboxRoot,outputRoot).split(import_node_path2.default.sep).join("/");const canonicalTarget=import_node_path2.default.posix.join("/sandbox",relative);return managedStartupStateRootMountTargets(expectedAgent).includes(canonicalTarget)}function validateExistingAncestors(target2,expectedAgent,options){const relative=relativeTarget(target2,options);const sandboxStat=requireDirectory(options.sandboxRoot,options);const outputRoot=agentRoot(expectedAgent,options.sandboxRoot);if(target2!==outputRoot&&!target2.startsWith(`${outputRoot}${import_node_path2.default.sep}`)){fail5(`transaction target escapes the ${expectedAgent} state root: ${target2}`)}let current=options.sandboxRoot;let expectedDevice=sandboxStat.dev;const segments=relative.split("/").slice(0,-1);for(const segment of segments){current=import_node_path2.default.join(current,segment);let stat;try{stat=import_node_fs2.default.lstatSync(current)}catch(error){if(error.code==="ENOENT")return;fail5(`could not inspect transaction path ancestor ${current}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`transaction path ancestor is unsafe: ${current}`)}if(current===outputRoot&&isDeclaredAgentStateRoot(expectedAgent,outputRoot,options)){expectedDevice=stat.dev}else if(stat.dev!==expectedDevice){fail5(`transaction path crosses a nested filesystem mount: ${current}`)}}}function managedOutputDevice(expectedAgent,options){const sandboxStat=requireDirectory(options.sandboxRoot,options);const outputRoot=agentRoot(expectedAgent,options.sandboxRoot);let stat;try{stat=import_node_fs2.default.lstatSync(outputRoot)}catch(error){if(error.code==="ENOENT")return sandboxStat.dev;fail5(`could not inspect managed output root ${outputRoot}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`managed output root is unsafe: ${outputRoot}`)}if(!isDeclaredAgentStateRoot(expectedAgent,outputRoot,options)&&stat.dev!==sandboxStat.dev){fail5(`managed output root crosses a nested filesystem mount: ${outputRoot}`)}return stat.dev}function agentRoot(agent,sandboxRoot){switch(agent){case"openclaw":return import_node_path2.default.join(sandboxRoot,".openclaw");case"hermes":return import_node_path2.default.join(sandboxRoot,".hermes");case"langchain-deepagents-code":return import_node_path2.default.join(sandboxRoot,".deepagents");case"pi":return import_node_path2.default.join(sandboxRoot,".pi")}}function resolveUnderAgentRoot(root,relativePath){const safe=safeRelativePath(relativePath);const target2=import_node_path2.default.resolve(root,safe);if(!target2.startsWith(`${root}${import_node_path2.default.sep}`)){fail5(`managed output escapes the agent root: ${relativePath}`)}return target2}function renderTarget(root,agent,target2){if(agent==="openclaw"&&target2==="openclaw.json"){return import_node_path2.default.join(root,"openclaw.json")}const prefix=agent==="openclaw"?"~/.openclaw/":agent==="hermes"?"~/.hermes/":null;if(!prefix||!target2.startsWith(prefix)){fail5(`unsupported managed messaging render target ${JSON.stringify(target2)}`)}return resolveUnderAgentRoot(root,target2.slice(prefix.length))}function managedOutputTargets(profile,options){const root=agentRoot(profile.agent,options.sandboxRoot);const files=new Set;const directories=new Set([root]);switch(profile.agent){case"openclaw":files.add(import_node_path2.default.join(root,"openclaw.json"));break;case"hermes":files.add(import_node_path2.default.join(root,"config.yaml"));files.add(import_node_path2.default.join(root,".env"));files.add(import_node_path2.default.join(root,".config-hash"));break;case"langchain-deepagents-code":files.add(import_node_path2.default.join(root,"config.toml"));directories.add(import_node_path2.default.join(root,".state"));break;case"pi":directories.add(import_node_path2.default.join(root,"agent"));files.add(import_node_path2.default.join(root,"agent","models.json"));files.add(import_node_path2.default.join(root,"agent","settings.json"));break}if(profile.messaging.plan!==null){const plan=parseSandboxMessagingPlan(profile.messaging.plan,{agent:profile.agent});if(!plan)fail5("managed messaging plan is invalid");for(const render of selectEnabledMessagingAgentRender(plan)){if(typeof render.target!=="string")continue;files.add(renderTarget(root,profile.agent,render.target))}for(const step of selectEnabledPostAgentInstallBuildFiles(plan)){if(typeof step.value!=="object"||step.value===null){continue}const outputPath=step.value.path;if(typeof outputPath==="string"){files.add(resolveUnderAgentRoot(root,outputPath))}}}for(const file of files){let parent=import_node_path2.default.dirname(file);while(parent!==options.sandboxRoot&&parent.startsWith(`${root}${import_node_path2.default.sep}`)){directories.add(parent);if(parent===root)break;parent=import_node_path2.default.dirname(parent)}}return{files:[...files].sort(),directories:[...directories].sort((left,right)=>left.split(import_node_path2.default.sep).length-right.split(import_node_path2.default.sep).length)}}function snapshotFile(target2,index,expectedAgent,options){validateExistingAncestors(target2,expectedAgent,options);let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT"){return{receipt:{path:relativeTarget(target2,options),state:"absent"},bytes:null}}fail5(`could not inspect managed output ${target2}`)}if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1){fail5(`managed output is not a safe regular file: ${target2}`)}if(stat.dev!==managedOutputDevice(expectedAgent,options)){fail5(`managed output crosses a nested filesystem mount: ${target2}`)}const stable=readStableFile(target2,MAX_TRANSACTION_FILE_BYTES);const size=Number(stable.stat.size);const backup=`${String(index).padStart(3,"0")}.bin`;return{receipt:{path:relativeTarget(target2,options),state:"file",backup,sha256:(0,import_node_crypto5.createHash)("sha256").update(stable.bytes).digest("hex"),size,uid:Number(stable.stat.uid),gid:Number(stable.stat.gid),mode:Number(stable.stat.mode&0o7777n)},bytes:stable.bytes}}function snapshotDirectory(target2,expectedAgent,options){validateExistingAncestors(import_node_path2.default.join(target2,".receipt"),expectedAgent,options);let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT"){return{path:relativeTarget(target2,options),state:"absent"}}fail5(`could not inspect managed output directory ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`managed output directory is unsafe: ${target2}`)}if(stat.dev!==managedOutputDevice(expectedAgent,options)){fail5(`managed output directory crosses a nested filesystem mount: ${target2}`)}return{path:relativeTarget(target2,options),state:"directory",uid:stat.uid,gid:stat.gid,mode:modeOf2(stat)}}function atomicWriteTrustedFile(target2,contents,mode,uid,gid){const parent=import_node_path2.default.dirname(target2);const temporary=import_node_path2.default.join(parent,`.${import_node_path2.default.basename(target2)}.${(0,import_node_crypto5.randomBytes)(12).toString("hex")}`);let descriptor;try{descriptor=import_node_fs2.default.openSync(temporary,import_node_fs2.default.constants.O_CREAT|import_node_fs2.default.constants.O_EXCL|import_node_fs2.default.constants.O_WRONLY|import_node_fs2.default.constants.O_NOFOLLOW,384);import_node_fs2.default.writeFileSync(descriptor,contents);import_node_fs2.default.fchownSync(descriptor,uid,gid);import_node_fs2.default.fchmodSync(descriptor,mode);import_node_fs2.default.fsyncSync(descriptor);import_node_fs2.default.closeSync(descriptor);descriptor=void 0;import_node_fs2.default.renameSync(temporary,target2)}catch(error){if(descriptor!==void 0)import_node_fs2.default.closeSync(descriptor);try{import_node_fs2.default.unlinkSync(temporary)}catch{}fail5(`could not atomically write ${target2}: ${error.message}`)}}function fsyncDirectory(directory){const descriptor=import_node_fs2.default.openSync(directory,import_node_fs2.default.constants.O_RDONLY);try{import_node_fs2.default.fsyncSync(descriptor)}finally{import_node_fs2.default.closeSync(descriptor)}}function canonicalManifest(manifest){return`${JSON.stringify(manifest,null,2)} `}function canonicalLegacyManifest(manifest){return`${JSON.stringify({schemaVersion:manifest.schemaVersion,agent:manifest.agent,profileFingerprint:manifest.profileFingerprint,files:manifest.files,directories:manifest.directories},null,2)} `}function canonicalCommitReceipt(receipt){return`${JSON.stringify(receipt,null,2)} -`}function requireExactKeys(record2,keys){if(Object.keys(record2).sort().join(",")!==[...keys].sort().join(",")){fail5("transaction manifest contains unexpected fields")}}function parseCommitReceipt(text){let parsed;try{parsed=JSON.parse(text)}catch{fail5("commit receipt is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail5("commit receipt must be an object")}const record2=parsed;requireExactKeys(record2,["agent","bootstrapIdentity","profileFingerprint","schemaVersion"]);if(record2.schemaVersion!==TRANSACTION_SCHEMA_VERSION||!MANAGED_STARTUP_AGENTS.includes(String(record2.agent))||typeof record2.profileFingerprint!=="string"||!/^[a-f0-9]{64}$/u.test(record2.profileFingerprint)||typeof record2.bootstrapIdentity!=="string"||!/^[a-f0-9]{64}$/u.test(record2.bootstrapIdentity)){fail5("commit receipt has an invalid envelope")}const receipt={schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:record2.agent,profileFingerprint:record2.profileFingerprint,bootstrapIdentity:record2.bootstrapIdentity};if(canonicalCommitReceipt(receipt)!==text){fail5("commit receipt is not canonical")}return receipt}function safeMetadata(value){return Number.isSafeInteger(value)&&value>=0}function parseManifest(text){let parsed;try{parsed=JSON.parse(text)}catch{fail5("transaction manifest is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail5("transaction manifest must be an object")}const record2=parsed;const hasBootstrapIdentity=Object.hasOwn(record2,"bootstrapIdentity");requireExactKeys(record2,hasBootstrapIdentity?["agent","bootstrapIdentity","directories","files","profileFingerprint","schemaVersion"]:["agent","directories","files","profileFingerprint","schemaVersion"]);const bootstrapIdentity=hasBootstrapIdentity?record2.bootstrapIdentity:null;if(record2.schemaVersion!==TRANSACTION_SCHEMA_VERSION||!MANAGED_STARTUP_AGENTS.includes(String(record2.agent))||typeof record2.profileFingerprint!=="string"||!/^[a-f0-9]{64}$/u.test(record2.profileFingerprint)||!(bootstrapIdentity===null||typeof bootstrapIdentity==="string"&&/^[a-f0-9]{64}$/u.test(bootstrapIdentity))||!Array.isArray(record2.files)||!Array.isArray(record2.directories)||record2.files.length>MAX_TRANSACTION_FILES||record2.directories.length>MAX_TRANSACTION_FILES*4){fail5("transaction manifest has an invalid envelope")}const files=record2.files.map(value=>{if(typeof value!=="object"||value===null||Array.isArray(value)){return fail5("transaction file receipt must be an object")}const receipt=value;if(typeof receipt.path!=="string"){return fail5("transaction file receipt path must be a string")}const receiptPath=safeRelativePath(receipt.path);if(receipt.state==="absent"){requireExactKeys(receipt,["path","state"]);return{path:receiptPath,state:"absent"}}requireExactKeys(receipt,["backup","gid","mode","path","sha256","size","state","uid"]);if(receipt.state!=="file"||typeof receipt.backup!=="string"||!/^[0-9]{3}\.bin$/u.test(receipt.backup)||typeof receipt.sha256!=="string"||!/^[a-f0-9]{64}$/u.test(receipt.sha256)||!safeMetadata(receipt.size)||receipt.size>MAX_TRANSACTION_FILE_BYTES||!safeMetadata(receipt.uid)||!safeMetadata(receipt.gid)||!safeMetadata(receipt.mode)||receipt.mode>4095){return fail5("transaction file receipt is invalid")}return{path:receiptPath,state:"file",backup:receipt.backup,sha256:receipt.sha256,size:receipt.size,uid:receipt.uid,gid:receipt.gid,mode:receipt.mode}});const directories=record2.directories.map(value=>{if(typeof value!=="object"||value===null||Array.isArray(value)){return fail5("transaction directory receipt must be an object")}const receipt=value;if(typeof receipt.path!=="string"){return fail5("transaction directory receipt path must be a string")}const receiptPath=safeRelativePath(receipt.path);if(receipt.state==="absent"){requireExactKeys(receipt,["path","state"]);return{path:receiptPath,state:"absent"}}requireExactKeys(receipt,["gid","mode","path","state","uid"]);if(receipt.state!=="directory"||!safeMetadata(receipt.uid)||!safeMetadata(receipt.gid)||!safeMetadata(receipt.mode)||receipt.mode>4095){return fail5("transaction directory receipt is invalid")}return{path:receiptPath,state:"directory",uid:receipt.uid,gid:receipt.gid,mode:receipt.mode}});const filePaths=files.map(receipt=>receipt.path);const directoryPaths=directories.map(receipt=>receipt.path);const backupNames=files.filter(receipt=>receipt.state==="file").map(receipt=>receipt.backup);if(new Set(filePaths).size!==filePaths.length||new Set(directoryPaths).size!==directoryPaths.length||new Set(backupNames).size!==backupNames.length){fail5("transaction manifest contains duplicate receipts")}const manifest={schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:record2.agent,profileFingerprint:record2.profileFingerprint,bootstrapIdentity,files,directories};const canonical=hasBootstrapIdentity?canonicalManifest(manifest):canonicalLegacyManifest(manifest);if(canonical!==text){fail5("transaction manifest is not canonical")}return manifest}function requireTrustedTransactionPath(target2,mode,options){const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||(mode===TRANSACTION_DIRECTORY_MODE?!stat.isDirectory():!stat.isFile())||!options.readOnlyReceipt&&(stat.uid!==options.trustedUid||stat.gid!==options.trustedGid)||modeOf2(stat)!==mode){fail5(`transaction artifact has unsafe metadata: ${target2}`)}}function requireReadOnlyReceiptMount(target2,options){if(!options.readOnlyReceipt)return;const probe=import_node_path2.default.join(target2,".nemoclaw-write-probe");let descriptor;try{descriptor=import_node_fs2.default.openSync(probe,import_node_fs2.default.constants.O_CREAT|import_node_fs2.default.constants.O_EXCL|import_node_fs2.default.constants.O_WRONLY|import_node_fs2.default.constants.O_NOFOLLOW,384);import_node_fs2.default.closeSync(descriptor);descriptor=void 0;import_node_fs2.default.unlinkSync(probe)}catch(error){if(descriptor!==void 0)import_node_fs2.default.closeSync(descriptor);if(error.code==="EROFS")return;fail5("copied receipt must be mounted on a read-only filesystem")}fail5("copied receipt mount is writable")}function loadManifest(options){requireTransactionBoundaries(options);if(!pathExistsNoFollow(options.transactionDirectory))return null;requireTrustedTransactionPath(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE,options);requireReadOnlyReceiptMount(options.transactionDirectory,options);requireTrustedTransactionPath(options.backupDirectory,TRANSACTION_DIRECTORY_MODE,options);requireTrustedTransactionPath(options.manifestFile,TRANSACTION_FILE_MODE,options);const stable=readStableFile(options.manifestFile,MAX_MANIFEST_BYTES);if(!options.readOnlyReceipt&&(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid)||Number(stable.stat.mode&0o7777n)!==TRANSACTION_FILE_MODE){fail5("transaction manifest ownership changed while it was read")}return parseManifest(stable.bytes.toString("utf8"))}function transactionOptionsAt(options,transactionDirectory){return{...options,transactionDirectory,backupDirectory:import_node_path2.default.join(transactionDirectory,"backups"),manifestFile:import_node_path2.default.join(transactionDirectory,"manifest.json")}}function loadCommitReceipt(options){requireTransactionBoundaries(options);if(!pathExistsNoFollow(options.commitReceiptDirectory))return null;requireTrustedTransactionPath(options.commitReceiptDirectory,TRANSACTION_DIRECTORY_MODE,options);if(pathExistsNoFollow(options.commitReceiptFile)){requireReadOnlyReceiptMount(options.commitReceiptDirectory,options);requireTrustedTransactionPath(options.commitReceiptFile,TRANSACTION_FILE_MODE,options);const stable=readStableFile(options.commitReceiptFile,MAX_COMMIT_RECEIPT_BYTES);if(!options.readOnlyReceipt&&(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid)||Number(stable.stat.mode&0o7777n)!==TRANSACTION_FILE_MODE){fail5("commit receipt ownership changed while it was read")}return{receipt:parseCommitReceipt(stable.bytes.toString("utf8")),compact:true}}const stagedOptions=transactionOptionsAt(options,options.commitReceiptDirectory);const staged=loadManifest(stagedOptions);if(!staged||staged.bootstrapIdentity===null){fail5("durable commit staging receipt is incomplete")}verifyAllBackups(staged.files,stagedOptions);return{receipt:{schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:staged.agent,profileFingerprint:staged.profileFingerprint,bootstrapIdentity:staged.bootstrapIdentity},compact:false}}function verifyBackup(receipt,options){const backupPath=import_node_path2.default.join(options.backupDirectory,receipt.backup);requireTrustedTransactionPath(backupPath,TRANSACTION_FILE_MODE,options);const stable=readStableFile(backupPath,MAX_TRANSACTION_FILE_BYTES);const digest=(0,import_node_crypto6.createHash)("sha256").update(stable.bytes).digest("hex");if(stable.bytes.length!==receipt.size||digest!==receipt.sha256){fail5(`transaction backup does not match its receipt: ${receipt.path}`)}return stable.bytes}function verifyAllBackups(receipts,options){const backups=new Map;for(const receipt of receipts){if(receipt.state==="file"){backups.set(receipt.path,verifyBackup(receipt,options))}}return backups}function fileMatchesReceipt(target2,receipt){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return false;fail5(`could not inspect managed output ${target2}`)}if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1)return false;const stable=readStableFile(target2,MAX_TRANSACTION_FILE_BYTES);return stable.bytes.length===receipt.size&&(0,import_node_crypto6.createHash)("sha256").update(stable.bytes).digest("hex")===receipt.sha256&&Number(stable.stat.uid)===receipt.uid&&Number(stable.stat.gid)===receipt.gid&&Number(stable.stat.mode&0o7777n)===receipt.mode}function directoryMatchesReceipt(target2,receipt){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return false;fail5(`could not inspect managed output directory ${target2}`)}return!stat.isSymbolicLink()&&stat.isDirectory()&&stat.uid===receipt.uid&&stat.gid===receipt.gid&&modeOf2(stat)===receipt.mode}function removeTransactionDirectory(options){requireTrustedTransactionPath(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE,options);import_node_fs2.default.rmSync(options.transactionDirectory,{force:false,recursive:true});fsyncDirectory(options.transactionParentDirectory);if(pathExistsNoFollow(options.transactionDirectory)){fail5("transaction directory remained after cleanup")}}function assertCommitReceiptMatches(receipt,expected){if(receipt.agent!==expected.agent||expected.profileFingerprint!==void 0&&receipt.profileFingerprint!==expected.profileFingerprint||receipt.bootstrapIdentity!==expected.bootstrapIdentity){fail5("durable commit receipt belongs to a different bootstrap attempt")}}function loadCommitStagingManifest(options){if(!pathExistsNoFollow(options.manifestFile))return null;requireTrustedTransactionPath(options.manifestFile,TRANSACTION_FILE_MODE,options);const stable=readStableFile(options.manifestFile,MAX_MANIFEST_BYTES);if(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid||Number(stable.stat.mode&0o7777n)!==TRANSACTION_FILE_MODE){fail5("durable commit staging manifest ownership changed while it was read")}return parseManifest(stable.bytes.toString("utf8"))}function retireInterruptedCommitReceiptWrites(receipt,options){const temporaryPattern=new RegExp(`^\\.${MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE.replace(".","\\.")}\\.[a-f0-9]{24}$`,"u");for(const entry of import_node_fs2.default.readdirSync(options.commitReceiptDirectory)){if(!temporaryPattern.test(entry))continue;const target2=import_node_path2.default.join(options.commitReceiptDirectory,entry);const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==options.trustedUid||stat.gid!==options.trustedGid||![ATOMIC_TEMPORARY_FILE_MODE,TRANSACTION_FILE_MODE].includes(modeOf2(stat))){fail5("interrupted durable commit receipt write has unsafe metadata")}const stable=readStableFile(target2,MAX_COMMIT_RECEIPT_BYTES);const mode=Number(stable.stat.mode&0o7777n);if(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid||![ATOMIC_TEMPORARY_FILE_MODE,TRANSACTION_FILE_MODE].includes(mode)){fail5("interrupted durable commit receipt write changed during verification")}if(stable.bytes.length>0){let interruptedReceipt=null;try{interruptedReceipt=parseCommitReceipt(stable.bytes.toString("utf8"))}catch{}if(interruptedReceipt)assertCommitReceiptMatches(interruptedReceipt,receipt)}import_node_fs2.default.unlinkSync(target2);fsyncDirectory(options.commitReceiptDirectory)}}function compactDurableCommitReceipt(state,options){if(!state.compact){atomicWriteTrustedFile(options.commitReceiptFile,canonicalCommitReceipt(state.receipt),TRANSACTION_FILE_MODE,options.trustedUid,options.trustedGid);fsyncDirectory(options.commitReceiptDirectory)}retireInterruptedCommitReceiptWrites(state.receipt,options);const stagedOptions=transactionOptionsAt(options,options.commitReceiptDirectory);const manifestExists=pathExistsNoFollow(stagedOptions.manifestFile);const backupsExist=pathExistsNoFollow(stagedOptions.backupDirectory);const unexpectedBeforeCleanup=import_node_fs2.default.readdirSync(options.commitReceiptDirectory).filter(entry=>![MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE,import_node_path2.default.basename(stagedOptions.backupDirectory),import_node_path2.default.basename(stagedOptions.manifestFile)].includes(entry));if(unexpectedBeforeCleanup.length!==0){fail5("durable commit receipt directory contains unexpected artifacts")}if(manifestExists){const staged=loadCommitStagingManifest(stagedOptions);if(!staged||staged.bootstrapIdentity===null){fail5("durable commit staging receipt disappeared during cleanup")}assertCommitReceiptMatches(state.receipt,{agent:staged.agent,profileFingerprint:staged.profileFingerprint,bootstrapIdentity:staged.bootstrapIdentity})}if(backupsExist){requireTrustedTransactionPath(stagedOptions.backupDirectory,TRANSACTION_DIRECTORY_MODE,options);import_node_fs2.default.rmSync(stagedOptions.backupDirectory,{force:false,recursive:true});fsyncDirectory(options.commitReceiptDirectory)}if(manifestExists){requireTrustedTransactionPath(stagedOptions.manifestFile,TRANSACTION_FILE_MODE,options);import_node_fs2.default.unlinkSync(stagedOptions.manifestFile);fsyncDirectory(options.commitReceiptDirectory)}const unexpected=import_node_fs2.default.readdirSync(options.commitReceiptDirectory).filter(entry=>entry!==MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE);if(unexpected.length!==0){fail5("durable commit receipt directory contains unexpected artifacts")}const verified=loadCommitReceipt(options);if(!verified?.compact)fail5("durable commit receipt did not compact successfully");assertCommitReceiptMatches(verified.receipt,state.receipt)}function beginManagedStartupSharedStateTransaction(profile,inputOptions={}){const options=resolveOptions(inputOptions);requireTransactionIdentity(options);if(options.readOnlyReceipt){fail5("cannot begin a transaction from a read-only rollback receipt")}requireTransactionBoundaries(options);const profileFingerprint=fingerprintManagedStartupProfile(profile);const committed=loadCommitReceipt(options);if(committed){if(options.bootstrapIdentity===null){fail5("a durable managed bootstrap commit receipt already exists")}assertCommitReceiptMatches(committed.receipt,{agent:profile.agent,profileFingerprint,bootstrapIdentity:options.bootstrapIdentity});fail5("this managed bootstrap attempt is already durably committed")}const pending=loadManifest(options);if(pending){if(pending.agent!==profile.agent||pending.profileFingerprint!==profileFingerprint||pending.bootstrapIdentity!==options.bootstrapIdentity){fail5("a pending managed startup transaction belongs to a different agent, profile fingerprint, or bootstrap attempt")}verifyAllBackups(pending.files,options);return false}const targets=managedOutputTargets(profile,options);if(targets.files.length>MAX_TRANSACTION_FILES){fail5("managed startup transaction has too many file targets")}const snapshots=targets.files.map((target2,index)=>snapshotFile(target2,index,profile.agent,options));const totalBytes=snapshots.reduce((sum,snapshot)=>sum+(snapshot.bytes?.length??0),0);if(totalBytes>MAX_TRANSACTION_TOTAL_BYTES){fail5("managed startup transaction backup exceeds the total size limit")}const directories=targets.directories.map(target2=>snapshotDirectory(target2,profile.agent,options));const manifest={schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:profile.agent,profileFingerprint,bootstrapIdentity:options.bootstrapIdentity,files:snapshots.map(({receipt})=>receipt),directories};let createdTransactionIdentity;try{import_node_fs2.default.mkdirSync(options.transactionDirectory,{mode:TRANSACTION_DIRECTORY_MODE});const created=import_node_fs2.default.lstatSync(options.transactionDirectory,{bigint:true});if(!created.isDirectory()||created.isSymbolicLink()){fail5("new transaction path is not a directory")}createdTransactionIdentity={dev:created.dev,ino:created.ino,uid:created.uid,gid:created.gid};import_node_fs2.default.chownSync(options.transactionDirectory,options.trustedUid,options.trustedGid);import_node_fs2.default.chmodSync(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE);fsyncDirectory(options.transactionParentDirectory);import_node_fs2.default.mkdirSync(options.backupDirectory,{mode:TRANSACTION_DIRECTORY_MODE});import_node_fs2.default.chownSync(options.backupDirectory,options.trustedUid,options.trustedGid);import_node_fs2.default.chmodSync(options.backupDirectory,TRANSACTION_DIRECTORY_MODE);fsyncDirectory(options.transactionDirectory);for(const snapshot of snapshots){if(snapshot.receipt.state!=="file"||snapshot.bytes===null)continue;atomicWriteTrustedFile(import_node_path2.default.join(options.backupDirectory,snapshot.receipt.backup),snapshot.bytes,TRANSACTION_FILE_MODE,options.trustedUid,options.trustedGid)}fsyncDirectory(options.backupDirectory);atomicWriteTrustedFile(options.manifestFile,canonicalManifest(manifest),TRANSACTION_FILE_MODE,options.trustedUid,options.trustedGid);fsyncDirectory(options.transactionDirectory);loadManifest(options)}catch(error){try{if(createdTransactionIdentity&&pathExistsNoFollow(options.transactionDirectory)){const current=import_node_fs2.default.lstatSync(options.transactionDirectory,{bigint:true});if(!current.isSymbolicLink()&¤t.isDirectory()&¤t.dev===createdTransactionIdentity.dev&¤t.ino===createdTransactionIdentity.ino&¤t.uid===createdTransactionIdentity.uid&¤t.gid===createdTransactionIdentity.gid){import_node_fs2.default.chmodSync(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE);import_node_fs2.default.chownSync(options.transactionDirectory,options.trustedUid,options.trustedGid)}requireTrustedTransactionPath(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE,options);import_node_fs2.default.rmSync(options.transactionDirectory,{force:true,recursive:true})}}catch{}throw error}return true}function ensureOriginalDirectories(receipts,expectedAgent,options){for(const receipt of receipts){if(receipt.state!=="directory")continue;const target2=absoluteTarget(receipt.path,options);validateExistingAncestors(import_node_path2.default.join(target2,".restore"),expectedAgent,options);let stat=null;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code!=="ENOENT"){fail5(`could not inspect restore directory ${target2}`)}}if(stat&&(stat.isSymbolicLink()||!stat.isDirectory())){fail5(`restore directory is unsafe: ${target2}`)}if(stat&&directoryMatchesReceipt(target2,receipt))continue;if(!stat)import_node_fs2.default.mkdirSync(target2,{mode:receipt.mode});import_node_fs2.default.chownSync(target2,receipt.uid,receipt.gid);import_node_fs2.default.chmodSync(target2,receipt.mode)}}function restoreFiles(receipts,backups,expectedAgent,options){for(const receipt of receipts){const target2=absoluteTarget(receipt.path,options);validateExistingAncestors(target2,expectedAgent,options);if(receipt.state==="absent"){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")continue;fail5(`could not inspect new managed output ${target2}`)}if(stat.isDirectory()){fail5(`new managed output unexpectedly became a directory: ${target2}`)}import_node_fs2.default.unlinkSync(target2);continue}if(fileMatchesReceipt(target2,receipt))continue;const bytes=backups.get(receipt.path);if(!bytes)fail5(`verified transaction backup is missing: ${receipt.path}`);let current=null;try{current=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code!=="ENOENT"){fail5(`could not inspect managed output before restore: ${target2}`)}}if(current?.isDirectory()){fail5(`managed output unexpectedly became a directory: ${target2}`)}atomicWriteTrustedFile(target2,bytes,receipt.mode,receipt.uid,receipt.gid)}}function restoreDirectoryMetadata(receipts,options){for(const receipt of[...receipts].reverse()){const target2=absoluteTarget(receipt.path,options);if(receipt.state==="absent"){try{import_node_fs2.default.rmdirSync(target2)}catch(error){if(error.code==="ENOENT")continue;fail5(`could not remove newly created managed directory ${target2}`)}continue}if(directoryMatchesReceipt(target2,receipt))continue;const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`managed directory changed type during restore: ${target2}`)}import_node_fs2.default.chownSync(target2,receipt.uid,receipt.gid);import_node_fs2.default.chmodSync(target2,receipt.mode)}}function verifyRestoration(manifest,options){for(const receipt of manifest.files){const target2=absoluteTarget(receipt.path,options);if(receipt.state==="absent"){if(pathExistsNoFollow(target2)){fail5(`new managed output remained after rollback: ${target2}`)}continue}const stable=readStableFile(target2,MAX_TRANSACTION_FILE_BYTES);if(stable.bytes.length!==receipt.size||(0,import_node_crypto6.createHash)("sha256").update(stable.bytes).digest("hex")!==receipt.sha256||Number(stable.stat.uid)!==receipt.uid||Number(stable.stat.gid)!==receipt.gid||Number(stable.stat.mode&0o7777n)!==receipt.mode){fail5(`managed output was not restored exactly: ${target2}`)}}for(const receipt of manifest.directories){const target2=absoluteTarget(receipt.path,options);if(receipt.state==="absent"){if(pathExistsNoFollow(target2)){fail5(`new managed directory remained after rollback: ${target2}`)}continue}const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isDirectory()||stat.uid!==receipt.uid||stat.gid!==receipt.gid||modeOf2(stat)!==receipt.mode){fail5(`managed directory metadata was not restored exactly: ${target2}`)}}}function rollbackManagedStartupSharedStateTransaction(expectedAgent,inputOptions={}){const options=resolveOptions(inputOptions);requireTransactionIdentity(options);const committed=loadCommitReceipt(options);if(committed){if(options.bootstrapIdentity===null){fail5("shared state is already durably committed")}assertCommitReceiptMatches(committed.receipt,{agent:expectedAgent,bootstrapIdentity:options.bootstrapIdentity});fail5("shared state is already durably committed and cannot be rolled back")}const manifest=loadManifest(options);if(!manifest)return false;if(manifest.agent!==expectedAgent){fail5(`pending transaction targets ${manifest.agent}, expected ${expectedAgent}`)}if(manifest.bootstrapIdentity!==options.bootstrapIdentity){fail5("pending transaction belongs to a different bootstrap attempt")}const backups=verifyAllBackups(manifest.files,options);ensureOriginalDirectories(manifest.directories,expectedAgent,options);restoreFiles(manifest.files,backups,expectedAgent,options);restoreDirectoryMetadata(manifest.directories,options);verifyRestoration(manifest,options);if(!options.readOnlyReceipt){removeTransactionDirectory(options)}return true}function commitManagedStartupSharedStateTransaction(expectedAgent,inputOptions={}){const options=resolveOptions(inputOptions);requireTransactionIdentity(options);if(options.readOnlyReceipt){fail5("cannot commit a read-only rollback receipt")}const committed=loadCommitReceipt(options);if(committed){if(options.bootstrapIdentity===null){fail5("durable commit receipt is missing its expected bootstrap identity")}assertCommitReceiptMatches(committed.receipt,{agent:expectedAgent,bootstrapIdentity:options.bootstrapIdentity});compactDurableCommitReceipt(committed,options);return true}const manifest=loadManifest(options);if(!manifest)return false;if(manifest.agent!==expectedAgent){fail5(`pending transaction targets ${manifest.agent}, expected ${expectedAgent}`)}if(manifest.bootstrapIdentity!==options.bootstrapIdentity){fail5("pending transaction belongs to a different bootstrap attempt")}if(manifest.bootstrapIdentity===null){removeTransactionDirectory(options);return true}verifyAllBackups(manifest.files,options);if(pathExistsNoFollow(options.commitReceiptDirectory)){fail5("durable commit receipt path appeared before transaction commit")}try{import_node_fs2.default.renameSync(options.transactionDirectory,options.commitReceiptDirectory);fsyncDirectory(options.transactionParentDirectory)}catch(error){fail5(`could not atomically establish durable commit state: ${error.message}`)}const renamed=loadCommitReceipt(options);if(!renamed)fail5("durable commit state disappeared after atomic rename");assertCommitReceiptMatches(renamed.receipt,{agent:expectedAgent,profileFingerprint:manifest.profileFingerprint,bootstrapIdentity:manifest.bootstrapIdentity});compactDurableCommitReceipt(renamed,options);return true}function clearManagedStartupSharedStateCommitReceipt(expectedAgent,inputOptions={}){const options=resolveOptions(inputOptions);requireTransactionIdentity(options);if(options.readOnlyReceipt){fail5("cannot clear a durable commit from a read-only receipt")}if(options.bootstrapIdentity===null){fail5("durable commit cleanup requires its bootstrap identity")}const committed=loadCommitReceipt(options);if(!committed)return false;assertCommitReceiptMatches(committed.receipt,{agent:expectedAgent,bootstrapIdentity:options.bootstrapIdentity});compactDurableCommitReceipt(committed,options);requireTrustedTransactionPath(options.commitReceiptDirectory,TRANSACTION_DIRECTORY_MODE,options);requireTrustedTransactionPath(options.commitReceiptFile,TRANSACTION_FILE_MODE,options);const entries=import_node_fs2.default.readdirSync(options.commitReceiptDirectory);if(entries.length!==1||entries[0]!==MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE){fail5("durable commit receipt directory contains unexpected artifacts")}import_node_fs2.default.rmSync(options.commitReceiptDirectory,{force:false,recursive:true});fsyncDirectory(options.transactionParentDirectory);if(pathExistsNoFollow(options.commitReceiptDirectory)){fail5("durable commit receipt remained after cleanup")}return true}function getManagedStartupSharedStateTransactionStatus(expected,inputOptions={}){const options=resolveOptions({...inputOptions,bootstrapIdentity:expected.bootstrapIdentity});requireTransactionIdentity(options);const manifest=loadManifest(options);if(manifest){if(manifest.agent!==expected.agent||manifest.profileFingerprint!==expected.profileFingerprint||manifest.bootstrapIdentity!==expected.bootstrapIdentity){fail5("pending transaction does not match the expected agent, profile fingerprint, or bootstrap identity")}verifyAllBackups(manifest.files,options);return"pending"}const committed=loadCommitReceipt(options);if(!committed)return"none";assertCommitReceiptMatches(committed.receipt,expected);return"committed"}var MANAGED_STARTUP_PROFILE_ENV="NEMOCLAW_STARTUP_PROFILE_B64";var MANAGED_STARTUP_CA_ENV="NEMOCLAW_CORPORATE_CA_B64";var MANAGED_STARTUP_EXCHANGE_DIRECTORY="/tmp";var MANAGED_STARTUP_RUNTIME_ENV_FILE=`${MANAGED_STARTUP_EXCHANGE_DIRECTORY}/nemoclaw-managed-startup-runtime.env`;var MANAGED_STARTUP_RUNTIME_EXECUTABLE="/usr/local/lib/nemoclaw/managed-startup-image-runtime.cjs";var MANAGED_STARTUP_MERGED_CA_FILE=`${MANAGED_STARTUP_EXCHANGE_DIRECTORY}/nemoclaw-managed-startup-ca-bundle.pem`;var MANAGED_STARTUP_COMPLETION_FILE=`${MANAGED_STARTUP_EXCHANGE_DIRECTORY}/nemoclaw-managed-startup-complete.json`;var MANAGED_STARTUP_CORPORATE_CA_FILE="/usr/local/share/nemoclaw/corporate-ca.pem";var MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY="/usr/local/share/ca-certificates";var MANAGED_STARTUP_SYSTEM_CA_ANCHOR_RE=/^nemoclaw-corporate-ca-[0-9]{2}\.crt$/u;var SYSTEM_CA_BUNDLE_FILE="/etc/ssl/certs/ca-certificates.crt";var UPDATE_CA_CERTIFICATES_EXECUTABLE="/usr/sbin/update-ca-certificates";var MANAGED_STARTUP_TLS_ENV_NAMES=new Set(["CURL_CA_BUNDLE","GIT_SSL_CAINFO","NODE_EXTRA_CA_CERTS","REQUESTS_CA_BUNDLE","SSL_CERT_FILE"]);var MESSAGING_RUNTIME_PLAN_FILE="/usr/local/share/nemoclaw/messaging-runtime-plan.json";var ROOT_STATE_PARENT="/var/lib/nemoclaw";var ROOT_RUNTIME_DIRECTORY="/run/nemoclaw";var ROOT_OWNED_DIRECTORY_MODE=493;var MAX_TRUST_BUNDLE_BYTES=4*1024*1024;var HERMES_MANAGED_CONFIG_FILES=["/sandbox/.hermes/config.yaml","/sandbox/.hermes/.env"];var HERMES_GENERATED_MANAGED_POLICY_FILE="/sandbox/.hermes/managed-policy.json";var HERMES_INSTALLED_MANAGED_POLICY_FILE="/usr/local/share/nemoclaw/hermes-managed-policy.json";var MAX_HERMES_MANAGED_POLICY_BYTES=4*1024*1024;var MAX_HERMES_SWITCHYARD_CONFIG_BYTES=128*1024;var MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES=4*Math.ceil(MAX_HERMES_MANAGED_POLICY_BYTES/3)+4096;var FIXED_PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin";var SHA256_RE4=/^[a-f0-9]{64}$/u;var MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION=1;var MAX_MANAGED_STARTUP_COMPLETION_BYTES=4096;var MAX_MANAGED_STARTUP_RUNTIME_ENVIRONMENT_BYTES=512*1024;var defaultWorkspaceRootApplyDeps={lstat:target2=>import_node_fs3.default.lstatSync(target2),chown:(target2,uid,gid)=>import_node_fs3.default.chownSync(target2,uid,gid),chmod:(target2,mode)=>import_node_fs3.default.chmodSync(target2,mode)};function normalizeManagedStartupWorkspaceRoot(agent,deps=defaultWorkspaceRootApplyDeps){const target2=managedImageRuntimeIdentity(agent).workdir;const before=deps.lstat(target2);if(before.isSymbolicLink()||!before.isDirectory()){fail6("managed startup workspace root must be one real directory")}const expected=managedStartupWorkspaceRoot({agent,agentIdentity:managedImageRuntimeIdentity(agent)});try{deps.chown(target2,expected.uid,expected.gid);deps.chmod(target2,expected.mode)}catch{fail6("managed startup workspace-root normalization failed")}const after=deps.lstat(target2);if(after.isSymbolicLink()||!after.isDirectory()||after.uid!==expected.uid||after.gid!==expected.gid||(after.mode&4095)!==expected.mode){fail6("managed startup workspace root does not match its declared posture")}}var ManagedStartupImageActionPlanError=class extends Error{constructor(message){super(`Cannot build managed startup image action plan: ${message}`);this.name="ManagedStartupImageActionPlanError"}};var ManagedStartupImageRuntimeError=class extends Error{constructor(message){super(`Managed startup image application failed: ${message}`);this.name="ManagedStartupImageRuntimeError"}};function failActionPlan(message){throw new ManagedStartupImageActionPlanError(message)}function exactActionPlanAgent(value){if(MANAGED_STARTUP_AGENTS.includes(value)){return value}return failActionPlan(`unsupported agent ${JSON.stringify(value)}`)}function fail6(message){throw new ManagedStartupImageRuntimeError(message)}function validateManagedStartupApplicationRuntimePlan(plan){if(typeof plan!=="object"||plan===null){return fail6("application runtime plan must be an object")}const exportEnvironment=plan.exportEnvironment;const unsetEnvironment=plan.unsetEnvironment;if(typeof exportEnvironment!=="object"||exportEnvironment===null||Array.isArray(exportEnvironment)||!Array.isArray(unsetEnvironment)){return fail6("application runtime plan must contain exports and unsets")}const exports2={};for(const[name,value]of Object.entries(exportEnvironment)){if(!/^[A-Za-z_][A-Za-z0-9_]*$/u.test(name)){return fail6(`invalid application runtime environment key ${JSON.stringify(name)}`)}if(typeof value!=="string"||value.includes("\0")||/[\r\n]/u.test(value)){return fail6(`application runtime environment value for ${name} must be single-line text`)}exports2[name]=value}const unsets=new Set;for(const name of unsetEnvironment){if(typeof name!=="string"||!/^[A-Za-z_][A-Za-z0-9_]*$/u.test(name)){return fail6(`invalid application runtime unset ${JSON.stringify(name)}`)}if(unsets.has(name)){return fail6(`duplicate application runtime unset ${name}`)}if(Object.hasOwn(exports2,name)){return fail6(`application runtime cannot both export and unset ${name}`)}unsets.add(name)}return Object.freeze({exportEnvironment:Object.freeze(Object.fromEntries(Object.entries(exports2).sort(([left],[right])=>left.localeCompare(right)))),unsetEnvironment:Object.freeze([...unsets].sort())})}function applyManagedStartupCommandEnvironmentPlan(environment,plan){const validated=validateManagedStartupApplicationRuntimePlan(plan);const applied={...environment};for(const name of[...Object.keys(validated.exportEnvironment),...validated.unsetEnvironment]){delete applied[name]}return applied}function exactAgent2(value){if(MANAGED_STARTUP_AGENTS.includes(value)){return value}return fail6(`unsupported agent ${JSON.stringify(value)}`)}function managedTransactionProfile(expectedAgentInput,env=process.env){requireRoot();const expectedAgent=exactAgent2(expectedAgentInput);if(env.NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION!=="1"){fail6("shared-state transactions require a complete managed image")}const encodedProfile=env[MANAGED_STARTUP_PROFILE_ENV];if(!encodedProfile)fail6(`${MANAGED_STARTUP_PROFILE_ENV} is required`);const profile=decodeManagedStartupProfile(encodedProfile);if(profile.agent!==expectedAgent){fail6(`shared-state transaction profile targets ${profile.agent}, expected ${expectedAgent}`)}return profile}function requireRoot(){if(process.geteuid?.()!==0){fail6("managed startup requires container effective uid 0")}}function modeOf3(stat){return stat.mode&511}function requireManagedStartupExchangeDirectory(exchangeDirectory=MANAGED_STARTUP_EXCHANGE_DIRECTORY){const stat=import_node_fs3.default.lstatSync(exchangeDirectory);if(stat.isSymbolicLink()||!stat.isDirectory()||stat.uid!==0||stat.gid!==0||(stat.mode&4095)!==1023){fail6("managed startup exchange directory must be root:root mode 1777")}}function requireRootOwnedDirectory(target2,mode){let stat;try{stat=import_node_fs3.default.lstatSync(target2)}catch{fail6(`required root-owned directory is missing: ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()||stat.uid!==0||stat.gid!==0||modeOf3(stat)!==mode){fail6(`${target2} must be a root:root directory with mode ${mode.toString(8)}`)}}function ensureRootOwnedDirectory(target2,mode=ROOT_OWNED_DIRECTORY_MODE){const parent=import_node_path3.default.dirname(target2);const parentStat=import_node_fs3.default.lstatSync(parent);if(parentStat.isSymbolicLink()||!parentStat.isDirectory()||parentStat.uid!==0||parentStat.gid!==0||(modeOf3(parentStat)&18)!==0){fail6(`refusing unsafe parent directory for ${target2}`)}try{import_node_fs3.default.mkdirSync(target2,{mode});import_node_fs3.default.chownSync(target2,0,0);import_node_fs3.default.chmodSync(target2,mode)}catch(error){if(error.code!=="EEXIST"){fail6(`could not create ${target2}`)}}requireRootOwnedDirectory(target2,mode)}function requireSafeExistingRootTarget(target2){let stat;try{stat=import_node_fs3.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return;fail6(`could not inspect ${target2}`)}if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==0||stat.gid!==0){fail6(`refusing to replace unsafe root-owned file ${target2}`)}}function atomicWriteRootFile(target2,contents,mode){const parent=import_node_path3.default.dirname(target2);const parentStat=import_node_fs3.default.lstatSync(parent);const trustedStickyExchange=parent===MANAGED_STARTUP_EXCHANGE_DIRECTORY&&(parentStat.mode&4095)===1023;if(parentStat.isSymbolicLink()||!parentStat.isDirectory()||parentStat.uid!==0||parentStat.gid!==0||(modeOf3(parentStat)&18)!==0&&!trustedStickyExchange){fail6(`refusing unsafe root-owned file parent ${parent}`)}requireSafeExistingRootTarget(target2);const temporary=import_node_path3.default.join(parent,`.${import_node_path3.default.basename(target2)}.${(0,import_node_crypto7.randomBytes)(12).toString("hex")}`);let descriptor;try{descriptor=import_node_fs3.default.openSync(temporary,import_node_fs3.default.constants.O_CREAT|import_node_fs3.default.constants.O_EXCL|import_node_fs3.default.constants.O_WRONLY|import_node_fs3.default.constants.O_NOFOLLOW,384);import_node_fs3.default.fchownSync(descriptor,0,0);import_node_fs3.default.writeFileSync(descriptor,contents);import_node_fs3.default.fchmodSync(descriptor,mode);import_node_fs3.default.fsyncSync(descriptor);import_node_fs3.default.closeSync(descriptor);descriptor=void 0;import_node_fs3.default.renameSync(temporary,target2)}catch(error){if(descriptor!==void 0)import_node_fs3.default.closeSync(descriptor);try{import_node_fs3.default.unlinkSync(temporary)}catch{}fail6(`could not atomically write ${target2}: ${error.message}`)}const stat=import_node_fs3.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==0||stat.gid!==0||modeOf3(stat)!==mode){fail6(`root-owned output failed metadata verification: ${target2}`)}}function removeSafeRootFile(target2){requireSafeExistingRootTarget(target2);try{import_node_fs3.default.unlinkSync(target2)}catch(error){if(error.code!=="ENOENT"){fail6(`could not remove ${target2}`)}}}function trustedExecutable(target2){try{const stat=import_node_fs3.default.lstatSync(target2);return!stat.isSymbolicLink()&&stat.isFile()&&stat.uid===0&&stat.gid===0&&(modeOf3(stat)&18)===0&&(modeOf3(stat)&73)!==0}catch{return false}}function readSandboxIdentity(){const readId=flag=>{const result=(0,import_node_child_process.spawnSync)("/usr/bin/id",[flag,"sandbox"],{encoding:"utf8",env:{PATH:FIXED_PATH}});const value=result.stdout.trim();if(result.status!==0||!/^[1-9][0-9]*$/u.test(value)){fail6("could not resolve the sandbox account")}return value};return{uid:readId("-u"),gid:readId("-g")}}function managedStartupSandboxPrefix(){if(trustedExecutable("/usr/bin/setpriv")){const identity=readSandboxIdentity();return["/usr/bin/setpriv",`--reuid=${identity.uid}`,`--regid=${identity.gid}`,"--init-groups","--"]}return fail6("a trusted setpriv executable is required")}function commandEnvironment(configurationEnvironment,applicationRuntime){const env=applyManagedStartupCommandEnvironmentPlan({...process.env,...configurationEnvironment,HOME:"/sandbox",PATH:FIXED_PATH,NPM_CONFIG_OFFLINE:"true",npm_config_offline:"true",PIP_DISABLE_PIP_VERSION_CHECK:"1",PIP_NO_INDEX:"1",UV_OFFLINE:"1"},applicationRuntime);delete env[MANAGED_STARTUP_PROFILE_ENV];delete env[MANAGED_STARTUP_CA_ENV];return env}function execute(argv,runAs,configurationEnvironment,applicationRuntime,capture=false){if(argv.length===0)fail6("refusing an empty managed startup command");const command=runAs==="sandbox"?[...managedStartupSandboxPrefix(),...argv]:[...argv];const result=(0,import_node_child_process.spawnSync)(command[0],command.slice(1),{encoding:"utf8",env:commandEnvironment(configurationEnvironment,applicationRuntime),stdio:capture?"pipe":"inherit"});if(result.error){fail6(`could not execute ${argv[0]}: ${result.error.message}`)}if(result.status!==0){const detail=capture?`: ${(result.stderr||result.stdout).trim()}`:"";fail6(`${argv[0]} exited with status ${String(result.status??"unknown")}${detail}`)}return{status:result.status,stdout:result.stdout??"",stderr:result.stderr??""}}function generatorCommand(agent){switch(agent){case"openclaw":return["/usr/local/bin/node","/scripts/generate-openclaw-config.mts"];case"hermes":return["/usr/local/bin/node","/opt/nemoclaw-hermes-config/generate-config.ts"];case"langchain-deepagents-code":return["/usr/local/bin/node","/opt/nemoclaw-deepagents-code/generate-config.ts"];case"pi":return["/usr/local/bin/node","/opt/nemoclaw-pi/generate-config.ts"]}}function messagingCommand(agent,phase,mode){return["/usr/local/bin/node","/src/lib/messaging/applier/build/messaging-build-applier.mts","--agent",agent,"--phase",phase,"--mode",mode,...phase==="post-agent-install"?["--managed-startup-runtime"]:[]]}function assertActionAgent(inputAgent,actionAgent){if(inputAgent!==actionAgent){failActionPlan(`action for ${actionAgent} cannot be used by ${inputAgent}`)}}function buildManagedStartupImageActionPlan(input){const inputAgent=exactActionPlanAgent(input.agent);const commands=[];let dashboardActions=0;let generateActions=0;let runtimeMessagingActions=0;let postMessagingActions=0;for(const action of input.actions){switch(action.kind){case"configure-dashboard":{if(action.dashboard.agent!==input.agent){failActionPlan(`dashboard for ${action.dashboard.agent} cannot be used by ${input.agent}`)}dashboardActions+=1;break}case"generate-agent-config":{assertActionAgent(inputAgent,exactActionPlanAgent(action.agent));if(action.runAs!=="sandbox"){failActionPlan("agent configuration generation must run as sandbox")}generateActions+=1;commands.push({action:"generate-agent-config",runAs:action.runAs,argv:generatorCommand(action.agent)});break}case"apply-messaging-plan":{assertActionAgent(inputAgent,exactActionPlanAgent(action.agent));if(action.mode!=="apply"&&action.mode!=="clear"){failActionPlan("messaging intent must be apply or clear")}if(action.phase==="runtime-setup"){if(action.runAs!=="root"){failActionPlan("messaging runtime setup must run as root")}runtimeMessagingActions+=1;commands.push({action:"messaging-runtime-setup",runAs:action.runAs,argv:messagingCommand(action.agent,action.phase,action.mode)})}else if(action.phase==="post-agent-install"){if(action.runAs!=="sandbox"){failActionPlan("messaging post-agent configuration must run as sandbox")}postMessagingActions+=1;commands.push({action:"messaging-post-agent-install",runAs:action.runAs,argv:messagingCommand(action.agent,action.phase,action.mode)})}else{failActionPlan("unsupported messaging construction phase")}break}default:failActionPlan("unsupported managed startup construction action")}}if(dashboardActions!==1){failActionPlan("exactly one dashboard construction action is required")}if(generateActions!==1){failActionPlan("exactly one agent config construction action is required")}const supportsMessaging=MANAGED_STARTUP_MESSAGING_AGENTS.includes(inputAgent);const expectedMessagingActions=supportsMessaging?1:0;if(runtimeMessagingActions!==expectedMessagingActions||postMessagingActions!==expectedMessagingActions){failActionPlan(`${inputAgent} requires ${String(expectedMessagingActions)} action for each messaging phase`)}const expectedOrder=supportsMessaging?["messaging-runtime-setup","generate-agent-config","messaging-post-agent-install"]:["generate-agent-config"];if(commands.some((command,index)=>command.action!==expectedOrder[index])){failActionPlan(`${inputAgent} image actions are not in the required construction order`)}return Object.freeze(commands.map(command=>Object.freeze({...command,argv:Object.freeze([...command.argv])})))}function prepareMessagingRuntimeTarget(mode){if(mode==="clear"){removeSafeRootFile(MESSAGING_RUNTIME_PLAN_FILE);return}requireSafeExistingRootTarget(MESSAGING_RUNTIME_PLAN_FILE);try{import_node_fs3.default.unlinkSync(MESSAGING_RUNTIME_PLAN_FILE)}catch(error){if(error.code!=="ENOENT"){fail6("could not prepare the messaging runtime-plan target")}}}function verifyMessagingRuntimeTarget(mode){if(mode==="clear"){if(import_node_fs3.default.existsSync(MESSAGING_RUNTIME_PLAN_FILE)){fail6("clear messaging profile left a runtime-plan artifact")}return}const stat=import_node_fs3.default.lstatSync(MESSAGING_RUNTIME_PLAN_FILE);if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==0||stat.gid!==0||modeOf3(stat)!==420){fail6("messaging runtime-plan artifact failed root ownership validation")}}function runInternalSandboxAction(action,configurationEnvironment,applicationRuntime,extraEnvironment={}){execute(["/usr/local/bin/node",MANAGED_STARTUP_RUNTIME_EXECUTABLE,`--internal-${action}`],"sandbox",{...configurationEnvironment,...extraEnvironment},applicationRuntime)}function validateOpenClawConfiguration(configurationEnvironment,applicationRuntime){const validation=execute(["/usr/local/bin/openclaw","config","validate","--json"],"sandbox",{...configurationEnvironment,OPENCLAW_CONFIG_PATH:"/sandbox/.openclaw/openclaw.json"},applicationRuntime,true);let parsed;try{parsed=JSON.parse(validation.stdout)}catch{fail6("OpenClaw config validation did not emit JSON")}if(typeof parsed!=="object"||parsed===null||parsed.valid!==true){fail6("OpenClaw rejected the generated managed startup config")}}function sameStableFileMetadata(left,right){return left.dev===right.dev&&left.ino===right.ino&&left.mode===right.mode&&left.nlink===right.nlink&&left.uid===right.uid&&left.gid===right.gid&&left.size===right.size&&left.mtimeNs===right.mtimeNs&&left.ctimeNs===right.ctimeNs}function readStableRegularFileSnapshot(target2,maxBytes){if(typeof import_node_fs3.default.constants.O_NOFOLLOW!=="number"){fail6("O_NOFOLLOW is unavailable for managed startup file reads")}const nonblock=typeof import_node_fs3.default.constants.O_NONBLOCK==="number"?import_node_fs3.default.constants.O_NONBLOCK:0;let descriptor;try{descriptor=import_node_fs3.default.openSync(target2,import_node_fs3.default.constants.O_RDONLY|import_node_fs3.default.constants.O_NOFOLLOW|nonblock)}catch(error){if(error.code==="ENOENT")throw error;fail6(`refusing unsafe or unreadable file ${target2}`)}try{const before=import_node_fs3.default.fstatSync(descriptor,{bigint:true});if(!before.isFile()||before.nlink!==1n||before.size<1n||before.size>BigInt(maxBytes)){fail6(`refusing unsafe or oversized file ${target2}`)}const bytes=Buffer.alloc(Number(before.size));let offset=0;while(offset{const identity=readSandboxIdentity();return{uid:Number(identity.uid),gid:Number(identity.gid)}})();const relaySource=expectedSandboxOwnedRoutingSource(paths.relayPluginsSource,expectedRelay,sandboxIdentity);const relayStateSource=expectedSandboxOwnedRoutingSource(paths.relayPluginStateSource,expectedRelayState,sandboxIdentity);const deploymentSource=expectedSandboxOwnedRoutingSource(paths.switchyardDeploymentSource,expectedDeployment,sandboxIdentity);atomicWriteRootFile(paths.switchyardDeploymentTarget,expectedDeployment,292);atomicWriteRootFile(paths.relayPluginStateTarget,expectedRelayState,292);atomicWriteRootFile(paths.relayPluginsTarget,expectedRelay,292);verifyCommittedRoutingFile(paths.switchyardDeploymentTarget,expectedDeployment);verifyCommittedRoutingFile(paths.relayPluginStateTarget,expectedRelayState);verifyCommittedRoutingFile(paths.relayPluginsTarget,expectedRelay);removeStableRoutingSource(paths.switchyardDeploymentSource,deploymentSource);removeStableRoutingSource(paths.relayPluginStateSource,relayStateSource);removeStableRoutingSource(paths.relayPluginsSource,relaySource)}function verifyHermesSwitchyardRoutingFiles(configurationEnvironment,paths=HERMES_SWITCHYARD_ROUTING_PATHS){const routing=readHermesSwitchyardRoutingEnvironment(configurationEnvironment);if(!routing){assertRoutingFileAbsent(paths.relayPluginsTarget);assertRoutingFileAbsent(paths.relayPluginStateTarget);assertRoutingFileAbsent(paths.switchyardDeploymentTarget);return}const serialized=serializeHermesSwitchyardRouting(routing);verifyCommittedRoutingFile(paths.relayPluginsTarget,Buffer.from(serialized.relayPluginsToml,"utf8"));verifyCommittedRoutingFile(paths.relayPluginStateTarget,Buffer.from(serialized.relayPluginStateJson,"utf8"));verifyCommittedRoutingFile(paths.switchyardDeploymentTarget,Buffer.from(serialized.switchyardDeploymentToml,"utf8"))}function normalizeHermesManagedConfigDescriptor(target2,sandboxIdentity){if(!Number.isSafeInteger(sandboxIdentity.uid)||sandboxIdentity.uid<=0||!Number.isSafeInteger(sandboxIdentity.gid)||sandboxIdentity.gid<=0){fail6("invalid sandbox identity for Hermes descriptor normalization")}if(typeof import_node_fs3.default.constants.O_NOFOLLOW!=="number"){fail6("O_NOFOLLOW is unavailable for Hermes descriptor normalization")}const nonblock=typeof import_node_fs3.default.constants.O_NONBLOCK==="number"?import_node_fs3.default.constants.O_NONBLOCK:0;let descriptor;try{descriptor=import_node_fs3.default.openSync(target2,import_node_fs3.default.constants.O_RDONLY|import_node_fs3.default.constants.O_NOFOLLOW|nonblock)}catch{fail6(`refusing unsafe Hermes managed config descriptor ${target2}`)}try{const before=import_node_fs3.default.fstatSync(descriptor,{bigint:true});const beforeMode=Number(before.mode&0o777n);const mutable=before.uid===BigInt(sandboxIdentity.uid)&&before.gid===BigInt(sandboxIdentity.gid)&&(beforeMode===384||beforeMode===416);if(!before.isFile()||before.nlink!==1n||!mutable){fail6(`refusing unexpected Hermes managed config descriptor ${target2}`)}const expectedMode=416;if(beforeMode===384){try{import_node_fs3.default.fchmodSync(descriptor,expectedMode)}catch{fail6(`could not normalize Hermes managed config descriptor ${target2}`)}}const after=import_node_fs3.default.fstatSync(descriptor,{bigint:true});let pathAfter;try{pathAfter=import_node_fs3.default.lstatSync(target2,{bigint:true})}catch{fail6(`Hermes managed config descriptor disappeared during normalization: ${target2}`)}const expectedUid=BigInt(sandboxIdentity.uid);const expectedGid=BigInt(sandboxIdentity.gid);if(!after.isFile()||after.nlink!==1n||after.dev!==before.dev||after.ino!==before.ino||after.uid!==expectedUid||after.gid!==expectedGid||Number(after.mode&0o777n)!==expectedMode||after.size!==before.size||after.mtimeNs!==before.mtimeNs||pathAfter.isSymbolicLink()||!pathAfter.isFile()||!sameStableFileMetadata(after,pathAfter)){fail6(`Hermes managed config descriptor changed during normalization: ${target2}`)}}finally{try{import_node_fs3.default.closeSync(descriptor)}catch{fail6(`could not close Hermes managed config descriptor ${target2}`)}}}function normalizeHermesManagedConfiguration(){const identity=readSandboxIdentity();const sandboxIdentity={uid:Number(identity.uid),gid:Number(identity.gid)};for(const target2 of HERMES_MANAGED_CONFIG_FILES){normalizeHermesManagedConfigDescriptor(target2,sandboxIdentity)}}function sealHermesConfiguration(configurationEnvironment,applicationRuntime){const configPath="/sandbox/.hermes/config.yaml";const envPath="/sandbox/.hermes/.env";const config=readStableRegularFile(configPath,4*1024*1024);const env=readStableRegularFile(envPath,512*1024);const hashText=[`${(0,import_node_crypto7.createHash)("sha256").update(config).digest("hex")} ${configPath}`,`${(0,import_node_crypto7.createHash)("sha256").update(env).digest("hex")} ${envPath}`,""].join("\n");atomicWriteRootFile("/etc/nemoclaw/hermes.config-hash",hashText,292);runInternalSandboxAction("write-hermes-compat-hash",configurationEnvironment,applicationRuntime,{NEMOCLAW_MANAGED_HERMES_HASH_B64:Buffer.from(hashText,"utf8").toString("base64")})}function installRootOwnedMaterials(materials){for(const material of materials){if(material.kind!=="root-owned-file")continue;if(material.owner!=="root"||material.group!=="root"||material.mode!==292){fail6(`unsupported root-owned material contract for ${material.path}`)}atomicWriteRootFile(material.path,material.contents,material.mode)}}function verifyRootOwnedMaterials(materials){for(const material of materials){if(material.kind!=="root-owned-file")continue;const expected=Buffer.from(material.contents,"utf8");const{bytes,stat}=readStableRegularFileSnapshot(material.path,expected.length);if(stat.nlink!==1n||stat.uid!==0n||stat.gid!==0n||Number(stat.mode&0o777n)!==material.mode||!bytes.equals(expected)){fail6(`committed root-owned material drifted: ${material.path}`)}}}function installCorporateCa(corporateCaPath){if(corporateCaPath===null){removeSafeRootFile(MANAGED_STARTUP_CORPORATE_CA_FILE);return}const bytes=readStableRegularFile(corporateCaPath,128*1024);atomicWriteRootFile(MANAGED_STARTUP_CORPORATE_CA_FILE,bytes,292)}function corporateCaCertificateBlocks(corporateCaPath){const corporate=readStableRegularFile(corporateCaPath,128*1024).toString("utf8");const blocks=corporate.match(PEM_CERTIFICATE_RE_GLOBAL);if(!blocks||blocks.length===0){fail6("corporate CA material contains no certificate")}for(const block of blocks){try{if(!new import_node_crypto7.X509Certificate(block).ca){fail6("corporate CA material contains a certificate that is not a CA")}}catch(error){if(error instanceof ManagedStartupImageRuntimeError)throw error;fail6("corporate CA material contains an invalid certificate")}}return blocks.map(block=>`${block.trim()} -`)}function managedSystemCaAnchorNames(){try{import_node_fs3.default.lstatSync(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY)}catch(error){if(error.code==="ENOENT")return[];fail6("could not inspect the managed system CA anchor directory")}requireRootOwnedDirectory(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,ROOT_OWNED_DIRECTORY_MODE);try{return import_node_fs3.default.readdirSync(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY).filter(name=>MANAGED_STARTUP_SYSTEM_CA_ANCHOR_RE.test(name)).sort()}catch(error){fail6("could not inspect the managed system CA anchors")}}function refreshSystemCaBundle(){if(!trustedExecutable(UPDATE_CA_CERTIFICATES_EXECUTABLE)){fail6(`a trusted ${UPDATE_CA_CERTIFICATES_EXECUTABLE} executable is required`)}const result=(0,import_node_child_process.spawnSync)(UPDATE_CA_CERTIFICATES_EXECUTABLE,[],{encoding:"utf8",env:{PATH:FIXED_PATH},stdio:"inherit"});if(result.error){fail6(`could not execute ${UPDATE_CA_CERTIFICATES_EXECUTABLE}: ${result.error.message}`)}if(result.status!==0){fail6(`${UPDATE_CA_CERTIFICATES_EXECUTABLE} exited with status ${String(result.status??"unknown")}`)}}function requireSystemCaBundleContains(blocks){const systemBundle=safeTrustBundle(SYSTEM_CA_BUNDLE_FILE);if(systemBundle===null)fail6("the refreshed system CA bundle is missing");const systemBlocks=systemBundle.toString("utf8").match(PEM_CERTIFICATE_RE_GLOBAL)??[];const systemFingerprints=new Set;for(const block of systemBlocks){try{systemFingerprints.add(new import_node_crypto7.X509Certificate(block).fingerprint256)}catch{fail6("the refreshed system CA bundle contains an invalid certificate")}}for(const block of blocks){if(!systemFingerprints.has(new import_node_crypto7.X509Certificate(block).fingerprint256)){fail6("the refreshed system CA bundle does not contain the corporate CA")}}}function installCorporateCaSystemAnchors(corporateCaPath){const existingNames=managedSystemCaAnchorNames();if(corporateCaPath===null){for(const name of existingNames){removeSafeRootFile(import_node_path3.default.join(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,name))}refreshSystemCaBundle();return}ensureRootOwnedDirectory(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY);const blocks=corporateCaCertificateBlocks(corporateCaPath);const expectedNames=blocks.map((_block,index)=>`nemoclaw-corporate-ca-${String(index+1).padStart(2,"0")}.crt`);for(const name of existingNames){if(!expectedNames.includes(name)){removeSafeRootFile(import_node_path3.default.join(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,name))}}for(const[index,name]of expectedNames.entries()){atomicWriteRootFile(import_node_path3.default.join(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,name),blocks[index],292)}refreshSystemCaBundle();requireSystemCaBundleContains(blocks)}function safeTrustBundle(target2){try{const{bytes,stat}=readStableRegularFileSnapshot(target2,MAX_TRUST_BUNDLE_BYTES);if(Number(stat.mode&0o022n)!==0){fail6(`refusing unsafe trust bundle ${target2}`)}return bytes}catch(error){if(error.code==="ENOENT")return null;throw error}}function mergeCorporateCa(corporateCaPath){if(corporateCaPath===null){removeSafeRootFile(MANAGED_STARTUP_MERGED_CA_FILE);return false}const corporate=readStableRegularFile(corporateCaPath,128*1024);const candidates=["/etc/openshell-tls/ca-bundle.pem",process.env.SSL_CERT_FILE??"","/etc/ssl/certs/ca-certificates.crt"].filter((candidate,index,values)=>candidate&&candidate!==MANAGED_STARTUP_MERGED_CA_FILE&&values.indexOf(candidate)===index);let base=null;for(const candidate of candidates){base=safeTrustBundle(candidate);if(base)break}const merged=Buffer.concat([...base?[base,Buffer.from("\n","utf8")]:[],corporate,...corporate.at(-1)===10?[]:[Buffer.from("\n","utf8")]]);atomicWriteRootFile(MANAGED_STARTUP_MERGED_CA_FILE,merged,292);return true}function shellSingleQuote(value){if(value.includes("\0")||/[\r\n]/u.test(value)){fail6("runtime environment values must be single-line text")}return`'${value.replaceAll("'",`'"'"'`)}'`}function serializeManagedStartupRuntimeEnvironment(environment,corporateCaMerged,configurationEnvironment={},applicationRuntime={exportEnvironment:{},unsetEnvironment:[]}){const{output,unsetNames}=materializeManagedStartupRuntimeEnvironment(environment,corporateCaMerged,configurationEnvironment,applicationRuntime);const unsetLines=unsetNames.map(name=>`unset ${name}`);const exportLines=Object.entries(output).sort(([left],[right])=>left.localeCompare(right)).map(([name,value])=>`export ${name}=${shellSingleQuote(value)}`);return`${[...unsetLines,...exportLines].join("\n")} +`}function requireExactKeys(record2,keys){if(Object.keys(record2).sort().join(",")!==[...keys].sort().join(",")){fail5("transaction manifest contains unexpected fields")}}function parseCommitReceipt(text){let parsed;try{parsed=JSON.parse(text)}catch{fail5("commit receipt is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail5("commit receipt must be an object")}const record2=parsed;requireExactKeys(record2,["agent","bootstrapIdentity","profileFingerprint","schemaVersion"]);if(record2.schemaVersion!==TRANSACTION_SCHEMA_VERSION||!MANAGED_STARTUP_AGENTS.includes(String(record2.agent))||typeof record2.profileFingerprint!=="string"||!/^[a-f0-9]{64}$/u.test(record2.profileFingerprint)||typeof record2.bootstrapIdentity!=="string"||!/^[a-f0-9]{64}$/u.test(record2.bootstrapIdentity)){fail5("commit receipt has an invalid envelope")}const receipt={schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:record2.agent,profileFingerprint:record2.profileFingerprint,bootstrapIdentity:record2.bootstrapIdentity};if(canonicalCommitReceipt(receipt)!==text){fail5("commit receipt is not canonical")}return receipt}function safeMetadata(value){return Number.isSafeInteger(value)&&value>=0}function parseManifest(text){let parsed;try{parsed=JSON.parse(text)}catch{fail5("transaction manifest is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail5("transaction manifest must be an object")}const record2=parsed;const hasBootstrapIdentity=Object.hasOwn(record2,"bootstrapIdentity");requireExactKeys(record2,hasBootstrapIdentity?["agent","bootstrapIdentity","directories","files","profileFingerprint","schemaVersion"]:["agent","directories","files","profileFingerprint","schemaVersion"]);const bootstrapIdentity=hasBootstrapIdentity?record2.bootstrapIdentity:null;if(record2.schemaVersion!==TRANSACTION_SCHEMA_VERSION||!MANAGED_STARTUP_AGENTS.includes(String(record2.agent))||typeof record2.profileFingerprint!=="string"||!/^[a-f0-9]{64}$/u.test(record2.profileFingerprint)||!(bootstrapIdentity===null||typeof bootstrapIdentity==="string"&&/^[a-f0-9]{64}$/u.test(bootstrapIdentity))||!Array.isArray(record2.files)||!Array.isArray(record2.directories)||record2.files.length>MAX_TRANSACTION_FILES||record2.directories.length>MAX_TRANSACTION_FILES*4){fail5("transaction manifest has an invalid envelope")}const files=record2.files.map(value=>{if(typeof value!=="object"||value===null||Array.isArray(value)){return fail5("transaction file receipt must be an object")}const receipt=value;if(typeof receipt.path!=="string"){return fail5("transaction file receipt path must be a string")}const receiptPath=safeRelativePath(receipt.path);if(receipt.state==="absent"){requireExactKeys(receipt,["path","state"]);return{path:receiptPath,state:"absent"}}requireExactKeys(receipt,["backup","gid","mode","path","sha256","size","state","uid"]);if(receipt.state!=="file"||typeof receipt.backup!=="string"||!/^[0-9]{3}\.bin$/u.test(receipt.backup)||typeof receipt.sha256!=="string"||!/^[a-f0-9]{64}$/u.test(receipt.sha256)||!safeMetadata(receipt.size)||receipt.size>MAX_TRANSACTION_FILE_BYTES||!safeMetadata(receipt.uid)||!safeMetadata(receipt.gid)||!safeMetadata(receipt.mode)||receipt.mode>4095){return fail5("transaction file receipt is invalid")}return{path:receiptPath,state:"file",backup:receipt.backup,sha256:receipt.sha256,size:receipt.size,uid:receipt.uid,gid:receipt.gid,mode:receipt.mode}});const directories=record2.directories.map(value=>{if(typeof value!=="object"||value===null||Array.isArray(value)){return fail5("transaction directory receipt must be an object")}const receipt=value;if(typeof receipt.path!=="string"){return fail5("transaction directory receipt path must be a string")}const receiptPath=safeRelativePath(receipt.path);if(receipt.state==="absent"){requireExactKeys(receipt,["path","state"]);return{path:receiptPath,state:"absent"}}requireExactKeys(receipt,["gid","mode","path","state","uid"]);if(receipt.state!=="directory"||!safeMetadata(receipt.uid)||!safeMetadata(receipt.gid)||!safeMetadata(receipt.mode)||receipt.mode>4095){return fail5("transaction directory receipt is invalid")}return{path:receiptPath,state:"directory",uid:receipt.uid,gid:receipt.gid,mode:receipt.mode}});const filePaths=files.map(receipt=>receipt.path);const directoryPaths=directories.map(receipt=>receipt.path);const backupNames=files.filter(receipt=>receipt.state==="file").map(receipt=>receipt.backup);if(new Set(filePaths).size!==filePaths.length||new Set(directoryPaths).size!==directoryPaths.length||new Set(backupNames).size!==backupNames.length){fail5("transaction manifest contains duplicate receipts")}const manifest={schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:record2.agent,profileFingerprint:record2.profileFingerprint,bootstrapIdentity,files,directories};const canonical=hasBootstrapIdentity?canonicalManifest(manifest):canonicalLegacyManifest(manifest);if(canonical!==text){fail5("transaction manifest is not canonical")}return manifest}function requireTrustedTransactionPath(target2,mode,options){const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||(mode===TRANSACTION_DIRECTORY_MODE?!stat.isDirectory():!stat.isFile())||!options.readOnlyReceipt&&(stat.uid!==options.trustedUid||stat.gid!==options.trustedGid)||modeOf2(stat)!==mode){fail5(`transaction artifact has unsafe metadata: ${target2}`)}}function requireReadOnlyReceiptMount(target2,options){if(!options.readOnlyReceipt)return;const probe=import_node_path2.default.join(target2,".nemoclaw-write-probe");let descriptor;try{descriptor=import_node_fs2.default.openSync(probe,import_node_fs2.default.constants.O_CREAT|import_node_fs2.default.constants.O_EXCL|import_node_fs2.default.constants.O_WRONLY|import_node_fs2.default.constants.O_NOFOLLOW,384);import_node_fs2.default.closeSync(descriptor);descriptor=void 0;import_node_fs2.default.unlinkSync(probe)}catch(error){if(descriptor!==void 0)import_node_fs2.default.closeSync(descriptor);if(error.code==="EROFS")return;fail5("copied receipt must be mounted on a read-only filesystem")}fail5("copied receipt mount is writable")}function loadManifest(options){requireTransactionBoundaries(options);if(!pathExistsNoFollow(options.transactionDirectory))return null;requireTrustedTransactionPath(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE,options);requireReadOnlyReceiptMount(options.transactionDirectory,options);requireTrustedTransactionPath(options.backupDirectory,TRANSACTION_DIRECTORY_MODE,options);requireTrustedTransactionPath(options.manifestFile,TRANSACTION_FILE_MODE,options);const stable=readStableFile(options.manifestFile,MAX_MANIFEST_BYTES);if(!options.readOnlyReceipt&&(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid)||Number(stable.stat.mode&0o7777n)!==TRANSACTION_FILE_MODE){fail5("transaction manifest ownership changed while it was read")}return parseManifest(stable.bytes.toString("utf8"))}function transactionOptionsAt(options,transactionDirectory){return{...options,transactionDirectory,backupDirectory:import_node_path2.default.join(transactionDirectory,"backups"),manifestFile:import_node_path2.default.join(transactionDirectory,"manifest.json")}}function loadCommitReceipt(options){requireTransactionBoundaries(options);if(!pathExistsNoFollow(options.commitReceiptDirectory))return null;requireTrustedTransactionPath(options.commitReceiptDirectory,TRANSACTION_DIRECTORY_MODE,options);if(pathExistsNoFollow(options.commitReceiptFile)){requireReadOnlyReceiptMount(options.commitReceiptDirectory,options);requireTrustedTransactionPath(options.commitReceiptFile,TRANSACTION_FILE_MODE,options);const stable=readStableFile(options.commitReceiptFile,MAX_COMMIT_RECEIPT_BYTES);if(!options.readOnlyReceipt&&(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid)||Number(stable.stat.mode&0o7777n)!==TRANSACTION_FILE_MODE){fail5("commit receipt ownership changed while it was read")}return{receipt:parseCommitReceipt(stable.bytes.toString("utf8")),compact:true}}const stagedOptions=transactionOptionsAt(options,options.commitReceiptDirectory);const staged=loadManifest(stagedOptions);if(!staged||staged.bootstrapIdentity===null){fail5("durable commit staging receipt is incomplete")}verifyAllBackups(staged.files,stagedOptions);return{receipt:{schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:staged.agent,profileFingerprint:staged.profileFingerprint,bootstrapIdentity:staged.bootstrapIdentity},compact:false}}function verifyBackup(receipt,options){const backupPath=import_node_path2.default.join(options.backupDirectory,receipt.backup);requireTrustedTransactionPath(backupPath,TRANSACTION_FILE_MODE,options);const stable=readStableFile(backupPath,MAX_TRANSACTION_FILE_BYTES);const digest=(0,import_node_crypto5.createHash)("sha256").update(stable.bytes).digest("hex");if(stable.bytes.length!==receipt.size||digest!==receipt.sha256){fail5(`transaction backup does not match its receipt: ${receipt.path}`)}return stable.bytes}function verifyAllBackups(receipts,options){const backups=new Map;for(const receipt of receipts){if(receipt.state==="file"){backups.set(receipt.path,verifyBackup(receipt,options))}}return backups}function fileMatchesReceipt(target2,receipt){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return false;fail5(`could not inspect managed output ${target2}`)}if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1)return false;const stable=readStableFile(target2,MAX_TRANSACTION_FILE_BYTES);return stable.bytes.length===receipt.size&&(0,import_node_crypto5.createHash)("sha256").update(stable.bytes).digest("hex")===receipt.sha256&&Number(stable.stat.uid)===receipt.uid&&Number(stable.stat.gid)===receipt.gid&&Number(stable.stat.mode&0o7777n)===receipt.mode}function directoryMatchesReceipt(target2,receipt){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return false;fail5(`could not inspect managed output directory ${target2}`)}return!stat.isSymbolicLink()&&stat.isDirectory()&&stat.uid===receipt.uid&&stat.gid===receipt.gid&&modeOf2(stat)===receipt.mode}function removeTransactionDirectory(options){requireTrustedTransactionPath(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE,options);import_node_fs2.default.rmSync(options.transactionDirectory,{force:false,recursive:true});fsyncDirectory(options.transactionParentDirectory);if(pathExistsNoFollow(options.transactionDirectory)){fail5("transaction directory remained after cleanup")}}function assertCommitReceiptMatches(receipt,expected){if(receipt.agent!==expected.agent||expected.profileFingerprint!==void 0&&receipt.profileFingerprint!==expected.profileFingerprint||receipt.bootstrapIdentity!==expected.bootstrapIdentity){fail5("durable commit receipt belongs to a different bootstrap attempt")}}function loadCommitStagingManifest(options){if(!pathExistsNoFollow(options.manifestFile))return null;requireTrustedTransactionPath(options.manifestFile,TRANSACTION_FILE_MODE,options);const stable=readStableFile(options.manifestFile,MAX_MANIFEST_BYTES);if(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid||Number(stable.stat.mode&0o7777n)!==TRANSACTION_FILE_MODE){fail5("durable commit staging manifest ownership changed while it was read")}return parseManifest(stable.bytes.toString("utf8"))}function retireInterruptedCommitReceiptWrites(receipt,options){const temporaryPattern=new RegExp(`^\\.${MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE.replace(".","\\.")}\\.[a-f0-9]{24}$`,"u");for(const entry of import_node_fs2.default.readdirSync(options.commitReceiptDirectory)){if(!temporaryPattern.test(entry))continue;const target2=import_node_path2.default.join(options.commitReceiptDirectory,entry);const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==options.trustedUid||stat.gid!==options.trustedGid||![ATOMIC_TEMPORARY_FILE_MODE,TRANSACTION_FILE_MODE].includes(modeOf2(stat))){fail5("interrupted durable commit receipt write has unsafe metadata")}const stable=readStableFile(target2,MAX_COMMIT_RECEIPT_BYTES);const mode=Number(stable.stat.mode&0o7777n);if(Number(stable.stat.uid)!==options.trustedUid||Number(stable.stat.gid)!==options.trustedGid||![ATOMIC_TEMPORARY_FILE_MODE,TRANSACTION_FILE_MODE].includes(mode)){fail5("interrupted durable commit receipt write changed during verification")}if(stable.bytes.length>0){let interruptedReceipt=null;try{interruptedReceipt=parseCommitReceipt(stable.bytes.toString("utf8"))}catch{}if(interruptedReceipt)assertCommitReceiptMatches(interruptedReceipt,receipt)}import_node_fs2.default.unlinkSync(target2);fsyncDirectory(options.commitReceiptDirectory)}}function compactDurableCommitReceipt(state,options){if(!state.compact){atomicWriteTrustedFile(options.commitReceiptFile,canonicalCommitReceipt(state.receipt),TRANSACTION_FILE_MODE,options.trustedUid,options.trustedGid);fsyncDirectory(options.commitReceiptDirectory)}retireInterruptedCommitReceiptWrites(state.receipt,options);const stagedOptions=transactionOptionsAt(options,options.commitReceiptDirectory);const manifestExists=pathExistsNoFollow(stagedOptions.manifestFile);const backupsExist=pathExistsNoFollow(stagedOptions.backupDirectory);const unexpectedBeforeCleanup=import_node_fs2.default.readdirSync(options.commitReceiptDirectory).filter(entry=>![MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE,import_node_path2.default.basename(stagedOptions.backupDirectory),import_node_path2.default.basename(stagedOptions.manifestFile)].includes(entry));if(unexpectedBeforeCleanup.length!==0){fail5("durable commit receipt directory contains unexpected artifacts")}if(manifestExists){const staged=loadCommitStagingManifest(stagedOptions);if(!staged||staged.bootstrapIdentity===null){fail5("durable commit staging receipt disappeared during cleanup")}assertCommitReceiptMatches(state.receipt,{agent:staged.agent,profileFingerprint:staged.profileFingerprint,bootstrapIdentity:staged.bootstrapIdentity})}if(backupsExist){requireTrustedTransactionPath(stagedOptions.backupDirectory,TRANSACTION_DIRECTORY_MODE,options);import_node_fs2.default.rmSync(stagedOptions.backupDirectory,{force:false,recursive:true});fsyncDirectory(options.commitReceiptDirectory)}if(manifestExists){requireTrustedTransactionPath(stagedOptions.manifestFile,TRANSACTION_FILE_MODE,options);import_node_fs2.default.unlinkSync(stagedOptions.manifestFile);fsyncDirectory(options.commitReceiptDirectory)}const unexpected=import_node_fs2.default.readdirSync(options.commitReceiptDirectory).filter(entry=>entry!==MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE);if(unexpected.length!==0){fail5("durable commit receipt directory contains unexpected artifacts")}const verified=loadCommitReceipt(options);if(!verified?.compact)fail5("durable commit receipt did not compact successfully");assertCommitReceiptMatches(verified.receipt,state.receipt)}function beginManagedStartupSharedStateTransaction(profile,inputOptions={}){const options=resolveOptions(inputOptions);requireTransactionIdentity(options);if(options.readOnlyReceipt){fail5("cannot begin a transaction from a read-only rollback receipt")}requireTransactionBoundaries(options);const profileFingerprint=fingerprintManagedStartupProfile(profile);const committed=loadCommitReceipt(options);if(committed){if(options.bootstrapIdentity===null){fail5("a durable managed bootstrap commit receipt already exists")}assertCommitReceiptMatches(committed.receipt,{agent:profile.agent,profileFingerprint,bootstrapIdentity:options.bootstrapIdentity});fail5("this managed bootstrap attempt is already durably committed")}const pending=loadManifest(options);if(pending){if(pending.agent!==profile.agent||pending.profileFingerprint!==profileFingerprint||pending.bootstrapIdentity!==options.bootstrapIdentity){fail5("a pending managed startup transaction belongs to a different agent, profile fingerprint, or bootstrap attempt")}verifyAllBackups(pending.files,options);return false}const{manifest,snapshots}=inputOptions.capturedState===void 0?captureSharedState(profile,options):parseCapturedSharedState(inputOptions.capturedState,profile,options);let createdTransactionIdentity;try{import_node_fs2.default.mkdirSync(options.transactionDirectory,{mode:TRANSACTION_DIRECTORY_MODE});const created=import_node_fs2.default.lstatSync(options.transactionDirectory,{bigint:true});if(!created.isDirectory()||created.isSymbolicLink()){fail5("new transaction path is not a directory")}createdTransactionIdentity={dev:created.dev,ino:created.ino,uid:created.uid,gid:created.gid};import_node_fs2.default.chownSync(options.transactionDirectory,options.trustedUid,options.trustedGid);import_node_fs2.default.chmodSync(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE);fsyncDirectory(options.transactionParentDirectory);import_node_fs2.default.mkdirSync(options.backupDirectory,{mode:TRANSACTION_DIRECTORY_MODE});import_node_fs2.default.chownSync(options.backupDirectory,options.trustedUid,options.trustedGid);import_node_fs2.default.chmodSync(options.backupDirectory,TRANSACTION_DIRECTORY_MODE);fsyncDirectory(options.transactionDirectory);for(const snapshot of snapshots){if(snapshot.receipt.state!=="file"||snapshot.bytes===null)continue;atomicWriteTrustedFile(import_node_path2.default.join(options.backupDirectory,snapshot.receipt.backup),snapshot.bytes,TRANSACTION_FILE_MODE,options.trustedUid,options.trustedGid)}fsyncDirectory(options.backupDirectory);atomicWriteTrustedFile(options.manifestFile,canonicalManifest(manifest),TRANSACTION_FILE_MODE,options.trustedUid,options.trustedGid);fsyncDirectory(options.transactionDirectory);loadManifest(options)}catch(error){try{if(createdTransactionIdentity&&pathExistsNoFollow(options.transactionDirectory)){const current=import_node_fs2.default.lstatSync(options.transactionDirectory,{bigint:true});if(!current.isSymbolicLink()&¤t.isDirectory()&¤t.dev===createdTransactionIdentity.dev&¤t.ino===createdTransactionIdentity.ino&¤t.uid===createdTransactionIdentity.uid&¤t.gid===createdTransactionIdentity.gid){import_node_fs2.default.chmodSync(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE);import_node_fs2.default.chownSync(options.transactionDirectory,options.trustedUid,options.trustedGid)}requireTrustedTransactionPath(options.transactionDirectory,TRANSACTION_DIRECTORY_MODE,options);import_node_fs2.default.rmSync(options.transactionDirectory,{force:true,recursive:true})}}catch{}throw error}return true}function captureSharedState(profile,options){const targets=managedOutputTargets(profile,options);if(targets.files.length>MAX_TRANSACTION_FILES){fail5("managed startup transaction has too many file targets")}const snapshots=targets.files.map((target2,index)=>snapshotFile(target2,index,profile.agent,options));const totalBytes=snapshots.reduce((sum,snapshot)=>sum+(snapshot.bytes?.length??0),0);if(totalBytes>MAX_TRANSACTION_TOTAL_BYTES){fail5("managed startup transaction backup exceeds the total size limit")}const directories=targets.directories.map(target2=>snapshotDirectory(target2,profile.agent,options));const manifest={schemaVersion:TRANSACTION_SCHEMA_VERSION,agent:profile.agent,profileFingerprint:fingerprintManagedStartupProfile(profile),bootstrapIdentity:options.bootstrapIdentity,files:snapshots.map(({receipt})=>receipt),directories};return{manifest,snapshots}}function captureManagedStartupSharedState(profile,inputOptions){const captured=captureSharedState(profile,resolveOptions(inputOptions));return serializeCapturedSharedState(captured.manifest,captured.snapshots.map(({bytes})=>bytes))}function serializeCapturedSharedState(manifest,contents){const text=`${JSON.stringify({manifest:canonicalManifest(manifest),contents:contents.map(bytes=>bytes?.toString("base64")??null)})} +`;if(Buffer.byteLength(text)>MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES){fail5("shared-state capture exceeds its bounded transport")}return text}function parseCapturedSharedState(text,profile,options){if(Buffer.byteLength(text)>MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES){fail5("shared-state capture exceeds its bounded transport")}let value;try{value=JSON.parse(text)}catch{fail5("shared-state capture is not valid JSON")}if(!value||typeof value!=="object"||Array.isArray(value))fail5("invalid shared-state capture");const record2=value;requireExactKeys(record2,["manifest","contents"]);if(typeof record2.manifest!=="string"||Buffer.byteLength(record2.manifest)>MAX_MANIFEST_BYTES||!Array.isArray(record2.contents)){fail5("invalid shared-state capture fields")}const manifest=parseManifest(record2.manifest);const targets=managedOutputTargets(profile,options);if(manifest.agent!==profile.agent||manifest.profileFingerprint!==fingerprintManagedStartupProfile(profile)||manifest.bootstrapIdentity!==options.bootstrapIdentity||options.bootstrapIdentity===null||JSON.stringify(manifest.files.map(({path:target2})=>target2))!==JSON.stringify(targets.files.map(target2=>relativeTarget(target2,options)))||JSON.stringify(manifest.directories.map(({path:target2})=>target2))!==JSON.stringify(targets.directories.map(target2=>relativeTarget(target2,options)))||record2.contents.length!==manifest.files.length){fail5("shared-state capture does not match the exact profile and bootstrap targets")}const contents=record2.contents;let total=0;const snapshots=manifest.files.map((receipt,index)=>{const encoded=contents[index];if(receipt.state==="absent"){if(encoded!==null)fail5("absent captured output has unexpected contents");return{receipt,bytes:null}}if(typeof encoded!=="string"||encoded.length>4*Math.ceil(MAX_TRANSACTION_FILE_BYTES/3)){fail5("captured output exceeds its bounded encoding")}const bytes=Buffer.from(encoded,"base64");total+=bytes.length;if(total>MAX_TRANSACTION_TOTAL_BYTES||bytes.toString("base64")!==encoded||bytes.length!==receipt.size||(0,import_node_crypto5.createHash)("sha256").update(bytes).digest("hex")!==receipt.sha256){fail5("captured output does not match its bounded receipt")}return{receipt,bytes}});return{manifest,snapshots}}function ensureOriginalDirectories(receipts,expectedAgent,options){for(const receipt of receipts){if(receipt.state!=="directory")continue;const target2=absoluteTarget(receipt.path,options);validateExistingAncestors(import_node_path2.default.join(target2,".restore"),expectedAgent,options);let stat=null;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code!=="ENOENT"){fail5(`could not inspect restore directory ${target2}`)}}if(stat&&(stat.isSymbolicLink()||!stat.isDirectory())){fail5(`restore directory is unsafe: ${target2}`)}if(stat&&directoryMatchesReceipt(target2,receipt))continue;if(!stat)import_node_fs2.default.mkdirSync(target2,{mode:receipt.mode});import_node_fs2.default.chownSync(target2,receipt.uid,receipt.gid);import_node_fs2.default.chmodSync(target2,receipt.mode)}}function restoreFiles(receipts,backups,expectedAgent,options){for(const receipt of receipts){const target2=absoluteTarget(receipt.path,options);validateExistingAncestors(target2,expectedAgent,options);if(receipt.state==="absent"){let stat;try{stat=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")continue;fail5(`could not inspect new managed output ${target2}`)}if(stat.isDirectory()){fail5(`new managed output unexpectedly became a directory: ${target2}`)}import_node_fs2.default.unlinkSync(target2);continue}if(fileMatchesReceipt(target2,receipt))continue;const bytes=backups.get(receipt.path);if(!bytes)fail5(`verified transaction backup is missing: ${receipt.path}`);let current=null;try{current=import_node_fs2.default.lstatSync(target2)}catch(error){if(error.code!=="ENOENT"){fail5(`could not inspect managed output before restore: ${target2}`)}}if(current?.isDirectory()){fail5(`managed output unexpectedly became a directory: ${target2}`)}atomicWriteTrustedFile(target2,bytes,receipt.mode,receipt.uid,receipt.gid)}}function restoreDirectoryMetadata(receipts,options){for(const receipt of[...receipts].reverse()){const target2=absoluteTarget(receipt.path,options);if(receipt.state==="absent"){try{import_node_fs2.default.rmdirSync(target2)}catch(error){if(error.code==="ENOENT")continue;fail5(`could not remove newly created managed directory ${target2}`)}continue}if(directoryMatchesReceipt(target2,receipt))continue;const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isDirectory()){fail5(`managed directory changed type during restore: ${target2}`)}import_node_fs2.default.chownSync(target2,receipt.uid,receipt.gid);import_node_fs2.default.chmodSync(target2,receipt.mode)}}function verifyRestoration(manifest,options){for(const receipt of manifest.files){const target2=absoluteTarget(receipt.path,options);if(receipt.state==="absent"){if(pathExistsNoFollow(target2)){fail5(`new managed output remained after rollback: ${target2}`)}continue}const stable=readStableFile(target2,MAX_TRANSACTION_FILE_BYTES);if(stable.bytes.length!==receipt.size||(0,import_node_crypto5.createHash)("sha256").update(stable.bytes).digest("hex")!==receipt.sha256||Number(stable.stat.uid)!==receipt.uid||Number(stable.stat.gid)!==receipt.gid||Number(stable.stat.mode&0o7777n)!==receipt.mode){fail5(`managed output was not restored exactly: ${target2}`)}}for(const receipt of manifest.directories){const target2=absoluteTarget(receipt.path,options);if(receipt.state==="absent"){if(pathExistsNoFollow(target2)){fail5(`new managed directory remained after rollback: ${target2}`)}continue}const stat=import_node_fs2.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isDirectory()||stat.uid!==receipt.uid||stat.gid!==receipt.gid||modeOf2(stat)!==receipt.mode){fail5(`managed directory metadata was not restored exactly: ${target2}`)}}}function rollbackManagedStartupSharedStateTransaction(expectedAgent,inputOptions={}){const options=resolveOptions(inputOptions);const manifest=loadRollbackManifest(expectedAgent,options);if(!manifest)return false;const backups=verifyAllBackups(manifest.files,options);restoreSharedState(manifest,backups,options);if(!options.readOnlyReceipt){removeTransactionDirectory(options)}return true}function loadRollbackManifest(expectedAgent,options){requireTransactionIdentity(options);const committed=loadCommitReceipt(options);if(committed){if(options.bootstrapIdentity===null){fail5("shared state is already durably committed")}assertCommitReceiptMatches(committed.receipt,{agent:expectedAgent,bootstrapIdentity:options.bootstrapIdentity});fail5("shared state is already durably committed and cannot be rolled back")}const manifest=loadManifest(options);if(!manifest)return null;if(manifest.agent!==expectedAgent){fail5(`pending transaction targets ${manifest.agent}, expected ${expectedAgent}`)}if(manifest.bootstrapIdentity!==options.bootstrapIdentity){fail5("pending transaction belongs to a different bootstrap attempt")}return manifest}function restoreSharedState(manifest,backups,options){ensureOriginalDirectories(manifest.directories,manifest.agent,options);restoreFiles(manifest.files,backups,manifest.agent,options);restoreDirectoryMetadata(manifest.directories,options);verifyRestoration(manifest,options)}function exportManagedStartupSharedStateRestoration(profile,inputOptions){const options=resolveOptions(inputOptions);const manifest=loadRollbackManifest(profile.agent,options);if(!manifest)return null;const backups=verifyAllBackups(manifest.files,options);const captured=serializeCapturedSharedState(manifest,manifest.files.map(receipt=>receipt.state==="file"?backups.get(receipt.path):null));parseCapturedSharedState(captured,profile,options);return captured}function restoreManagedStartupCapturedSharedState(profile,capturedState,inputOptions){const options=resolveOptions(inputOptions);const{manifest,snapshots}=parseCapturedSharedState(capturedState,profile,options);for(const receipt of[...manifest.files,...manifest.directories]){if(receipt.state==="absent")continue;if(receipt.uid!==process.geteuid?.()||receipt.gid!==process.getegid?.()){fail5("captured output requires a different restoration identity")}}const backups=new Map;for(const snapshot of snapshots){if(snapshot.bytes!==null)backups.set(snapshot.receipt.path,snapshot.bytes)}restoreSharedState(manifest,backups,options);return captureManagedStartupSharedState(profile,inputOptions)}function finishManagedStartupSharedStateRestoration(profile,observedState,inputOptions){const options=resolveOptions(inputOptions);if(options.readOnlyReceipt)fail5("cannot retire a read-only rollback receipt");const original=loadRollbackManifest(profile.agent,options);if(!original)return false;verifyAllBackups(original.files,options);const observed=parseCapturedSharedState(observedState,profile,options).manifest;if(canonicalManifest(original)!==canonicalManifest(observed)){fail5("restored state does not match the protected original snapshot")}removeTransactionDirectory(options);return true}function commitManagedStartupSharedStateTransaction(expectedAgent,inputOptions={}){const options=resolveOptions(inputOptions);requireTransactionIdentity(options);if(options.readOnlyReceipt){fail5("cannot commit a read-only rollback receipt")}const committed=loadCommitReceipt(options);if(committed){if(options.bootstrapIdentity===null){fail5("durable commit receipt is missing its expected bootstrap identity")}assertCommitReceiptMatches(committed.receipt,{agent:expectedAgent,bootstrapIdentity:options.bootstrapIdentity});compactDurableCommitReceipt(committed,options);return true}const manifest=loadManifest(options);if(!manifest)return false;if(manifest.agent!==expectedAgent){fail5(`pending transaction targets ${manifest.agent}, expected ${expectedAgent}`)}if(manifest.bootstrapIdentity!==options.bootstrapIdentity){fail5("pending transaction belongs to a different bootstrap attempt")}if(manifest.bootstrapIdentity===null){removeTransactionDirectory(options);return true}verifyAllBackups(manifest.files,options);if(pathExistsNoFollow(options.commitReceiptDirectory)){fail5("durable commit receipt path appeared before transaction commit")}try{import_node_fs2.default.renameSync(options.transactionDirectory,options.commitReceiptDirectory);fsyncDirectory(options.transactionParentDirectory)}catch(error){fail5(`could not atomically establish durable commit state: ${error.message}`)}const renamed=loadCommitReceipt(options);if(!renamed)fail5("durable commit state disappeared after atomic rename");assertCommitReceiptMatches(renamed.receipt,{agent:expectedAgent,profileFingerprint:manifest.profileFingerprint,bootstrapIdentity:manifest.bootstrapIdentity});compactDurableCommitReceipt(renamed,options);return true}function clearManagedStartupSharedStateCommitReceipt(expectedAgent,inputOptions={}){const options=resolveOptions(inputOptions);requireTransactionIdentity(options);if(options.readOnlyReceipt){fail5("cannot clear a durable commit from a read-only receipt")}if(options.bootstrapIdentity===null){fail5("durable commit cleanup requires its bootstrap identity")}const committed=loadCommitReceipt(options);if(!committed)return false;assertCommitReceiptMatches(committed.receipt,{agent:expectedAgent,bootstrapIdentity:options.bootstrapIdentity});compactDurableCommitReceipt(committed,options);requireTrustedTransactionPath(options.commitReceiptDirectory,TRANSACTION_DIRECTORY_MODE,options);requireTrustedTransactionPath(options.commitReceiptFile,TRANSACTION_FILE_MODE,options);const entries=import_node_fs2.default.readdirSync(options.commitReceiptDirectory);if(entries.length!==1||entries[0]!==MANAGED_STARTUP_SHARED_COMMIT_RECEIPT_FILE){fail5("durable commit receipt directory contains unexpected artifacts")}import_node_fs2.default.rmSync(options.commitReceiptDirectory,{force:false,recursive:true});fsyncDirectory(options.transactionParentDirectory);if(pathExistsNoFollow(options.commitReceiptDirectory)){fail5("durable commit receipt remained after cleanup")}return true}function getManagedStartupSharedStateTransactionStatus(expected,inputOptions={}){const options=resolveOptions({...inputOptions,bootstrapIdentity:expected.bootstrapIdentity});requireTransactionIdentity(options);const manifest=loadManifest(options);if(manifest){if(manifest.agent!==expected.agent||manifest.profileFingerprint!==expected.profileFingerprint||manifest.bootstrapIdentity!==expected.bootstrapIdentity){fail5("pending transaction does not match the expected agent, profile fingerprint, or bootstrap identity")}verifyAllBackups(manifest.files,options);return"pending"}const committed=loadCommitReceipt(options);if(!committed)return"none";assertCommitReceiptMatches(committed.receipt,expected);return"committed"}var MANAGED_STARTUP_PROFILE_ENV="NEMOCLAW_STARTUP_PROFILE_B64";var MANAGED_STARTUP_CA_ENV="NEMOCLAW_CORPORATE_CA_B64";var MANAGED_STARTUP_EXCHANGE_DIRECTORY="/tmp";var MANAGED_STARTUP_RUNTIME_ENV_FILE=`${MANAGED_STARTUP_EXCHANGE_DIRECTORY}/nemoclaw-managed-startup-runtime.env`;var MANAGED_STARTUP_RUNTIME_EXECUTABLE="/usr/local/lib/nemoclaw/managed-startup-image-runtime.cjs";var MANAGED_STARTUP_MERGED_CA_FILE=`${MANAGED_STARTUP_EXCHANGE_DIRECTORY}/nemoclaw-managed-startup-ca-bundle.pem`;var MANAGED_STARTUP_COMPLETION_FILE=`${MANAGED_STARTUP_EXCHANGE_DIRECTORY}/nemoclaw-managed-startup-complete.json`;var MANAGED_STARTUP_CORPORATE_CA_FILE="/usr/local/share/nemoclaw/corporate-ca.pem";var MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY="/usr/local/share/ca-certificates";var MANAGED_STARTUP_SYSTEM_CA_ANCHOR_RE=/^nemoclaw-corporate-ca-[0-9]{2}\.crt$/u;var SYSTEM_CA_BUNDLE_FILE="/etc/ssl/certs/ca-certificates.crt";var UPDATE_CA_CERTIFICATES_EXECUTABLE="/usr/sbin/update-ca-certificates";var MANAGED_STARTUP_TLS_ENV_NAMES=new Set(["CURL_CA_BUNDLE","GIT_SSL_CAINFO","NODE_EXTRA_CA_CERTS","REQUESTS_CA_BUNDLE","SSL_CERT_FILE"]);var MESSAGING_RUNTIME_PLAN_FILE="/usr/local/share/nemoclaw/messaging-runtime-plan.json";var ROOT_STATE_PARENT="/var/lib/nemoclaw";var ROOT_RUNTIME_DIRECTORY="/run/nemoclaw";var ROOT_OWNED_DIRECTORY_MODE=493;var MAX_TRUST_BUNDLE_BYTES=4*1024*1024;var HERMES_MANAGED_CONFIG_FILES=["/sandbox/.hermes/config.yaml","/sandbox/.hermes/.env"];var HERMES_GENERATED_MANAGED_POLICY_FILE="/sandbox/.hermes/managed-policy.json";var HERMES_INSTALLED_MANAGED_POLICY_FILE="/usr/local/share/nemoclaw/hermes-managed-policy.json";var MAX_HERMES_MANAGED_POLICY_BYTES=4*1024*1024;var MAX_HERMES_SWITCHYARD_CONFIG_BYTES=128*1024;var MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES=4*Math.ceil(MAX_HERMES_MANAGED_POLICY_BYTES/3)+4096;var FIXED_PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin";var SHA256_RE4=/^[a-f0-9]{64}$/u;var MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION=1;var MAX_MANAGED_STARTUP_COMPLETION_BYTES=4096;var MAX_MANAGED_STARTUP_RUNTIME_ENVIRONMENT_BYTES=512*1024;var defaultWorkspaceRootApplyDeps={lstat:target2=>import_node_fs3.default.lstatSync(target2),chown:(target2,uid,gid)=>import_node_fs3.default.chownSync(target2,uid,gid),chmod:(target2,mode)=>import_node_fs3.default.chmodSync(target2,mode)};function normalizeManagedStartupWorkspaceRoot(agent,deps=defaultWorkspaceRootApplyDeps){const target2=managedImageRuntimeIdentity(agent).workdir;const before=deps.lstat(target2);if(before.isSymbolicLink()||!before.isDirectory()){fail6("managed startup workspace root must be one real directory")}const expected=managedStartupWorkspaceRoot({agent,agentIdentity:managedImageRuntimeIdentity(agent)});try{if(before.uid!==expected.uid||before.gid!==expected.gid||(before.mode&4095)!==expected.mode){deps.chown(target2,expected.uid,expected.gid);deps.chmod(target2,expected.mode)}}catch{fail6("managed startup workspace-root normalization failed")}const after=deps.lstat(target2);if(after.isSymbolicLink()||!after.isDirectory()||after.uid!==expected.uid||after.gid!==expected.gid||(after.mode&4095)!==expected.mode){fail6("managed startup workspace root does not match its declared posture")}}var ManagedStartupImageActionPlanError=class extends Error{constructor(message){super(`Cannot build managed startup image action plan: ${message}`);this.name="ManagedStartupImageActionPlanError"}};var ManagedStartupImageRuntimeError=class extends Error{constructor(message){super(`Managed startup image application failed: ${message}`);this.name="ManagedStartupImageRuntimeError"}};function failActionPlan(message){throw new ManagedStartupImageActionPlanError(message)}function exactActionPlanAgent(value){if(MANAGED_STARTUP_AGENTS.includes(value)){return value}return failActionPlan(`unsupported agent ${JSON.stringify(value)}`)}function fail6(message){throw new ManagedStartupImageRuntimeError(message)}function validateManagedStartupApplicationRuntimePlan(plan){if(typeof plan!=="object"||plan===null){return fail6("application runtime plan must be an object")}const exportEnvironment=plan.exportEnvironment;const unsetEnvironment=plan.unsetEnvironment;if(typeof exportEnvironment!=="object"||exportEnvironment===null||Array.isArray(exportEnvironment)||!Array.isArray(unsetEnvironment)){return fail6("application runtime plan must contain exports and unsets")}const exports2={};for(const[name,value]of Object.entries(exportEnvironment)){if(!/^[A-Za-z_][A-Za-z0-9_]*$/u.test(name)){return fail6(`invalid application runtime environment key ${JSON.stringify(name)}`)}if(typeof value!=="string"||value.includes("\0")||/[\r\n]/u.test(value)){return fail6(`application runtime environment value for ${name} must be single-line text`)}exports2[name]=value}const unsets=new Set;for(const name of unsetEnvironment){if(typeof name!=="string"||!/^[A-Za-z_][A-Za-z0-9_]*$/u.test(name)){return fail6(`invalid application runtime unset ${JSON.stringify(name)}`)}if(unsets.has(name)){return fail6(`duplicate application runtime unset ${name}`)}if(Object.hasOwn(exports2,name)){return fail6(`application runtime cannot both export and unset ${name}`)}unsets.add(name)}return Object.freeze({exportEnvironment:Object.freeze(Object.fromEntries(Object.entries(exports2).sort(([left],[right])=>left.localeCompare(right)))),unsetEnvironment:Object.freeze([...unsets].sort())})}function applyManagedStartupCommandEnvironmentPlan(environment,plan){const validated=validateManagedStartupApplicationRuntimePlan(plan);const applied={...environment};for(const name of[...Object.keys(validated.exportEnvironment),...validated.unsetEnvironment]){delete applied[name]}return applied}function exactAgent2(value){if(MANAGED_STARTUP_AGENTS.includes(value)){return value}return fail6(`unsupported agent ${JSON.stringify(value)}`)}function managedTransactionProfile(expectedAgentInput,env=process.env){requireRoot();const expectedAgent=exactAgent2(expectedAgentInput);if(env.NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION!=="1"){fail6("shared-state transactions require a complete managed image")}const encodedProfile=env[MANAGED_STARTUP_PROFILE_ENV];if(!encodedProfile)fail6(`${MANAGED_STARTUP_PROFILE_ENV} is required`);const profile=decodeManagedStartupProfile(encodedProfile);if(profile.agent!==expectedAgent){fail6(`shared-state transaction profile targets ${profile.agent}, expected ${expectedAgent}`)}return profile}function requireRoot(){if(process.geteuid?.()!==0){fail6("managed startup requires container effective uid 0")}}function modeOf3(stat){return stat.mode&511}function requireManagedStartupExchangeDirectory(exchangeDirectory=MANAGED_STARTUP_EXCHANGE_DIRECTORY){const stat=import_node_fs3.default.lstatSync(exchangeDirectory);if(stat.isSymbolicLink()||!stat.isDirectory()||stat.uid!==0||stat.gid!==0||(stat.mode&4095)!==1023){fail6("managed startup exchange directory must be root:root mode 1777")}}function requireRootOwnedDirectory(target2,mode){let stat;try{stat=import_node_fs3.default.lstatSync(target2)}catch{fail6(`required root-owned directory is missing: ${target2}`)}if(stat.isSymbolicLink()||!stat.isDirectory()||stat.uid!==0||stat.gid!==0||modeOf3(stat)!==mode){fail6(`${target2} must be a root:root directory with mode ${mode.toString(8)}`)}}function ensureRootOwnedDirectory(target2,mode=ROOT_OWNED_DIRECTORY_MODE){const parent=import_node_path3.default.dirname(target2);const parentStat=import_node_fs3.default.lstatSync(parent);if(parentStat.isSymbolicLink()||!parentStat.isDirectory()||parentStat.uid!==0||parentStat.gid!==0||(modeOf3(parentStat)&18)!==0){fail6(`refusing unsafe parent directory for ${target2}`)}try{import_node_fs3.default.mkdirSync(target2,{mode});import_node_fs3.default.chownSync(target2,0,0);import_node_fs3.default.chmodSync(target2,mode)}catch(error){if(error.code!=="EEXIST"){fail6(`could not create ${target2}`)}}requireRootOwnedDirectory(target2,mode)}function requireSafeExistingRootTarget(target2){let stat;try{stat=import_node_fs3.default.lstatSync(target2)}catch(error){if(error.code==="ENOENT")return;fail6(`could not inspect ${target2}`)}if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==0||stat.gid!==0){fail6(`refusing to replace unsafe root-owned file ${target2}`)}}function atomicWriteRootFile(target2,contents,mode){const parent=import_node_path3.default.dirname(target2);const parentStat=import_node_fs3.default.lstatSync(parent);const trustedStickyExchange=parent===MANAGED_STARTUP_EXCHANGE_DIRECTORY&&(parentStat.mode&4095)===1023;if(parentStat.isSymbolicLink()||!parentStat.isDirectory()||parentStat.uid!==0||parentStat.gid!==0||(modeOf3(parentStat)&18)!==0&&!trustedStickyExchange){fail6(`refusing unsafe root-owned file parent ${parent}`)}requireSafeExistingRootTarget(target2);const temporary=import_node_path3.default.join(parent,`.${import_node_path3.default.basename(target2)}.${(0,import_node_crypto6.randomBytes)(12).toString("hex")}`);let descriptor;try{descriptor=import_node_fs3.default.openSync(temporary,import_node_fs3.default.constants.O_CREAT|import_node_fs3.default.constants.O_EXCL|import_node_fs3.default.constants.O_WRONLY|import_node_fs3.default.constants.O_NOFOLLOW,384);import_node_fs3.default.fchownSync(descriptor,0,0);import_node_fs3.default.writeFileSync(descriptor,contents);import_node_fs3.default.fchmodSync(descriptor,mode);import_node_fs3.default.fsyncSync(descriptor);import_node_fs3.default.closeSync(descriptor);descriptor=void 0;import_node_fs3.default.renameSync(temporary,target2)}catch(error){if(descriptor!==void 0)import_node_fs3.default.closeSync(descriptor);try{import_node_fs3.default.unlinkSync(temporary)}catch{}fail6(`could not atomically write ${target2}: ${error.message}`)}const stat=import_node_fs3.default.lstatSync(target2);if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==0||stat.gid!==0||modeOf3(stat)!==mode){fail6(`root-owned output failed metadata verification: ${target2}`)}}function removeSafeRootFile(target2){requireSafeExistingRootTarget(target2);try{import_node_fs3.default.unlinkSync(target2)}catch(error){if(error.code!=="ENOENT"){fail6(`could not remove ${target2}`)}}}function trustedExecutable(target2){try{const stat=import_node_fs3.default.lstatSync(target2);return!stat.isSymbolicLink()&&stat.isFile()&&stat.uid===0&&stat.gid===0&&(modeOf3(stat)&18)===0&&(modeOf3(stat)&73)!==0}catch{return false}}function readSandboxIdentity(){const readId=flag=>{const result=(0,import_node_child_process.spawnSync)("/usr/bin/id",[flag,"sandbox"],{encoding:"utf8",env:{PATH:FIXED_PATH}});const value=result.stdout.trim();if(result.status!==0||!/^[1-9][0-9]*$/u.test(value)){fail6("could not resolve the sandbox account")}return value};return{uid:readId("-u"),gid:readId("-g")}}function managedStartupSandboxPrefix(){if(trustedExecutable("/usr/bin/setpriv")){const identity=readSandboxIdentity();if(process.geteuid?.()===Number(identity.uid)&&process.getegid?.()===Number(identity.gid))return[];return["/usr/bin/setpriv",`--reuid=${identity.uid}`,`--regid=${identity.gid}`,"--init-groups","--"]}return fail6("a trusted setpriv executable is required")}function commandEnvironment(configurationEnvironment,applicationRuntime){const env=applyManagedStartupCommandEnvironmentPlan({...process.env,...configurationEnvironment,HOME:"/sandbox",PATH:FIXED_PATH,NPM_CONFIG_OFFLINE:"true",npm_config_offline:"true",PIP_DISABLE_PIP_VERSION_CHECK:"1",PIP_NO_INDEX:"1",UV_OFFLINE:"1"},applicationRuntime);delete env[MANAGED_STARTUP_PROFILE_ENV];delete env[MANAGED_STARTUP_CA_ENV];return env}function execute(argv,runAs,configurationEnvironment,applicationRuntime,capture=false){if(argv.length===0)fail6("refusing an empty managed startup command");const command=runAs==="sandbox"?[...managedStartupSandboxPrefix(),...argv]:[...argv];const result=(0,import_node_child_process.spawnSync)(command[0],command.slice(1),{encoding:"utf8",env:commandEnvironment(configurationEnvironment,applicationRuntime),stdio:capture?"pipe":"inherit"});if(result.error){fail6(`could not execute ${argv[0]}: ${result.error.message}`)}if(result.status!==0){const detail=capture===true?`: ${(result.stderr||result.stdout).trim()}`:"";fail6(`${argv[0]} exited with status ${String(result.status??"unknown")}${detail}`)}return{status:result.status,stdout:result.stdout??"",stderr:result.stderr??""}}function generatorCommand(agent){switch(agent){case"openclaw":return["/usr/local/bin/node","/scripts/generate-openclaw-config.mts"];case"hermes":return["/usr/local/bin/node","/opt/nemoclaw-hermes-config/generate-config.ts"];case"langchain-deepagents-code":return["/usr/local/bin/node","/opt/nemoclaw-deepagents-code/generate-config.ts"];case"pi":return["/usr/local/bin/node","/opt/nemoclaw-pi/generate-config.ts"]}}function messagingCommand(agent,phase,mode){return["/usr/local/bin/node","/src/lib/messaging/applier/build/messaging-build-applier.mts","--agent",agent,"--phase",phase,"--mode",mode,...phase==="post-agent-install"?["--managed-startup-runtime"]:[]]}function assertActionAgent(inputAgent,actionAgent){if(inputAgent!==actionAgent){failActionPlan(`action for ${actionAgent} cannot be used by ${inputAgent}`)}}function buildManagedStartupImageActionPlan(input){const inputAgent=exactActionPlanAgent(input.agent);const commands=[];let dashboardActions=0;let generateActions=0;let runtimeMessagingActions=0;let postMessagingActions=0;for(const action of input.actions){switch(action.kind){case"configure-dashboard":{if(action.dashboard.agent!==input.agent){failActionPlan(`dashboard for ${action.dashboard.agent} cannot be used by ${input.agent}`)}dashboardActions+=1;break}case"generate-agent-config":{assertActionAgent(inputAgent,exactActionPlanAgent(action.agent));if(action.runAs!=="sandbox"){failActionPlan("agent configuration generation must run as sandbox")}generateActions+=1;commands.push({action:"generate-agent-config",runAs:action.runAs,argv:generatorCommand(action.agent)});break}case"apply-messaging-plan":{assertActionAgent(inputAgent,exactActionPlanAgent(action.agent));if(action.mode!=="apply"&&action.mode!=="clear"){failActionPlan("messaging intent must be apply or clear")}if(action.phase==="runtime-setup"){if(action.runAs!=="root"){failActionPlan("messaging runtime setup must run as root")}runtimeMessagingActions+=1;commands.push({action:"messaging-runtime-setup",runAs:action.runAs,argv:messagingCommand(action.agent,action.phase,action.mode)})}else if(action.phase==="post-agent-install"){if(action.runAs!=="sandbox"){failActionPlan("messaging post-agent configuration must run as sandbox")}postMessagingActions+=1;commands.push({action:"messaging-post-agent-install",runAs:action.runAs,argv:messagingCommand(action.agent,action.phase,action.mode)})}else{failActionPlan("unsupported messaging construction phase")}break}default:failActionPlan("unsupported managed startup construction action")}}if(dashboardActions!==1){failActionPlan("exactly one dashboard construction action is required")}if(generateActions!==1){failActionPlan("exactly one agent config construction action is required")}const supportsMessaging=MANAGED_STARTUP_MESSAGING_AGENTS.includes(inputAgent);const expectedMessagingActions=supportsMessaging?1:0;if(runtimeMessagingActions!==expectedMessagingActions||postMessagingActions!==expectedMessagingActions){failActionPlan(`${inputAgent} requires ${String(expectedMessagingActions)} action for each messaging phase`)}const expectedOrder=supportsMessaging?["messaging-runtime-setup","generate-agent-config","messaging-post-agent-install"]:["generate-agent-config"];if(commands.some((command,index)=>command.action!==expectedOrder[index])){failActionPlan(`${inputAgent} image actions are not in the required construction order`)}return Object.freeze(commands.map(command=>Object.freeze({...command,argv:Object.freeze([...command.argv])})))}function prepareMessagingRuntimeTarget(mode){if(mode==="clear"){removeSafeRootFile(MESSAGING_RUNTIME_PLAN_FILE);return}requireSafeExistingRootTarget(MESSAGING_RUNTIME_PLAN_FILE);try{import_node_fs3.default.unlinkSync(MESSAGING_RUNTIME_PLAN_FILE)}catch(error){if(error.code!=="ENOENT"){fail6("could not prepare the messaging runtime-plan target")}}}function verifyMessagingRuntimeTarget(mode){if(mode==="clear"){if(import_node_fs3.default.existsSync(MESSAGING_RUNTIME_PLAN_FILE)){fail6("clear messaging profile left a runtime-plan artifact")}return}const stat=import_node_fs3.default.lstatSync(MESSAGING_RUNTIME_PLAN_FILE);if(stat.isSymbolicLink()||!stat.isFile()||stat.nlink!==1||stat.uid!==0||stat.gid!==0||modeOf3(stat)!==420){fail6("messaging runtime-plan artifact failed root ownership validation")}}function runInternalSandboxAction(action,configurationEnvironment,applicationRuntime,extraEnvironment={}){execute(["/usr/local/bin/node",MANAGED_STARTUP_RUNTIME_EXECUTABLE,`--internal-${action}`],"sandbox",{...configurationEnvironment,...extraEnvironment},applicationRuntime)}function validateOpenClawConfiguration(configurationEnvironment,applicationRuntime,privateOutput=false){const validation=execute(["/usr/local/bin/openclaw","config","validate","--json"],"sandbox",{...configurationEnvironment,OPENCLAW_CONFIG_PATH:"/sandbox/.openclaw/openclaw.json"},applicationRuntime,privateOutput?"private":true);let parsed;try{parsed=JSON.parse(validation.stdout)}catch{fail6("OpenClaw config validation did not emit JSON")}if(typeof parsed!=="object"||parsed===null||parsed.valid!==true){fail6("OpenClaw rejected the generated managed startup config")}}function sameStableFileMetadata(left,right){return left.dev===right.dev&&left.ino===right.ino&&left.mode===right.mode&&left.nlink===right.nlink&&left.uid===right.uid&&left.gid===right.gid&&left.size===right.size&&left.mtimeNs===right.mtimeNs&&left.ctimeNs===right.ctimeNs}function readStableRegularFileSnapshot(target2,maxBytes){if(typeof import_node_fs3.default.constants.O_NOFOLLOW!=="number"){fail6("O_NOFOLLOW is unavailable for managed startup file reads")}const nonblock=typeof import_node_fs3.default.constants.O_NONBLOCK==="number"?import_node_fs3.default.constants.O_NONBLOCK:0;let descriptor;try{descriptor=import_node_fs3.default.openSync(target2,import_node_fs3.default.constants.O_RDONLY|import_node_fs3.default.constants.O_NOFOLLOW|nonblock)}catch(error){if(error.code==="ENOENT")throw error;fail6(`refusing unsafe or unreadable file ${target2}`)}try{const before=import_node_fs3.default.fstatSync(descriptor,{bigint:true});if(!before.isFile()||before.nlink!==1n||before.size<1n||before.size>BigInt(maxBytes)){fail6(`refusing unsafe or oversized file ${target2}`)}const bytes=Buffer.alloc(Number(before.size));let offset=0;while(offset{const identity=readSandboxIdentity();return{uid:Number(identity.uid),gid:Number(identity.gid)}})();const relaySource=expectedSandboxOwnedRoutingSource(paths.relayPluginsSource,expectedRelay,sandboxIdentity);const relayStateSource=expectedSandboxOwnedRoutingSource(paths.relayPluginStateSource,expectedRelayState,sandboxIdentity);const deploymentSource=expectedSandboxOwnedRoutingSource(paths.switchyardDeploymentSource,expectedDeployment,sandboxIdentity);atomicWriteRootFile(paths.switchyardDeploymentTarget,expectedDeployment,292);atomicWriteRootFile(paths.relayPluginStateTarget,expectedRelayState,292);atomicWriteRootFile(paths.relayPluginsTarget,expectedRelay,292);verifyCommittedRoutingFile(paths.switchyardDeploymentTarget,expectedDeployment);verifyCommittedRoutingFile(paths.relayPluginStateTarget,expectedRelayState);verifyCommittedRoutingFile(paths.relayPluginsTarget,expectedRelay);removeStableRoutingSource(paths.switchyardDeploymentSource,deploymentSource);removeStableRoutingSource(paths.relayPluginStateSource,relayStateSource);removeStableRoutingSource(paths.relayPluginsSource,relaySource)}function verifyHermesSwitchyardRoutingFiles(configurationEnvironment,paths=HERMES_SWITCHYARD_ROUTING_PATHS){const routing=readHermesSwitchyardRoutingEnvironment(configurationEnvironment);if(!routing){assertRoutingFileAbsent(paths.relayPluginsTarget);assertRoutingFileAbsent(paths.relayPluginStateTarget);assertRoutingFileAbsent(paths.switchyardDeploymentTarget);return}const serialized=serializeHermesSwitchyardRouting(routing);verifyCommittedRoutingFile(paths.relayPluginsTarget,Buffer.from(serialized.relayPluginsToml,"utf8"));verifyCommittedRoutingFile(paths.relayPluginStateTarget,Buffer.from(serialized.relayPluginStateJson,"utf8"));verifyCommittedRoutingFile(paths.switchyardDeploymentTarget,Buffer.from(serialized.switchyardDeploymentToml,"utf8"))}function normalizeHermesManagedConfigDescriptor(target2,sandboxIdentity){if(!Number.isSafeInteger(sandboxIdentity.uid)||sandboxIdentity.uid<=0||!Number.isSafeInteger(sandboxIdentity.gid)||sandboxIdentity.gid<=0){fail6("invalid sandbox identity for Hermes descriptor normalization")}if(typeof import_node_fs3.default.constants.O_NOFOLLOW!=="number"){fail6("O_NOFOLLOW is unavailable for Hermes descriptor normalization")}const nonblock=typeof import_node_fs3.default.constants.O_NONBLOCK==="number"?import_node_fs3.default.constants.O_NONBLOCK:0;let descriptor;try{descriptor=import_node_fs3.default.openSync(target2,import_node_fs3.default.constants.O_RDONLY|import_node_fs3.default.constants.O_NOFOLLOW|nonblock)}catch{fail6(`refusing unsafe Hermes managed config descriptor ${target2}`)}try{const before=import_node_fs3.default.fstatSync(descriptor,{bigint:true});const beforeMode=Number(before.mode&0o777n);const mutable=before.uid===BigInt(sandboxIdentity.uid)&&before.gid===BigInt(sandboxIdentity.gid)&&(beforeMode===384||beforeMode===416);if(!before.isFile()||before.nlink!==1n||!mutable){fail6(`refusing unexpected Hermes managed config descriptor ${target2}`)}const expectedMode=416;if(beforeMode===384){try{import_node_fs3.default.fchmodSync(descriptor,expectedMode)}catch{fail6(`could not normalize Hermes managed config descriptor ${target2}`)}}const after=import_node_fs3.default.fstatSync(descriptor,{bigint:true});let pathAfter;try{pathAfter=import_node_fs3.default.lstatSync(target2,{bigint:true})}catch{fail6(`Hermes managed config descriptor disappeared during normalization: ${target2}`)}const expectedUid=BigInt(sandboxIdentity.uid);const expectedGid=BigInt(sandboxIdentity.gid);if(!after.isFile()||after.nlink!==1n||after.dev!==before.dev||after.ino!==before.ino||after.uid!==expectedUid||after.gid!==expectedGid||Number(after.mode&0o777n)!==expectedMode||after.size!==before.size||after.mtimeNs!==before.mtimeNs||pathAfter.isSymbolicLink()||!pathAfter.isFile()||!sameStableFileMetadata(after,pathAfter)){fail6(`Hermes managed config descriptor changed during normalization: ${target2}`)}}finally{try{import_node_fs3.default.closeSync(descriptor)}catch{fail6(`could not close Hermes managed config descriptor ${target2}`)}}}function normalizeHermesManagedConfiguration(){const identity=readSandboxIdentity();const sandboxIdentity={uid:Number(identity.uid),gid:Number(identity.gid)};for(const target2 of HERMES_MANAGED_CONFIG_FILES){normalizeHermesManagedConfigDescriptor(target2,sandboxIdentity)}}function sealHermesConfiguration(configurationEnvironment,applicationRuntime){const hashText=hermesConfigurationHash();atomicWriteRootFile("/etc/nemoclaw/hermes.config-hash",hashText,292);runInternalSandboxAction("write-hermes-compat-hash",configurationEnvironment,applicationRuntime,{NEMOCLAW_MANAGED_HERMES_HASH_B64:Buffer.from(hashText,"utf8").toString("base64")})}function hermesConfigurationHash(){const configPath="/sandbox/.hermes/config.yaml";const envPath="/sandbox/.hermes/.env";const config=readStableRegularFile(configPath,4*1024*1024);const env=readStableRegularFile(envPath,512*1024);return[`${(0,import_node_crypto6.createHash)("sha256").update(config).digest("hex")} ${configPath}`,`${(0,import_node_crypto6.createHash)("sha256").update(env).digest("hex")} ${envPath}`,""].join("\n")}function installRootOwnedMaterials(materials){for(const material of materials){if(material.kind!=="root-owned-file")continue;if(material.owner!=="root"||material.group!=="root"||material.mode!==292){fail6(`unsupported root-owned material contract for ${material.path}`)}atomicWriteRootFile(material.path,material.contents,material.mode)}}function verifyRootOwnedMaterials(materials){for(const material of materials){if(material.kind!=="root-owned-file")continue;const expected=Buffer.from(material.contents,"utf8");const{bytes,stat}=readStableRegularFileSnapshot(material.path,expected.length);if(stat.nlink!==1n||stat.uid!==0n||stat.gid!==0n||Number(stat.mode&0o777n)!==material.mode||!bytes.equals(expected)){fail6(`committed root-owned material drifted: ${material.path}`)}}}function installCorporateCa(corporateCaPath){if(corporateCaPath===null){removeSafeRootFile(MANAGED_STARTUP_CORPORATE_CA_FILE);return}const bytes=readStableRegularFile(corporateCaPath,128*1024);atomicWriteRootFile(MANAGED_STARTUP_CORPORATE_CA_FILE,bytes,292)}function corporateCaCertificateBlocks(corporateCaPath){const corporate=readStableRegularFile(corporateCaPath,128*1024).toString("utf8");const blocks=corporate.match(PEM_CERTIFICATE_RE_GLOBAL);if(!blocks||blocks.length===0){fail6("corporate CA material contains no certificate")}for(const block of blocks){try{if(!new import_node_crypto6.X509Certificate(block).ca){fail6("corporate CA material contains a certificate that is not a CA")}}catch(error){if(error instanceof ManagedStartupImageRuntimeError)throw error;fail6("corporate CA material contains an invalid certificate")}}return blocks.map(block=>`${block.trim()} +`)}function managedSystemCaAnchorNames(){try{import_node_fs3.default.lstatSync(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY)}catch(error){if(error.code==="ENOENT")return[];fail6("could not inspect the managed system CA anchor directory")}requireRootOwnedDirectory(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,ROOT_OWNED_DIRECTORY_MODE);try{return import_node_fs3.default.readdirSync(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY).filter(name=>MANAGED_STARTUP_SYSTEM_CA_ANCHOR_RE.test(name)).sort()}catch(error){fail6("could not inspect the managed system CA anchors")}}function refreshSystemCaBundle(){if(!trustedExecutable(UPDATE_CA_CERTIFICATES_EXECUTABLE)){fail6(`a trusted ${UPDATE_CA_CERTIFICATES_EXECUTABLE} executable is required`)}const result=(0,import_node_child_process.spawnSync)(UPDATE_CA_CERTIFICATES_EXECUTABLE,[],{encoding:"utf8",env:{PATH:FIXED_PATH},stdio:"inherit"});if(result.error){fail6(`could not execute ${UPDATE_CA_CERTIFICATES_EXECUTABLE}: ${result.error.message}`)}if(result.status!==0){fail6(`${UPDATE_CA_CERTIFICATES_EXECUTABLE} exited with status ${String(result.status??"unknown")}`)}}function requireSystemCaBundleContains(blocks){const systemBundle=safeTrustBundle(SYSTEM_CA_BUNDLE_FILE);if(systemBundle===null)fail6("the refreshed system CA bundle is missing");const systemBlocks=systemBundle.toString("utf8").match(PEM_CERTIFICATE_RE_GLOBAL)??[];const systemFingerprints=new Set;for(const block of systemBlocks){try{systemFingerprints.add(new import_node_crypto6.X509Certificate(block).fingerprint256)}catch{fail6("the refreshed system CA bundle contains an invalid certificate")}}for(const block of blocks){if(!systemFingerprints.has(new import_node_crypto6.X509Certificate(block).fingerprint256)){fail6("the refreshed system CA bundle does not contain the corporate CA")}}}function installCorporateCaSystemAnchors(corporateCaPath){const existingNames=managedSystemCaAnchorNames();if(corporateCaPath===null){for(const name of existingNames){removeSafeRootFile(import_node_path3.default.join(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,name))}refreshSystemCaBundle();return}ensureRootOwnedDirectory(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY);const blocks=corporateCaCertificateBlocks(corporateCaPath);const expectedNames=blocks.map((_block,index)=>`nemoclaw-corporate-ca-${String(index+1).padStart(2,"0")}.crt`);for(const name of existingNames){if(!expectedNames.includes(name)){removeSafeRootFile(import_node_path3.default.join(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,name))}}for(const[index,name]of expectedNames.entries()){atomicWriteRootFile(import_node_path3.default.join(MANAGED_STARTUP_SYSTEM_CA_ANCHOR_DIRECTORY,name),blocks[index],292)}refreshSystemCaBundle();requireSystemCaBundleContains(blocks)}function safeTrustBundle(target2){try{const{bytes,stat}=readStableRegularFileSnapshot(target2,MAX_TRUST_BUNDLE_BYTES);if(Number(stat.mode&0o022n)!==0){fail6(`refusing unsafe trust bundle ${target2}`)}return bytes}catch(error){if(error.code==="ENOENT")return null;throw error}}function mergeCorporateCa(corporateCaPath){if(corporateCaPath===null){removeSafeRootFile(MANAGED_STARTUP_MERGED_CA_FILE);return false}const corporate=readStableRegularFile(corporateCaPath,128*1024);const candidates=["/etc/openshell-tls/ca-bundle.pem",process.env.SSL_CERT_FILE??"","/etc/ssl/certs/ca-certificates.crt"].filter((candidate,index,values)=>candidate&&candidate!==MANAGED_STARTUP_MERGED_CA_FILE&&values.indexOf(candidate)===index);let base=null;for(const candidate of candidates){base=safeTrustBundle(candidate);if(base)break}const merged=Buffer.concat([...base?[base,Buffer.from("\n","utf8")]:[],corporate,...corporate.at(-1)===10?[]:[Buffer.from("\n","utf8")]]);atomicWriteRootFile(MANAGED_STARTUP_MERGED_CA_FILE,merged,292);return true}function shellSingleQuote(value){if(value.includes("\0")||/[\r\n]/u.test(value)){fail6("runtime environment values must be single-line text")}return`'${value.replaceAll("'",`'"'"'`)}'`}function serializeManagedStartupRuntimeEnvironment(environment,corporateCaMerged,configurationEnvironment={},applicationRuntime={exportEnvironment:{},unsetEnvironment:[]}){const{output,unsetNames}=materializeManagedStartupRuntimeEnvironment(environment,corporateCaMerged,configurationEnvironment,applicationRuntime);const unsetLines=unsetNames.map(name=>`unset ${name}`);const exportLines=Object.entries(output).sort(([left],[right])=>left.localeCompare(right)).map(([name,value])=>`export ${name}=${shellSingleQuote(value)}`);return`${[...unsetLines,...exportLines].join("\n")} `}function materializeManagedStartupRuntimeEnvironment(environment,corporateCaMerged,configurationEnvironment={},applicationRuntime={exportEnvironment:{},unsetEnvironment:[]}){const validatedApplicationRuntime=validateManagedStartupApplicationRuntimePlan(applicationRuntime);const output={...environment,...validatedApplicationRuntime.exportEnvironment,NEMOCLAW_MANAGED_STARTUP_APPLIED:"1"};if(corporateCaMerged){for(const name of MANAGED_STARTUP_TLS_ENV_NAMES){delete output[name]}output._NEMOCLAW_CORPORATE_CA_MERGED="1"}for(const name of[...Object.keys(configurationEnvironment),...Object.keys(output)]){if(!/^[A-Za-z_][A-Za-z0-9_]*$/u.test(name)){fail6(`invalid runtime environment key ${JSON.stringify(name)}`)}}const unsetNames=new Set([...Object.keys(configurationEnvironment).filter(name=>!Object.hasOwn(output,name)&&(!corporateCaMerged||!MANAGED_STARTUP_TLS_ENV_NAMES.has(name))),...validatedApplicationRuntime.unsetEnvironment.filter(name=>!corporateCaMerged||!MANAGED_STARTUP_TLS_ENV_NAMES.has(name))]);for(const name of validatedApplicationRuntime.unsetEnvironment){if(Object.hasOwn(output,name)){fail6(`runtime environment cannot both export and unset ${name}`)}}for(const name of unsetNames){if(!/^[A-Za-z_][A-Za-z0-9_]*$/u.test(name)){fail6(`invalid runtime environment key ${JSON.stringify(name)}`)}}return{output,unsetNames:[...unsetNames].sort()}}function serializeManagedStartupCompletionMarker(marker){if(marker.schemaVersion!==MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION||!MANAGED_STARTUP_AGENTS.includes(marker.agent)||!SHA256_RE4.test(marker.profileFingerprint)||!SHA256_RE4.test(marker.runtimeEnvironmentSha256)||typeof marker.corporateCaMerged!=="boolean"){fail6("managed startup completion marker is invalid")}return`${JSON.stringify({agent:marker.agent,corporateCaMerged:marker.corporateCaMerged,profileFingerprint:marker.profileFingerprint,runtimeEnvironmentSha256:marker.runtimeEnvironmentSha256,schemaVersion:marker.schemaVersion})} -`}function publishManagedStartupImageCompletion(agent,profileFingerprint,runtimeEnvironment,corporateCaMerged,completionFile=MANAGED_STARTUP_COMPLETION_FILE){atomicWriteRootFile(completionFile,serializeManagedStartupCompletionMarker({schemaVersion:MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,agent,profileFingerprint,runtimeEnvironmentSha256:(0,import_node_crypto7.createHash)("sha256").update(runtimeEnvironment).digest("hex"),corporateCaMerged}),292)}function parseManagedStartupCompletionMarker(text){let parsed;try{parsed=JSON.parse(text)}catch{fail6("managed startup completion marker is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail6("managed startup completion marker must be an object")}const record2=parsed;const expectedKeys=["agent","corporateCaMerged","profileFingerprint","runtimeEnvironmentSha256","schemaVersion"];if(Object.keys(record2).sort().join(",")!==expectedKeys.sort().join(",")||record2.schemaVersion!==MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION||typeof record2.agent!=="string"||!MANAGED_STARTUP_AGENTS.includes(record2.agent)||typeof record2.profileFingerprint!=="string"||!SHA256_RE4.test(record2.profileFingerprint)||typeof record2.runtimeEnvironmentSha256!=="string"||!SHA256_RE4.test(record2.runtimeEnvironmentSha256)||typeof record2.corporateCaMerged!=="boolean"){fail6("managed startup completion marker has an invalid schema")}const marker={schemaVersion:MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,agent:record2.agent,profileFingerprint:record2.profileFingerprint,runtimeEnvironmentSha256:record2.runtimeEnvironmentSha256,corporateCaMerged:record2.corporateCaMerged};if(serializeManagedStartupCompletionMarker(marker)!==text){fail6("managed startup completion marker is not canonical")}return marker}function verifyManagedStartupImageCompletion(expectedAgentInput,expectedFingerprint,completionFile=MANAGED_STARTUP_COMPLETION_FILE,runtimeEnvironmentFile=MANAGED_STARTUP_RUNTIME_ENV_FILE){const exchangeDirectory=import_node_path3.default.dirname(completionFile);if(import_node_path3.default.dirname(runtimeEnvironmentFile)!==exchangeDirectory){fail6("managed startup handoff files must share one exchange directory")}requireManagedStartupExchangeDirectory(exchangeDirectory);const expectedAgent=exactAgent2(expectedAgentInput);if(!SHA256_RE4.test(expectedFingerprint)){fail6("startup completion expected profile fingerprint is invalid")}const{bytes,stat}=readStableRegularFileSnapshot(completionFile,MAX_MANAGED_STARTUP_COMPLETION_BYTES);if(stat.nlink!==1n||stat.uid!==0n||stat.gid!==0n||Number(stat.mode&0o777n)!==292){fail6("managed startup completion marker must be root:root mode 0444")}const marker=parseManagedStartupCompletionMarker(bytes.toString("utf8"));if(marker.agent!==expectedAgent||marker.profileFingerprint!==expectedFingerprint){fail6("managed startup completion marker does not match the requested profile")}const runtimeEnvironment=readStableRegularFileSnapshot(runtimeEnvironmentFile,MAX_MANAGED_STARTUP_RUNTIME_ENVIRONMENT_BYTES);if(runtimeEnvironment.stat.nlink!==1n||runtimeEnvironment.stat.uid!==0n||runtimeEnvironment.stat.gid!==0n||Number(runtimeEnvironment.stat.mode&0o777n)!==292){fail6("managed startup runtime environment must be root:root mode 0444")}const runtimeEnvironmentSha256=(0,import_node_crypto7.createHash)("sha256").update(runtimeEnvironment.bytes).digest("hex");if(runtimeEnvironmentSha256!==marker.runtimeEnvironmentSha256){fail6("managed startup completion marker runtime environment digest mismatch")}return{agent:expectedAgent,fingerprint:expectedFingerprint}}function waitForManagedStartupImageCompletion(expectedAgentInput,expectedFingerprint,timeoutSeconds=600){if(!Number.isSafeInteger(timeoutSeconds)||timeoutSeconds<1||timeoutSeconds>3600){fail6("startup completion wait timeout must be an integer from 1 to 3600 seconds")}const deadline=Date.now()+timeoutSeconds*1e3;while(true){try{return verifyManagedStartupImageCompletion(expectedAgentInput,expectedFingerprint)}catch(error){if(error.code!=="ENOENT")throw error;if(Date.now()>=deadline){fail6(`startup completion was not published within ${String(timeoutSeconds)} seconds`)}Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,250)}}}function publishManagedStartupCompletionAfterCommit(expectedAgentInput,expectedFingerprint,bootstrapIdentity,completionFile=MANAGED_STARTUP_COMPLETION_FILE,runtimeEnvironmentFile=MANAGED_STARTUP_RUNTIME_ENV_FILE){requireRoot();requireManagedStartupExchangeDirectory();const expectedAgent=exactAgent2(expectedAgentInput);if(getManagedStartupSharedStateTransactionStatus({agent:expectedAgent,profileFingerprint:expectedFingerprint,bootstrapIdentity})!=="committed"){fail6("managed startup completion requires a committed shared-state transaction")}try{verifyManagedStartupImageCompletion(expectedAgent,expectedFingerprint,completionFile,runtimeEnvironmentFile);return}catch(error){if(error.code!=="ENOENT")throw error}const runtimeEnvironment=readStableRegularFileSnapshot(runtimeEnvironmentFile,MAX_MANAGED_STARTUP_RUNTIME_ENVIRONMENT_BYTES);if(runtimeEnvironment.stat.nlink!==1n||runtimeEnvironment.stat.uid!==0n||runtimeEnvironment.stat.gid!==0n||Number(runtimeEnvironment.stat.mode&0o777n)!==292){fail6("managed startup runtime environment must be root:root mode 0444")}const corporateCaMerged=runtimeEnvironment.bytes.toString("utf8").split("\n").includes("export _NEMOCLAW_CORPORATE_CA_MERGED='1'");publishManagedStartupImageCompletion(expectedAgent,expectedFingerprint,runtimeEnvironment.bytes,corporateCaMerged,completionFile);verifyManagedStartupImageCompletion(expectedAgent,expectedFingerprint,completionFile,runtimeEnvironmentFile)}function applyAdapter(context,mapped){if(mapped.agent!==context.agent){fail6(`mapped ${mapped.agent} environment for ${context.agent}`)}const commandPlan=buildManagedStartupImageActionPlan({agent:mapped.agent,actions:mapped.actions});let commandIndex=0;for(const action of mapped.actions){if(action.kind==="configure-dashboard")continue;const command=commandPlan[commandIndex];if(!command)fail6(`missing image command for ${action.kind}`);commandIndex+=1;if(action.kind==="apply-messaging-plan"){if(action.phase==="runtime-setup"){prepareMessagingRuntimeTarget(action.mode)}execute(command.argv,command.runAs,mapped.configurationEnvironment,mapped.applicationRuntime);if(action.phase==="runtime-setup"){verifyMessagingRuntimeTarget(action.mode)}continue}execute(command.argv,command.runAs,mapped.configurationEnvironment,mapped.applicationRuntime)}if(commandIndex!==commandPlan.length){fail6("image action plan contains an unmatched command")}switch(context.agent){case"openclaw":validateOpenClawConfiguration(mapped.configurationEnvironment,mapped.applicationRuntime);break;case"hermes":installHermesManagedPolicy();installHermesSwitchyardRoutingFiles(mapped.configurationEnvironment);sealHermesConfiguration(mapped.configurationEnvironment,mapped.applicationRuntime);normalizeHermesManagedConfiguration();break;case"langchain-deepagents-code":break}installRootOwnedMaterials(mapped.materials);installCorporateCa(context.corporateCaPath);installCorporateCaSystemAnchors(context.corporateCaPath);mergeCorporateCa(context.corporateCaPath)}function adapters(mapped){return MANAGED_STARTUP_AGENTS.map(agent=>({agent,apply:context=>applyAdapter(context,mapped)}))}async function applyManagedStartupImageProfile(expectedAgentInput,env=process.env,options={}){requireRoot();const expectedAgent=exactAgent2(expectedAgentInput);if(env.NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION!=="1"){fail6("startup profiles require a complete managed image")}const encodedProfile=env[MANAGED_STARTUP_PROFILE_ENV];if(!encodedProfile)fail6(`${MANAGED_STARTUP_PROFILE_ENV} is required`);let profile;try{profile=decodeManagedStartupProfile(encodedProfile)}catch(error){fail6(error.message)}if(profile.agent!==expectedAgent){fail6(`managed startup profile targets ${profile.agent}, expected ${expectedAgent}`)}requireManagedStartupExchangeDirectory();const mapped=mapManagedStartupProfileToAgentEnvironment(profile,env);validateManagedStartupApplicationRuntimePlan(mapped.applicationRuntime);ensureRootOwnedDirectory(ROOT_STATE_PARENT);ensureRootOwnedDirectory(ROOT_RUNTIME_DIRECTORY);const result=await coordinateManagedStartupApplication({encodedProfile,expectedAgent,...env[MANAGED_STARTUP_CA_ENV]===void 0?{}:{corporateCaB64:env[MANAGED_STARTUP_CA_ENV]}},adapters(mapped));if(mapped.agent!==result.application.profile.agent){fail6(`mapped ${mapped.agent} environment for ${result.application.profile.agent}`)}if(expectedAgent==="hermes"&&!result.adapterApplied){normalizeHermesManagedConfiguration();verifyHermesSwitchyardRoutingFiles(mapped.configurationEnvironment)}let corporateCaMerged;if(result.adapterApplied){corporateCaMerged=result.application.corporateCaPath!==null}else{verifyRootOwnedMaterials(mapped.materials);if(result.application.corporateCaPath===null){if(import_node_fs3.default.existsSync(MANAGED_STARTUP_CORPORATE_CA_FILE)){fail6("committed profile without a corporate CA has a stale CA material")}}else{const expected=readStableRegularFile(result.application.corporateCaPath,128*1024);const installed=readStableRegularFile(MANAGED_STARTUP_CORPORATE_CA_FILE,128*1024);if(!expected.equals(installed)){fail6("committed corporate CA material drifted")}}installCorporateCaSystemAnchors(result.application.corporateCaPath);corporateCaMerged=mergeCorporateCa(result.application.corporateCaPath)}const runtimeEnvironment=serializeManagedStartupRuntimeEnvironment(mapped.runtimeEnvironment,corporateCaMerged,mapped.configurationEnvironment,mapped.applicationRuntime);atomicWriteRootFile(MANAGED_STARTUP_RUNTIME_ENV_FILE,runtimeEnvironment,292);if(options.publishCompletion!==false){publishManagedStartupImageCompletion(expectedAgent,result.application.fingerprint,runtimeEnvironment,corporateCaMerged)}return{agent:expectedAgent,adapterApplied:result.adapterApplied,fingerprint:result.application.fingerprint,runtimeEnvironmentFile:MANAGED_STARTUP_RUNTIME_ENV_FILE}}function completionAlreadyPublished(request){try{verifyManagedStartupImageCompletion(request.agent,request.profileFingerprint);return true}catch(error){if(error.code==="ENOENT")return false;throw error}}async function applyManagedStartupRootRequest(request,env=process.env,options={}){requireRoot();const profile=decodeManagedStartupProfile(request.encodedProfile);if(profile.agent!==request.agent||fingerprintManagedStartupProfile(profile)!==request.profileFingerprint){fail6("root application request identity does not match its profile")}const imageEnvironment={HOME:"/root",PATH:FIXED_PATH,NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION:"1",...selectManagedStartupApplicationRuntimeEnvironment(env),[MANAGED_STARTUP_PROFILE_ENV]:request.encodedProfile,...request.corporateCaB64===null?{}:{[MANAGED_STARTUP_CA_ENV]:request.corporateCaB64}};mapManagedStartupProfileToAgentEnvironment(profile,imageEnvironment);normalizeManagedStartupWorkspaceRoot(request.agent);const alreadyPublished=completionAlreadyPublished(request);const bootstrapIdentity=options.bootstrapIdentity??null;const transactionStatus=alreadyPublished&&bootstrapIdentity!==null?getManagedStartupSharedStateTransactionStatus({agent:request.agent,profileFingerprint:request.profileFingerprint,bootstrapIdentity}):null;if(transactionStatus==="none"){fail6("completed startup profile has no shared-state authority for this bootstrap attempt")}if(!alreadyPublished){ensureRootOwnedDirectory(ROOT_STATE_PARENT);beginManagedStartupSharedStateTransaction(profile,{bootstrapIdentity})}const result=await applyManagedStartupImageProfile(request.agent,imageEnvironment,{publishCompletion:bootstrapIdentity===null||alreadyPublished});return{...result,transactionPending:!alreadyPublished||transactionStatus==="pending"}}function readBoundedRootApplyStdin(){const chunks=[];let total=0;while(true){const chunk=Buffer.alloc(16*1024);const read=import_node_fs3.default.readSync(0,chunk,0,chunk.length,null);if(read===0)break;total+=read;if(total>MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES){fail6("root application stdin exceeds its bounded transport")}chunks.push(chunk.subarray(0,read))}const bytes=Buffer.concat(chunks,total);const text=bytes.toString("utf8");if(text.includes("\0")||!Buffer.from(text,"utf8").equals(bytes)){fail6("root application stdin must be valid UTF-8 without NUL bytes")}return text}function writeSandboxFileAtomically(target2,contents,mode){const parent=import_node_path3.default.dirname(target2);const parentStat=import_node_fs3.default.lstatSync(parent);if(parentStat.isSymbolicLink()||!parentStat.isDirectory()||parentStat.uid!==process.geteuid?.()||parentStat.gid!==process.getegid?.()){fail6(`refusing unsafe sandbox-owned directory ${parent}`)}const temporary=import_node_path3.default.join(parent,`.${import_node_path3.default.basename(target2)}.${(0,import_node_crypto7.randomBytes)(12).toString("hex")}`);let descriptor;try{descriptor=import_node_fs3.default.openSync(temporary,import_node_fs3.default.constants.O_CREAT|import_node_fs3.default.constants.O_EXCL|import_node_fs3.default.constants.O_WRONLY|import_node_fs3.default.constants.O_NOFOLLOW,384);import_node_fs3.default.writeFileSync(descriptor,contents);import_node_fs3.default.fchmodSync(descriptor,mode);import_node_fs3.default.fsyncSync(descriptor);import_node_fs3.default.closeSync(descriptor);descriptor=void 0;import_node_fs3.default.renameSync(temporary,target2)}catch(error){if(descriptor!==void 0)import_node_fs3.default.closeSync(descriptor);try{import_node_fs3.default.unlinkSync(temporary)}catch{}fail6(`could not write sandbox-owned file ${target2}: ${error.message}`)}}function internalWriteHermesCompatHash(){if(process.geteuid?.()===0)fail6("sandbox hash writer must not run as root");const encoded=process.env.NEMOCLAW_MANAGED_HERMES_HASH_B64??"";if(encoded.length===0||encoded.length>4096||!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u.test(encoded)){fail6("Hermes compatibility hash transport is invalid")}const decoded=Buffer.from(encoded,"base64");if(decoded.toString("base64")!==encoded){fail6("Hermes compatibility hash transport is non-canonical")}writeSandboxFileAtomically("/sandbox/.hermes/.config-hash",decoded.toString("utf8"),416)}function readCliAgent(argv,expectedLength=2){const index=argv.indexOf("--agent");if(index<0||index+1>=argv.length||argv.length!==expectedLength){fail6("usage: managed-startup-image-runtime [--apply-root-stdin|--release-startup-hold|--wait-for-completion|--verify-completion|--begin-shared-state-transaction|--commit-shared-state-transaction|--rollback-shared-state-transaction|--clear-shared-state-commit-receipt|--shared-state-transaction-status] --agent ")}return argv[index+1]}function readCliFingerprint(argv){const index=argv.indexOf("--profile-fingerprint");if(index<0||index+1>=argv.length){fail6("managed startup profile fingerprint argument is missing")}return argv[index+1]}function readCliBootstrapIdentity(argv){const index=argv.indexOf("--bootstrap-identity");if(index<0||index+1>=argv.length||!SHA256_RE4.test(String(argv[index+1]??""))){fail6("managed bootstrap identity argument is missing or invalid")}return argv[index+1]}async function main(argv=process.argv.slice(2)){if(argv.length===1&&argv[0]==="--internal-write-hermes-compat-hash"){internalWriteHermesCompatHash();return}if((argv.length===3||argv.length===5)&&argv[0]==="--apply-root-stdin"){const expectedAgent=exactAgent2(readCliAgent(argv,argv.length));const request=parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin());if(request.agent!==expectedAgent){fail6(`root application request targets ${request.agent}, expected ${expectedAgent}`)}const result2=await applyManagedStartupRootRequest(request,process.env,{bootstrapIdentity:argv.length===5?readCliBootstrapIdentity(argv):null});console.log(result2.transactionPending?`[managed-startup] applied ${result2.agent} profile ${result2.fingerprint}; transaction pending`:`[managed-startup] ${result2.agent} profile ${result2.fingerprint} was already complete`);return}if(argv.length===7&&argv[0]==="--release-startup-hold"){const agent=readCliAgent(argv,7);const fingerprint=readCliFingerprint(argv);const bootstrapIdentity=readCliBootstrapIdentity(argv);publishManagedStartupCompletionAfterCommit(agent,fingerprint,bootstrapIdentity);console.log(`[managed-startup] released ${agent} profile ${fingerprint} startup hold`);return}if((argv.length===5||argv.length===7)&&(argv[0]==="--verify-completion"||argv[0]==="--wait-for-completion")){const agent=readCliAgent(argv,argv.length);const fingerprint=readCliFingerprint(argv);if(argv.length===7)readCliBootstrapIdentity(argv);const result2=argv[0]==="--wait-for-completion"?waitForManagedStartupImageCompletion(agent,fingerprint):verifyManagedStartupImageCompletion(agent,fingerprint);console.log(`[managed-startup] verified ${result2.agent} profile ${result2.fingerprint} completion`);return}if(argv.length===3&&argv[0]==="--begin-shared-state-transaction"){const profile=managedTransactionProfile(readCliAgent(argv,3));ensureRootOwnedDirectory(ROOT_STATE_PARENT);const created=beginManagedStartupSharedStateTransaction(profile);process.stdout.write(created?"created\n":"pending\n");return}if((argv.length===4||argv.length===6)&&argv[0]==="--rollback-shared-state-transaction"&&argv[argv.length-1]==="--read-only-receipt"){requireRoot();const agent=exactAgent2(readCliAgent(argv,argv.length));const rolledBack=rollbackManagedStartupSharedStateTransaction(agent,{transactionDirectory:MANAGED_STARTUP_SHARED_ROLLBACK_RECEIPT_DIRECTORY,readOnlyReceipt:true,bootstrapIdentity:argv.length===6?readCliBootstrapIdentity(argv):null});if(!rolledBack)fail6("read-only shared-state rollback receipt is missing");console.log(`[managed-startup] verified and restored ${agent} shared state`);return}if((argv.length===3||argv.length===5)&&argv[0]==="--rollback-shared-state-transaction"){requireRoot();const agent=exactAgent2(readCliAgent(argv,argv.length));const rolledBack=rollbackManagedStartupSharedStateTransaction(agent,{bootstrapIdentity:argv.length===5?readCliBootstrapIdentity(argv):null});if(!rolledBack)fail6("managed startup transaction is missing at rollback");console.log(`[managed-startup] verified and restored ${agent} shared state`);return}if((argv.length===3||argv.length===5)&&argv[0]==="--commit-shared-state-transaction"){requireRoot();const agent=exactAgent2(readCliAgent(argv,argv.length));if(!commitManagedStartupSharedStateTransaction(agent,{bootstrapIdentity:argv.length===5?readCliBootstrapIdentity(argv):null})){fail6("managed startup transaction is missing at commit")}console.log(`[managed-startup] committed ${agent} shared state`);return}if(argv.length===5&&argv[0]==="--clear-shared-state-commit-receipt"){requireRoot();const agent=exactAgent2(readCliAgent(argv,5));const bootstrapIdentity=readCliBootstrapIdentity(argv);if(!clearManagedStartupSharedStateCommitReceipt(agent,{bootstrapIdentity})){fail6("managed startup durable commit receipt is missing at cleanup")}console.log(`[managed-startup] cleared ${agent} durable shared-state commit receipt`);return}if(argv.length===7&&argv[0]==="--shared-state-transaction-status"){requireRoot();const agent=exactAgent2(readCliAgent(argv,7));const profileFingerprint=readCliFingerprint(argv);const bootstrapIdentity=readCliBootstrapIdentity(argv);process.stdout.write(`${getManagedStartupSharedStateTransactionStatus({agent,profileFingerprint,bootstrapIdentity})} +`}function publishManagedStartupImageCompletion(agent,profileFingerprint,runtimeEnvironment,corporateCaMerged,completionFile=MANAGED_STARTUP_COMPLETION_FILE){atomicWriteRootFile(completionFile,serializeManagedStartupCompletionMarker({schemaVersion:MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,agent,profileFingerprint,runtimeEnvironmentSha256:(0,import_node_crypto6.createHash)("sha256").update(runtimeEnvironment).digest("hex"),corporateCaMerged}),292)}function parseManagedStartupCompletionMarker(text){let parsed;try{parsed=JSON.parse(text)}catch{fail6("managed startup completion marker is not valid JSON")}if(typeof parsed!=="object"||parsed===null||Array.isArray(parsed)){fail6("managed startup completion marker must be an object")}const record2=parsed;const expectedKeys=["agent","corporateCaMerged","profileFingerprint","runtimeEnvironmentSha256","schemaVersion"];if(Object.keys(record2).sort().join(",")!==expectedKeys.sort().join(",")||record2.schemaVersion!==MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION||typeof record2.agent!=="string"||!MANAGED_STARTUP_AGENTS.includes(record2.agent)||typeof record2.profileFingerprint!=="string"||!SHA256_RE4.test(record2.profileFingerprint)||typeof record2.runtimeEnvironmentSha256!=="string"||!SHA256_RE4.test(record2.runtimeEnvironmentSha256)||typeof record2.corporateCaMerged!=="boolean"){fail6("managed startup completion marker has an invalid schema")}const marker={schemaVersion:MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,agent:record2.agent,profileFingerprint:record2.profileFingerprint,runtimeEnvironmentSha256:record2.runtimeEnvironmentSha256,corporateCaMerged:record2.corporateCaMerged};if(serializeManagedStartupCompletionMarker(marker)!==text){fail6("managed startup completion marker is not canonical")}return marker}function verifyManagedStartupImageCompletion(expectedAgentInput,expectedFingerprint,completionFile=MANAGED_STARTUP_COMPLETION_FILE,runtimeEnvironmentFile=MANAGED_STARTUP_RUNTIME_ENV_FILE){const exchangeDirectory=import_node_path3.default.dirname(completionFile);if(import_node_path3.default.dirname(runtimeEnvironmentFile)!==exchangeDirectory){fail6("managed startup handoff files must share one exchange directory")}requireManagedStartupExchangeDirectory(exchangeDirectory);const expectedAgent=exactAgent2(expectedAgentInput);if(!SHA256_RE4.test(expectedFingerprint)){fail6("startup completion expected profile fingerprint is invalid")}const{bytes,stat}=readStableRegularFileSnapshot(completionFile,MAX_MANAGED_STARTUP_COMPLETION_BYTES);if(stat.nlink!==1n||stat.uid!==0n||stat.gid!==0n||Number(stat.mode&0o777n)!==292){fail6("managed startup completion marker must be root:root mode 0444")}const marker=parseManagedStartupCompletionMarker(bytes.toString("utf8"));if(marker.agent!==expectedAgent||marker.profileFingerprint!==expectedFingerprint){fail6("managed startup completion marker does not match the requested profile")}const runtimeEnvironment=readStableRegularFileSnapshot(runtimeEnvironmentFile,MAX_MANAGED_STARTUP_RUNTIME_ENVIRONMENT_BYTES);if(runtimeEnvironment.stat.nlink!==1n||runtimeEnvironment.stat.uid!==0n||runtimeEnvironment.stat.gid!==0n||Number(runtimeEnvironment.stat.mode&0o777n)!==292){fail6("managed startup runtime environment must be root:root mode 0444")}const runtimeEnvironmentSha256=(0,import_node_crypto6.createHash)("sha256").update(runtimeEnvironment.bytes).digest("hex");if(runtimeEnvironmentSha256!==marker.runtimeEnvironmentSha256){fail6("managed startup completion marker runtime environment digest mismatch")}return{agent:expectedAgent,fingerprint:expectedFingerprint}}function waitForManagedStartupImageCompletion(expectedAgentInput,expectedFingerprint,timeoutSeconds=600){if(!Number.isSafeInteger(timeoutSeconds)||timeoutSeconds<1||timeoutSeconds>3600){fail6("startup completion wait timeout must be an integer from 1 to 3600 seconds")}const deadline=Date.now()+timeoutSeconds*1e3;while(true){try{return verifyManagedStartupImageCompletion(expectedAgentInput,expectedFingerprint)}catch(error){if(error.code!=="ENOENT")throw error;if(Date.now()>=deadline){fail6(`startup completion was not published within ${String(timeoutSeconds)} seconds`)}Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,250)}}}function publishManagedStartupCompletionAfterCommit(expectedAgentInput,expectedFingerprint,bootstrapIdentity,completionFile=MANAGED_STARTUP_COMPLETION_FILE,runtimeEnvironmentFile=MANAGED_STARTUP_RUNTIME_ENV_FILE){requireRoot();requireManagedStartupExchangeDirectory();const expectedAgent=exactAgent2(expectedAgentInput);if(getManagedStartupSharedStateTransactionStatus({agent:expectedAgent,profileFingerprint:expectedFingerprint,bootstrapIdentity})!=="committed"){fail6("managed startup completion requires a committed shared-state transaction")}try{verifyManagedStartupImageCompletion(expectedAgent,expectedFingerprint,completionFile,runtimeEnvironmentFile);return}catch(error){if(error.code!=="ENOENT")throw error}const runtimeEnvironment=readStableRegularFileSnapshot(runtimeEnvironmentFile,MAX_MANAGED_STARTUP_RUNTIME_ENVIRONMENT_BYTES);if(runtimeEnvironment.stat.nlink!==1n||runtimeEnvironment.stat.uid!==0n||runtimeEnvironment.stat.gid!==0n||Number(runtimeEnvironment.stat.mode&0o777n)!==292){fail6("managed startup runtime environment must be root:root mode 0444")}const corporateCaMerged=runtimeEnvironment.bytes.toString("utf8").split("\n").includes("export _NEMOCLAW_CORPORATE_CA_MERGED='1'");publishManagedStartupImageCompletion(expectedAgent,expectedFingerprint,runtimeEnvironment.bytes,corporateCaMerged,completionFile);verifyManagedStartupImageCompletion(expectedAgent,expectedFingerprint,completionFile,runtimeEnvironmentFile)}function applyAdapterCommands(mapped,identity){const commandPlan=buildManagedStartupImageActionPlan({agent:mapped.agent,actions:mapped.actions});let commandIndex=0;for(const action of mapped.actions){if(action.kind==="configure-dashboard")continue;const command=commandPlan[commandIndex];if(!command)fail6(`missing image command for ${action.kind}`);commandIndex+=1;if(identity!==void 0&&command.runAs!==identity)continue;if(action.kind==="apply-messaging-plan"){if(action.phase==="runtime-setup"){prepareMessagingRuntimeTarget(action.mode)}execute(command.argv,command.runAs,mapped.configurationEnvironment,mapped.applicationRuntime,identity===void 0?false:"private");if(action.phase==="runtime-setup"){verifyMessagingRuntimeTarget(action.mode)}continue}execute(command.argv,command.runAs,mapped.configurationEnvironment,mapped.applicationRuntime,identity===void 0?false:"private")}if(commandIndex!==commandPlan.length){fail6("image action plan contains an unmatched command")}}function applyAdapter(context,mapped,ownerResult){if(mapped.agent!==context.agent)fail6(`mapped ${mapped.agent} environment for ${context.agent}`);if(ownerResult){for(const action of mapped.actions){if(action.kind==="apply-messaging-plan"&&action.phase==="runtime-setup"){verifyMessagingRuntimeTarget(action.mode)}}if(context.agent==="hermes"){atomicWriteRootFile(HERMES_INSTALLED_MANAGED_POLICY_FILE,Buffer.from(ownerResult.hermesPolicyB64,"base64"),292);atomicWriteRootFile("/etc/nemoclaw/hermes.config-hash",ownerResult.hermesConfigHash,292);installHermesSwitchyardRoutingFiles(mapped.configurationEnvironment)}}else{applyAdapterCommands(mapped);switch(context.agent){case"openclaw":validateOpenClawConfiguration(mapped.configurationEnvironment,mapped.applicationRuntime);break;case"hermes":installHermesManagedPolicy();installHermesSwitchyardRoutingFiles(mapped.configurationEnvironment);sealHermesConfiguration(mapped.configurationEnvironment,mapped.applicationRuntime);normalizeHermesManagedConfiguration();break;case"langchain-deepagents-code":break}}installRootOwnedMaterials(mapped.materials);installCorporateCa(context.corporateCaPath);installCorporateCaSystemAnchors(context.corporateCaPath);mergeCorporateCa(context.corporateCaPath)}function adapters(mapped,ownerResult){return MANAGED_STARTUP_AGENTS.map(agent=>({agent,apply:context=>applyAdapter(context,mapped,ownerResult)}))}async function applyManagedStartupImageProfile(expectedAgentInput,env=process.env,options={}){requireRoot();const expectedAgent=exactAgent2(expectedAgentInput);if(env.NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION!=="1"){fail6("startup profiles require a complete managed image")}const encodedProfile=env[MANAGED_STARTUP_PROFILE_ENV];if(!encodedProfile)fail6(`${MANAGED_STARTUP_PROFILE_ENV} is required`);let profile;try{profile=decodeManagedStartupProfile(encodedProfile)}catch(error){fail6(error.message)}if(profile.agent!==expectedAgent){fail6(`managed startup profile targets ${profile.agent}, expected ${expectedAgent}`)}requireManagedStartupExchangeDirectory();const mapped=mapManagedStartupProfileToAgentEnvironment(profile,env);validateManagedStartupApplicationRuntimePlan(mapped.applicationRuntime);ensureRootOwnedDirectory(ROOT_STATE_PARENT);ensureRootOwnedDirectory(ROOT_RUNTIME_DIRECTORY);const result=await coordinateManagedStartupApplication({encodedProfile,expectedAgent,...env[MANAGED_STARTUP_CA_ENV]===void 0?{}:{corporateCaB64:env[MANAGED_STARTUP_CA_ENV]}},adapters(mapped,options.ownerResult));if(mapped.agent!==result.application.profile.agent){fail6(`mapped ${mapped.agent} environment for ${result.application.profile.agent}`)}if(expectedAgent==="hermes"&&!result.adapterApplied){if(!options.ownerResult){normalizeHermesManagedConfiguration()}verifyHermesSwitchyardRoutingFiles(mapped.configurationEnvironment)}let corporateCaMerged;if(result.adapterApplied){corporateCaMerged=result.application.corporateCaPath!==null}else{verifyRootOwnedMaterials(mapped.materials);if(result.application.corporateCaPath===null){if(import_node_fs3.default.existsSync(MANAGED_STARTUP_CORPORATE_CA_FILE)){fail6("committed profile without a corporate CA has a stale CA material")}}else{const expected=readStableRegularFile(result.application.corporateCaPath,128*1024);const installed=readStableRegularFile(MANAGED_STARTUP_CORPORATE_CA_FILE,128*1024);if(!expected.equals(installed)){fail6("committed corporate CA material drifted")}}installCorporateCaSystemAnchors(result.application.corporateCaPath);corporateCaMerged=mergeCorporateCa(result.application.corporateCaPath)}const runtimeEnvironment=serializeManagedStartupRuntimeEnvironment(mapped.runtimeEnvironment,corporateCaMerged,mapped.configurationEnvironment,mapped.applicationRuntime);atomicWriteRootFile(MANAGED_STARTUP_RUNTIME_ENV_FILE,runtimeEnvironment,292);if(options.publishCompletion!==false){publishManagedStartupImageCompletion(expectedAgent,result.application.fingerprint,runtimeEnvironment,corporateCaMerged)}return{agent:expectedAgent,adapterApplied:result.adapterApplied,fingerprint:result.application.fingerprint,runtimeEnvironmentFile:MANAGED_STARTUP_RUNTIME_ENV_FILE}}function requestImageEnvironment(request,env){return{NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION:"1",...selectManagedStartupApplicationRuntimeEnvironment(env),[MANAGED_STARTUP_PROFILE_ENV]:request.encodedProfile,...request.corporateCaB64===null?{}:{[MANAGED_STARTUP_CA_ENV]:request.corporateCaB64}}}function prepareDelegatedApplication(request,bootstrapIdentity){requireRoot();if(!SHA256_RE4.test(bootstrapIdentity)||getManagedStartupSharedStateTransactionStatus({agent:request.agent,profileFingerprint:request.profileFingerprint,bootstrapIdentity})==="none")fail6("delegated startup requires its protected shared-state transaction");requireManagedStartupExchangeDirectory();ensureRootOwnedDirectory(ROOT_STATE_PARENT);ensureRootOwnedDirectory(ROOT_RUNTIME_DIRECTORY);return prepareManagedStartupApplication({encodedProfile:request.encodedProfile,expectedAgent:request.agent,...request.corporateCaB64===null?{}:{corporateCaB64:request.corporateCaB64}})}function prepareManagedStartupDelegatedRequest(request,bootstrapIdentity,env=process.env){requireRoot();const profile=decodeManagedStartupProfile(request.encodedProfile);const mapped=mapManagedStartupProfileToAgentEnvironment(profile,requestImageEnvironment(request,env));validateManagedStartupApplicationRuntimePlan(mapped.applicationRuntime);const prepared=prepareDelegatedApplication(request,bootstrapIdentity);if(prepared.status==="prepared")applyAdapterCommands(mapped,"root");return prepared.status}function applyManagedStartupOwnerRequest(request,bootstrapIdentity,replay=false,env=process.env){requireNativeStateOwner(request.agent);if(!SHA256_RE4.test(bootstrapIdentity))fail6("invalid owner bootstrap identity");const profile=decodeManagedStartupProfile(request.encodedProfile);const mapped=mapManagedStartupProfileToAgentEnvironment(profile,requestImageEnvironment(request,env));validateManagedStartupApplicationRuntimePlan(mapped.applicationRuntime);if(!replay)applyAdapterCommands(mapped,"sandbox");let hermesPolicyB64=null;let hermesConfigHash=null;if(request.agent==="openclaw"&&!replay)validateOpenClawConfiguration(mapped.configurationEnvironment,mapped.applicationRuntime,true);if(request.agent==="hermes"){if(!replay){const policy=readStableRegularFileSnapshot(HERMES_GENERATED_MANAGED_POLICY_FILE,MAX_HERMES_MANAGED_POLICY_BYTES);hermesPolicyB64=policy.bytes.toString("base64");hermesConfigHash=hermesConfigurationHash();writeSandboxFileAtomically("/sandbox/.hermes/.config-hash",hermesConfigHash,416);const routing=readHermesSwitchyardRoutingEnvironment(mapped.configurationEnvironment);if(routing){const serialized=serializeHermesSwitchyardRouting(routing);writeSandboxFileAtomically(`/sandbox/.hermes/${HERMES_RELAY_PLUGINS_STAGING_FILE}`,serialized.relayPluginsToml,384);writeSandboxFileAtomically(`/sandbox/.hermes/${HERMES_RELAY_PLUGIN_STATE_STAGING_FILE}`,serialized.relayPluginStateJson,384);writeSandboxFileAtomically(`/sandbox/.hermes/${HERMES_SWITCHYARD_DEPLOYMENT_STAGING_FILE}`,serialized.switchyardDeploymentToml,384)}if(!sameStableFileMetadata(policy.stat,import_node_fs3.default.lstatSync(HERMES_GENERATED_MANAGED_POLICY_FILE,{bigint:true})))fail6("Hermes policy changed before owner cleanup");import_node_fs3.default.unlinkSync(HERMES_GENERATED_MANAGED_POLICY_FILE)}normalizeHermesManagedConfiguration()}return{agent:request.agent,profileFingerprint:request.profileFingerprint,bootstrapIdentity,applied:!replay,hermesPolicyB64,hermesConfigHash}}function parseOwnerResult(text,request,bootstrapIdentity,applied){if(Buffer.byteLength(text)>MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES)fail6("owner result exceeds its bounded transport");let value;try{value=JSON.parse(text)}catch{fail6("owner result is not valid JSON")}if(!value||typeof value!=="object"||Array.isArray(value))fail6("invalid owner result");const result=value;if(Object.keys(result).sort().join(",")!=="agent,applied,bootstrapIdentity,hermesConfigHash,hermesPolicyB64,profileFingerprint"||result.agent!==request.agent||result.profileFingerprint!==request.profileFingerprint||result.bootstrapIdentity!==bootstrapIdentity||result.applied!==applied)fail6("owner result does not match prepared startup");if(request.agent==="hermes"&&applied){if(typeof result.hermesPolicyB64!=="string"||result.hermesPolicyB64.length===0||result.hermesPolicyB64.length>4*Math.ceil(MAX_HERMES_MANAGED_POLICY_BYTES/3)||Buffer.from(result.hermesPolicyB64,"base64").toString("base64")!==result.hermesPolicyB64||typeof result.hermesConfigHash!=="string"||!/^[a-f0-9]{64} \/sandbox\/\.hermes\/config\.yaml\n[a-f0-9]{64} \/sandbox\/\.hermes\/\.env\n$/u.test(result.hermesConfigHash))fail6("invalid Hermes owner materials")}else if(result.hermesPolicyB64!==null||result.hermesConfigHash!==null)fail6("unexpected owner materials");return result}async function finishManagedStartupDelegatedRequest(request,bootstrapIdentity,ownerResultText,env=process.env){const prepared=prepareDelegatedApplication(request,bootstrapIdentity);const ownerResult=parseOwnerResult(ownerResultText,request,bootstrapIdentity,prepared.status==="prepared");return applyManagedStartupImageProfile(request.agent,requestImageEnvironment(request,env),{publishCompletion:false,ownerResult})}function completionAlreadyPublished(request){try{verifyManagedStartupImageCompletion(request.agent,request.profileFingerprint);return true}catch(error){if(error.code==="ENOENT")return false;throw error}}async function applyManagedStartupRootRequest(request,env=process.env,options={}){requireRoot();const profile=decodeManagedStartupProfile(request.encodedProfile);if(profile.agent!==request.agent||fingerprintManagedStartupProfile(profile)!==request.profileFingerprint){fail6("root application request identity does not match its profile")}const imageEnvironment={HOME:"/root",PATH:FIXED_PATH,NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION:"1",...selectManagedStartupApplicationRuntimeEnvironment(env),[MANAGED_STARTUP_PROFILE_ENV]:request.encodedProfile,...request.corporateCaB64===null?{}:{[MANAGED_STARTUP_CA_ENV]:request.corporateCaB64}};mapManagedStartupProfileToAgentEnvironment(profile,imageEnvironment);normalizeManagedStartupWorkspaceRoot(request.agent);const alreadyPublished=completionAlreadyPublished(request);const bootstrapIdentity=options.bootstrapIdentity??null;const transactionStatus=alreadyPublished&&bootstrapIdentity!==null?getManagedStartupSharedStateTransactionStatus({agent:request.agent,profileFingerprint:request.profileFingerprint,bootstrapIdentity}):null;if(transactionStatus==="none"){fail6("completed startup profile has no shared-state authority for this bootstrap attempt")}if(!alreadyPublished){ensureRootOwnedDirectory(ROOT_STATE_PARENT);beginManagedStartupSharedStateTransaction(profile,{bootstrapIdentity})}const result=await applyManagedStartupImageProfile(request.agent,imageEnvironment,{publishCompletion:bootstrapIdentity===null||alreadyPublished});return{...result,transactionPending:!alreadyPublished||transactionStatus==="pending"}}function readBoundedRootApplyStdin(maxBytes=MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES){const chunks=[];let total=0;while(true){const chunk=Buffer.alloc(16*1024);const read=import_node_fs3.default.readSync(0,chunk,0,chunk.length,null);if(read===0)break;total+=read;if(total>maxBytes){fail6("root application stdin exceeds its bounded transport")}chunks.push(chunk.subarray(0,read))}const bytes=Buffer.concat(chunks,total);const text=bytes.toString("utf8");if(text.includes("\0")||!Buffer.from(text,"utf8").equals(bytes)){fail6("root application stdin must be valid UTF-8 without NUL bytes")}return text}function writeSandboxFileAtomically(target2,contents,mode){const parent=import_node_path3.default.dirname(target2);const parentStat=import_node_fs3.default.lstatSync(parent);if(parentStat.isSymbolicLink()||!parentStat.isDirectory()||parentStat.uid!==process.geteuid?.()||parentStat.gid!==process.getegid?.()){fail6(`refusing unsafe sandbox-owned directory ${parent}`)}const temporary=import_node_path3.default.join(parent,`.${import_node_path3.default.basename(target2)}.${(0,import_node_crypto6.randomBytes)(12).toString("hex")}`);let descriptor;try{descriptor=import_node_fs3.default.openSync(temporary,import_node_fs3.default.constants.O_CREAT|import_node_fs3.default.constants.O_EXCL|import_node_fs3.default.constants.O_WRONLY|import_node_fs3.default.constants.O_NOFOLLOW,384);import_node_fs3.default.writeFileSync(descriptor,contents);import_node_fs3.default.fchmodSync(descriptor,mode);import_node_fs3.default.fsyncSync(descriptor);import_node_fs3.default.closeSync(descriptor);descriptor=void 0;import_node_fs3.default.renameSync(temporary,target2)}catch(error){if(descriptor!==void 0)import_node_fs3.default.closeSync(descriptor);try{import_node_fs3.default.unlinkSync(temporary)}catch{}fail6(`could not write sandbox-owned file ${target2}: ${error.message}`)}}function internalWriteHermesCompatHash(){if(process.geteuid?.()===0)fail6("sandbox hash writer must not run as root");const encoded=process.env.NEMOCLAW_MANAGED_HERMES_HASH_B64??"";if(encoded.length===0||encoded.length>4096||!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u.test(encoded)){fail6("Hermes compatibility hash transport is invalid")}const decoded=Buffer.from(encoded,"base64");if(decoded.toString("base64")!==encoded){fail6("Hermes compatibility hash transport is non-canonical")}writeSandboxFileAtomically("/sandbox/.hermes/.config-hash",decoded.toString("utf8"),416)}function readCliAgent(argv,expectedLength=2){const index=argv.indexOf("--agent");if(index<0||index+1>=argv.length||argv.length!==expectedLength){fail6("usage: managed-startup-image-runtime [--apply-root-stdin|--release-startup-hold|--wait-for-completion|--verify-completion|--begin-shared-state-transaction|--commit-shared-state-transaction|--rollback-shared-state-transaction|--clear-shared-state-commit-receipt|--shared-state-transaction-status] --agent ")}return argv[index+1]}function readCliFingerprint(argv){const index=argv.indexOf("--profile-fingerprint");if(index<0||index+1>=argv.length){fail6("managed startup profile fingerprint argument is missing")}return argv[index+1]}function readCliBootstrapIdentity(argv){const index=argv.indexOf("--bootstrap-identity");if(index<0||index+1>=argv.length||!SHA256_RE4.test(String(argv[index+1]??""))){fail6("managed bootstrap identity argument is missing or invalid")}return argv[index+1]}function requireNativeStateOwner(agent){const identity=managedImageRuntimeIdentity(agent);if(process.geteuid?.()!==identity.uid||process.getegid?.()!==identity.gid){fail6("shared-state capture requires the exact native state owner")}}function readSharedStateEnvelope(agent){let value;try{value=JSON.parse(readBoundedRootApplyStdin(MANAGED_STARTUP_SHARED_CAPTURE_MAX_BYTES+MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES))}catch{fail6("shared-state capture envelope is unreadable")}if(!value||typeof value!=="object"||Array.isArray(value))fail6("invalid shared-state capture envelope");const envelope=value;if(Object.keys(envelope).sort().join(",")!=="capturedState,encodedProfile"||typeof envelope.capturedState!=="string"||typeof envelope.encodedProfile!=="string"){fail6("invalid shared-state capture envelope fields")}const profile=decodeManagedStartupProfile(envelope.encodedProfile);if(profile.agent!==agent)fail6("shared-state capture envelope targets another agent");return{profile,capturedState:envelope.capturedState}}async function main(argv=process.argv.slice(2)){if(argv.length===1&&argv[0]==="--startup-protocol"){requireRoot();process.stdout.write("delegated-owner-v1\n");return}if(argv.length===5&&argv[0]==="--prepare-delegated-transaction-stdin"){requireRoot();const agent=exactAgent2(readCliAgent(argv,5));const bootstrapIdentity=readCliBootstrapIdentity(argv);const request=parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin());if(request.agent!==agent)fail6("transaction preparation targets another agent");ensureRootOwnedDirectory(ROOT_STATE_PARENT);process.stdout.write(`${getManagedStartupSharedStateTransactionStatus({agent,profileFingerprint:request.profileFingerprint,bootstrapIdentity})} +`);return}if(argv.length===5&&argv[0]==="--export-shared-restoration-profile-stdin"){requireRoot();const agent=exactAgent2(readCliAgent(argv,5));const profile=decodeManagedStartupProfile(readBoundedRootApplyStdin().trim());if(profile.agent!==agent)fail6("shared-state restoration profile targets another agent");const bootstrapIdentity=readCliBootstrapIdentity(argv);process.stdout.write(exportManagedStartupSharedStateRestoration(profile,{bootstrapIdentity})??"absent\n");return}if((argv.length===5||argv.length===6&&argv[5]==="--replay")&&["--prepare-delegated-startup-stdin","--apply-owner-startup-stdin"].includes(argv[0]??"")){const agent=exactAgent2(readCliAgent(argv,argv.length));const owner=argv[0]==="--apply-owner-startup-stdin";if(owner)requireNativeStateOwner(agent);else{requireRoot();if(argv.length!==5)fail6("root preparation cannot request owner replay")}const bootstrapIdentity=readCliBootstrapIdentity(argv);const request=parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin());if(request.agent!==agent)fail6("delegated startup request targets another agent");if(owner)process.stdout.write(`${JSON.stringify(applyManagedStartupOwnerRequest(request,bootstrapIdentity,argv.length===6))} +`);else process.stdout.write(`${prepareManagedStartupDelegatedRequest(request,bootstrapIdentity)} +`);return}if(argv.length===5&&argv[0]==="--finish-delegated-startup-stdin"){requireRoot();const agent=exactAgent2(readCliAgent(argv,5));const bootstrapIdentity=readCliBootstrapIdentity(argv);let value;try{value=JSON.parse(readBoundedRootApplyStdin(MANAGED_STARTUP_ROOT_APPLY_MAX_BYTES+MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES))}catch{fail6("delegated completion envelope is unreadable")}if(!value||typeof value!=="object"||Array.isArray(value))fail6("invalid delegated completion envelope");const envelope=value;if(Object.keys(envelope).sort().join(",")!=="ownerResult,request"||typeof envelope.request!=="string"||typeof envelope.ownerResult!=="string")fail6("invalid delegated completion fields");const request=parseManagedStartupRootApplyRequest(envelope.request);if(request.agent!==agent)fail6("delegated completion targets another agent");await finishManagedStartupDelegatedRequest(request,bootstrapIdentity,envelope.ownerResult);process.stdout.write("applied\n");return}if(argv.length===5&&(argv[0]==="--capture-shared-state-stdin"||argv[0]==="--export-shared-restoration-stdin")){const agent=exactAgent2(readCliAgent(argv,5));const exporting=argv[0]==="--export-shared-restoration-stdin";if(exporting)requireRoot();else requireNativeStateOwner(agent);const request=parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin());if(request.agent!==agent)fail6("shared-state capture request targets another agent");const profile=decodeManagedStartupProfile(request.encodedProfile);const options={bootstrapIdentity:readCliBootstrapIdentity(argv)};process.stdout.write(exporting?exportManagedStartupSharedStateRestoration(profile,options)??"absent\n":captureManagedStartupSharedState(profile,options));return}if(argv.length===5&&["--begin-shared-state-from-capture-stdin","--restore-shared-state-stdin","--finish-shared-restoration-stdin"].includes(argv[0]??"")){const agent=exactAgent2(readCliAgent(argv,5));const restoring=argv[0]==="--restore-shared-state-stdin";if(restoring)requireNativeStateOwner(agent);else requireRoot();const bootstrapIdentity=readCliBootstrapIdentity(argv);const{profile,capturedState}=readSharedStateEnvelope(agent);if(restoring){process.stdout.write(restoreManagedStartupCapturedSharedState(profile,capturedState,{bootstrapIdentity}));return}if(argv[0]==="--finish-shared-restoration-stdin"){const restored=finishManagedStartupSharedStateRestoration(profile,capturedState,{bootstrapIdentity});process.stdout.write(restored?"restored\n":"absent\n");return}ensureRootOwnedDirectory(ROOT_STATE_PARENT);const created=beginManagedStartupSharedStateTransaction(profile,{bootstrapIdentity,capturedState});process.stdout.write(created?"created\n":"pending\n");return}if(argv.length===1&&argv[0]==="--internal-write-hermes-compat-hash"){internalWriteHermesCompatHash();return}if((argv.length===3||argv.length===5)&&argv[0]==="--apply-root-stdin"){const expectedAgent=exactAgent2(readCliAgent(argv,argv.length));const request=parseManagedStartupRootApplyRequest(readBoundedRootApplyStdin());if(request.agent!==expectedAgent){fail6(`root application request targets ${request.agent}, expected ${expectedAgent}`)}const result2=await applyManagedStartupRootRequest(request,process.env,{bootstrapIdentity:argv.length===5?readCliBootstrapIdentity(argv):null});console.log(result2.transactionPending?`[managed-startup] applied ${result2.agent} profile ${result2.fingerprint}; transaction pending`:`[managed-startup] ${result2.agent} profile ${result2.fingerprint} was already complete`);return}if(argv.length===7&&argv[0]==="--release-startup-hold"){const agent=readCliAgent(argv,7);const fingerprint=readCliFingerprint(argv);const bootstrapIdentity=readCliBootstrapIdentity(argv);publishManagedStartupCompletionAfterCommit(agent,fingerprint,bootstrapIdentity);console.log(`[managed-startup] released ${agent} profile ${fingerprint} startup hold`);return}if((argv.length===5||argv.length===7)&&(argv[0]==="--verify-completion"||argv[0]==="--wait-for-completion")){const agent=readCliAgent(argv,argv.length);const fingerprint=readCliFingerprint(argv);if(argv.length===7)readCliBootstrapIdentity(argv);const result2=argv[0]==="--wait-for-completion"?waitForManagedStartupImageCompletion(agent,fingerprint):verifyManagedStartupImageCompletion(agent,fingerprint);console.log(`[managed-startup] verified ${result2.agent} profile ${result2.fingerprint} completion`);return}if(argv.length===3&&argv[0]==="--begin-shared-state-transaction"){const profile=managedTransactionProfile(readCliAgent(argv,3));ensureRootOwnedDirectory(ROOT_STATE_PARENT);const created=beginManagedStartupSharedStateTransaction(profile);process.stdout.write(created?"created\n":"pending\n");return}if((argv.length===4||argv.length===6)&&argv[0]==="--rollback-shared-state-transaction"&&argv[argv.length-1]==="--read-only-receipt"){requireRoot();const agent=exactAgent2(readCliAgent(argv,argv.length));const rolledBack=rollbackManagedStartupSharedStateTransaction(agent,{transactionDirectory:MANAGED_STARTUP_SHARED_ROLLBACK_RECEIPT_DIRECTORY,readOnlyReceipt:true,bootstrapIdentity:argv.length===6?readCliBootstrapIdentity(argv):null});if(!rolledBack)fail6("read-only shared-state rollback receipt is missing");console.log(`[managed-startup] verified and restored ${agent} shared state`);return}if((argv.length===3||argv.length===5)&&argv[0]==="--rollback-shared-state-transaction"){requireRoot();const agent=exactAgent2(readCliAgent(argv,argv.length));const rolledBack=rollbackManagedStartupSharedStateTransaction(agent,{bootstrapIdentity:argv.length===5?readCliBootstrapIdentity(argv):null});if(!rolledBack)fail6("managed startup transaction is missing at rollback");console.log(`[managed-startup] verified and restored ${agent} shared state`);return}if((argv.length===3||argv.length===5)&&argv[0]==="--commit-shared-state-transaction"){requireRoot();const agent=exactAgent2(readCliAgent(argv,argv.length));if(!commitManagedStartupSharedStateTransaction(agent,{bootstrapIdentity:argv.length===5?readCliBootstrapIdentity(argv):null})){fail6("managed startup transaction is missing at commit")}console.log(`[managed-startup] committed ${agent} shared state`);return}if(argv.length===5&&argv[0]==="--clear-shared-state-commit-receipt"){requireRoot();const agent=exactAgent2(readCliAgent(argv,5));const bootstrapIdentity=readCliBootstrapIdentity(argv);if(!clearManagedStartupSharedStateCommitReceipt(agent,{bootstrapIdentity})){fail6("managed startup durable commit receipt is missing at cleanup")}console.log(`[managed-startup] cleared ${agent} durable shared-state commit receipt`);return}if(argv.length===7&&argv[0]==="--shared-state-transaction-status"){requireRoot();const agent=exactAgent2(readCliAgent(argv,7));const profileFingerprint=readCliFingerprint(argv);const bootstrapIdentity=readCliBootstrapIdentity(argv);process.stdout.write(`${getManagedStartupSharedStateTransactionStatus({agent,profileFingerprint,bootstrapIdentity})} `);return}if(argv.length===8&&argv[0]==="--shared-state-transaction-status"&&argv[7]==="--read-only-receipt"){requireRoot();const agent=exactAgent2(readCliAgent(argv,8));const profileFingerprint=readCliFingerprint(argv);const bootstrapIdentity=readCliBootstrapIdentity(argv);process.stdout.write(`${getManagedStartupSharedStateTransactionStatus({agent,profileFingerprint,bootstrapIdentity},{readOnlyReceipt:true})} -`);return}const result=await applyManagedStartupImageProfile(readCliAgent(argv));console.log(result.adapterApplied?`[managed-startup] applied ${result.agent} profile ${result.fingerprint}`:`[managed-startup] ${result.agent} profile ${result.fingerprint} is already committed`)}if(require.main===module){void main().catch(error=>{console.error(error instanceof Error?error.message:String(error));process.exitCode=1})}0&&(module.exports={MANAGED_STARTUP_CA_ENV,MANAGED_STARTUP_COMPLETION_FILE,MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,MANAGED_STARTUP_MERGED_CA_FILE,MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES,MANAGED_STARTUP_PROFILE_ENV,MANAGED_STARTUP_RUNTIME_ENV_FILE,MANAGED_STARTUP_RUNTIME_EXECUTABLE,ManagedStartupImageActionPlanError,ManagedStartupImageRuntimeError,NVIDIA_HOSTED_CREDENTIAL_ENV,NVIDIA_HOSTED_NATIVE_ENDPOINT,applyManagedStartupCommandEnvironmentPlan,applyManagedStartupImageProfile,applyManagedStartupRootRequest,atomicWriteRootFile,buildManagedStartupImageActionPlan,hostedNativeProvider,installCorporateCaSystemAnchors,installHermesManagedPolicy,installHermesSwitchyardRoutingFiles,main,managedStartupSandboxPrefix,normalizeHermesManagedConfigDescriptor,normalizeManagedStartupWorkspaceRoot,publishManagedStartupCompletionAfterCommit,readStableRegularFile,readStableRegularFileSnapshot,serializeManagedStartupCompletionMarker,serializeManagedStartupRuntimeEnvironment,validateManagedStartupApplicationRuntimePlan,verifyHermesSwitchyardRoutingFiles,verifyManagedStartupImageCompletion,waitForManagedStartupImageCompletion}); +`);return}const result=await applyManagedStartupImageProfile(readCliAgent(argv));console.log(result.adapterApplied?`[managed-startup] applied ${result.agent} profile ${result.fingerprint}`:`[managed-startup] ${result.agent} profile ${result.fingerprint} is already committed`)}if(require.main===module){void main().catch(error=>{console.error(error instanceof Error?error.message:String(error));process.exitCode=1})}0&&(module.exports={MANAGED_STARTUP_CA_ENV,MANAGED_STARTUP_COMPLETION_FILE,MANAGED_STARTUP_COMPLETION_SCHEMA_VERSION,MANAGED_STARTUP_MERGED_CA_FILE,MANAGED_STARTUP_OWNER_RESULT_MAX_BYTES,MANAGED_STARTUP_PROFILE_ENV,MANAGED_STARTUP_RUNTIME_ENV_FILE,MANAGED_STARTUP_RUNTIME_EXECUTABLE,ManagedStartupImageActionPlanError,ManagedStartupImageRuntimeError,NVIDIA_HOSTED_CREDENTIAL_ENV,NVIDIA_HOSTED_NATIVE_ENDPOINT,applyManagedStartupCommandEnvironmentPlan,applyManagedStartupImageProfile,applyManagedStartupOwnerRequest,applyManagedStartupRootRequest,atomicWriteRootFile,buildManagedStartupImageActionPlan,finishManagedStartupDelegatedRequest,fixedHostedNativeProvider,installCorporateCaSystemAnchors,installHermesManagedPolicy,installHermesSwitchyardRoutingFiles,main,managedStartupSandboxPrefix,normalizeHermesManagedConfigDescriptor,normalizeManagedStartupWorkspaceRoot,prepareManagedStartupDelegatedRequest,publishManagedStartupCompletionAfterCommit,readStableRegularFile,readStableRegularFileSnapshot,serializeManagedStartupCompletionMarker,serializeManagedStartupRuntimeEnvironment,validateManagedStartupApplicationRuntimePlan,verifyHermesSwitchyardRoutingFiles,verifyManagedStartupImageCompletion,waitForManagedStartupImageCompletion});